Installing CouchDB in Ubuntu

I have tested the below procedure in Ubuntu 10.10 (Maverick) to install CouchDB 1.1.1 candidate release.

$ ps -ef|grep couch|awk '{print $2}'|xargs sudo kill -9
$ sudo apt-get update
$ sudo apt-get autoremove
$ sudo apt-get remove couchdb
$ sudo apt-get build-dep couchdb
$ sudo apt-get install libtool zip
$ cd
$ curl -O http://ftp.mozilla.org/pub/mozilla.org/js/js185-1.0.0.tar.gz
$ tar xvzf js185-1.0.0.tar.gz 
$ cd js-1.8.5/js/src
$ ./configure
$ make
$ sudo make install
$ cd
$ curl -O http://www.erlang.org/download/otp_src_R14B04.tar.gz
$ tar xvzf otp_src_R14B04.tar.gz 
$ cd otp_src_R14B04
$ ./configure --enable-smp-support --enable-dynamic-ssl-lib --enable-kernel-poll
$ make
$ sudo make install
$ cd
$ svn co http://svn.apache.org/repos/asf/couchdb/branches/1.1.x/ couchdb1.1.x
$ cd couchdb1.1.x
$ ./bootstrap
$ prefix='/usr/local'
$ ./configure --prefix=${prefix} 
$ make
$ sudo make install
$ sudo useradd -d /var/lib/couchdb couchdb
$ sudo chown -R couchdb: ${prefix}/var/{lib,log,run}/couchdb ${prefix}/etc/couchdb
$ for dir in `whereis couchdb | sed 's/couchdb: //'`; do echo $dir | xargs sudo chown couchdb; done
$ export xulrunnerversion=`xulrunner -v 2>&1 >  /dev/null | egrep -o "([0-9]{1,2})(\.[0-9]{1,2})+"`
$ echo $xulrunnerversion
$ echo "/usr/lib/xulrunner-$xulrunnerversion" > /etc/ld.so.conf.d/xulrunner.conf
$ echo "/usr/lib/xulrunner-devel-$xulrunnerversion" >> /etc/ld.so.conf.d/xulrunner.conf
$ sudo ln -s /usr/local/etc/init.d/couchdb /etc/init.d/couchdb
$ update-rc.d couchdb defaults
$ /etc/init.d/couchdb start
$ curl -X GET http://localhost:5984
{"couchdb":"Welcome","version":"1.1.1a1187726"}

Monitoring CouchDB with Monit

Just add the below in monitrc and reload config. I am expecting you hardened CouchDB and so it is listening in an SSL port. Note that couchdb is already monitored by heart so you can probably just not monitor it as well ;-)

#################################################################
# couchdb
################################################################

#check process couchdb
#  with pidfile /usr/local/var/run/couchdb/couchdb.pid
#  start program = "/usr/local/etc/init.d/couchdb start"
#  stop program = "/usr/local/etc/init.d/couchdb stop"
#  if failed port 6984 then restart
#  if failed url https://localhost:6984/ and content == '"couchdb"' then restart

#couchdb does not save the parent pid of the starting process so the above would serve no purpose
check host couchdb with address localhost
  if failed port 6984 then alert
  if failed url https://localhost:6984/ and content == '"couchdb"' then alert
group couchdb

Hardening CouchDB: A more secure distributed database

Here are the steps to follow in order to harden CouchDB. I am currently using candidate release 1.1.1 and these are instructions for a local environment. For a production environment you have to locate the non dev config files local.ini and default.ini (Just run "ps -ef | grep couchdb" and look for the path, in my case /usr/local/etc/couchdb/local.ini and /usr/local/etc/couchdb/default.ini).

1. Install a primary key and generated pem certificate:
   

   $ export DOMAIN=couchdb.nestorurquiza.com
   $ openssl genrsa -des3 -out ${DOMAIN}.pem 1024
   $ openssl req -new -key ${DOMAIN}.pem -out ${DOMAIN}.csr
   $ openssl x509 -req -days 3650 -in ${DOMAIN}.csr -signkey ${DOMAIN}.pem -out ${DOMAIN}.cert.pem
   $ openssl rsa -in ${DOMAIN}.pem -out ${DOMAIN}.pem
   $ sudo mkdir -p /opt/couchdb/certs/
   $ sudo cp ${DOMAIN}.pem /opt/couchdb/certs/
   $ sudo cp ${DOMAIN}.cert.pem /opt/couchdb/certs/
   

2. 
   Configure couchDB for *SSL only* and restart the server
   

   $ vi /Users/nestor/Downloads/couchdb1.1.x/etc/couchdb/local.ini
   [admins]
   ;admin = mysecretpassword
   nestorurquizaadmin = secret1
   ...
   [daemons]
   ; enable SSL support by uncommenting the following line and supply the PEM's below.
   ; the default ssl port CouchDB listens on is 6984
   httpsd = {couch_httpd, start_link, [https]}
   ...
   [ssl]
   cert_file = /opt/couchdb/certs/couchdb.nestorurquiza.com.cert.pem
   key_file = /opt/couchdb/certs/couchdb.nestorurquiza.com.pem
   $ vi /Users/nestor/Downloads/couchdb1.1.x/etc/couchdb/default.ini
   [httpd]
   bind_address = 0.0.0.0 #Use no loopback address only if the server will be exposed to a different machine
   ...
   [daemons]
   ...
   ; httpd={couch_httpd, start_link, []}
   ...
   

3. Restart
   

   sudo couchdb #local OSX
   sudo /usr/local/etc/init.d/couchdb restart #Ubuntu
   

4. 
   Now we can access the secured CouchDB instance
   

   HOST="https://nestorurquizaadmin:mysecret@127.0.0.1:6984"
   curl -k -X GET $HOST
   

Note that at this time to remove the insecure http you have to use /usr/local/etc/couchdb/default.ini which gets overwritten after any upgrade.

Of course we should sign our certificate with a Certificate Authority (CA) to make this really secure but you can also add some security if you play with firewall rules and allow access to the SSL port to certain IP only. I can do so because I do use a middle tear between CouchDB and the browser. If you directly hitting CouchDB from the wild you better use a CA.

CouchDB filtered replication

One of the greatest features of CouchDB is its replication which allows for great distributed computing. It reminds me when in 1999 I met Erlang language for the first time (Working for a Telco). Erlang is made for distributed computing and so CouchDB which of course is built in Erlang.

I have to say I have successfully tested this in upcoming version 1.1.1 (built from a branch) Do not try this in 1.1.0.

The example below is based on the document that I have been discussing in the three part tutorial about building a Document Management System (DMS) with CouchDB.

Filtered or selective replication is a two step process:

1. First create a filter named for example "clientFilter" in a new document called "replicateFilter". This sample filter will reject any client not matching the clientId parameter (step 2 explains what this parameter is about). Any deleted documents will be deleted from the target as well.
   

   curl -H 'Content-Type: application/json' -X PUT http://127.0.0.1:5984/dms4/_design/replicateFilter -d \
   '{
     "filters":{
       "clientFilter":"function(doc, req) {
         if (doc._deleted) {
           return true;
         }
    
         if(!doc.clientId) {
           return false;
         }
    
         if(!req.query.clientId) {
           throw(\"Please provide a query parameter clientId.\");
         }
    
         if(doc.clientId == req.query.clientId) {
           return true;
         }
         return false;
       }"
     }
   }'
   

2. Create a replication document called "by_clientId". This example passes clientId=1 as a parameter to the filter we created in step number 1 ("replicateFilter/clientFilter"). You figured we will end up replicating documents for that client.
   

   curl -H 'Content-Type: application/json' -X POST http://127.0.0.1:5984/_replicator -d \
   '{
     "_id":"by_clientId",
     "source":"dms4",
     "target":"http://couchdb.nestorurquiza.com:5984/dms4",
     "create_target":true,
     "continuous":true,
     "filter":"replicateFilter/clientFilter",
     "query_params":{"clientId":1}
   }'
   

Deleting a replication document is how you turn off that replication. This is not any different than deleting any other document:

nestor:~ nestor$ curl -X GET http://127.0.0.1:5984/_replicator/by_clientId
{"_id":"by_clientId","_rev":"5-e177ca7f79d9ba6f91b803a2cb2abc1e","source":"dms4","target":"http://couchdb.nestorurquiza.com:5984/dms4","create_target":true,"continuous":true,"filter":"replicateFilter/clientFilter","query_params":{"clientId":1},"_replication_state":"triggered","_replication_state_time":"2011-10-20T13:09:56-04:00","_replication_id":"d8dc09e97f4948de0294260dda19fc6f"}
nestor:~ nestor$ curl -X DELETE http://127.0.0.1:5984/_replicator/by_clientId?rev=5-e177ca7f79d9ba6f91b803a2cb2abc1e
{"ok":true,"id":"by_clientId","rev":"6-0d20d90cbed22837eb3233e2bd8dfb2c"}

The same applies for getting a list of the current defined "selective replicators". You can use a temporary view like I show here or create a permanent view to list all the replicators:

$ curl -X POST http://127.0.0.1:5984/_replicator/_temp_view -H "Content-Type: application/json" -d '{
  "map": "function(doc) {
            emit(null, doc);
          }"
}'

OSX homebrew update: unexpected token near HOMEBREW_BREW_FILE

Update: I have seen this issue with other packages as well. Here is a recipe to make sure a specific version of ruby is used:

cp -p /usr/local/bin/brew /usr/local/bin/brew.old
echo  '#!'/Users/`whoami`/.rvm/rubies/ruby-1.8.7-p160/bin/ruby > /tmp/f1
sed '1d' /usr/local/bin/brew > /tmp/f2
cat /tmp/f1 /tmp/f2 > /usr/local/bin/brew

Original Post: After upgrading homebrew:
$ brew update

I got errors:
$ brew install mozilla-js
/usr/local/bin/brew: line 4: syntax error near unexpected token `('
/usr/local/bin/brew: line 4: `HOMEBREW_BREW_FILE = ENV['HOMEBREW_BREW_FILE'] = File.expand_path(__FILE__)'

Here is a temporary solution which you will need to do everytime you update brew:
$ sudo vi /usr/local/bin/brew
#!/Users/nestor/.rvm/rubies/ruby-1.9.2-p290/bin/ruby
##!/usr/bin/ruby

Upgrading couchDB in OSX

While trying to use couchDB filtered replication I had some problems as documented in gist.

I had to upgrade then couchDB to a non released version. Here are the steps I followed.

Delete all files from previous installations:

$ sudo find /usr/local -name couchdb | sudo xargs rm -fR

Install ICU from http://download.icu-project.org

$ tar xvzf icu4c-4_8_1-src.tgz 
$ cd icu/source/
$ ./runConfigureICU MacOSX --with-library-bits=64 --disable-samples --enable-static # if this fails for you try: ./configure --enable-64bit-libs
$ make
$ sudo make install

Install Spidermonkey

$ curl -O http://ftp.mozilla.org/pub/mozilla.org/js/js185-1.0.0.tar.gz
$ tar xvzf js185-1.0.0.tar.gz 
$ cd js-1.8.5/js/src
./configure 
$ make
$ sudo make install

Then install latest Erlang:

$ curl -O http://www.erlang.org/download/otp_src_R14B04.tar.gz
$ tar xvzf otp_src_R14B04.tar.gz 
$ cd otp_src_R14B04
$ ./configure --enable-smp-support --enable-dynamic-ssl-lib --enable-kernel-poll --enable-darwin-64bit
$ make
$ sudo make install

Then checkout the needed version. I tried from git (http://git-wip-us.apache.org/repos/asf/couchdb.git) first but I had several problems with autoconf and beyond (.configure was not available so I needed to go with automake -a; autoconf; autoheader) so I then built from svn:

$ svn co http://svn.apache.org/repos/asf/couchdb/branches/1.1.x/ couchdb1.1.x
$ cd couchdb1.1.x
$ ./bootstrap
$ ./configure
$ make
$ sudo make install
$ sudo couchdb

Document Management System with CouchDB - Third Part

This is the last part of my attempt to cover how to use CouchDB to build a DMS. We will be using Java and the Ektorp library for the implementation.

Using Ektorp

A good place to start is downloading the BlogPost application (org.ektorp.sample-1.7-project) that shows some of the most important concepts around the Ektorp API. Follow Ektorp Tutorial and of course do not miss the reference documentation.

I will show here the steps on how to make your existing spring project able to interact with CouchDB using the Ektorp project. The code presented here is an implementation of the ideas exposed in the second part of this project.

Here the dependencies for your project. Note that I am adding file upload dependencies because I am building a DMS and I will provide an upload module:

<!-- Ektorp for CouchDB -->
        <dependency>
            <groupId>org.ektorp</groupId>
            <artifactId>org.ektorp</artifactId>
            <version>${ektorp.version}</version>
            <exclusions>
             <exclusion>
              <artifactId>slf4j-api</artifactId>
              <groupId>org.slf4j</groupId>
             </exclusion>
            </exclusions>
        </dependency>
        <dependency>
            <groupId>org.ektorp</groupId>
            <artifactId>org.ektorp.spring</artifactId>
            <version>${ektorp.version}</version>
        </dependency>

        <!-- File Upload -->
        <dependency>
            <groupId>commons-fileupload</groupId>
            <artifactId>commons-fileupload</artifactId>
            <version>1.2.2</version>
        </dependency>

In spring application context xml (Note the upload component which is only needed because again we need it for the DMS upload functionality):

...
xmlns:util="http://www.springframework.org/schema/util"
xmlns:couchdb="http://www.ektorp.org/schema/couchdb"
...
http://www.springframework.org/schema/util http://www.springframework.org/schema/util/spring-util-3.0.xsd
http://www.ektorp.org/schema/couchdb http://www.ektorp.org/schema/couchdb/couchdb.xsd
...
<context:component-scan base-package="com.nu.dms.couchdb.ektorp.model"/>
<context:component-scan base-package="com.nu.dms.couchdb.ektorp.dao"/>
...
<util:properties id="couchdbProperties" location="classpath:/couchdb.properties"/>
<couchdb:instance id="dmsCouchdb" url="${couchdb.url}" properties="couchdbProperties" />
<couchdb:database id="dmsDatabase" name="${couchdb.db}" instance-ref="dmsCouchdb" />
...
<!-- File Upload -->
<bean id="multipartResolver" class="org.springframework.web.multipart.commons.CommonsMultipartResolver">
        <property name="maxUploadSize" value="2000000"/>
</bean>

In classpath environment properties file:

couchdb.url=http://localhost:5984
couchdb.db=dms4 #my database name

File classpath couchdb.properties file:

host=localhost
port=5984
maxConnections=20
connectionTimeout=1000
socketTimeout=10000
autoUpdateViewOnChange=true
caching=false

A Document POJO following the BlogPost POJO from the Ektorp example:

package com.nu.dms.couchdb.ektorp.model;

import org.ektorp.Attachment;
import org.ektorp.support.CouchDbDocument;

public class CustomCouchDbDocument extends CouchDbDocument {

    private static final long serialVersionUID = -9012014877538917152L;

    @Override
    public void addInlineAttachment(Attachment a) {
        super.addInlineAttachment(a);
    }   
}


package com.nu.dms.couchdb.ektorp.model;

import java.util.Date;

import javax.validation.constraints.NotNull;

import org.ektorp.support.TypeDiscriminator;

public class Document extends CustomCouchDbDocument {

    private static final long serialVersionUID = 59516215253102057L;
    
    public Document() {
        super();
    }
    
    public Document(String title) {
        this.title = title;
    }
    
    /**
     * @TypeDiscriminator is used to mark properties that makes this class's documents unique in the database. 
     */
    @TypeDiscriminator
    @NotNull
    private String title;
    
    private int clientId;
    private int createdByEmployeeId;
    private int reviewedByEmployeeId;
    private int approvedByManagerId;
    private Date dateEffective;
    private Date dateCreated;
    private Date dateReviewed;
    private Date dateApproved;
    private int investorId;
    private int categoryId;
    private int subCategoryId;
    private int statusId;
    
    public String getTitle() {
        return title;
    }
    public void setTitle(String title) {
        this.title = title;
    }
    public int getClientId() {
        return clientId;
    }
    public void setClientId(int clientId) {
        this.clientId = clientId;
    }
    public int getCreatedByEmployeeId() {
        return createdByEmployeeId;
    }
    public void setCreatedByEmployeeId(int createdByEmployeeId) {
        this.createdByEmployeeId = createdByEmployeeId;
    }
    public int getReviewedByEmployeeId() {
        return reviewedByEmployeeId;
    }
    public void setReviewedByEmployeeId(int reviewedByEmployeeId) {
        this.reviewedByEmployeeId = reviewedByEmployeeId;
    }
    public int getApprovedByManagerId() {
        return approvedByManagerId;
    }
    public void setApprovedByManagerId(int approvedByManagerId) {
        this.approvedByManagerId = approvedByManagerId;
    }
  
    public Date getDateEffective() {
        return dateEffective;
    }

    public void setDateEffective(Date dateEffective) {
        this.dateEffective = dateEffective;
    }

    public Date getDateCreated() {
        return dateCreated;
    }
    public void setDateCreated(Date dateCreated) {
        this.dateCreated = dateCreated;
    }
    public Date getDateReviewed() {
        return dateReviewed;
    }
    public void setDateReviewed(Date dateReviewed) {
        this.dateReviewed = dateReviewed;
    }
    public Date getDateApproved() {
        return dateApproved;
    }
    public void setDateApproved(Date dateApproved) {
        this.dateApproved = dateApproved;
    }
    public int getInvestorId() {
        return investorId;
    }
    public void setInvestorId(int investorId) {
        this.investorId = investorId;
    }
    public int getCategoryId() {
        return categoryId;
    }
    public void setCategoryId(int categoryId) {
        this.categoryId = categoryId;
    }
    public int getSubCategoryId() {
        return subCategoryId;
    }

    public void setSubCategoryId(int subCategoryId) {
        this.subCategoryId = subCategoryId;
    }
    public int getStatusId() {
        return statusId;
    }
    public void setStatusId(int statusId) {
        this.statusId = statusId;
    }
    @Override
    public void setRevision(String s) {
        // downstream code does not like revision set to emtpy string, which Spring does when binding
        if (s != null && !s.isEmpty()) super.setRevision(s);
    }
    
    public boolean isNew() {
        return getId() == null;
    }
}

A Document Repository:

package com.nu.dms.couchdb.ektorp.dao;

import org.ektorp.CouchDbConnector;
import org.ektorp.support.CouchDbRepositorySupport;

public class CustomCouchDbRepositorySupport<T> extends CouchDbRepositorySupport<T> {


    protected CustomCouchDbRepositorySupport(Class<T> type, CouchDbConnector db) {
        super(type, db);
    }

    public CouchDbConnector getDb() {
        return super.db;
    }   
}

package com.nu.dms.couchdb.ektorp.dao;

import java.io.InputStream;

@Component
public class DocumentRepository  extends CustomCouchDbRepositorySupport<Document> {
    
    private static final Logger log = LoggerFactory.getLogger(DocumentRepository.class);
    
    @Autowired
    public DocumentRepository(@Qualifier("dmsDatabase") CouchDbConnector db) {
        super(Document.class, db);
        initStandardDesignDocument();
    }

    @GenerateView @Override
    public List<Document> getAll() {
        ViewQuery q = createQuery("all")
                        .includeDocs(true);
        return db.queryView(q, Document.class);
    }
    
    public Page<Document> getAll(PageRequest pr) {
        ViewQuery q = createQuery("all")
                        .includeDocs(true);
        return db.queryForPage(q, pr, Document.class);
    }
    
    @View( name = "tree", map = "classpath:/couchdb/tree_map.js", reduce = "classpath:/couchdb/tree_reduce.js")
    public InputStream getTree(String startKey, String endKey, int groupLevel) {
        ViewQuery q = createQuery("tree")
        .startKey(startKey)
        .endKey(endKey)
        .groupLevel(groupLevel)
        .group(true);
        InputStream is = db.queryForStream(q);
        return is;
    }

}

Map and Reduce javascript functions in src/main/resources/couchdb in other words in the classpath:

//by_categoryId_map.js
function(doc) { 
 if(doc.title && doc.categoryId) {
  emit(doc.categoryId, doc._id)
 } 
}

//by_categoryId_reduce.js
_count

//tree_map.js
function(doc) {
  var tokens = doc.dateEffective.split("-");
  var year = null;
  var month = null;
  if(tokens.length == 3) {
    year = tokens[0];
    month = tokens[1];
  }
  var key = [doc.clientId, doc.categoryId, doc.subCategoryId, year, month].concat(doc.title);
  var value = null;
  emit(key, value);
}

//tree_reduce.js
_count

In web.xml we need to allow flash because we will use the swfUpload to upload multiple files:

<url-pattern>*.swf</url-pattern>
</servlet-mapping>

Keeping things short I am not using a Service layer so I provide a unique Controller that allows creating a document including the attachment and metadata, uploading multiple documents (using swfupload flash) which follow a strong naming convention to produce all necessary metadata our of their names, viewing the documents as explained in part two in a tree view (using jquery.treeview.async.js)

package com.nu.web.controller.dms;

import java.io.ByteArrayOutputStream;
import java.io.InputStream;
import java.net.URLConnection;
import java.text.SimpleDateFormat;
import java.util.Date;
import java.util.Iterator;
import java.util.Locale;
import java.util.Map;
import java.util.Scanner;

import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;

import org.apache.commons.codec.binary.Base64;
import org.apache.commons.fileupload.FileItemFactory;
import org.apache.commons.fileupload.FileUploadException;
import org.apache.commons.fileupload.disk.DiskFileItemFactory;
import org.apache.commons.fileupload.servlet.ServletFileUpload;
import org.apache.commons.lang.StringUtils;
import org.apache.poi.util.IOUtils;
import org.codehaus.jackson.JsonNode;
import org.codehaus.jackson.map.ObjectMapper;
import org.codehaus.jackson.node.ObjectNode;
import org.ektorp.Attachment;
import org.ektorp.AttachmentInputStream;
import org.ektorp.PageRequest;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Controller;
import org.springframework.ui.Model;
import org.springframework.validation.BindingResult;
import org.springframework.validation.ObjectError;
import org.springframework.web.bind.WebDataBinder;
import org.springframework.web.bind.annotation.InitBinder;
import org.springframework.web.bind.annotation.ModelAttribute;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.multipart.MultipartFile;
import org.springframework.web.multipart.support.ByteArrayMultipartFileEditor;
import org.springframework.web.servlet.ModelAndView;

import com.nu.dms.couchdb.ektorp.dao.DocumentRepository;
import com.nu.dms.couchdb.ektorp.model.Document;
import com.nu.web.ControllerContext;
import com.nu.web.RootController;
import com.nu.web.WebConstants;
import com.nu.web.validator.BeanValidator;
import com.windriver.gson.extension.GeneralObjectDeserializer;

@Controller
@RequestMapping("/dms/document/*")
public class DocumentController extends RootController {
    private static final Logger log = LoggerFactory.getLogger(DocumentController.class);
    
    @Autowired
    DocumentRepository documentRepository;
    
    @Autowired
    private BeanValidator validator;
    
    private static final String LIST_PATH = "/dms/document/list";
    private static final String FORM_PATH = "/dms/document/form";
    private static final String TREE_PATH = "/dms/document/tree";
    public static final long UPLOAD_MAX_FILE_SIZE = 20 * 1024 * 1024; //10 MB
    public static final long UPLOAD_MAX_TOTAL_FILES_SIZE = 1 * 1024 * 1024 * 1024; //1 GB
    private static final int DOCUMENT_LEVEL = 5;
    
    @RequestMapping("/list")
    public ModelAndView list(   HttpServletRequest request, 
                                HttpServletResponse response, 
                                Model m, 
                                @RequestParam(value = "p", required = false) String pageLink) {
        ControllerContext ctx = new ControllerContext(request, response);
        init(ctx);

        if (!isValidCsrfToken(ctx)) {
            return getModelAndView(ctx, LIST_PATH);
        }
        
        PageRequest pr = pageLink != null ? PageRequest.fromLink(pageLink) : PageRequest.firstPage(5);
        m.addAttribute(documentRepository.getAll(pr));
        return getModelAndView(ctx, LIST_PATH);
    }
    
    @RequestMapping("/add")
    public ModelAndView add(HttpServletRequest request,
            HttpServletResponse response,
            @RequestParam(value = "attachment", required = false) MultipartFile multipartFile,
            @ModelAttribute("document") Document document,
            BindingResult result) {

        //Store constants for JSP
        request.setAttribute("UPLOAD_MAX_FILE_SIZE", UPLOAD_MAX_FILE_SIZE); 
        
        ControllerContext ctx = new ControllerContext(request, response);
        init(ctx);

        if (!isValidCsrfToken(ctx)) {
            return getModelAndView(ctx, FORM_PATH);
        }

        if (!isSubmission(ctx)) {
            return getModelAndView(ctx, FORM_PATH);
        } else {
            
            validator.validate(document, result);
            
            if (result.hasErrors()) {
                return getModelAndView(ctx, FORM_PATH);
            } else {
                if(multipartFile == null) {
                    result.addError(new ObjectError("document", getMessage("error.add", new String[] {"document"})));
                    return getModelAndView(ctx, FORM_PATH);
                }
                
                try {
                    String title = document.getTitle();
                    if(StringUtils.isEmpty(title)) throw new Exception("Empty title");
                    document.setId(title);
                    String contentType = multipartFile.getContentType();
                    String base64 = new String (Base64.encodeBase64(multipartFile.getBytes()));
                    if(StringUtils.isEmpty(base64)) throw new Exception("Empty attachment");
                    Attachment a = new Attachment(title, base64, contentType);
                    document.addInlineAttachment(a);
                } catch (Exception ex) {
                    result.addError(new ObjectError("attachmentError", getMessage("error.attachingDocument")));
                    log.error(null, ex);
                    return getModelAndView(ctx, FORM_PATH);
                }   
                
                try {
                    document.setDateCreated(new Date());
                    documentRepository.add(document);
                } catch (Exception ex) {
                    result.addError(new ObjectError("document", getMessage("error.add", new String[] {"document"})));
                    log.error(null, ex);
                    return getModelAndView(ctx, FORM_PATH);
                }
                return getModelAndView(ctx, LIST_PATH, true, true);
            }
        }
    }
    
    
    @RequestMapping("/{id}/edit")
    public ModelAndView edit(HttpServletRequest request,
            HttpServletResponse response,
            @ModelAttribute("document") Document document,
            BindingResult result,
            @PathVariable("id") String id,
            Model model) {

        ControllerContext ctx = new ControllerContext(request, response);
        init(ctx);

        if (!isValidCsrfToken(ctx)) {
            return getModelAndView(ctx, FORM_PATH);
        }

        try {
            Document storedDocument = documentRepository.get(id);
            if(storedDocument == null) {
                throw new Exception("No document found with id '" + id + "'");
            }
           
            if (!isSubmission(ctx)) {
                model.addAttribute("document", storedDocument);
                return getModelAndView(ctx, FORM_PATH);
            } else {
                validator.validate(document, result);
                
                if (result.hasErrors()) {
                    return getModelAndView(ctx, FORM_PATH);
                } else {
                    String title = document.getTitle();
                    if(StringUtils.isEmpty(title)) throw new Exception("Empty title");
                    document.setId(title);
                    document.setDateCreated(storedDocument.getDateCreated());
                    document.setRevision(storedDocument.getRevision());
                    documentRepository.update(document);
                    return getModelAndView(ctx, LIST_PATH, true, true);
                }
            }
        } catch (Exception ex) {
            result.addError(new ObjectError("document", getMessage("error.edit", new String[] {"document"}) + "." + ex.getMessage()));
            log.error(null, ex);
            return getModelAndView(ctx, FORM_PATH);
        }
    }
    
    
    @RequestMapping("/{id}/delete")
    public ModelAndView delete(HttpServletRequest request,
            HttpServletResponse response,
            @ModelAttribute("document") Document document,
            BindingResult result,
            @PathVariable("id") String id,
            Model model) {
        ControllerContext ctx = new ControllerContext(request, response);
        init(ctx);

        if (!isValidCsrfToken(ctx)) {
            return getModelAndView(ctx, LIST_PATH);
        }

        try {
            document = documentRepository.get(id);
            if(document == null) {
                throw new Exception("No document found with id '" + id + "'");
            }
            documentRepository.remove(document);
        } catch (Exception ex) {
            result.addError(new ObjectError("document", getMessage("error.add", new String[] {"document"})));
            log.error(null, ex);
            return getModelAndView(ctx, LIST_PATH);
        }
        return getModelAndView(ctx, LIST_PATH, true, true);
    }
    
    @RequestMapping("/{id}/show")
    public ModelAndView show(HttpServletRequest request,
            HttpServletResponse response,
            @ModelAttribute("document") Document document,
            BindingResult result,
            @PathVariable("id") String id,
            Model model) {
        ControllerContext ctx = new ControllerContext(request, response);
        init(ctx);

        if (!isValidCsrfToken(ctx)) {
            return getModelAndView(ctx, LIST_PATH);
        }

        try {
            document = documentRepository.get(id);
            if(document == null) {
                throw new Exception("No document found with id '" + id + "'");
            }
            Map<String, Attachment> attachments = document.getAttachments();
            if(attachments == null || attachments.size() == 0) {
                throw new Exception("No attachment found for id '" + id + "'");
            }
            for(Map.Entry<String, Attachment> entry : attachments.entrySet()) {
                String attachmentId = entry.getKey();
                Attachment attachment = entry.getValue();
                //long contentLength = attachment.getContentLength();
                String contentType = attachment.getContentType();
                AttachmentInputStream ais = documentRepository.getDb().getAttachment(id, attachmentId);
                response.setHeader("Content-Disposition", "attachment; filename=\"" + document.getTitle() + "\"");
                response.setContentType(contentType);
                final ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
                IOUtils.copy(ais, outputStream);
                render(response, outputStream);
            }
            return getModelAndView(ctx, LIST_PATH, true, true);
        } catch (Exception ex) {
            result.addError(new ObjectError("document", getMessage("error.internal", new String[] {"document"})));
            log.error(null, ex);
            return getModelAndView(ctx, LIST_PATH);
        }
        
    }
    
    @RequestMapping("/tree")
    public ModelAndView tree(HttpServletRequest request,
            HttpServletResponse response,
            @RequestParam(value = "root", required = false) String root) {
        ControllerContext ctx = new ControllerContext(request, response);
        init(ctx);

        if (!isValidCsrfToken(ctx)) {
            return getModelAndView(ctx, TREE_PATH);
        }
        if(root == null) {
            return getModelAndView(ctx, TREE_PATH);
        }
        try {
            Object objTree = getTreeObject(root);
            if(objTree == null) {
                ctx.setRequestAttribute("treeInfo", getMessage("noItemFound", new String[] {"record"}));
            }
            ctx.setRequestAttribute("tree", objTree);
            return getModelAndView(ctx, TREE_PATH);
        } catch (Exception ex) {
            ctx.setRequestViewAttribute("treeError", ex.getMessage());
            log.error(null, ex);
            return getModelAndView(ctx, TREE_PATH);
        }
        
    }
    
    /**
     * The needs for the current treeview plugin makes mandatory certain json structure so parsing the /document/tree service is not an option at the moment
     * @param request
     * @param response
     * @param root
     * @return
     */
    @RequestMapping("/ajaxTree")
    public ModelAndView ajaxTree(HttpServletRequest request,
            HttpServletResponse response,
            @RequestParam(value = "root", required = true) String root) {
        ControllerContext ctx = new ControllerContext(request, response);
        init(ctx);

        if (!isValidCsrfToken(ctx)) {
            return getModelAndView(ctx, LIST_PATH);
        }

        try {
            InputStream is = getTreeInputStream(root);
            ObjectMapper mapper = new ObjectMapper();
            JsonNode rootNode = mapper.readValue(is, JsonNode.class);
            JsonNode rowsNode = rootNode.get("rows");
            Iterator<JsonNode> iter = rowsNode.getElements();
            while (iter.hasNext()) {
                JsonNode row = iter.next();
                JsonNode keyNode = row.get("key");
                String[] key = mapper.readValue(keyNode, String[].class);
                if(key == null) {
                    continue;
                }
                String name = key[key.length - 1];
                String classes = null;
                if(key.length == DOCUMENT_LEVEL + 1) {
                    //Listing files
                    String extension = "unknownExtension";
                    String fileName = key[DOCUMENT_LEVEL];
                    String[] tokens = fileName.split("\\.");
                    if(tokens.length >= 2) {
                        extension = tokens[tokens.length - 1];
                    }
                    classes = "file " + extension;
                } else {
                    //Listing folders
                    classes = "folder";
                }
                
                ((ObjectNode)row).put("classes", classes);
                ((ObjectNode)row).put("name", name);
                boolean hasChildren = false;
                //To use the key as id for next request
                if(key.length != DOCUMENT_LEVEL + 1) {
                    int value = mapper.readValue(row.get("value"), Integer.class);
                    if(value > 0) {
                        hasChildren = true;
                    }
                } else {
                    ((ObjectNode)row).put("url", request.getContextPath() + "/dms/document/" + name + "/show?ctoken=" + ctx.getSessionAttribute(WebConstants.CSRF_TOKEN, ""));
                }
                ((ObjectNode)row).put("hasChildren", hasChildren);
                ((ObjectNode)row).put("id", keyNode.toString().replaceAll("[\\[\\]]", ""));
                
            }
            
            response.setContentType("application/json");
            final ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
            mapper.writeValue(outputStream, rowsNode);
            render(response, outputStream);
            return null;
        } catch (Exception ex) {
            log.error(null, ex);
            return null;
        }
        
    }
    
    private Object getTreeObject(String root) {
        InputStream is = getTreeInputStream(root);
        String tree = new Scanner(is).useDelimiter("\\A").next();
        Object objTree = GeneralObjectDeserializer.fromJson(tree);
        return objTree;
    }
    
    private InputStream getTreeInputStream(String root) {
        //Making it compatible with the jquery tree view in use by Portal in Liferay
        String endKey = null;
        if("0".equals(root)) {
            endKey = "[{}]";
        } else {
            endKey = "[" + root.substring(0,root.length()) + ",{}]";
        }
         
        String[] tokens = endKey.replaceAll("[\\[\\]]", "").split(",");
        int groupLevel = tokens.length;
        String startKey = "[1]";
        if(!"{}".equals(tokens[0])) {
            startKey = "[" + root + "]";
        }
        InputStream is = documentRepository.getTree(startKey, endKey, groupLevel);
        return is;
    }
    
    @InitBinder
    public void initBinder(WebDataBinder binder) {
        binder.registerCustomEditor(byte[].class, new ByteArrayMultipartFileEditor());
    }

    
    /**
     * To be consumed by swfupload.swf which is in charge of uploading multiple files
     * 
     * @param request
     * @param response
     * @return
     */
    @RequestMapping("/addBatch")
    public void addBatch(HttpServletRequest request,
            HttpServletResponse response,
            @RequestParam(value = "Filedata", required = false) MultipartFile multipartFile,
            @ModelAttribute("document") Document document,
            BindingResult result) {

        String message = "Completed";
        ControllerContext ctx = new ControllerContext(request, response);
        init(ctx);
        
        try {
            if (!isValidCsrfToken(ctx)) {
                message = error("Invalid session token");
            } else {
                uploadFile(request, multipartFile);
            }
            
        } catch (FileUploadException e) {
            log.error("FileUploadException:", e);
            message = error(e.getMessage());
        } catch (Exception e) {
            String errorMessage = e.toString();
            log.error("FileUploadException:", e);
            if (errorMessage == null) {
                errorMessage = "Internal Error. Please look at server logs for more detail";
            }
            message = error(errorMessage);
        }
        
        try {
            render(response, message.getBytes());
        } catch (Exception ex) {
            log.error(null, ex);
        }
    }
    
    /**
     * File must follow this convention: clientId_categoryId_subCategoryId_year_month_investorId.extension
     * @param req
     * @param multipartFile
     * @throws Exception
     */
    private void uploadFile(HttpServletRequest req, MultipartFile multipartFile) throws Exception {

            // Create a new file upload handler
            FileItemFactory factory = new DiskFileItemFactory();
            ServletFileUpload upload = new ServletFileUpload(factory);
            upload.setFileSizeMax(UPLOAD_MAX_FILE_SIZE);
            upload.setSizeMax(UPLOAD_MAX_TOTAL_FILES_SIZE);

            if(multipartFile == null) {
                throw new FileUploadException(getMessage("error.add", new String[] {"document"}));
            }
            
            String fileName = multipartFile.getOriginalFilename();
            String[] tokens = fileName.split("_");
            if(tokens.length != 6) {
                throw new Exception("Filename must have 6 tokens");
            }
            int clientId = Integer.parseInt(tokens[0]);
            int categoryId = Integer.parseInt(tokens[1]);
            int subCategoryId = Integer.parseInt(tokens[2]);
            String year = tokens[3];
            String month = tokens[4];
            //Use whole filename as title
            int investorId = Integer.parseInt(tokens[5].split("\\.")[0]);
            String title = fileName;
            //Using swfupload we almost always get "application/octet-stream"
            String contentType = multipartFile.getContentType();
            String guessedContentType = URLConnection.guessContentTypeFromName(fileName);
            if (guessedContentType != null) {
                contentType = guessedContentType;
            }
            String base64 = new String (Base64.encodeBase64(multipartFile.getBytes()));
            if(StringUtils.isEmpty(base64)) throw new Exception("Empty attachment");
            Attachment a = new Attachment(fileName, base64, contentType);
            Document document = new Document(title);
            document.setId(fileName);
            document.addInlineAttachment(a);
            document.setDateCreated(new Date());
            document.setClientId(clientId);
            document.setCategoryId(categoryId);
            document.setSubCategoryId(subCategoryId);
            document.setDateEffective(new SimpleDateFormat("yyyy-MM-dd", Locale.ENGLISH).parse(year + "-" + month + "-" + 1));
            document.setInvestorId(investorId);
            documentRepository.add(document);
            
            //Within Spring the below does not work
            /*
            List<FileItem> items = (List<FileItem>) upload.parseRequest(req);
            Iterator<FileItem> iter = items.iterator();
            while (iter.hasNext()) {
                FileItem item = iter.next();
                if (item.isFormField()) {
                    String name = item.getFieldName();
                    String value = item.getString();
                    log.debug("Form field " + name + " with value " + value);
                } else {
                    String fileName = item.getName();
                    String contentType = item.getContentType();
                    Document document = new Document(fileName);
                    String base64 = new String (Base64.encodeBase64(item.get()));
                    if(StringUtils.isEmpty(base64)) throw new Exception("Empty attachment");
                    Attachment a = new Attachment(fileName, base64, contentType);
                    document.addInlineAttachment(a);
                    document.setDateCreated(new Date());
                    documentRepository.add(document);
                }
            }
            */
    }
    /*
     * To format the message so sfwupload understands it
     */
    private String error(String error) {
        return "ERROR: " + error;
    }
    
    
}

Below are some screenshots of our simple user interface:

Why CouchDB? and not a SQL database?

Relational Database Management Systems (RDBMS) are good to store tabular data, enforce relationship, remove duplicated information, ensure data consistency and the list goes on and on. There is one thing though that makes relational databases not ideal for distributed computing and that is locking. The need for an alternative comes from impediments related to replication but also from storing hierarchical structures in RDBMS which is not natural. Finally it is difficult to manage inheritance and schema changes can easily become a big problem when the system grows and new simple necessities emerge from business requirements. RDBMS are also slow if you want an index for all fields in a table or multiple complex indexes. If your content management system has a need for distributed computing, fast storage and you can afford the compromise about losing the ability for easy normalization (for example your documents once created are stamped with metadata available at that time and which does not change over time)

CouchDB is a noSql type database which stores data structures as documents (JSON Strings). Schema less, based on b-tree and with no locking (but Multi Version Concurrency Control) design it is a really fast alternative when you look for a solution to store hierarchical non-strict schema data like for example storing web pages or binary files. All this robustness is exposed with a HTTP REST API where JSON is used to send messages to the server as well as receive messages from it. This makes it really attractive for those looking for lightweight solutions. One of the "trade-offs" in couchDB is that the only way to query CouchDB without creating indexes is a temporary View and that is not an option for production as the mapped result will not be stored in a B-Tree hitting performance in your server.

I have no other option than considering CouchDB the logical pick for my BHUB Document Management functionality. CouchDB provides fast access to data specified by stored keys. Using Map functions in your Views there is no limit on the efficiency you can get out of the fact that you can create at any point a key composed of several document fields.

CouchDB has been engineered with replication in mind and that means you get distributed computing on top of the advantages I already discussed above. You just run a command specifying URLs for the source and the destination server and the replication is done. By default latest changes will be favored and if there are conflicts you will get the differences so you can update with changes that resolve the changes. Think about any versioning system like subversion for a comparison on how it works. You can replicate in both directions of course.