Sunday, October 16, 2011

Document Management System with CouchDB - Third Part

This is the last part of my attempt to cover how to use CouchDB to build a DMS. We will be using Java and the Ektorp library for the implementation.

Using Ektorp

A good place to start is downloading the BlogPost application (org.ektorp.sample-1.7-project) that shows some of the most important concepts around the Ektorp API. Follow Ektorp Tutorial and of course do not miss the reference documentation.

I will show here the steps on how to make your existing spring project able to interact with CouchDB using the Ektorp project. The code presented here is an implementation of the ideas exposed in the second part of this project.

Here the dependencies for your project. Note that I am adding file upload dependencies because I am building a DMS and I will provide an upload module:
<!-- Ektorp for CouchDB -->
        <dependency>
            <groupId>org.ektorp</groupId>
            <artifactId>org.ektorp</artifactId>
            <version>${ektorp.version}</version>
            <exclusions>
             <exclusion>
              <artifactId>slf4j-api</artifactId>
              <groupId>org.slf4j</groupId>
             </exclusion>
            </exclusions>
        </dependency>
        <dependency>
            <groupId>org.ektorp</groupId>
            <artifactId>org.ektorp.spring</artifactId>
            <version>${ektorp.version}</version>
        </dependency>

        <!-- File Upload -->
        <dependency>
            <groupId>commons-fileupload</groupId>
            <artifactId>commons-fileupload</artifactId>
            <version>1.2.2</version>
        </dependency>

In spring application context xml (Note the upload component which is only needed because again we need it for the DMS upload functionality):
...
xmlns:util="http://www.springframework.org/schema/util"
xmlns:couchdb="http://www.ektorp.org/schema/couchdb"
...
http://www.springframework.org/schema/util http://www.springframework.org/schema/util/spring-util-3.0.xsd
http://www.ektorp.org/schema/couchdb http://www.ektorp.org/schema/couchdb/couchdb.xsd
...
<context:component-scan base-package="com.nu.dms.couchdb.ektorp.model"/>
<context:component-scan base-package="com.nu.dms.couchdb.ektorp.dao"/>
...
<util:properties id="couchdbProperties" location="classpath:/couchdb.properties"/>
<couchdb:instance id="dmsCouchdb" url="${couchdb.url}" properties="couchdbProperties" />
<couchdb:database id="dmsDatabase" name="${couchdb.db}" instance-ref="dmsCouchdb" />
...
<!-- File Upload -->
<bean id="multipartResolver" class="org.springframework.web.multipart.commons.CommonsMultipartResolver">
        <property name="maxUploadSize" value="2000000"/>
</bean>

In classpath environment properties file:
couchdb.url=http://localhost:5984
couchdb.db=dms4 #my database name

File classpath couchdb.properties file:
host=localhost
port=5984
maxConnections=20
connectionTimeout=1000
socketTimeout=10000
autoUpdateViewOnChange=true
caching=false

A Document POJO following the BlogPost POJO from the Ektorp example:
package com.nu.dms.couchdb.ektorp.model;

import org.ektorp.Attachment;
import org.ektorp.support.CouchDbDocument;

public class CustomCouchDbDocument extends CouchDbDocument {

    private static final long serialVersionUID = -9012014877538917152L;

    @Override
    public void addInlineAttachment(Attachment a) {
        super.addInlineAttachment(a);
    }   
}


package com.nu.dms.couchdb.ektorp.model;

import java.util.Date;

import javax.validation.constraints.NotNull;

import org.ektorp.support.TypeDiscriminator;

public class Document extends CustomCouchDbDocument {

    private static final long serialVersionUID = 59516215253102057L;
    
    public Document() {
        super();
    }
    
    public Document(String title) {
        this.title = title;
    }
    
    /**
     * @TypeDiscriminator is used to mark properties that makes this class's documents unique in the database. 
     */
    @TypeDiscriminator
    @NotNull
    private String title;
    
    private int clientId;
    private int createdByEmployeeId;
    private int reviewedByEmployeeId;
    private int approvedByManagerId;
    private Date dateEffective;
    private Date dateCreated;
    private Date dateReviewed;
    private Date dateApproved;
    private int investorId;
    private int categoryId;
    private int subCategoryId;
    private int statusId;
    
    public String getTitle() {
        return title;
    }
    public void setTitle(String title) {
        this.title = title;
    }
    public int getClientId() {
        return clientId;
    }
    public void setClientId(int clientId) {
        this.clientId = clientId;
    }
    public int getCreatedByEmployeeId() {
        return createdByEmployeeId;
    }
    public void setCreatedByEmployeeId(int createdByEmployeeId) {
        this.createdByEmployeeId = createdByEmployeeId;
    }
    public int getReviewedByEmployeeId() {
        return reviewedByEmployeeId;
    }
    public void setReviewedByEmployeeId(int reviewedByEmployeeId) {
        this.reviewedByEmployeeId = reviewedByEmployeeId;
    }
    public int getApprovedByManagerId() {
        return approvedByManagerId;
    }
    public void setApprovedByManagerId(int approvedByManagerId) {
        this.approvedByManagerId = approvedByManagerId;
    }
  
    public Date getDateEffective() {
        return dateEffective;
    }

    public void setDateEffective(Date dateEffective) {
        this.dateEffective = dateEffective;
    }

    public Date getDateCreated() {
        return dateCreated;
    }
    public void setDateCreated(Date dateCreated) {
        this.dateCreated = dateCreated;
    }
    public Date getDateReviewed() {
        return dateReviewed;
    }
    public void setDateReviewed(Date dateReviewed) {
        this.dateReviewed = dateReviewed;
    }
    public Date getDateApproved() {
        return dateApproved;
    }
    public void setDateApproved(Date dateApproved) {
        this.dateApproved = dateApproved;
    }
    public int getInvestorId() {
        return investorId;
    }
    public void setInvestorId(int investorId) {
        this.investorId = investorId;
    }
    public int getCategoryId() {
        return categoryId;
    }
    public void setCategoryId(int categoryId) {
        this.categoryId = categoryId;
    }
    public int getSubCategoryId() {
        return subCategoryId;
    }

    public void setSubCategoryId(int subCategoryId) {
        this.subCategoryId = subCategoryId;
    }
    public int getStatusId() {
        return statusId;
    }
    public void setStatusId(int statusId) {
        this.statusId = statusId;
    }
    @Override
    public void setRevision(String s) {
        // downstream code does not like revision set to emtpy string, which Spring does when binding
        if (s != null && !s.isEmpty()) super.setRevision(s);
    }
    
    public boolean isNew() {
        return getId() == null;
    }
}

A Document Repository:
package com.nu.dms.couchdb.ektorp.dao;

import org.ektorp.CouchDbConnector;
import org.ektorp.support.CouchDbRepositorySupport;

public class CustomCouchDbRepositorySupport<T> extends CouchDbRepositorySupport<T> {


    protected CustomCouchDbRepositorySupport(Class<T> type, CouchDbConnector db) {
        super(type, db);
    }

    public CouchDbConnector getDb() {
        return super.db;
    }   
}

package com.nu.dms.couchdb.ektorp.dao;

import java.io.InputStream;

@Component
public class DocumentRepository  extends CustomCouchDbRepositorySupport<Document> {
    
    private static final Logger log = LoggerFactory.getLogger(DocumentRepository.class);
    
    @Autowired
    public DocumentRepository(@Qualifier("dmsDatabase") CouchDbConnector db) {
        super(Document.class, db);
        initStandardDesignDocument();
    }

    @GenerateView @Override
    public List<Document> getAll() {
        ViewQuery q = createQuery("all")
                        .includeDocs(true);
        return db.queryView(q, Document.class);
    }
    
    public Page<Document> getAll(PageRequest pr) {
        ViewQuery q = createQuery("all")
                        .includeDocs(true);
        return db.queryForPage(q, pr, Document.class);
    }
    
    @View( name = "tree", map = "classpath:/couchdb/tree_map.js", reduce = "classpath:/couchdb/tree_reduce.js")
    public InputStream getTree(String startKey, String endKey, int groupLevel) {
        ViewQuery q = createQuery("tree")
        .startKey(startKey)
        .endKey(endKey)
        .groupLevel(groupLevel)
        .group(true);
        InputStream is = db.queryForStream(q);
        return is;
    }

}

Map and Reduce javascript functions in src/main/resources/couchdb in other words in the classpath:
//by_categoryId_map.js
function(doc) { 
 if(doc.title && doc.categoryId) {
  emit(doc.categoryId, doc._id)
 } 
}

//by_categoryId_reduce.js
_count

//tree_map.js
function(doc) {
  var tokens = doc.dateEffective.split("-");
  var year = null;
  var month = null;
  if(tokens.length == 3) {
    year = tokens[0];
    month = tokens[1];
  }
  var key = [doc.clientId, doc.categoryId, doc.subCategoryId, year, month].concat(doc.title);
  var value = null;
  emit(key, value);
}

//tree_reduce.js
_count

In web.xml we need to allow flash because we will use the swfUpload to upload multiple files:
<url-pattern>*.swf</url-pattern>
</servlet-mapping>

Keeping things short I am not using a Service layer so I provide a unique Controller that allows creating a document including the attachment and metadata, uploading multiple documents (using swfupload flash) which follow a strong naming convention to produce all necessary metadata our of their names, viewing the documents as explained in part two in a tree view (using jquery.treeview.async.js)

package com.nu.web.controller.dms;

import java.io.ByteArrayOutputStream;
import java.io.InputStream;
import java.net.URLConnection;
import java.text.SimpleDateFormat;
import java.util.Date;
import java.util.Iterator;
import java.util.Locale;
import java.util.Map;
import java.util.Scanner;

import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;

import org.apache.commons.codec.binary.Base64;
import org.apache.commons.fileupload.FileItemFactory;
import org.apache.commons.fileupload.FileUploadException;
import org.apache.commons.fileupload.disk.DiskFileItemFactory;
import org.apache.commons.fileupload.servlet.ServletFileUpload;
import org.apache.commons.lang.StringUtils;
import org.apache.poi.util.IOUtils;
import org.codehaus.jackson.JsonNode;
import org.codehaus.jackson.map.ObjectMapper;
import org.codehaus.jackson.node.ObjectNode;
import org.ektorp.Attachment;
import org.ektorp.AttachmentInputStream;
import org.ektorp.PageRequest;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Controller;
import org.springframework.ui.Model;
import org.springframework.validation.BindingResult;
import org.springframework.validation.ObjectError;
import org.springframework.web.bind.WebDataBinder;
import org.springframework.web.bind.annotation.InitBinder;
import org.springframework.web.bind.annotation.ModelAttribute;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.multipart.MultipartFile;
import org.springframework.web.multipart.support.ByteArrayMultipartFileEditor;
import org.springframework.web.servlet.ModelAndView;

import com.nu.dms.couchdb.ektorp.dao.DocumentRepository;
import com.nu.dms.couchdb.ektorp.model.Document;
import com.nu.web.ControllerContext;
import com.nu.web.RootController;
import com.nu.web.WebConstants;
import com.nu.web.validator.BeanValidator;
import com.windriver.gson.extension.GeneralObjectDeserializer;

@Controller
@RequestMapping("/dms/document/*")
public class DocumentController extends RootController {
    private static final Logger log = LoggerFactory.getLogger(DocumentController.class);
    
    @Autowired
    DocumentRepository documentRepository;
    
    @Autowired
    private BeanValidator validator;
    
    private static final String LIST_PATH = "/dms/document/list";
    private static final String FORM_PATH = "/dms/document/form";
    private static final String TREE_PATH = "/dms/document/tree";
    public static final long UPLOAD_MAX_FILE_SIZE = 20 * 1024 * 1024; //10 MB
    public static final long UPLOAD_MAX_TOTAL_FILES_SIZE = 1 * 1024 * 1024 * 1024; //1 GB
    private static final int DOCUMENT_LEVEL = 5;
    
    @RequestMapping("/list")
    public ModelAndView list(   HttpServletRequest request, 
                                HttpServletResponse response, 
                                Model m, 
                                @RequestParam(value = "p", required = false) String pageLink) {
        ControllerContext ctx = new ControllerContext(request, response);
        init(ctx);

        if (!isValidCsrfToken(ctx)) {
            return getModelAndView(ctx, LIST_PATH);
        }
        
        PageRequest pr = pageLink != null ? PageRequest.fromLink(pageLink) : PageRequest.firstPage(5);
        m.addAttribute(documentRepository.getAll(pr));
        return getModelAndView(ctx, LIST_PATH);
    }
    
    @RequestMapping("/add")
    public ModelAndView add(HttpServletRequest request,
            HttpServletResponse response,
            @RequestParam(value = "attachment", required = false) MultipartFile multipartFile,
            @ModelAttribute("document") Document document,
            BindingResult result) {

        //Store constants for JSP
        request.setAttribute("UPLOAD_MAX_FILE_SIZE", UPLOAD_MAX_FILE_SIZE); 
        
        ControllerContext ctx = new ControllerContext(request, response);
        init(ctx);

        if (!isValidCsrfToken(ctx)) {
            return getModelAndView(ctx, FORM_PATH);
        }

        if (!isSubmission(ctx)) {
            return getModelAndView(ctx, FORM_PATH);
        } else {
            
            validator.validate(document, result);
            
            if (result.hasErrors()) {
                return getModelAndView(ctx, FORM_PATH);
            } else {
                if(multipartFile == null) {
                    result.addError(new ObjectError("document", getMessage("error.add", new String[] {"document"})));
                    return getModelAndView(ctx, FORM_PATH);
                }
                
                try {
                    String title = document.getTitle();
                    if(StringUtils.isEmpty(title)) throw new Exception("Empty title");
                    document.setId(title);
                    String contentType = multipartFile.getContentType();
                    String base64 = new String (Base64.encodeBase64(multipartFile.getBytes()));
                    if(StringUtils.isEmpty(base64)) throw new Exception("Empty attachment");
                    Attachment a = new Attachment(title, base64, contentType);
                    document.addInlineAttachment(a);
                } catch (Exception ex) {
                    result.addError(new ObjectError("attachmentError", getMessage("error.attachingDocument")));
                    log.error(null, ex);
                    return getModelAndView(ctx, FORM_PATH);
                }   
                
                try {
                    document.setDateCreated(new Date());
                    documentRepository.add(document);
                } catch (Exception ex) {
                    result.addError(new ObjectError("document", getMessage("error.add", new String[] {"document"})));
                    log.error(null, ex);
                    return getModelAndView(ctx, FORM_PATH);
                }
                return getModelAndView(ctx, LIST_PATH, true, true);
            }
        }
    }
    
    
    @RequestMapping("/{id}/edit")
    public ModelAndView edit(HttpServletRequest request,
            HttpServletResponse response,
            @ModelAttribute("document") Document document,
            BindingResult result,
            @PathVariable("id") String id,
            Model model) {

        ControllerContext ctx = new ControllerContext(request, response);
        init(ctx);

        if (!isValidCsrfToken(ctx)) {
            return getModelAndView(ctx, FORM_PATH);
        }

        try {
            Document storedDocument = documentRepository.get(id);
            if(storedDocument == null) {
                throw new Exception("No document found with id '" + id + "'");
            }
           
            if (!isSubmission(ctx)) {
                model.addAttribute("document", storedDocument);
                return getModelAndView(ctx, FORM_PATH);
            } else {
                validator.validate(document, result);
                
                if (result.hasErrors()) {
                    return getModelAndView(ctx, FORM_PATH);
                } else {
                    String title = document.getTitle();
                    if(StringUtils.isEmpty(title)) throw new Exception("Empty title");
                    document.setId(title);
                    document.setDateCreated(storedDocument.getDateCreated());
                    document.setRevision(storedDocument.getRevision());
                    documentRepository.update(document);
                    return getModelAndView(ctx, LIST_PATH, true, true);
                }
            }
        } catch (Exception ex) {
            result.addError(new ObjectError("document", getMessage("error.edit", new String[] {"document"}) + "." + ex.getMessage()));
            log.error(null, ex);
            return getModelAndView(ctx, FORM_PATH);
        }
    }
    
    
    @RequestMapping("/{id}/delete")
    public ModelAndView delete(HttpServletRequest request,
            HttpServletResponse response,
            @ModelAttribute("document") Document document,
            BindingResult result,
            @PathVariable("id") String id,
            Model model) {
        ControllerContext ctx = new ControllerContext(request, response);
        init(ctx);

        if (!isValidCsrfToken(ctx)) {
            return getModelAndView(ctx, LIST_PATH);
        }

        try {
            document = documentRepository.get(id);
            if(document == null) {
                throw new Exception("No document found with id '" + id + "'");
            }
            documentRepository.remove(document);
        } catch (Exception ex) {
            result.addError(new ObjectError("document", getMessage("error.add", new String[] {"document"})));
            log.error(null, ex);
            return getModelAndView(ctx, LIST_PATH);
        }
        return getModelAndView(ctx, LIST_PATH, true, true);
    }
    
    @RequestMapping("/{id}/show")
    public ModelAndView show(HttpServletRequest request,
            HttpServletResponse response,
            @ModelAttribute("document") Document document,
            BindingResult result,
            @PathVariable("id") String id,
            Model model) {
        ControllerContext ctx = new ControllerContext(request, response);
        init(ctx);

        if (!isValidCsrfToken(ctx)) {
            return getModelAndView(ctx, LIST_PATH);
        }

        try {
            document = documentRepository.get(id);
            if(document == null) {
                throw new Exception("No document found with id '" + id + "'");
            }
            Map<String, Attachment> attachments = document.getAttachments();
            if(attachments == null || attachments.size() == 0) {
                throw new Exception("No attachment found for id '" + id + "'");
            }
            for(Map.Entry<String, Attachment> entry : attachments.entrySet()) {
                String attachmentId = entry.getKey();
                Attachment attachment = entry.getValue();
                //long contentLength = attachment.getContentLength();
                String contentType = attachment.getContentType();
                AttachmentInputStream ais = documentRepository.getDb().getAttachment(id, attachmentId);
                response.setHeader("Content-Disposition", "attachment; filename=\"" + document.getTitle() + "\"");
                response.setContentType(contentType);
                final ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
                IOUtils.copy(ais, outputStream);
                render(response, outputStream);
            }
            return getModelAndView(ctx, LIST_PATH, true, true);
        } catch (Exception ex) {
            result.addError(new ObjectError("document", getMessage("error.internal", new String[] {"document"})));
            log.error(null, ex);
            return getModelAndView(ctx, LIST_PATH);
        }
        
    }
    
    @RequestMapping("/tree")
    public ModelAndView tree(HttpServletRequest request,
            HttpServletResponse response,
            @RequestParam(value = "root", required = false) String root) {
        ControllerContext ctx = new ControllerContext(request, response);
        init(ctx);

        if (!isValidCsrfToken(ctx)) {
            return getModelAndView(ctx, TREE_PATH);
        }
        if(root == null) {
            return getModelAndView(ctx, TREE_PATH);
        }
        try {
            Object objTree = getTreeObject(root);
            if(objTree == null) {
                ctx.setRequestAttribute("treeInfo", getMessage("noItemFound", new String[] {"record"}));
            }
            ctx.setRequestAttribute("tree", objTree);
            return getModelAndView(ctx, TREE_PATH);
        } catch (Exception ex) {
            ctx.setRequestViewAttribute("treeError", ex.getMessage());
            log.error(null, ex);
            return getModelAndView(ctx, TREE_PATH);
        }
        
    }
    
    /**
     * The needs for the current treeview plugin makes mandatory certain json structure so parsing the /document/tree service is not an option at the moment
     * @param request
     * @param response
     * @param root
     * @return
     */
    @RequestMapping("/ajaxTree")
    public ModelAndView ajaxTree(HttpServletRequest request,
            HttpServletResponse response,
            @RequestParam(value = "root", required = true) String root) {
        ControllerContext ctx = new ControllerContext(request, response);
        init(ctx);

        if (!isValidCsrfToken(ctx)) {
            return getModelAndView(ctx, LIST_PATH);
        }

        try {
            InputStream is = getTreeInputStream(root);
            ObjectMapper mapper = new ObjectMapper();
            JsonNode rootNode = mapper.readValue(is, JsonNode.class);
            JsonNode rowsNode = rootNode.get("rows");
            Iterator<JsonNode> iter = rowsNode.getElements();
            while (iter.hasNext()) {
                JsonNode row = iter.next();
                JsonNode keyNode = row.get("key");
                String[] key = mapper.readValue(keyNode, String[].class);
                if(key == null) {
                    continue;
                }
                String name = key[key.length - 1];
                String classes = null;
                if(key.length == DOCUMENT_LEVEL + 1) {
                    //Listing files
                    String extension = "unknownExtension";
                    String fileName = key[DOCUMENT_LEVEL];
                    String[] tokens = fileName.split("\\.");
                    if(tokens.length >= 2) {
                        extension = tokens[tokens.length - 1];
                    }
                    classes = "file " + extension;
                } else {
                    //Listing folders
                    classes = "folder";
                }
                
                ((ObjectNode)row).put("classes", classes);
                ((ObjectNode)row).put("name", name);
                boolean hasChildren = false;
                //To use the key as id for next request
                if(key.length != DOCUMENT_LEVEL + 1) {
                    int value = mapper.readValue(row.get("value"), Integer.class);
                    if(value > 0) {
                        hasChildren = true;
                    }
                } else {
                    ((ObjectNode)row).put("url", request.getContextPath() + "/dms/document/" + name + "/show?ctoken=" + ctx.getSessionAttribute(WebConstants.CSRF_TOKEN, ""));
                }
                ((ObjectNode)row).put("hasChildren", hasChildren);
                ((ObjectNode)row).put("id", keyNode.toString().replaceAll("[\\[\\]]", ""));
                
            }
            
            response.setContentType("application/json");
            final ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
            mapper.writeValue(outputStream, rowsNode);
            render(response, outputStream);
            return null;
        } catch (Exception ex) {
            log.error(null, ex);
            return null;
        }
        
    }
    
    private Object getTreeObject(String root) {
        InputStream is = getTreeInputStream(root);
        String tree = new Scanner(is).useDelimiter("\\A").next();
        Object objTree = GeneralObjectDeserializer.fromJson(tree);
        return objTree;
    }
    
    private InputStream getTreeInputStream(String root) {
        //Making it compatible with the jquery tree view in use by Portal in Liferay
        String endKey = null;
        if("0".equals(root)) {
            endKey = "[{}]";
        } else {
            endKey = "[" + root.substring(0,root.length()) + ",{}]";
        }
         
        String[] tokens = endKey.replaceAll("[\\[\\]]", "").split(",");
        int groupLevel = tokens.length;
        String startKey = "[1]";
        if(!"{}".equals(tokens[0])) {
            startKey = "[" + root + "]";
        }
        InputStream is = documentRepository.getTree(startKey, endKey, groupLevel);
        return is;
    }
    
    @InitBinder
    public void initBinder(WebDataBinder binder) {
        binder.registerCustomEditor(byte[].class, new ByteArrayMultipartFileEditor());
    }

    
    /**
     * To be consumed by swfupload.swf which is in charge of uploading multiple files
     * 
     * @param request
     * @param response
     * @return
     */
    @RequestMapping("/addBatch")
    public void addBatch(HttpServletRequest request,
            HttpServletResponse response,
            @RequestParam(value = "Filedata", required = false) MultipartFile multipartFile,
            @ModelAttribute("document") Document document,
            BindingResult result) {

        String message = "Completed";
        ControllerContext ctx = new ControllerContext(request, response);
        init(ctx);
        
        try {
            if (!isValidCsrfToken(ctx)) {
                message = error("Invalid session token");
            } else {
                uploadFile(request, multipartFile);
            }
            
        } catch (FileUploadException e) {
            log.error("FileUploadException:", e);
            message = error(e.getMessage());
        } catch (Exception e) {
            String errorMessage = e.toString();
            log.error("FileUploadException:", e);
            if (errorMessage == null) {
                errorMessage = "Internal Error. Please look at server logs for more detail";
            }
            message = error(errorMessage);
        }
        
        try {
            render(response, message.getBytes());
        } catch (Exception ex) {
            log.error(null, ex);
        }
    }
    
    /**
     * File must follow this convention: clientId_categoryId_subCategoryId_year_month_investorId.extension
     * @param req
     * @param multipartFile
     * @throws Exception
     */
    private void uploadFile(HttpServletRequest req, MultipartFile multipartFile) throws Exception {

            // Create a new file upload handler
            FileItemFactory factory = new DiskFileItemFactory();
            ServletFileUpload upload = new ServletFileUpload(factory);
            upload.setFileSizeMax(UPLOAD_MAX_FILE_SIZE);
            upload.setSizeMax(UPLOAD_MAX_TOTAL_FILES_SIZE);

            if(multipartFile == null) {
                throw new FileUploadException(getMessage("error.add", new String[] {"document"}));
            }
            
            String fileName = multipartFile.getOriginalFilename();
            String[] tokens = fileName.split("_");
            if(tokens.length != 6) {
                throw new Exception("Filename must have 6 tokens");
            }
            int clientId = Integer.parseInt(tokens[0]);
            int categoryId = Integer.parseInt(tokens[1]);
            int subCategoryId = Integer.parseInt(tokens[2]);
            String year = tokens[3];
            String month = tokens[4];
            //Use whole filename as title
            int investorId = Integer.parseInt(tokens[5].split("\\.")[0]);
            String title = fileName;
            //Using swfupload we almost always get "application/octet-stream"
            String contentType = multipartFile.getContentType();
            String guessedContentType = URLConnection.guessContentTypeFromName(fileName);
            if (guessedContentType != null) {
                contentType = guessedContentType;
            }
            String base64 = new String (Base64.encodeBase64(multipartFile.getBytes()));
            if(StringUtils.isEmpty(base64)) throw new Exception("Empty attachment");
            Attachment a = new Attachment(fileName, base64, contentType);
            Document document = new Document(title);
            document.setId(fileName);
            document.addInlineAttachment(a);
            document.setDateCreated(new Date());
            document.setClientId(clientId);
            document.setCategoryId(categoryId);
            document.setSubCategoryId(subCategoryId);
            document.setDateEffective(new SimpleDateFormat("yyyy-MM-dd", Locale.ENGLISH).parse(year + "-" + month + "-" + 1));
            document.setInvestorId(investorId);
            documentRepository.add(document);
            
            //Within Spring the below does not work
            /*
            List<FileItem> items = (List<FileItem>) upload.parseRequest(req);
            Iterator<FileItem> iter = items.iterator();
            while (iter.hasNext()) {
                FileItem item = iter.next();
                if (item.isFormField()) {
                    String name = item.getFieldName();
                    String value = item.getString();
                    log.debug("Form field " + name + " with value " + value);
                } else {
                    String fileName = item.getName();
                    String contentType = item.getContentType();
                    Document document = new Document(fileName);
                    String base64 = new String (Base64.encodeBase64(item.get()));
                    if(StringUtils.isEmpty(base64)) throw new Exception("Empty attachment");
                    Attachment a = new Attachment(fileName, base64, contentType);
                    document.addInlineAttachment(a);
                    document.setDateCreated(new Date());
                    documentRepository.add(document);
                }
            }
            */
    }
    /*
     * To format the message so sfwupload understands it
     */
    private String error(String error) {
        return "ERROR: " + error;
    }
    
    
}

Below are some screenshots of our simple user interface:



Why CouchDB? and not a SQL database?

Relational Database Management Systems (RDBMS) are good to store tabular data, enforce relationship, remove duplicated information, ensure data consistency and the list goes on and on. There is one thing though that makes relational databases not ideal for distributed computing and that is locking. The need for an alternative comes from impediments related to replication but also from storing hierarchical structures in RDBMS which is not natural. Finally it is difficult to manage inheritance and schema changes can easily become a big problem when the system grows and new simple necessities emerge from business requirements. RDBMS are also slow if you want an index for all fields in a table or multiple complex indexes. If your content management system has a need for distributed computing, fast storage and you can afford the compromise about losing the ability for easy normalization (for example your documents once created are stamped with metadata available at that time and which does not change over time)

CouchDB is a noSql type database which stores data structures as documents (JSON Strings). Schema less, based on b-tree and with no locking (but Multi Version Concurrency Control) design it is a really fast alternative when you look for a solution to store hierarchical non-strict schema data like for example storing web pages or binary files. All this robustness is exposed with a HTTP REST API where JSON is used to send messages to the server as well as receive messages from it. This makes it really attractive for those looking for lightweight solutions. One of the "trade-offs" in couchDB is that the only way to query CouchDB without creating indexes is a temporary View and that is not an option for production as the mapped result will not be stored in a B-Tree hitting performance in your server.

I have no other option than considering CouchDB the logical pick for my BHUB Document Management functionality. CouchDB provides fast access to data specified by stored keys. Using Map functions in your Views there is no limit on the efficiency you can get out of the fact that you can create at any point a key composed of several document fields.

CouchDB has been engineered with replication in mind and that means you get distributed computing on top of the advantages I already discussed above. You just run a command specifying URLs for the source and the destination server and the replication is done. By default latest changes will be favored and if there are conflicts you will get the differences so you can update with changes that resolve the changes. Think about any versioning system like subversion for a comparison on how it works. You can replicate in both directions of course.

6 comments:

Amit Parashar said...

Thanks. This was helpful.

Amit Parashar said...
This comment has been removed by a blog administrator.
benze said...

Do you have the code somewhere downloadable by any chance? On github or elsewhere?

Thanks,

Eric

Nestor Urquiza said...

No code for download but definitely working code posted here which was the starting point for a DMS still up and running today. Cheers,
- Nestor

softeng said...

Hi, your article is very inetresting
and very helpfull. but i can't run it! due to missing classes:
com.nu.web.ControllerContext
com.nu.web.RootController
com.nu.web.WebConstants
com.nu.web.validator.BeanValidator
may be you send thems to me! or the zip of the project in all.
Best regards,
Frederic.

Nestor Urquiza said...

Hi softeng,

The whole code cannot be shared but it should be not difficult to adapt the above to run without the missing classes.

Best,
- Nestor

Followers