Thinking In Software

Portable PGP: Encryption with Bouncy Castle

2012-01-25T14:26:00.000-08:00

I came across the PPGP project Today and I thought about sharing it not only because it is a nice tool that runs in any OS (Java) which allows to encrypt/decrypt files and text using PGP, sign the result, generate keys but also because being Open Source Java Project the whole source code is available so you could use proven code that actually works to perform PGP operations in your own project.

Below is simple code almost entirely extracted from this project that just encrypts a file with a PGP public key.

package com.nestorurquiza.utils;

import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.InputStream;
import java.io.OutputStream;
import java.security.SecureRandom;
import java.security.Security;
import java.util.Date;
import java.util.Iterator;

import org.bouncycastle.bcpg.ArmoredOutputStream;
import org.bouncycastle.jce.provider.BouncyCastleProvider;
import org.bouncycastle.openpgp.PGPCompressedData;
import org.bouncycastle.openpgp.PGPCompressedDataGenerator;
import org.bouncycastle.openpgp.PGPEncryptedData;
import org.bouncycastle.openpgp.PGPEncryptedDataGenerator;
import org.bouncycastle.openpgp.PGPLiteralData;
import org.bouncycastle.openpgp.PGPLiteralDataGenerator;
import org.bouncycastle.openpgp.PGPPublicKey;
import org.bouncycastle.openpgp.PGPPublicKeyRing;
import org.bouncycastle.openpgp.PGPPublicKeyRingCollection;
import org.bouncycastle.openpgp.PGPUtil;

/**
 * 
 * Code in part extracted from http://sourceforge.net/projects/ppgp/ 
 * A nice GUI to generate your own keys, encrypt and sign documents using PGP
 * 
 * @author nestor
 *
 */
public class PgpUtils {
    private final static int BUFFER_SIZE = 1 << 16;

    private static PGPPublicKey readPublicKeyFromCollection2(InputStream in) throws Exception {
        in = PGPUtil.getDecoderStream(in);
        PGPPublicKeyRing pkRing = null;
        PGPPublicKeyRingCollection pkCol = new PGPPublicKeyRingCollection(in);
        Iterator it = pkCol.getKeyRings();
        while (it.hasNext()) {
             pkRing = (PGPPublicKeyRing) it.next();
             Iterator pkIt = pkRing.getPublicKeys();
             while (pkIt.hasNext()) {
                     PGPPublicKey key = (PGPPublicKey) pkIt.next();
                     if (key.isEncryptionKey())
                             return key;
             }
        }
        return null;
    }
    

    /**
     * Encrypts a file using a public key
     * 
     * @param decryptedFilePath
     * @param encryptedFilePath
     * @param encKeyPath
     * @param armor
     * @param withIntegrityCheck
     * @throws Exception
     */
    public static void encryptFile(String decryptedFilePath,
            String encryptedFilePath,
            String encKeyPath,
            boolean armor,
            boolean withIntegrityCheck)            
            throws Exception{

        OutputStream out = new FileOutputStream(encryptedFilePath);
        FileInputStream pubKey = new FileInputStream(encKeyPath);
        PGPPublicKey encKey = readPublicKeyFromCollection2(pubKey);
        Security.addProvider(new BouncyCastleProvider());
        
        if (armor) 
            out = new ArmoredOutputStream(out);
            
        // Init encrypted data generator
        PGPEncryptedDataGenerator encryptedDataGenerator =
                new PGPEncryptedDataGenerator(PGPEncryptedData.CAST5, withIntegrityCheck, new SecureRandom(),"BC");
        
        encryptedDataGenerator.addMethod(encKey);
        
        
        OutputStream encryptedOut = encryptedDataGenerator.open(out, new byte[BUFFER_SIZE]);

        // Init compression  
        PGPCompressedDataGenerator compressedDataGenerator = new PGPCompressedDataGenerator(PGPCompressedData.ZIP);
        OutputStream compressedOut = compressedDataGenerator.open(encryptedOut);  

        PGPLiteralDataGenerator literalDataGenerator = new PGPLiteralDataGenerator();
        OutputStream literalOut = literalDataGenerator.open(compressedOut, PGPLiteralData.BINARY, decryptedFilePath, new Date(), new byte[BUFFER_SIZE]);
        FileInputStream inputFileStream = new FileInputStream(decryptedFilePath);
        byte[] buf = new byte[BUFFER_SIZE];  
        int len;
        while((len = inputFileStream.read(buf))>0){
            literalOut.write(buf,0,len);
        }
         
        literalOut.close();
        literalDataGenerator.close();
        
        compressedOut.close();
        compressedDataGenerator.close();
        encryptedOut.close();
        encryptedDataGenerator.close();
        inputFileStream.close();
        out.close();
        
    }
}

The Unit test showing how to use it:

package com.nestorurquiza.utils;

import java.io.IOException;

import org.junit.Test;
import org.xml.sax.SAXException;

public class PgpUtilsTest {
    @Ignore
    @Test    
    public void encryptTest() throws IOException, Exception, SAXException {
        String publicKeyFilePath = "/Users/nestor/Downloads/somePublicKey.pgp";
        String decryptedFilePath = "/Users/nestor/Downloads/someFile.csv";
        String encryptedFilePath = "/Users/nestor/Downloads/someFile.csv.enc";
        
        PgpUtils.encryptFile(decryptedFilePath, encryptedFilePath, publicKeyFilePath, false, true);
        
    }
}

Commons VFS: SFTP from Java the simple way

2012-01-24T17:52:00.000-08:00

I have always used jsch library to SCP (or what is commonly known as SFTP) files in remote servers from Java.

I was tempted to use my old code when I decide to look around for something cleaner just to came across this great post about sftp using Apache VFS. I made few changes to be able to sftp any input stream as a file to a remote SFTP site which I am sharing below.

Dependencies:


                org.apache.commons
                commons-vfs2
                2.0
            
            
                com.jcraft
                jsch
                0.1.45

The upload:

package com.nestorurquiza.utils;

package com.nestorurquiza.utils;

import java.io.InputStream;
import java.io.OutputStream;

import org.apache.commons.io.IOUtils;
import org.apache.commons.vfs2.FileObject;
import org.apache.commons.vfs2.FileSystemException;
import org.apache.commons.vfs2.FileSystemOptions;
import org.apache.commons.vfs2.Selectors;
import org.apache.commons.vfs2.impl.StandardFileSystemManager;
import org.apache.commons.vfs2.provider.sftp.SftpFileSystemConfigBuilder;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
/**
 * All credits for http://www.memorylack.com/2011/06/apache-commons-vfs-for-sftp.html
 * 
 * nestoru - 2012/01/24: Small changes:
 *   1. Using logging instead of System.out.println()
 *   2. The initial directory is not the user home directory 
 *   3. No additional slashes when adding remoteFilePath.
 *   4. Using InputStream instead of path to a local file
 * 
 * If you need extra functionality check out the url above
 * 
 * 
 *
 */
public class SftpUtils {
    private final static Logger log = LoggerFactory.getLogger(SftpUtils.class);
    
    public static void upload(String hostName, String username,
            String password, InputStream localInputStream, String localInputStreamName, String remoteFilePath) {
        
        StandardFileSystemManager manager = new StandardFileSystemManager();

        try {
            manager.init();

            FileSystemOptions fileSystemOptions = createDefaultOptions();
            
            // Create local file object
            //FileObject localFile = manager.resolveFile(f.getAbsolutePath());
            FileObject localFile = manager.resolveFile("ram://path/needed/" + localInputStreamName);
            localFile.createFile();
            OutputStream localOutputStream = localFile.getContent().getOutputStream();
            IOUtils.copy(localInputStream, localOutputStream);
            localOutputStream.flush();
            
            // Create remote file object
            FileObject remoteFile = manager.resolveFile(
                    createConnectionString(hostName, username, password,
                            remoteFilePath), fileSystemOptions);

            // Copy local file to sftp server
            remoteFile.copyFrom(localFile, Selectors.SELECT_SELF);

            log.debug("File upload success");
        } catch (Exception e) {
            throw new RuntimeException(e);
        } finally {
            manager.close();
        }
    }
    
    public static String createConnectionString(String hostName,
            String username, String password, String remoteFilePath) {
        // result: "sftp://user:123456@domainname.com/resume.pdf
        return "sftp://" + username + ":" + password + "@" + hostName
                + remoteFilePath;
    }

    public static FileSystemOptions createDefaultOptions() throws FileSystemException {
        // Create SFTP options
        FileSystemOptions opts = new FileSystemOptions();
        
        // SSH Key checking
        SftpFileSystemConfigBuilder.getInstance().setStrictHostKeyChecking(
                opts, "no");
        
        // Root directory set to user home
        SftpFileSystemConfigBuilder.getInstance().setUserDirIsRoot(opts, false);
        
        // Timeout is count by Milliseconds
        SftpFileSystemConfigBuilder.getInstance().setTimeout(opts, 10000);
        
        return opts;
    }

}

Then a quick unit test that I comment out Just because I am not interested really in regression test but rather just testing if the method works as expected (I do not like main() methods inside the classes)

package com.nestorurquiza.utils;

import java.io.File;
import java.io.FileInputStream;
import java.io.IOException;

import org.junit.Test;
import org.xml.sax.SAXException;


public class SftpUtilsTest {
    //@Ignore
    @Test    
    public void uploadTest() throws IOException, Exception, SAXException {
        String hostName = "bhubint.nestorurquiza.com";
        String userame = "user";
        String password = "pass";
        String localFilePath = "/tmp/test.txt";
        String remoteFilePath = "/home/report/report/test.txt";
        //SftpUtils.upload(hostName, userame, password, localFilePath, remoteFilePath);
        File f = new File(localFilePath);
        if (!f.exists())
            throw new RuntimeException("Error. Local file not found");
        FileInputStream fileInputStream = new FileInputStream(f);
        SftpUtils.upload(hostName, userame, password, fileInputStream, "test.txt", remoteFilePath);
    }
}

maven downloads the pom.xml but not the jar file

2012-01-24T10:11:00.000-08:00

Today I spent sometime troubleshooting one particular library I uploaded to Artifactory and which was not downloaded by Maven.

Maven was downloading just the pom file instead of downloading the jar file.

I had to reupload the jar and while doing it edit the pom file to include the pacjaging=jar

jar

Complex development environments with VirtualBox

2012-01-17T19:25:00.000-08:00

Having a local environment independent of any other existing environments has a big advantage: Development is not stopped when other environments are down and it even stops the "need" to hit production just because it is the only environment currently available. But perhaps the biggest advantage of all is to be able to work literally anywhere even if you have no internet connection at all.

Virtualization can help with this and here is how to host your whole environment locally in your machine using VirtualBox.

In an Enterprise you will find Solaris, Windows, OSX, Unix and what not. I personally use a Mac Book Pro with VirtualBox. In VirtualBox I can run as many VMs as I need to recreate the surrounded environment.

Ideally we should have an environment following the next features:

The guest should have connectivity to internet taking advantage of your network configuration in the host
The guest should communicate to the host even if no internet connection is available
The guest should automatically work independent what network you are using, from work, from home or even on the road with no local network at all

Virtual box will allow these features if we just add a first network interface as NAT (for internet) and the second as host-only (for connectivity with the box)

We will show this process using a virtualized instance of an Advent Geneva Server (A Solaris Box with some proprietary Accounting Software). It is useful to have this server locally to be able to mess with RSL reports for example without affecting other members of the dev or operations team.

In the host find the vboxnet0 interface to be sure you use an IP in the same network later on in the guest:

$ ifconfig -a
...
vboxnet0: flags=8843 mtu 1500
 ether 0a:00:27:00:00:00 
 inet 192.168.56.1 netmask 0xffffff00 broadcast 192.168.56.255
...

In the Solaris guest ...

Configure first interface (NAT) to use DHCP

$ touch /etc/dhcp.e1000g0
$ rm /etc/hostname.e1000g0
$ touch /etc/hostname.e1000g0
$ ifconfig e1000g0 plumb
$ ifconfig e1000g0 up
$ ifconfig e1000g0 dhcp start

And second interface (host-only) to use a static IP

$ vi /etc/hosts
...
192.168.56.2 genevadev loghost
...
$ rm /etc/dhcp.e1000g1
$ vi /etc/hostname.e1000g0
192.168.56.2
$ ifconfig e1000g1 plumb
$ ifconfig e1000g1 genevadev up
...

Be sure to define DNS available from any network to connect to the internet (only to be used in the case you do need internet in the guest) If you really need to access local resources you might want to add local DNS as well.

$ vi /etc/resolv.conf
nameserver 72.45.32.34
nameserver 72.45.32.37

Make sure you see both interfaces (the NAT and host-only). They should show up as e1000g0 and e1000g1. The first with an internal 10.0.xxx.xxx IP and the second with the static IP you picked (for this example 192.168.56.2)

$ ifconfig -a

Clearly the static IP must be in the same subnetwork (192.168.56.xxx).

You should be able to connect to the internet from the guest and at the same time ssh into it using its 192.168.56.2 IP. Do not forget to add in your host the new created server resolution:

$ vi /etc/hosts
genevadev 192.168.56.2

The question about sharing this environment or building it from scratch is one interesting topic. An even more interesting is left as homework and it deals with devops. In reality regardless the path you follow you should automate your actions and there is where things like http://vagrantup.com/ and puppet/chef can help you further.

Remove duplicate classes the agile way: Maven Duplicate Finder Plugin

2012-01-12T08:16:00.000-08:00

Without discussing the alternatives I think a Java project cannot live without checking for dependency conflicts. The conflicts can be buried in external jar files so the check must be performed by fully qualified name on absolutely all classes included in the project.

Of course such a check should stop the build until a resolution.

The Maven Duplicate Finder Plugin does exactly that.

Just clone the repo, build the maven project and upload the plugin jar file to your Maven repository. Then configure your application to use the plugin:

...
   
              com.ning.maven.plugins
              maven-duplicate-finder-plugin
              
                false
              
              
                
                  verify
                  
                    check
                  
                
              
            
...

If you run it with the failBuildInCaseOfConflict flag turned off as above you can get the messages as WARNING without stopping the project from building.

[INFO] [duplicate-finder:check {execution: default}]
[INFO] Checking compile classpath
[WARNING] Found duplicate classes in [commons-beanutils:commons-beanutils:1.8.3,commons-collections:commons-collections:3.2.1] :
[WARNING]   org.apache.commons.collections.ArrayStack
[WARNING]   org.apache.commons.collections.Buffer
[WARNING]   org.apache.commons.collections.BufferUnderflowException
[WARNING]   org.apache.commons.collections.FastHashMap
[WARNING] Found duplicate classes in [net.objectlab.kit:datecalc-common:1.2.0,net.objectlab.kit:datecalc-jdk:1.2.0] :
[WARNING]   net.objectlab.kit.datecalc.common.AbstractDateCalculator
[WARNING]   net.objectlab.kit.datecalc.common.AbstractIMMDateCalculator
[WARNING]   net.objectlab.kit.datecalc.common.AbstractKitCalculatorsFactory
[WARNING]   net.objectlab.kit.datecalc.common.DateCalculator
[WARNING]   net.objectlab.kit.datecalc.common.DefaultHolidayCalendar
[WARNING]   net.objectlab.kit.datecalc.common.ExcelDateUtil
[WARNING]   net.objectlab.kit.datecalc.common.HolidayCalendar
[WARNING]   net.objectlab.kit.datecalc.common.HolidayHandler
[WARNING]   net.objectlab.kit.datecalc.common.HolidayHandlerType
[WARNING]   net.objectlab.kit.datecalc.common.IMMDateCalculator
[WARNING]   net.objectlab.kit.datecalc.common.IMMPeriod
[WARNING]   net.objectlab.kit.datecalc.common.ImmutableHolidayCalendar
[WARNING]   net.objectlab.kit.datecalc.common.KitCalculatorsFactory
[WARNING]   net.objectlab.kit.datecalc.common.PeriodCountBasis
[WARNING]   net.objectlab.kit.datecalc.common.PeriodCountCalculator
[WARNING]   net.objectlab.kit.datecalc.common.StandardTenor
[WARNING]   net.objectlab.kit.datecalc.common.Tenor
[WARNING]   net.objectlab.kit.datecalc.common.TenorCode
[WARNING]   net.objectlab.kit.datecalc.common.Utils
[WARNING]   net.objectlab.kit.datecalc.common.WorkingWeek
[WARNING] Found duplicate resources in [net.sf.jasperreports:jasperreports:4.1.3,net.sf.jasperreports:jasperreports-fonts:4.1.3] :
[WARNING]   jasperreports_extension.properties
[WARNING] Found duplicate resources in [org.bouncycastle:bcmail-jdk14:1.38,org.bouncycastle:bcprov-jdk14:1.38,org.bouncycastle:bctsp-jdk14:1.38] :
[WARNING]   META-INF/BCKEY.DSA
[WARNING]   META-INF/BCKEY.SF
[INFO] Checking runtime classpath
[WARNING] Found duplicate classes in [commons-beanutils:commons-beanutils:1.8.3,commons-collections:commons-collections:3.2.1] :
[WARNING]   org.apache.commons.collections.ArrayStack
[WARNING]   org.apache.commons.collections.Buffer
[WARNING]   org.apache.commons.collections.BufferUnderflowException
[WARNING]   org.apache.commons.collections.FastHashMap
[WARNING] Found duplicate classes in [net.objectlab.kit:datecalc-common:1.2.0,net.objectlab.kit:datecalc-jdk:1.2.0] :
[WARNING]   net.objectlab.kit.datecalc.common.AbstractDateCalculator
[WARNING]   net.objectlab.kit.datecalc.common.AbstractIMMDateCalculator
[WARNING]   net.objectlab.kit.datecalc.common.AbstractKitCalculatorsFactory
[WARNING]   net.objectlab.kit.datecalc.common.DateCalculator
[WARNING]   net.objectlab.kit.datecalc.common.DefaultHolidayCalendar
[WARNING]   net.objectlab.kit.datecalc.common.ExcelDateUtil
[WARNING]   net.objectlab.kit.datecalc.common.HolidayCalendar
[WARNING]   net.objectlab.kit.datecalc.common.HolidayHandler
[WARNING]   net.objectlab.kit.datecalc.common.HolidayHandlerType
[WARNING]   net.objectlab.kit.datecalc.common.IMMDateCalculator
[WARNING]   net.objectlab.kit.datecalc.common.IMMPeriod
[WARNING]   net.objectlab.kit.datecalc.common.ImmutableHolidayCalendar
[WARNING]   net.objectlab.kit.datecalc.common.KitCalculatorsFactory
[WARNING]   net.objectlab.kit.datecalc.common.PeriodCountBasis
[WARNING]   net.objectlab.kit.datecalc.common.PeriodCountCalculator
[WARNING]   net.objectlab.kit.datecalc.common.StandardTenor
[WARNING]   net.objectlab.kit.datecalc.common.Tenor
[WARNING]   net.objectlab.kit.datecalc.common.TenorCode
[WARNING]   net.objectlab.kit.datecalc.common.Utils
[WARNING]   net.objectlab.kit.datecalc.common.WorkingWeek
[WARNING] Found duplicate resources in [net.sf.jasperreports:jasperreports:4.1.3,net.sf.jasperreports:jasperreports-fonts:4.1.3] :
[WARNING]   jasperreports_extension.properties
[WARNING] Found duplicate resources in [org.bouncycastle:bcmail-jdk14:1.38,org.bouncycastle:bcprov-jdk14:1.38,org.bouncycastle:bctsp-jdk14:1.38] :
[WARNING]   META-INF/BCKEY.DSA
[WARNING]   META-INF/BCKEY.SF
[INFO] Checking test classpath
[WARNING] Found duplicate classes in [commons-beanutils:commons-beanutils:1.8.3,commons-collections:commons-collections:3.2.1] :
[WARNING]   org.apache.commons.collections.ArrayStack
[WARNING]   org.apache.commons.collections.Buffer
[WARNING]   org.apache.commons.collections.BufferUnderflowException
[WARNING]   org.apache.commons.collections.FastHashMap
[WARNING] Found duplicate classes in [net.objectlab.kit:datecalc-common:1.2.0,net.objectlab.kit:datecalc-jdk:1.2.0] :
[WARNING]   net.objectlab.kit.datecalc.common.AbstractDateCalculator
[WARNING]   net.objectlab.kit.datecalc.common.AbstractIMMDateCalculator
[WARNING]   net.objectlab.kit.datecalc.common.AbstractKitCalculatorsFactory
[WARNING]   net.objectlab.kit.datecalc.common.DateCalculator
[WARNING]   net.objectlab.kit.datecalc.common.DefaultHolidayCalendar
[WARNING]   net.objectlab.kit.datecalc.common.ExcelDateUtil
[WARNING]   net.objectlab.kit.datecalc.common.HolidayCalendar
[WARNING]   net.objectlab.kit.datecalc.common.HolidayHandler
[WARNING]   net.objectlab.kit.datecalc.common.HolidayHandlerType
[WARNING]   net.objectlab.kit.datecalc.common.IMMDateCalculator
[WARNING]   net.objectlab.kit.datecalc.common.IMMPeriod
[WARNING]   net.objectlab.kit.datecalc.common.ImmutableHolidayCalendar
[WARNING]   net.objectlab.kit.datecalc.common.KitCalculatorsFactory
[WARNING]   net.objectlab.kit.datecalc.common.PeriodCountBasis
[WARNING]   net.objectlab.kit.datecalc.common.PeriodCountCalculator
[WARNING]   net.objectlab.kit.datecalc.common.StandardTenor
[WARNING]   net.objectlab.kit.datecalc.common.Tenor
[WARNING]   net.objectlab.kit.datecalc.common.TenorCode
[WARNING]   net.objectlab.kit.datecalc.common.Utils
[WARNING]   net.objectlab.kit.datecalc.common.WorkingWeek
[WARNING] Found duplicate resources in [net.sf.jasperreports:jasperreports:4.1.3,net.sf.jasperreports:jasperreports-fonts:4.1.3] :
[WARNING]   jasperreports_extension.properties
[WARNING] Found duplicate resources in [org.bouncycastle:bcmail-jdk14:1.38,org.bouncycastle:bcprov-jdk14:1.38,org.bouncycastle:bctsp-jdk14:1.38] :
[WARNING]   META-INF/BCKEY.DSA
[WARNING]   META-INF/BCKEY.SF

The above is the result of a project that I cleaned up so actually the errors you are seeing are the result of "accepted" conflicts. Since we are aware of them then we proceed (at our own risk) to exclude them from being checked as exceptions while setting failBuildInCaseOfConflict to true. That way no developer will be able to include a conflicting dependency without resolving it first.

...
  
              com.ning.maven.plugins
              maven-duplicate-finder-plugin
              
                true
                
                  META-INF.*
                
                
                  
                    
                      
                        commons-beanutils
                        commons-beanutils
                      
                      
                        commons-collections
                        commons-collections
                      
                    
                    
                      org.apache.commons.collections
                    
                  
                  
                    
                      
                        net.objectlab.kit
                        datecalc-common
                      
                      
                        net.objectlab.kit
                        datecalc-jdk
                      
                    
                    
                      net.objectlab.kit.datecalc.common
                    
                  
                  
                    
                      
                        net.sf.jasperreports
                        jasperreports
                      
                      
                        net.sf.jasperreports
                        jasperreports-fonts
                      
                    
                    
                      jasperreports_extension.properties
                    
                  
                
              
              
                
                  verify
                  
                    check
                  
                
              
     
...

Note how I have globally ignored any resource inside META-INF and I have ignored the properties file explicitly from the artifacts where they were found.

Also I have not used the classes node which would allow me to specify the individual classes that I approve to ignore. This can be bad but I feel lazy today ;-)

Finally I did not specify the version number for the conflicting dependencies. This can be bad as well.

Forgot Password with LDAP, CAS and Spring: Credentials expired

2012-01-12T06:39:00.000-08:00

I have already documented the steps I followed to integrate existing Spring applications that happen to be using LDAP with CAS including how to support "Forgot Password" functionality.

Some CAS client applications are currently using certain complicated logic to handle Terms and Conditions, Forgot Password and other features through custom filters and they were failing with a 401 ERROR "User credentials have expired". Basically the user hits the Forgot Password link in CAS which shows a CAS Client page where the email is given and after submission the temporary password is mailed. Any attempt to move to any page from that moment on results in a 401.

The problem here is that the default implementation when using Spring CAS client is to test the forcePasswordChange LDAP attribute and it currently throws an Exception rather than easily allowing to hook into the current implementation let us say with a property like "IGNORE_CREDENTIALS_EXPIRATION". As the project is already handling password expiration it does not need that extra check from Spring CAS Client so after some research I had to hack into Spring code.

I found that a possible solution is to use a CustomCasAuthenticationProvider to override CasAuthenticationProvider#authenticateNow() however I had to copy the whole code from the latter because of scope problems. Basically the original class is not designed to be inherited (at least for this feature). Below is the only method I changed in the class:

    private CasAuthenticationToken authenticateNow(final Authentication authentication) throws AuthenticationException {
        try {
            final Assertion assertion = this.ticketValidator.validate(authentication.getCredentials().toString(), serviceProperties.getService());
            final UserDetails userDetails = loadUserByAssertion(assertion);
            try {
                userDetailsChecker.check(userDetails);
            } catch (CredentialsExpiredException e) {
                if (log.isDebugEnabled()) {
                    log.debug("Credentials expired but we handle that case outside of the CAS logic.");
                }
            }
            return new CasAuthenticationToken(this.key, userDetails, authentication.getCredentials(), userDetails.getAuthorities(), userDetails, assertion);
        } catch (final TicketValidationException e) {
            throw new BadCredentialsException(e.getMessage(), e);
        }
    }

SSO with LDAP Spring and CAS

2012-01-09T18:51:00.000-08:00

This is a step by step documentation on how to setup a CAS Server and use Single Sign On (SSO) plus Single Sign Off from Spring client applications using LDAP to store authentication and authorization information. CAS support can be enabled or disabled commenting sections of the code.

Our Use Case for this exercise is to have a centralized server that handles user login and logout operations allowing the user to sign in just once and enjoy multiple applications thanks to that centralization.

We then achieve Federation because thanks to SSO via CAS we obtain Authentication while we pull the attributes from LDAP to achieve Authorization. At the end the user is trusted after the first login and the attributes can be obtained from any application.

Before we review Server and Client configuration be sure you understand CAS needs SSL so in order to correctly follow this post you will need to have the CAS Server and two Spring client applications serving on https. SSL needs an IP per domain and if you are running OSX by default you have only one loopback IP (127.0.0.1). This means you will need to add other two:

sudo ifconfig lo0 alias 127.0.0.2 up
sudo ifconfig lo0 alias 127.0.0.3 up

Then set in /etc/hosts a domain per loopback IP:

$ vi /etc/hosts
127.0.0.2   bhubdev.nestorurquiza.com
127.0.0.2   portaldev.nestorurquiza.com
127.0.0.3   casdev.nestorurquiza.com

You should be able to get the certificate from each URL and assert they are correct, for example:

$ echo | openssl s_client -connect bhubdev.nestorurquiza.com:443 2>/dev/null | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > ~/Downloads/bhubdev.nestorurquiza.com.cer
$ echo | openssl s_client -connect portaldev.nestorurquiza.com:443 2>/dev/null | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > ~/Downloads/portaldev.nestorurquiza.com.cer
$ echo | openssl s_client -connect casdev.nestorurquiza.com:443 2>/dev/null | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > ~/Downloads/casldev.nestorurquiza.com.cer
$ openssl x509 -text -in ~/Downloads/bhubdev.nestorurquiza.com.cer
$ openssl x509 -text -in ~/Downloads/portaldev.nestorurquiza.com.cer
$ openssl x509 -text -in ~/Downloads/casdev.nestorurquiza.com.cer

Once you are sure the certificates coming from the different URLs are correct (they belong to the specific domain) then you should add them to your keystore:

$ sudo keytool -delete -alias bhubdev.nestorurquiza.com -keystore  /Library/Java/Home/lib/security/cacerts
$ sudo keytool -delete -alias portaldev.nestorurquiza.com -keystore  /Library/Java/Home/lib/security/cacerts
$ sudo keytool -delete -alias casdev.nestorurquiza.com -keystore  /Library/Java/Home/lib/security/cacerts
$ sudo keytool -import -alias bhubdev.nestorurquiza.com -keystore  /Library/Java/Home/lib/security/cacerts -file ~/Downloads/bhubdev.nestorurquiza.com.cer
$ sudo keytool -import -alias portaldev.nestorurquiza.com -keystore  /Library/Java/Home/lib/security/cacerts -file ~/Downloads/portaldev.nestorurquiza.com.cer
$ sudo keytool -import -alias casdev.nestorurquiza.com -keystore  /Library/Java/Home/lib/security/cacerts -file ~/Downloads/casdev.nestorurquiza.com.cer

Of course after you do all the above tomcat must be restarted and if you use Apache for virtual hosting like I explain below you owe to restart it as well.

CAS Server installation and configuration

Download CAS Server from http://www.jasig.org/cas and get cas.war deployed in your server. At this point http://localhost:8080/cas should prompt for credentials. I commonly use apache in front of tomcat so below are the steps using virtual hosts as I have documented before:

$ mkdir /Users/nestor/Downloads/apache-tomcat-7.0.22/cas
$ mkdir /Users/nestor/Downloads/apache-tomcat-7.0.22/conf/Catalina/casdev.nestorurquiza.com
$ cp /Users/nestor/Downloads/cas-server-3.4.11/modules/cas-server-webapp-3.4.11.war /Users/nestor/Downloads/apache-tomcat-7.0.22/cas/ROOT.war

Prepare CAS to use LDAP authentication. This is about copying two files to the WEB-INF/lib directory and editing a configuration file in that same directory:

$ cp /Users/nestor/Downloads/cas-server-3.4.11/modules/cas-server-support-ldap-3.4.11.jar /Users/nestor/Downloads/apache-tomcat-7.0.22/cas/ROOT/WEB-INF/lib 
$ cp /Users/nestor/Downloads/spring-ldap-1.3.0.RELEASE/dist/spring-ldap-1.3.0.RELEASE-all.jar /Users/nestor/Downloads/apache-tomcat-7.0.22/cas/ROOT/WEB-INF/lib
$ vi /Users/nestor/Downloads/apache-tomcat-7.0.22/cas/ROOT/WEB-INF/deployerConfigContext.xml
…
   class="org.jasig.cas.authentication.AuthenticationManagerImpl">
 ...
  
   
    
         p:httpClient-ref="httpClient" />
    
    
    
    
      
      
      
    
   
  
Get spring-security source code

…

   class="org.springframework.ldap.core.support.LdapContextSource">







ldaps://ldapint.nestorurquiza.com:10636





















…

While at this point you should be able to login from the CAS landing page using your LDAP credentials CAS needs to run from an SSL URL in order to be used as Single Sign On (SSO). I use Apache and mod-jk with Tomcat so my SSL configuration is achieved on the Apache side. Here are the steps to generate the self signed certificate and configure Apache virtual host in OSX to serve the CAS service:

$ cd ~/Downloads/
$ sudo mkdir /private/etc/apache2/certs/
$ export DOMAIN=casdev.nestorurquiza.com
$ openssl genrsa -des3 -out ${DOMAIN}.key 1024
$ openssl req -new -key ${DOMAIN}.key -out  ${DOMAIN}.csr
$ openssl x509 -req -days 365 -in ${DOMAIN}.csr -signkey ${DOMAIN}.key -out ${DOMAIN}.crt
$ openssl rsa -in ${DOMAIN}.key -out ${DOMAIN}.key
$ sudo cp ${DOMAIN}.key /private/etc/apache2/certs/
$ sudo cp ${DOMAIN}.crt /private/etc/apache2/certs/
$ sudo vi /private/etc/apache2/extra/httpd-vhosts-ssl.conf
Listen 443
AddType application/x-x509-ca-cert .crt
AddType application/x-pkcs7-crl    .crl
SSLPassPhraseDialog  builtin
SSLSessionCache        "shmcb:/private/var/run/ssl_scache(512000)"
SSLSessionCacheTimeout  300
SSLMutex  "file:/private/var/run/ssl_mutex"


DocumentRoot "/usr/docs/casdev.nestorurquiza.com"
ServerName casdev.nestorurquiza.com:443
ServerAdmin nurquiza@nestorurquiza.com
ErrorLog "/private/var/log/apache2/casdev.nestorurquiza.com.log"
TransferLog "/private/var/log/apache2/casdev.nestorurquiza.com.log"
SSLEngine on
SSLCipherSuite ALL:!ADH:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP:+eNULL
SSLCertificateFile "/private/etc/apache2/certs/casdev.nestorurquiza.com.crt"
SSLCertificateKeyFile "/private/etc/apache2/certs/casdev.nestorurquiza.com.key"

    Options +FollowSymLinks
    AllowOverride All
    Order deny,allow
    Allow from all

JkMount "/*" nestorurquiza-app1

$ sudo vi /etc/apache2/httpd.conf
...
LoadModule ssl_module libexec/apache2/mod_ssl.so
...
Include /private/etc/apache2/extra/httpd-vhosts-ssl.conf
…
$ sudo apachectl configtest
$ sudo apachectl graceful

Logout and login from SSL. Note that you do not get any alerts stating SSO won't work:
```
https://casdev.nestorurquiza.com/logout
https://casdev.nestorurquiza.com
```
Change the look and feel for the CAS login page editing the file /WEB-INF/view/jsp/default/ui/casLogoutView.jsp and dependencies like the logo from /css/cas.css #header background url and #header h1#app-name background color
Further customize the login page for example as I needed to provide a forgot password functionality I came up with the below implementation in casLogoutView.jsp:
```
  
   
    
   
                 <a href="<c:out value="${schema}://${domain}/forgotPassword"/>">Forgot Password</a>
         
```

Import the client website certificate in the CAS server for example:

$ echo | openssl s_client -connect bhubdev.nestorurquiza.com:443 2>/dev/null | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > ~/Downloads/bhubdev.nestorurquiza.com.cer
$ sudo keytool -import -alias bhubdev.nestorurquiza.com -keystore  /Library/Java/Home/lib/security/cacerts -file ~/Downloads/bhubdev.nestorurquiza.com.cer

Spring CAS Client

Import the CAS certificate in the client

$ echo | openssl s_client -connect casdev.nestorurquiza.com:443 2>/dev/null | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > ~/Downloads/casdev.nestorurquiza.com.cer
$ sudo keytool -import -alias casdev.nestorurquiza.com -keystore  /Library/Java/Home/lib/security/cacerts -file ~/Downloads/casdev.nestorurquiza.com.cer

Declare some properties in the classpath of the client application:

...
#
# CAS
#
cas.loginURL=https://casdev.nestorurquiza.com/login
cas.homeURL=https://casdev.nestorurquiza.com
cas.serviceURL=https://bhubdev.nestorurquiza.com/j_spring_cas_security_check
cas.logoutURL=https://casdev.nestorurquiza.com/logout?service=https%3A%2F%2Fbhubdev.nestorurquiza.com%2Flogout
…

Then configure the client to perform the Single Sign On and Single Sign Off through CAS. Here is the Spring security configuration for such a use case:

...
    
    
    
    
    
        
        
        
…
 
        
        
 
        
…
    
        
        
        
        
                   
      
…
    
    
        
        
    
    
              class="org.springframework.security.cas.web.CasAuthenticationFilter">
      
    
    
         class="org.springframework.security.web.access.ExceptionTranslationFilter">
      
      
    
    
             class="org.springframework.security.web.access.AccessDeniedHandlerImpl">
      
    
    
            class="org.springframework.security.cas.web.CasAuthenticationEntryPoint">
      
      
    
    
          class="org.springframework.security.cas.authentication.CasAuthenticationProvider">

A simple JSP View for the client application Logout Controller

<%@ include file="/WEB-INF/jsp/includes.jsp" %>
<%@ include file="/WEB-INF/jsp/header.jsp" %>

<%@ include file="/WEB-INF/jsp/footer.jsp" %>

The LogoutController

package com.nestorurquiza.web;

import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;

import org.springframework.security.core.context.SecurityContextHolder;
import org.springframework.stereotype.Controller;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.servlet.ModelAndView;

@Controller
public class LogoutController extends RootController {

    @RequestMapping("/logout")
    public ModelAndView welcomeHandler(HttpServletRequest request,
            HttpServletResponse response) {
        SecurityContextHolder.getContext().setAuthentication(null);
        
        ...
        
        return getModelAndView(ctx, "logout", null);
    }
}

The logout link in JSP:

At least one dependency is needed in the client application:


            org.springframework.security
            spring-security-cas-client
            ${spring-security.version}
            compile

At this point hitting any client application URL (for example https://bhubdev.nestorurquiza.com/client/home) will redirect the user to the CAS login URL passing the service parameter which contains the encoded servciceURL (https://casdev.nestorurquiza.com/login?service=https%3A%2F%2Fbhubdev.nestorurquiza.com%2Fj_spring_cas_security_check). When the user provides valid credentials CAS will redirect her back to the j_spring_cas_security_check client resource and Spring will take care to pull from the session the originally requested URL. When the user clicks on the logout URL (https://casdev.nestorurquiza.com/logout?service=https%3A%2F%2Fbhubdev.nestorurquiza.com%2Flogout) the CAS server will invalidate the security token and the user will return to a URL where a custom Logout Controller invalidates the Spring Security Context. At this point any attempt to access a protected resource will result in a redirect to the CAS login page as explained.

Probably you will need in your custom code to do extra logic with the saved request (the original requested URL). In Spring 3.0.3 at least you use the below snippet to gain access to it:
```
SavedRequest savedRequest = new DefaultSavedRequest(request, new PortResolverImpl());
if( savedRequest != null ) {
  String redirectUrl = savedRequest.getRedirectUrl();
```

Clearly you can proceed with the final step from the Server and the steps from the Client to configure extra applications (CAS Services). When you logout from one application or directly from CAS interface you will be logged out from all applications that are integrated with CAS (Single Sign Out or Single Sign Off functionality)

Other than a little issue I reported here the whole integration went smooth. CAS is well supported in Spring and both Single Sign On and Single Sign Off will work out of the box.

By default CAS is shipped with a 4 hours expiration ticket expiration policy. This can be configured as explained in the documentation https://wiki.jasig.org/display/CASUM/Ticket+Expiration+Policy

Access a properties file from JSP/JSTL EL

2012-01-05T09:37:00.000-08:00

Sometimes you need to show information in a JSP from a properties file. Typical scenario is configuration data.

With spring you could do:

Then from a Listener or even from a Controller you could Autowire and inject the variable as a context or request attribute respectively:

...
@Autowired
public Properties applicationProperties;
...
request.setAttribute("applicationProperties", applicationProperties);

Suppose app.properties has this content:

...
bhub.environment=LOCAL
...

Then from JSP/JSTL you can use it as:

Subversive Authentication error svn OPTIONS 403 Forbidden

2012-01-05T09:05:00.000-08:00

Suddenly Eclipse started to complaint with an error like the below when I tried to update certain local svn copies:

Authentication error.
svn: OPTIONS of '/repos/reporting': 403 Forbidden (http://subversion.nestorurquiza.com)

While someone could be tempted to grant more permisions on the SVN server the reality is that from command line I did not get the same experience. Everything was working just fine from there.

To correct this I used the Subversive Update site from Eclipse Help/Install Software Menu. After an Eclipse restart I had to pick the latest 1.6 svn connector (not native) for it to work:

http://download.eclipse.org/technology/subversive/0.7/update-site/ - [required] Subversive plug-in

Two hours with Rasmus Lerdorf, the creator of PHP

2011-12-15T18:21:00.000-08:00

I had the pleasure to meet Rasmus Lerdorf, the creator of PHP in a two hour session organized by South Florida PHP Users Group in Nova South Eastern University.

As one of the earliest PHP adopters I share a lot of feelings for the language and it was great to see so many young people in the room together with not so young like me ;-) and some older than me. It is great to see a language going on for so long. I would say it was a celebration of more than 15 years of PHP coding.

Even though I have been working more with python, ruby and Java for the last 5 years PHP always comes back: I need to patch a built in PHP software, configure it or even get a quick snippet and modify it to get my work done.

So with a lot of quotations here is an extract of the introductory ideas this MIT awarded "top 100 innovators in the world" shared with us.

The idea behind PHP was always "the need for speed and performance". Rasmus "did not worry about correctness but about resolving the problem". He is a guy that "does not love to program" and that is precisely why you want to use PHP because instead he actually "loves to solve problems".

"You have to add non scalable features in order to make php not scalable". The "documentation was always done before the function was implemented". He constantly "insisted in having lot of examples as part of the documentation"

PHP was born while "thinking in the ecosystem". Basically shared hosting demanded control on memory and CPU (QoS) to adequate existing servers to the needs of multiple applications.

PHP "runs crappy code extremely fast" so it is better for startups than other languages like Java. It simply scales well.

I asked the question about why he considered Java not scalable and he clarified:
-It does scale but it comes out of the box with features that allow the requests to be sticked in memory, a thing php by default does not allow. Also when it comes to go beyond one server it is then more difficult in java just because of this.

Of course one can argue sticky sessions are enough for most projects. Yes the user session will expire and then you will need to re login but things like remember-me will help (not that I like it really). Not big deal considering how many times this actually happens, I would say nobody complained to me on the clusters I have setup for which in really weird circumstances a shared clustered session was used.

He described some features that make PHP even faster today:

libevent allows event driven programming ala nodejs.
zeromq for better and simplest socket programming. I have to add that messaging is in fact something I would say will gain more and more momentum within the software community. The world is asynchronous as the creator of Erlang language said said.
FastCGI Process Manager (FPM) allows dividing the workload while sending jobs to workers. This is the PHP way to say no to threads.

Rasmus talked about performance and brought some examples providing some guidance to improve it:

Use wisely your datastores. Do not try to replace sql with nosql, it simply does not work that way.
Turn off logging in production (error_reporting(-1) is expensive).
Use strace: Look for ENOENT, excessive stats (lstat), check your realpath_cache_size.
Use a profiler: callgrind, xdebug, xhprof, xhgui.
Set default timezone (look at phpinfo).
Watch for inclusions. Too many inclusions degrade performance. An example of a bad application is Magento which includes thousand pf files.
HipHop-PHP can be used to do static analysis. They need to parse php better than the php parser itself because their goal is to translate PHP to native code. Be prepared to wait for the compiler but this is a good exercise to find problems in your code.

Here are some architecture reminders/suggestions from Rasmus:

Use different domain/subdomain for static assets.
Keep cookies short (MTU size maters and will translate in more roundtrips when it is overflown).
Multiply by 5 times the amount of cores and that is the amount of real concurrent users. I have to say I have been calculating allowed concurrent users for years and the metric about just the cores is not that simple. It even depends on what the application is using of course so a word of advice from my end would be take a look at your system metrics for example with vmstat, top and other system commands. The use of automated stress tests driven by for example jmeter are a must do in my opinion.
Use out-of-band processing with Gearman, PHP-FPM or custom via ZeroMQ.
Tweak ORM and caching.

Finally he talked about the PHP 5.3 features, how PHP 6 development was stopped, the problems inherent to support Unicode and the efforts that are still being made in the current version 5 to support little by little more Unicode functionality. Here are some of those new features:

Better performance through lot of code optimization.
Support for closures.
Namespaces. There is a big debate around the decision to use a backslash instead of a dot for the namespaces. To be honest I do feel that as awkward but as Rasmus said "we will need to get used to it".
Late Static Binding support.
Garbage Colector which you should not need in web apps but if you have long running scripts then ii will definitely help.
NEWDOC which is like HEREDOC but does not perform any parsing inside the string block.
Remember "Go To Statement Considered Harmful", the famous letter from Edsger Dijkstra? The controversy is still on as PHP allows "goto" to eliminate verbosity or the use of break. In my opinion the discussion about the name is secondary. You can use break or continue to a a label in Java for example so if a goto has just the meaning of breaking the loop to a variable I think is awkward but but I am fine with it. However allowing goto to go to any part of the code block is not precisely a multi level break or continue, it is something more than that. If that is harmful, confusing, miss leading or not I leave it to the discussion. Perhaps just adjusting the multi level sysntax from PHP to accept a letter instead of a "magic" number would be a better approach.
DateInterval/DatePeriod classes.
date_create_from_format
FastCGI Process manager (FPM)
For people moving to ngnix from apache .php_ini allows for custom php directives just like .htaccess for Apache.
Traits: Compiler assisted copy and paste to resolve the lack of multiple inheritance.

Thank you Rasmus for PHP. Let's celebrate how lucky we are that not a single language could ever win all battles. Different languages exist to resolve similar problems unders different scenarios. This is not any different than we, human beings. We need different skills in a team as much as we need different languages in modern computing.

and different human beings exist to reso

Workspace defines a VM that does not contain a valid jre/lib/rt.jar in Snow Leopard OSX

2011-12-15T14:12:00.000-08:00

I was getting this error today after trying to use "mvn eclipse:eclipse" to generate Eclipse project files out of a maven project:

[WARNING] Workspace defines a VM that does not contain a valid jre/lib/rt.jar: /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home

There is a proposed patch for this bug but at the time of this writing I could not figure out a better way than running the below commands:

cd /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home
sudo mkdir -p jre/lib
cd jre/lib
sudo ln -s ../../../Classes/classes.jar rt.jar

Remove context from url from Tomcat applications

2011-12-15T12:25:00.000-08:00

If you are like me you love tomcat simplicity and you are used to deploy your WAR files or exploded directories in webapps folder directly. However in order to serve URLs without the context included you will need to stop that practice.

You might come with a hack like I discover a while back, but as you can read there that will result in the same application being deployed several times.

Here is how to configure tomcat (Tested in tomcat 7) to serve content from two different URLs

Make your websites resolve to real IPs. For production you rely on external DNS, for other environments you rely on internal DNS and many times in you /etc/hosts:
```
$ vi /etc/hosts
...
127.0.0.1   bhubdev.nestorurquiza.com
127.0.0.1   bhubdev2.nestorurquiza.com
...
```

You need to add them both to Engine section in conf/server.xml:

...
      unpackWARs="true" autoDeploy="true"
    xmlValidation="false" xmlNamespaceAware="false" deployOnStartup="true"/>
      unpackWARs="true" autoDeploy="true"
    xmlValidation="false" xmlNamespaceAware="false" deployOnStartup="true"/>

Create these folders (two per domain as you can see). Be sure to change to proper paths in your system:

$ mkdir /Users/nestor/Downloads/apache-tomcat-7.0.22/bhub-app
$ mkdir /Users/nestor/Downloads/apache-tomcat-7.0.22/conf/Catalina/bhubdev.nestorurquiza.com
$ mkdir /Users/nestor/Downloads/apache-tomcat-7.0.22/bhub2-app
$ mkdir /Users/nestor/Downloads/apache-tomcat-7.0.22/conf/Catalina/bhub2dev.nestorurquiza.com

Restart tomcat.

Deploy the WAR files as:

/Users/nestor/Downloads/apache-tomcat-7.0.22/bhub-app/ROOT.war
/Users/nestor/Downloads/apache-tomcat-7.0.22/bhub2-app/ROOT.war

Alternatively deploy the exploded WAR file as:

unzip bhub-app.war -d /Users/nestor/Downloads/apache-tomcat-7.0.22/bhub-app/ROOT/
unzip bhub-app2.war -d /Users/nestor/Downloads/apache-tomcat-7.0.22/bhub2-app/ROOT/

Now if you use maven and you want to deploy from there you probably have something like the below if you are still deploying to webapp folder (using ant tasks to deploy either the WAR file or the exploded directory)

...

    ${MVN_TOMCAT_HOME_DEPLOY}
...

    
    
        
    

...

    Deploying WAR locally
    

...

You will need to update that to point to the new directories and not to webapps anymore.

...

  ${CATALINA_HOME}
...

    
    
        
    

...

    
    

...

Do not forget to add to your profile the needed for Maven environment variable:

$ vi ~/.profile:
#old needed variable
#export MVN_TOMCAT_HOME_DEPLOY=/opt/tomcat/webapps
#new needed variable
export CATALINA_HOME=/opt/tomcat
$ source ~/.profile

Monitor Event Log in Windows 2008

2011-12-14T08:27:00.000-08:00

I already described how we can monitor event logs from windows but that procedure will not work for Windows 2008 and Windows 7 because eventtriggers.exe has been deprecated.

Here is a new script you will need to use in Windows 7/2008 together with "Windows Task Scheduler". The comments on the top of the script should be straightforward to understand how to get an email alert every time an application ERROR event is registered in the Event logs.

'''''''''''''''''''''''''''''''''''''''''''''''
'
' c:\scripts\events\sendEventErrorByEmail.vbs
'
' @Author: Nestor Urquiza
' @Created: 12/14/2011
'
'
' @Description: Alerts a Windows Admin there are errors in Event Viewer. 
' It could be scheduled to run every maxMinutes but
' Using it as an action for a custom Scheduled Task with a trigger on event filters:
'
' Task Scheduler Library: Create Task | Triggers | New Trigger | Begin the Task On an Event | Settings Custom | New Event Filter | Event Level Error | By Log | Event Logs | Windows Logs | Application
'
'
'
'

'
'
' @Compatibility: Tested so far in WindowsXP, Vista, 7, 2000, 2003, 2008
'
'
' @Parameters
' 1. A prefix body message in case specific errors are to be sent 
'    (a combination of batch and eventtriggers will do the trick)
'
'
' @Filters: I am filtering only "Application" events. Change the SQL query if you want to apply a different filter or not filter at all
'
'
'
''''''''''''''''''''''''''''''''''''''''''''''''

'Constants
strSmartHost = "krms.krco.com"
strSmartPort = 25
maxMinutes = 1
strComputer = "."
emailFrom = "donotreply@nestorurquiza.com"
emailTo = "nurquiza@nestorurquiza.com"

'System config
Set wshShell = WScript.CreateObject( "WScript.Shell" )
strComputerName = wshShell.ExpandEnvironmentStrings( "%COMPUTERNAME%" )
Set objSWbemServices = GetObject("winmgmts:" _
 & "{impersonationLevel=impersonate}!\\" & strComputer & "\root\cimv2")
Set colTimeZone = objSWbemServices.ExecQuery _
 ("SELECT * FROM Win32_TimeZone")
For Each objTimeZone in colTimeZone
 offset = objTimeZone.Bias
Next

'Parameters
Dim strBody
If (Wscript.Arguments.Count > 0) Then
  strBody = Wscript.Arguments(0)
End If

'Start date to look for events
dtmDate = DateAdd("n",-maxMinutes,Now())
dateToWMIDateString = Year(dtmDate) & padZeros(Month(dtmDate)) & padZeros(Day(dtmDate)) & padZeros(Hour(dtmDate)) & padZeros(Minute(dtmDate)) & padZeros(Second(dtmDate)) & ".000000" & offset

'Get events matching the query
Set objWMIService = GetObject("winmgmts:" _
    & "{impersonationLevel=impersonate}//" & _
        strComputer & "\root\cimv2")
Set colLogFiles = objWMIService.ExecQuery _
    ("Select * from Win32_NTLogEvent " _
        & "Where Logfile='Application' and Type='Error' and TimeGenerated > '" &  dateToWMIDateString & "'" )

'Accumulate all events dates and details
For Each objLogFile in colLogFiles
    dtmInstallDate = objLogFile.TimeGenerated
    WMIDateStringToDate = CDate(Mid(dtmInstallDate, 5, 2) & "/" & _
     Mid(dtmInstallDate, 7, 2) & "/" & Left(dtmInstallDate, 4) _
         & " " & Mid (dtmInstallDate, 9, 2) & ":" & _
             Mid(dtmInstallDate, 11, 2) & ":" & Mid(dtmInstallDate, _
                 13, 2))
    WMIDateStringToDate = DateAdd("n", offset, WMIDateStringToDate)
    details = details & vbCrLf & WMIDateStringToDate  & " - [" & _
                   objLogFile.Type & "] " & _
                   objLogFile.Message
    'Wscript.Echo details
Next

'Send email with details about matching events
If (Not IsNull(details) And details <> "") Then
    'Prepare email
    Set objEmail = CreateObject("CDO.Message")
    objEmail.From = emailFrom
    objEmail.To = emailTo
    objEmail.Subject = "[" & strComputerName & "] " & "Event Viewer Alert"
    If (Not IsNull(strBody) And strBody <> "") Then
        objEmail.Textbody = strBody & ". "
    End If
    objEmail.Textbody = objEmail.Textbody & details

    'Custom server
    objEmail.Configuration.Fields.Item("http://schemas.microsoft.com/cdo/configuration/sendusing") = 2
    objEmail.Configuration.Fields.Item("http://schemas.microsoft.com/cdo/configuration/smtpserver") = strSmartHost
    objEmail.Configuration.Fields.Item("http://schemas.microsoft.com/cdo/configuration/smtpserverport") = strSmartPort
    objEmail.Configuration.Fields.Update

    'Send it
    objEmail.Send
End If

Function padZeros(dtmDate)
If Len(dtmDate) = 1 Then
    padZeros = "0" & dtmDate
Else
    padZeros = dtmDate
End If
End Function

BTW you will notice the Event filter contains the below which could give you some hints to research even more powerful ways to control different event alerts:

Arial Font from JasperReports for free and legally

2011-12-07T12:42:00.000-08:00

If you are planning to ship documents to your clients and you need to use one of the commercial most used fonts like Arial you are in your way to pay a good chunk of money. There is not even a clear price on what they might cost because basically it depends on your application (read the license goes with how much money you are making with your product).

Thanks to the information I found in this post it took me little time to legally use a close to the proprietary Arial font in our application. If you plan to distribute or license your application you will need to buy Arial font or find a different project with a more commercial friendly license (like BSD) as the license for Liberation fonts is GPL. must then

Download and extract the Red Hat Open Source GPL licensed Liberation Fonts
From iReport Preferences on a Mac or Options for the rest of the mortals navigate through "iReport | Fonts | Install Font", then import "LiberationSans-Regular.ttf" from the exploded directory created as part of the above step. Click next and pick the bold, Italic and BoldItalic versions for LiberationSans. LiberationSans is the closest font to Arial. The rest of the Liberation Fonts are closer to Times New Roman and Courier New fonts
Pick a good family name that will later use in your style fontName property, for example "LiberationSans". Check the "Embed this font in the PDF document"

Include your font in a hello world jrxml like I show below and run it from iReport. Play with the properties and you should see bold and italic rendered as well (I am in this example striking through and underlining as well).

<?xml version="1.0" encoding="UTF-8"?>
<jasperReport xmlns="http://jasperreports.sourceforge.net/jasperreports" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://jasperreports.sourceforge.net/jasperreports http://jasperreports.sourceforge.net/xsd/jasperreport.xsd" name="hello" language="groovy" pageWidth="595" pageHeight="842" whenNoDataType="AllSectionsNoDetail" columnWidth="555" leftMargin="20" rightMargin="20" topMargin="20" bottomMargin="20">
 <property name="ireport.zoom" value="1.0"/>
 <property name="ireport.x" value="0"/>
 <property name="ireport.y" value="0"/>
 <style name="Default" isDefault="true" fontName="LiberationSans" fontSize="20" isBold="true" isItalic="true" isUnderline="true" isStrikeThrough="true"/>
 <background>
  <band splitType="Stretch"/>
 </background>
 <title>
  <band height="79" splitType="Stretch">
   <staticText>
    <reportElement x="12" y="10" width="531" height="57"/>
    <textElement/>
    <text><![CDATA[Hello JasperReports!!!]]></text>
   </staticText>
  </band>
 </title>
 <pageHeader>
  <band height="35" splitType="Stretch"/>
 </pageHeader>
 <columnHeader>
  <band height="61" splitType="Stretch"/>
 </columnHeader>
 <detail>
  <band height="125" splitType="Stretch"/>
 </detail>
 <columnFooter>
  <band height="45" splitType="Stretch"/>
 </columnFooter>
 <pageFooter>
  <band height="54" splitType="Stretch"/>
 </pageFooter>
 <summary>
  <band height="42" splitType="Stretch"/>
 </summary>
</jasperReport>

Now you should see from iReport your font correctly embedding in the resulting PDF
From iReport navigate to the Fonts settings as I explained before and then pick "LiberationSans" and export it as liberation-sans.jar
Include the jar in your application classpath
Now your app is serving close to Arial font styled content

svn dump stderr output causing false positives

2011-11-30T09:52:00.000-08:00

We were getting these annoying alerts every time the svn backup was running. Reason was 'svn dump' by default sends feedback like the below to the stderr:

* Dumped revision 0.
* Dumped revision 1.
* Dumped revision 2.
...

From the help command the reason and the workaround came clear:

# svnadmin dump --help
dump: usage: svnadmin dump REPOS_PATH [-r LOWER[:UPPER] [--incremental]]

Dump the contents of filesystem to stdout in a 'dumpfile'
portable format, sending feedback to stderr.  Dump revisions
LOWER rev through UPPER rev.  If no revisions are given, dump all
revision trees.  If only LOWER is given, dump that one revision tree.
If --incremental is passed, the first revision dumped will describe
only the paths changed in that revision; otherwise it will describe
every path present in the repository as of that revision.  (In either
case, the second and subsequent revisions, if any, describe only paths
changed in those revisions.)

Valid options:
  -r [--revision] ARG      : specify revision number ARG (or X:Y range)
  --incremental            : dump incrementally
  --deltas                 : use deltas in dump output
  -q [--quiet]             : no progress (only errors) to stderr

Just using the --quiet option does the trick:

svnadmin dump -q $MASTER_REPO >  $dump_file_path/dumpfile

Jasper Reports Excel Cross Sheet Formulas

2011-11-28T12:30:00.000-08:00

A member of the Data team brought my attention to an iReports bug affecting Excel Output containing formulas which reference a following sheet. If the formula is let us say referring Sheet1 from Sheet2 there is no problem but if it refers Sheet2 from Sheet1 Jasper will fail to generate a correct XLS file. This is not the case when we try to generate an XLSX file. In that case it does work as expected.

When running with "Excel 2007 (XLSX) Preview" the cell does get what we expect:

=mySheet2!A1

However when we pick "XLS Preview" we get:

=#REF!A1

Here is the screen shot showing a successful rendering when using XLSX:

Here is the failure when using XLS:

This is clearly a problem related to different exporters. I still need to test this from Java with different exporter options but it would be ideal if a simple cross sheets formula like this could work from either XLS or XLSX files.

Below is the source code for this report. I have picked one jrxm posted in forums and I have added a couple of lines just to make sure I generate two different spreadsheets.

The output contains two sheets which reference each other through a simple formula that brings the content of the A1 cell from one sheet to the other in both directions. As stated at the beginning the formula in Sheet1 will fail when exporting to XLS but will succeed when exporting to XLSX. The formula in Sheet 2 will always succeed.

<?xml version="1.0" encoding="UTF-8"?>
<jasperReport xmlns="http://jasperreports.sourceforge.net/jasperreports" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://jasperreports.sourceforge.net/jasperreports http://jasperreports.sourceforge.net/xsd/jasperreport.xsd" name="formulaSample" pageWidth="595" pageHeight="842" whenNoDataType="AllSectionsNoDetail" columnWidth="535" leftMargin="0" rightMargin="0" topMargin="0" bottomMargin="0">
 <property name="net.sf.jasperreports.export.xls.detect.cell.type" value="true"/>
 <property name="net.sf.jasperreports.export.xls.one.page.per.sheet" value="true"/>
 <property name="net.sf.jasperreports.export.xls.sheet.names.all values" value="mySheet1/mySheet2"/>
 <property name="ireport.zoom" value="1.0"/>
 <property name="ireport.x" value="0"/>
 <property name="ireport.y" value="0"/>
 <import value="net.sf.jasperreports.engine.*"/>
 <import value="java.util.*"/>
 <import value="net.sf.jasperreports.engine.data.*"/>
 <title>
  <band height="262" splitType="Stretch">
   <textField>
    <reportElement key="textField-1" x="0" y="0" width="68" height="23"/>
    <textElement/>
    <textFieldExpression><![CDATA[new Integer(2)]]></textFieldExpression>
   </textField>
   <textField>
    <reportElement key="textField-2" x="0" y="23" width="68" height="23"/>
    <textElement/>
    <textFieldExpression><![CDATA[new Integer(1)]]></textFieldExpression>
   </textField>
   <textField>
    <reportElement key="textField-3" x="0" y="46" width="68" height="23">
     <propertyExpression name="net.sf.jasperreports.export.xls.formula"><![CDATA["mySheet2!A1"]]></propertyExpression>
    </reportElement>
    <textElement/>
    <textFieldExpression><![CDATA[]]></textFieldExpression>
   </textField>
   <break>
    <reportElement x="0" y="116" width="1" height="1"/>
   </break>
   <textField>
    <reportElement key="textField-3" x="0" y="163" width="68" height="23">
     <propertyExpression name="net.sf.jasperreports.export.xls.formula"><![CDATA["mySheet1!A1"]]></propertyExpression>
    </reportElement>
    <textElement/>
    <textFieldExpression><![CDATA[]]></textFieldExpression>
   </textField>
   <textField>
    <reportElement key="textField-2" x="0" y="140" width="68" height="23"/>
    <textElement/>
    <textFieldExpression><![CDATA[new Integer(9)]]></textFieldExpression>
   </textField>
   <textField>
    <reportElement key="textField-1" x="0" y="117" width="68" height="23"/>
    <textElement/>
    <textFieldExpression><![CDATA[new Integer(7)]]></textFieldExpression>
   </textField>
  </band>
 </title>
</jasperReport>

Note that this example contains a useful hint that allows to quickly come up with a test case that can be shared with the community. This is just using java.lang.* package to include hard coded values in text fields (that later become Excel cells) without the need to having a datasource defined (Use "Empty Datasource" to run this jrxml)

Last but not least we set the property "whenNoDataType="AllSectionsNoDetail"" because we are not using any datasource. If you are rendering the jrxml with no datasource from Java you will need it otherwise you will just get blank pages or corrupted XML files depending on the options you use.

The plugin 'org.apache.maven.plugins:maven-jetty-plugin' does not exist or no valid version could be found

2011-11-23T14:24:00.000-08:00

This issue happens when the repository does reach the Mortbay release repo http://jetty.mortbay.org/maven2/release so to be safe I always add the below to the build/plugins node in the pom.xml.

<plugin>
        <groupId>org.mortbay.jetty</groupId>
        <artifactId>maven-jetty-plugin</artifactId>
        <version>6.0.1</version>
      </plugin>

SWFUpload plugin in Self Signed Certificate websites

2011-11-22T13:44:00.000-08:00

When you are developing you will not buy an SSL certificate but instead you will use a self signed one. Adobe Flash is used to provide cool multiple file upload capabilities through the use of the SWFPlugin but it will fail if the Certificate is self signed.

This was affecting our QA team today and thanks to this post I figured a solution for the problem:

In IE double click the "Certificate Error" icon
Pick Install Certificate. Pick the defaults and finish

For some people this was not working still so I suggested to restart browsers and even Windows, you know that kind of stuff you need to do when suspicious right? At least one of my team members reported cleaning cookies resolved the problem for him (of course after following the above steps)

Cannot determine realm for numeric host address

2011-11-22T09:00:00.000-08:00

I could not connect today to one of our servers using password less public key ssh login. Using -vvv ssh option would show:

debug1: An invalid name was supplied 
Cannot determine realm for numeric host address

So from the Red Hat target I inspected the secure log:

$ tail -f /var/log/secure
...
Nov 22 11:37:03 t2215179 sshd[23123]: Authentication tried for admin with correct key but not from a permitted host (host=tools, ip=xxx.xxx.xxx.xxx).
...

This was resolved adding the IP to /etc/hosts

$ tail -f /var/log/secure
...
xxx.xxx.xxx.xxx tools
...

Android on the HP Touchpad

2011-11-18T05:09:00.000-08:00

Just follow instructions from cyanogenmod touchpad port team, Download section. Be sure you install everything including ClockworkMod,

Then install GoogleApps (http://wiki.cyanogenmod.com/index.php?title=Latest_Version/Google_Apps) so you get Market application:

Boot into Android mode
Connect TouchPad to Computer
Touch the notification icon so you are able to turn on USB
From the computer drop the GoogleApps zip file (for example gapps-gb-20110828-signed.zip) into the TouchPad
Reboot the TouchPad (Power + Home buttons for 15-20sec will do it in case just Power does not give you the reboot option)
In the boot menu quickly move using the volume keys to option "ClockworkMod". Press Home button to accept
choose "install zip from sdcard", then "choose zip from sdcard" and pick the GoogleApps zip file.
Reboot

Hide link or anchor after click with jquery

2011-11-17T11:37:00.000-08:00

This is a common need when you provide what I call a naked user interface meaning, well I am waiting for my front end engineer and I provide just the bare minimums so the whole functionality can be used. But this is also a need for complicated user interfaces where sometimes a link is just supposed to trigger something, you want to stay in the page and you do not want the link to be available again (avoiding double submission for example).

So here is the JQuery code that will keep the link text but will change it by just a spanned text:

    //
    // Hide link after click
    //
    $('.hideAfterClickLink').click(function(e) {
     $(this).replaceWith('' + $(this).text() + '')
    });

Tomcat Servlet Context initialized twice or multiple times

2011-11-11T14:12:00.000-08:00

Tomcat Servlet Context initialized twice or multiple times
I have seen this issue both in JBoss and Tomcat and in both cases this is about a miss configuration. It is worth to be said this issue affects the agility of the developer because extra time needs to be spent just waiting for tomcat to finish its initialization. This is also an issue that hits production systems many of which have very slow startup just because of miss configurations.

The first thing that you need to do is to include a log INFO level message when the servlet context is initialized like below:

public class ApplicationServletContextListener implements
        ServletContextListener {
...
    Logger log = LoggerFactory.getLogger(ApplicationServletContextListener.class.getName());
    @Override
    public void contextInitialized(ServletContextEvent event) {
       ...
       log.info("Servlet Context initialized");
...

Now whenever you restart your server you should get only one line:

2011-11-11 16:29:38,027 INFO [com.nestorurquiza.utils.web.ApplicationServletContextListener] -   Servlet Context initialized

If you get more than one look for errors in server.xml. Just to show an example of something that actually hit me Today this was the configuration in one of my servers:

...
<Host name="bhubdev.nestorurquiza.com"">
  <Context path="" docBase="/opt/tomcat/webapps/nestorurquiza-app"/>
...

Just changing for the below fixed the issue:

...
<Host name="bhubdev.nestorurquiza.com" appBase="webapps/nestorurquiza-app"/>
...

However it also screwed the root context you get with the previous configuration. If you still need the application to respond to the root context you will need to deployed as ROOT.war as explained in the official documentation.

Excel Jasper Report with default Zoom using a Macro

2011-11-10T09:44:00.000-08:00

I could not find a way to specify a default zoom for an Excel Report generated from iReport/Jasper Reports so I had to apply the Macro hack I recently documented.

Here is the Visual Basic for Applications code that make this possible. It zoom out the whole workbook to 75%:

Private Sub Workbook_Open()
    If Application.Sheets.Count > 1 Then
        Worksheets("Macro").Visible = False
    End If
    Application.ScreenUpdating = False

    For Each ws In Worksheets
        ws.Activate
        ActiveWindow.Zoom = 75
    Next

    Application.ScreenUpdating = True
End Sub

Excel Jasper Reports With Macros

2011-11-09T14:46:00.000-08:00

I am a big fan of separation of concerns. Did I say this before? :-)

When it comes to View technologies there is a miss conception to think about View part as "passive" when in reality we have seen "active" Views for years with Javascript in the center of our HTML pages. Well, the same applies for PDF and Excel.

So I don't understand why providing Macro support in a Jasper Report should be considered wrong and since especially in the Financial Industry Excel is a core technology where Macros are simply needed I decided to present here a solution to provide Excel Jasper Reports containing Macros.

Some times the macro is needed by business while other times the macro can save time with formatting features that are unavailable in the iReport/Jasper last version. For the second case I do recommend to fill a feature request because it is a hack to force a formatting capability using a Macro and your users might not be OK with code running in their Excel which imposes a big security concern. But for sure this hack allows to move while keeping separation of concerns, in other words liberating Java developers from the need to touch code, recompile and redeploy (The Controller) just to fulfill a need in formatting that is not available where it should be available (The View).

Create an Excel Workbook containing the Macro to be applied to the Excel Jasper Report

Create a brand new Excel Workbook
Rename Sheet1 as 'Macro' and delete the rest of them
Select from the Menu: Tools | Macro | Visual Basic Editor

Right click on "This Workbook" and select "View Code". Paste the below in the code pane:

Private Sub Workbook_Open()
    If Application.Sheets.Count > 1 Then
        Worksheets("Macro").Visible = False
    End If
    MsgBox "Hello Excel Jasper Report!"
End Sub

Save the Workbook as "myTemplate.xls", close and open. A pop up should come up automatically. This code as you already noticed has a hack which basically allows to hide the unique sheet we are forced to leave in the workbook because of an Excel imposed limitation.
Now we are ready to use myTemplate.xls as the template out of which our Jasper Report will be built.

Use the Excel Template from jasper API

First, you will need to extend the exporter. I am showing here how to do it with JRXlsExporter which uses POI API but the same idea should be applicable to other exporters.

package com.nestorurquiza.utils;

import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;

import net.sf.jasperreports.engine.export.JRXlsExporter;

import org.apache.poi.hssf.usermodel.HSSFWorkbook;
import org.apache.poi.hssf.util.HSSFColor;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

public class CustomJRXlsExporter extends JRXlsExporter {
    private static final Logger log = LoggerFactory.getLogger(CustomJRXlsExporter.class);
    
    private InputStream is;
    
    public InputStream getIs() {
        return is;
    }

    public void setIs(InputStream is) {
        this.is = is;
    }

    @Override
    protected void openWorkbook(OutputStream os) {
        if(is != null) {
            try {
                workbook = new HSSFWorkbook(is);
                emptyCellStyle = workbook.createCellStyle();
                emptyCellStyle.setFillForegroundColor((new HSSFColor.WHITE()).getIndex());
                emptyCellStyle.setFillPattern(backgroundMode);
                dataFormat = workbook.createDataFormat();
                createHelper = workbook.getCreationHelper();
                return;
            } catch (IOException e) {
                log.error("Creating a new Workbook when I was supposed to use an existing one.", e);
            }
        }
        super.openWorkbook(os);
    }
}

Second just use the Custom Exporter passing the input stream to your Excel Template file.

...
CustomJRXlsExporter exporter = new CustomJRXlsExporter();
String xlsTemplate = ctx.getParameter("xlsTemplate");
if( xlsTemplate != null ) {
    String xlsTemplateFilePath = getReportsPath() + "/xlsTemplate/" + ctx.getParameter("xlsTemplate");
    exporter.setIs(new FileInputStream(xlsTemplateFilePath));
}
...

You end up with your report and a message box popping up from the macro code "Hello Excel Jasper Report!".

A more realistic example? I will probably start documenting some of them in the near future so search this blog for jasper excel macro.

This technique is simple but powerful. I hope it is integrated somehow in the different exporters as it was proposed 5 years ago.

As I have said there I think iReport should be able to use a hint so the front end report designer can get the formatting they want using an Excel Template without the need to wait for an external API implementation.

error running shared postrotate script for /var/log/mysql

2011-11-07T09:17:00.000-08:00

I am sure the error below can be related to many different circumstances:

/etc/cron.daily/logrotate:
error: error running shared postrotate script for '/var/log/mysql.log /var/log/mysql/mysql.log /var/log/mysql/mysql-slow.log '
run-parts: /etc/cron.daily/logrotate exited with return code 1

In my case after cloning an ESX Ubuntu VM where MySQL was installed into some other servers where we did not need mysql and after uninstalling mysql I could still see mysqld running.

Just running the below command was enough to stop the alert:

$ sudo apt-get autoremove
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages will be REMOVED:
  mysql-server-5.1 mysql-server-core-5.1
...

User impersonation with Spring and LDAP

2011-11-06T23:06:00.000-08:00

Impersonating a user (also known as "switch a user" or "do as user") in a web application is an important feature for doing Quality Assurance (QA) or any Manual User Acceptance Test (UAT).

With Spring you can achieve it using the SwitchUserFilter. If you are using LDAP here are the relevant pieces.

...
<beans:bean id="ldapUserDetailsService" class="org.springframework.security.ldap.userdetails.LdapUserDetailsService">
  <beans:constructor-arg><beans:ref bean="ldapUserSearch"/></beans:constructor-arg>
  <beans:constructor-arg><beans:ref bean="ldapAuthoritiesPopulator"/></beans:constructor-arg>
  <beans:property name="userDetailsMapper" ref="customUserDetailsContextMapper" />
</beans:bean>
...
<beans:bean id="ldapUserSearch" class="org.springframework.security.ldap.search.FilterBasedLdapUserSearch">
    <beans:constructor-arg type="String"><beans:value>ou=people,o=nestorurquiza</beans:value></beans:constructor-arg>
    <beans:constructor-arg type="String"><beans:value>mail={0}</beans:value></beans:constructor-arg>
    <beans:constructor-arg><beans:ref bean="ldapContextSource"/></beans:constructor-arg>
</beans:bean>
 ...                            
<beans:bean id="switchUserProcessingFilter" class="org.springframework.security.web.authentication.switchuser.SwitchUserFilter">
        <beans:property name="userDetailsService" ref="ldapUserDetailsService" />
        <beans:property name="switchUserUrl" value="/admin/switchUser" />
        <beans:property name="exitUserUrl" value="/admin/switchUserExit" />
        <beans:property name="targetUrl" value="/" />
</beans:bean>
...
<http auto-config="true" use-expressions="true" access-decision-manager-ref="accessDecisionManager" disable-url-rewriting="true">
    ...
 <custom-filter  after="FILTER_SECURITY_INTERCEPTOR" ref="switchUserProcessingFilter" />
    ...
    <intercept-url pattern="/admin/**" access="hasAnyRole('ROLE_ADMIN','ROLE_PREVIOUS_ADMINISTRATOR')" />
... 
<beans:bean id="ldapContextSource" class="org.springframework.ldap.core.support.LdapContextSource">

    <beans:property name="url" value="${ldap.url}" />
    <beans:property name="userDn" value="${ldap.userDn}" />
    <beans:property name="password" value="${ldap.password}" />
</beans:bean>
...
<beans:bean id="ldapAuthProvider"
        class="org.springframework.security.ldap.authentication.LdapAuthenticationProvider">
  <beans:constructor-arg>
    <beans:bean class="org.springframework.security.ldap.authentication.BindAuthenticator">
      <beans:constructor-arg ref="ldapContextSource"/>
      <beans:property name="userDnPatterns">
        <beans:list><beans:value>mail={0},ou=people,o=nestorurquiza</beans:value></beans:list>
      </beans:property>
    </beans:bean>
  </beans:constructor-arg>
  <beans:constructor-arg ref="ldapAuthoritiesPopulator"/>
</beans:bean>
...
<beans:bean id="ldapAuthoritiesPopulator" class="org.springframework.security.ldap.userdetails.DefaultLdapAuthoritiesPopulator">
      <beans:constructor-arg ref="ldapContextSource"/>
      <beans:constructor-arg value="ou=groups,o=nestorurquiza"/>
      <beans:property name="groupSearchFilter" value="uniquemember={0}" />
</beans:bean>
...    
<beans:bean id="ldapTemplate" class="org.springframework.ldap.core.LdapTemplate">
    <beans:constructor-arg ref="ldapContextSource" />
</beans:bean>
...

I18n messages:

switchUser=Impersonate
switchUserExit=Be yourself

A link to impersonate in the users listing page should be available only for admins:

<security:authorize access="hasRole('ROLE_ADMIN')">
                        | <a href="<spring:url value="/admin/switchUser?j_username=${employee.email}" htmlEscape="true" />">
                            <spring:message code="switchUser"/></a>
                </security:authorize>

A link to switch back to admin user should be available only if the user is being impersonated:

<security:authorize access="hasRole('ROLE_PREVIOUS_ADMINISTRATOR')">
                        | <a href="<spring:url value="/admin/switchUserExit" htmlEscape="true" />">
                            <spring:message code="switchUserExit"/></a>
                    </security:authorize>

The way you use it is hitting the below URLs to switch the user and come back to the original admin user context:

http://localhost:8080/nestorurquiza-app/admin/switchUser?j_username=nurquiza@nestorurquiza.com
http://localhost:8080/nestorurquiza-app/admin/switchUserExit

Spring One 2011 Notes

2011-10-31T17:43:00.000-07:00

It was quite a week of learning with the folks from Spring Framework in the SpringOne Conference in Chicago.

Not only I had the opportunity to hear Spring explained from the code committers but I managed to discuss some common architectural, managerial and leadership issues with other attendees.

The amount of work Spring helps to solve could be enough for four weeks of conference so there were mainly 4 slices (or tracks) of very related conferences from which you had to pick just one. Here are some comments about those that I attended.

Javascript application engineering

It is clear javascript is now stronger that ever. It was a nice look into how to do Object Oriented Programming with Javascript, Inheritance through prototypes (Object.create(Object.prototype)), closures and more. Between others websites like Javascript weekly and hacker news were recommended to be followed up.

Spring Integration. Practical Tips and tricks

The Spring Integration project is an attempt to implement the patterns described in the book Enterprise Integration Patterns Book by Gregor Hohpe and Bobby Wolf
It was a discussion about Consumers, Producers and the Publish/Subscribe pattern using AMQP channels and comparing them with JMS channels. Of course SOAP, REST and others are supported. Channel redundancy was presented Rabitt MQ and JSM. The project reminds me of ESB and indeed there was a mention to it. Basically Spring Integration is a different approach than ESB because there is no server needed but just patterns applied.

Messaging

Messaging was presented through the Rabbit MQ which is the reference implementation of the Advanced Messaging Queue Protocol (AMQP). It was even proposed to start sending messages between Controllers and Services to prepare the application to scale in the future. I have to say I won't buy into that even though for sure I would consider using it to implement future distributed needs in my projects. Messaging was presented as a way around problems inherent to the use of shared class loaders (OSGI). In reality that is what any distributed technique does after all, being more than anything a consequence of the paradigm in use.

Neo4J

This is a graph database and the presentation was useful especially to eliminate any of the ongoing discussions about what is the best database out there in the NoSQL domain. It is clear they all address different problems while of course there is more than one choice within each possible domain. Basically there is a trade between size and complexity and in that order we have different options like key-value, bigtable, document, graph.

Spring Insight Plugin Development

This is a project with a mission that sounds familiar: Inspect how the application is behaving performance wise. Classes are annotated with @InsightOperation and @InsightEndPoint to allow performance inspection: Basically how long the method gets to execute. In other words where the time is spent in the application. While this could be definitely useful in many scenarios we have tools doing this already so it looked to me like an effort to build a lightweight profiler focused on vFabric.

Spring Integration. Implementing Scalable Architectures with Spring Integration

A simple implementation based on a simple relational database locked row was presented to provide a messaging cluster implementation.

Spring Batch

In a glance this is a project to manipulate data. We learned how Spring Batch works while defining and instantiating jobs (Job and Job Instance) that basically act on data. I will not start a war about if spring is better or worst that etl for this task. It is just another way of doing the same and depending on your architecture and team one will be favored over the other. I tend to separate concerns as much as possible and I think data is better processed and tackle by data centric developers and existing tools. In my experience most of people dealing with Spring actually love to implement business rules that are not necessarily related to data intensive processing. But again this discussion should end here, there is not correct way just different ways.

Deliver Performance and scalability with ehcache and spring

JSR107 will standardize caching while some of us still use some caching api from Google code to cache data at method level. However terracotta ehCache and the new standard add more flexibility to caching. For simple caching you can use the Google code API but for more complicated situations you will use this approach which involves @Cacheable and @CacheEvict annotations. When the source code is not available @Cacheable can be used from XML which is great you will agree. AOP helps here to cache certain methods. Different providers like terracotta backed open source Ehcache offer different possibilities which go beyond the basis of the default ConcurrentMap implementation. Automatic Resource Control now in beta will allow ehacahe to use cache sizing in terms of bytes or percentage besides the existing object count. Then of course you can even go for enterprise class monitoring with "terracotta Developer Console".

Eventing Data with RabbitMQ and Riak

Riak was discussed as the unique NoSQL database that would allow hooking into the data insertion. This is of advantage to create a trigger for RabitMQ when new data gets stored. A very cool example was presented where WebSocket was used to notify the client directly from the server. Again the CAP theorem was discussed: You must pick one from Consistency, Availability and Performance triangle and that would ultimately drive your decision towards one database or another.

Where did my architecture go

It was disscused how to control the gap between architecture and codebase. Here you have code analysis with JDepend and Sonar, SonarGraph (formerly SonarJ). Suggestions were made about the usage of ServiceLoader or OSGI for externalizing class creation.

Tailoring Spring for Custom Usage

As we all know Spring is about DI, IOP and Enterprise Service Abstractions (libraries). This lecture went through each of the components giving real case utilization of them. We saw @Configuration and @Bean to provide configuration from Java code instead of XML, @Value to allow the injection of a system property as a string member of a class, Spring scopes to be used for example to eliminate the need to pass custom context objects from method to method (A technique that I still use in my projects for maintaining ControllerContext), even though Singleton and Prototype are the most common scopes there are several others like Request and Session, @Profile annotation to define production or development. The Spring expression language in general was introduced.

Improving Java with Groovy

There were several trcks of Groovy and Grails. I attended just this one from the latest slice because there was nothing for Spring Java available and even though it was basic stuff we saw examples that did work from an excellent demo showing how to use both Java and Groovy in your project. It was basically a practical presentation of the ideas behind the book "Making Java Groovy". Heavy use of closures showed how the code got smaller while avoiding Interfaces and Inner Classes.

Cron error email notification in Ubuntu

2011-10-24T09:10:00.000-07:00

A member of the team realized one of the cron processes was failing but we were not getting any alerts.

While I favor the use of LogMonitor for absolutely everything there are simple commands you use in cron jobs for which ideally you would like to get notifications without the need of logging.

If you design your script carefully respecting the fact that error messages mut go to stderr this procedure should work:

Be sure sSMTP is installed and properly configured:

$ sudo apt-get install ssmtp
$ sudo vi /etc/ssmtp/ssmtp.conf
...
root=nurquiza@nestorurquiza.com
...
mailhub=mail.nestorurquiza.com
..
hostname=apache1
...

Add a line at the beginning of crontab stating who the error email should me sent to:
```
$ crontab -e 
MAILTO=alerts@nestorurquiza.com
0 22 * * * /usr/sbin/sftp_backup.sh
...
```

Upgrading CouchDB in Ubuntu

2011-10-22T17:42:00.001-07:00

I had to upgrade CouchDB to candidate release 1.1.1 and I did it just following the same instructions I followed to install it from scratch.

Installing CouchDB in Ubuntu

2011-10-22T16:54:00.000-07:00

I have tested the below procedure in Ubuntu 10.10 (Maverick) to install CouchDB 1.1.1 candidate release.

$ ps -ef|grep couch|awk '{print $2}'|xargs sudo kill -9
$ sudo apt-get update
$ sudo apt-get autoremove
$ sudo apt-get remove couchdb
$ sudo apt-get build-dep couchdb
$ sudo apt-get install libtool zip
$ cd
$ curl -O http://ftp.mozilla.org/pub/mozilla.org/js/js185-1.0.0.tar.gz
$ tar xvzf js185-1.0.0.tar.gz 
$ cd js-1.8.5/js/src
$ ./configure
$ make
$ sudo make install
$ cd
$ curl -O http://www.erlang.org/download/otp_src_R14B04.tar.gz
$ tar xvzf otp_src_R14B04.tar.gz 
$ cd otp_src_R14B04
$ ./configure --enable-smp-support --enable-dynamic-ssl-lib --enable-kernel-poll
$ make
$ sudo make install
$ cd
$ svn co http://svn.apache.org/repos/asf/couchdb/branches/1.1.x/ couchdb1.1.x
$ cd couchdb1.1.x
$ ./bootstrap
$ prefix='/usr/local'
$ ./configure --prefix=${prefix} 
$ make
$ sudo make install
$ sudo useradd -d /var/lib/couchdb couchdb
$ sudo chown -R couchdb: ${prefix}/var/{lib,log,run}/couchdb ${prefix}/etc/couchdb
$ for dir in `whereis couchdb | sed 's/couchdb: //'`; do echo $dir | xargs sudo chown couchdb; done
$ xulrunner -v #take note of the version x.x.x.x (in my case 1.9.2.24)
$ sudo vi /etc/ld.so.conf.d/xulrunner.conf #change x for the version below:
/usr/lib/xulrunner-x.x.x.x
/usr/lib/xulrunner-devel-x.x.x.x
$ sudo /etc/init.d/couchdb start
$ curl -X GET http://localhost:5984
{"couchdb":"Welcome","version":"1.1.1a1187726"}

Monitoring CouchDB with Monit

2011-10-21T16:03:00.000-07:00

Just add the below in monitrc and reload config. I am expecting you hardened CouchDB and so it is listening in an SSL port.

#################################################################
# couchdb
################################################################

check process couchdb
  with pidfile /usr/local/var/run/couchdb/couchdb.pid
  start program = "/usr/local/etc/init.d//couchdb start"
  stop program = "/usr/local/etc/init.d//couchdb stop"
  if failed host couchdb.nestorurquiza.com port 6984 type TCPSSL proto http then restart
  if failed url https://couchdb.nestorurquiza.com:6984/ and content == '"couchdb"' then restart
group couchdb

Hardening CouchDB: A more secure distributed database

2011-10-21T03:48:00.000-07:00

Here are the steps to follow in order to harden CouchDB. I am currently using candidate release 1.1.1 and these are instructions for a local environment. For a production environment you have to locate the non dev config files local.ini and default.ini (Just run "ps -ef | grep couchdb" and look for the path, in my case /usr/local/etc/couchdb/local.ini and /usr/local/etc/couchdb/default.ini).

Install a primary key and generated pem certificate:

$ export DOMAIN=couchdb.nestorurquiza.com
$ openssl genrsa -des3 -out ${DOMAIN}.pem 1024
$ openssl req -new -key ${DOMAIN}.pem -out ${DOMAIN}.csr
$ openssl x509 -req -days 3650 -in ${DOMAIN}.csr -signkey ${DOMAIN}.pem -out ${DOMAIN}.cert.pem
$ sudo mkdir -p /opt/couchdb/certs/
$ sudo cp ${DOMAIN}.pem /opt/couchdb/certs/
$ sudo cp ${DOMAIN}.cert.pem /opt/couchdb/certs/

Configure couchDB for *SSL only* and restart the server

$ vi /Users/nestor/Downloads/couchdb1.1.x/etc/couchdb/local_dev.ini
[admins]
;admin = mysecretpassword
nestorurquizaadmin = secret1
...
[daemons]
; enable SSL support by uncommenting the following line and supply the PEM's below.
; the default ssl port CouchDB listens on is 6984
httpsd = {couch_httpd, start_link, [https]}
...
[ssl]
cert_file = /opt/couchdb/certs/couchdb.nestorurquiza.com.cert.pem
key_file = /opt/couchdb/certs/couchdb.nestorurquiza.com.pem
$ vi /Users/nestor/Downloads/couchdb1.1.x/etc/couchdb/default_dev.ini
[httpd]
bind_address = a.non.loopback.ip #Use no loopback address only if the server will be exposed to a different machine
...
[daemons]
...
; httpd={couch_httpd, start_link, []}
...

Restart

sudo couchdb #local OSX
sudo /usr/local/etc/init.d/couchdb restart #Ubuntu

Now we can access the secured CouchDB instance

HOST="https://nestorurquizaadmin:mysecret@127.0.0.1:6984"
curl -k -X GET $HOST

Note that at this time to remove the insecure http you have to use /usr/local/etc/couchdb/default.ini which gets overwritten after any upgrade.

Of course we should sign our certificate with a Certificate Authority (CA) to make this really secure but you can also add some security if you play with firewall rules and allow access to the SSL port to certain IP only. I can do so because I do use a middle tear between CouchDB and the browser. If you directly hitting CouchDB from the wild you better use a CA.

CouchDB filtered replication

2011-10-20T13:10:00.000-07:00

One of the greatest features of CouchDB is its replication which allows for great distributed computing. It reminds me when in 1999 I met Erlang language for the first time (Working for a Telco). Erlang is made for distributed computing and so CouchDB which of course is built in Erlang.

I have to say I have successfully tested this in upcoming version 1.1.1 (built from a branch) Do not try this in 1.1.0.

The example below is based on the document that I have been discussing in the three part tutorial about building a Document Management System (DMS) with CouchDB.

Filtered or selective replication is a two step process:

First create a filter named for example "clientFilter" in a new document called "replicateFilter". This sample filter will reject any client not matching the clientId parameter (step 2 explains what this parameter is about). Any deleted documents will be deleted from the target as well.

curl -H 'Content-Type: application/json' -X PUT http://127.0.0.1:5984/dms4/_design/replicateFilter -d \
'{
  "filters":{
    "clientFilter":"function(doc, req) {
      if (doc._deleted) {
        return true;
      }
 
      if(!doc.clientId) {
        return false;
      }
 
      if(!req.query.clientId) {
        throw(\"Please provide a query parameter clientId.\");
      }
 
      if(doc.clientId == req.query.clientId) {
        return true;
      }
      return false;
    }"
  }
}'

Create a replication document called "by_clientId". This example passes clientId=1 as a parameter to the filter we created in step number 1 ("replicateFilter/clientFilter"). You figured we will end up replicating documents for that client.

curl -H 'Content-Type: application/json' -X POST http://127.0.0.1:5984/_replicator -d \
'{
  "_id":"by_clientId",
  "source":"dms4",
  "target":"http://couchdb.nestorurquiza.com:5984/dms4",
  "create_target":true,
  "continuous":true,
  "filter":"replicateFilter/clientFilter",
  "query_params":{"clientId":1}
}'

Deleting a replication document is how you turn off that replication. This is not any different than deleting any other document:

nestor:~ nestor$ curl -X GET http://127.0.0.1:5984/_replicator/by_clientId
{"_id":"by_clientId","_rev":"5-e177ca7f79d9ba6f91b803a2cb2abc1e","source":"dms4","target":"http://couchdb.nestorurquiza.com:5984/dms4","create_target":true,"continuous":true,"filter":"replicateFilter/clientFilter","query_params":{"clientId":1},"_replication_state":"triggered","_replication_state_time":"2011-10-20T13:09:56-04:00","_replication_id":"d8dc09e97f4948de0294260dda19fc6f"}
nestor:~ nestor$ curl -X DELETE http://127.0.0.1:5984/_replicator/by_clientId?rev=5-e177ca7f79d9ba6f91b803a2cb2abc1e
{"ok":true,"id":"by_clientId","rev":"6-0d20d90cbed22837eb3233e2bd8dfb2c"}

The same applies for getting a list of the current defined "selective replicators". You can use a temporary view like I show here or create a permanent view to list all the replicators:

$ curl -X POST http://127.0.0.1:5984/_replicator/_temp_view -H "Content-Type: application/json" -d '{
  "map": "function(doc) {
            emit(null, doc);
          }"
}'

OSX homebrew update: unexpected token near HOMEBREW_BREW_FILE

2011-10-20T09:59:00.000-07:00

After upgrading homebrew:
$ brew update

I got errors:
$ brew install mozilla-js
/usr/local/bin/brew: line 4: syntax error near unexpected token `('
/usr/local/bin/brew: line 4: `HOMEBREW_BREW_FILE = ENV['HOMEBREW_BREW_FILE'] = File.expand_path(__FILE__)'

Here is a temporary solution which you will need to do everytime you update brew:
$ sudo vi /usr/local/bin/brew
#!/Users/nestor/.rvm/rubies/ruby-1.9.2-p290/bin/ruby
##!/usr/bin/ruby

Upgrading couchDB in OSX

2011-10-20T07:51:00.001-07:00

While trying to use couchDB filtered replication I had some problems as documented in gist.

I had to upgrade then couchDB to a non released version. Here are the steps I followed.

Delete all files from previous installations:

$ sudo find /usr/local -name couchdb | sudo xargs rm -fR

Install ICU from http://download.icu-project.org

$ tar xvzf icu4c-4_8_1-src.tgz 
$ cd icu/source/
$ ./runConfigureICU MacOSX --with-library-bits=64 --disable-samples --enable-static # if this fails for you try: ./configure --enable-64bit-libs
$ make
$ sudo make install

Install Spidermonkey

$ curl -O http://ftp.mozilla.org/pub/mozilla.org/js/js185-1.0.0.tar.gz
$ tar xvzf js185-1.0.0.tar.gz 
$ cd js-1.8.5/js/src
./configure 
$ make
$ sudo make install

Then install latest Erlang:

$ curl -O http://www.erlang.org/download/otp_src_R14B04.tar.gz
$ tar xvzf otp_src_R14B04.tar.gz 
$ cd otp_src_R14B04
$ ./configure --enable-smp-support --enable-dynamic-ssl-lib --enable-kernel-poll --enable-darwin-64bit
$ make
$ sudo make install

Then checkout the needed version. I tried from git (http://git-wip-us.apache.org/repos/asf/couchdb.git) first but I had several problems with autoconf and beyond (.configure was not available so I needed to go with automake -a; autoconf; autoheader) so I then built from svn:

$ svn co http://svn.apache.org/repos/asf/couchdb/branches/1.1.x/ couchdb1.1.x
$ cd couchdb1.1.x
$ ./bootstrap
$ ./configure
$ make
$ sudo make install
$ sudo couchdb

Document Management System with CouchDB - Third Part

2011-10-16T21:12:00.000-07:00

This is the last part of my attempt to cover how to use CouchDB to build a DMS. We will be using Java and the Ektorp library for the implementation.

Using Ektorp

A good place to start is downloading the BlogPost application (org.ektorp.sample-1.7-project) that shows some of the most important concepts around the Ektorp API. Follow Ektorp Tutorial and of course do not miss the reference documentation.

I will show here the steps on how to make your existing spring project able to interact with CouchDB using the Ektorp project. The code presented here is an implementation of the ideas exposed in the second part of this project.

Here the dependencies for your project. Note that I am adding file upload dependencies because I am building a DMS and I will provide an upload module:

<!-- Ektorp for CouchDB -->
        <dependency>
            <groupId>org.ektorp</groupId>
            <artifactId>org.ektorp</artifactId>
            <version>${ektorp.version}</version>
            <exclusions>
             <exclusion>
              <artifactId>slf4j-api</artifactId>
              <groupId>org.slf4j</groupId>
             </exclusion>
            </exclusions>
        </dependency>
        <dependency>
            <groupId>org.ektorp</groupId>
            <artifactId>org.ektorp.spring</artifactId>
            <version>${ektorp.version}</version>
        </dependency>

        <!-- File Upload -->
        <dependency>
            <groupId>commons-fileupload</groupId>
            <artifactId>commons-fileupload</artifactId>
            <version>1.2.2</version>
        </dependency>

In spring application context xml (Note the upload component which is only needed because again we need it for the DMS upload functionality):

...
xmlns:util="http://www.springframework.org/schema/util"
xmlns:couchdb="http://www.ektorp.org/schema/couchdb"
...
http://www.springframework.org/schema/util http://www.springframework.org/schema/util/spring-util-3.0.xsd
http://www.ektorp.org/schema/couchdb http://www.ektorp.org/schema/couchdb/couchdb.xsd
...
<context:component-scan base-package="com.nu.dms.couchdb.ektorp.model"/>
<context:component-scan base-package="com.nu.dms.couchdb.ektorp.dao"/>
...
<util:properties id="couchdbProperties" location="classpath:/couchdb.properties"/>
<couchdb:instance id="dmsCouchdb" url="${couchdb.url}" properties="couchdbProperties" />
<couchdb:database id="dmsDatabase" name="${couchdb.db}" instance-ref="dmsCouchdb" />
...
<!-- File Upload -->
<bean id="multipartResolver" class="org.springframework.web.multipart.commons.CommonsMultipartResolver">
        <property name="maxUploadSize" value="2000000"/>
</bean>

In classpath environment properties file:

couchdb.url=http://localhost:5984
couchdb.db=dms4 #my database name

File classpath couchdb.properties file:

host=localhost
port=5984
maxConnections=20
connectionTimeout=1000
socketTimeout=10000
autoUpdateViewOnChange=true
caching=false

A Document POJO following the BlogPost POJO from the Ektorp example:

package com.nu.dms.couchdb.ektorp.model;

import org.ektorp.Attachment;
import org.ektorp.support.CouchDbDocument;

public class CustomCouchDbDocument extends CouchDbDocument {

    private static final long serialVersionUID = -9012014877538917152L;

    @Override
    public void addInlineAttachment(Attachment a) {
        super.addInlineAttachment(a);
    }   
}


package com.nu.dms.couchdb.ektorp.model;

import java.util.Date;

import javax.validation.constraints.NotNull;

import org.ektorp.support.TypeDiscriminator;

public class Document extends CustomCouchDbDocument {

    private static final long serialVersionUID = 59516215253102057L;
    
    public Document() {
        super();
    }
    
    public Document(String title) {
        this.title = title;
    }
    
    /**
     * @TypeDiscriminator is used to mark properties that makes this class's documents unique in the database. 
     */
    @TypeDiscriminator
    @NotNull
    private String title;
    
    private int clientId;
    private int createdByEmployeeId;
    private int reviewedByEmployeeId;
    private int approvedByManagerId;
    private Date dateEffective;
    private Date dateCreated;
    private Date dateReviewed;
    private Date dateApproved;
    private int investorId;
    private int categoryId;
    private int subCategoryId;
    private int statusId;
    
    public String getTitle() {
        return title;
    }
    public void setTitle(String title) {
        this.title = title;
    }
    public int getClientId() {
        return clientId;
    }
    public void setClientId(int clientId) {
        this.clientId = clientId;
    }
    public int getCreatedByEmployeeId() {
        return createdByEmployeeId;
    }
    public void setCreatedByEmployeeId(int createdByEmployeeId) {
        this.createdByEmployeeId = createdByEmployeeId;
    }
    public int getReviewedByEmployeeId() {
        return reviewedByEmployeeId;
    }
    public void setReviewedByEmployeeId(int reviewedByEmployeeId) {
        this.reviewedByEmployeeId = reviewedByEmployeeId;
    }
    public int getApprovedByManagerId() {
        return approvedByManagerId;
    }
    public void setApprovedByManagerId(int approvedByManagerId) {
        this.approvedByManagerId = approvedByManagerId;
    }
  
    public Date getDateEffective() {
        return dateEffective;
    }

    public void setDateEffective(Date dateEffective) {
        this.dateEffective = dateEffective;
    }

    public Date getDateCreated() {
        return dateCreated;
    }
    public void setDateCreated(Date dateCreated) {
        this.dateCreated = dateCreated;
    }
    public Date getDateReviewed() {
        return dateReviewed;
    }
    public void setDateReviewed(Date dateReviewed) {
        this.dateReviewed = dateReviewed;
    }
    public Date getDateApproved() {
        return dateApproved;
    }
    public void setDateApproved(Date dateApproved) {
        this.dateApproved = dateApproved;
    }
    public int getInvestorId() {
        return investorId;
    }
    public void setInvestorId(int investorId) {
        this.investorId = investorId;
    }
    public int getCategoryId() {
        return categoryId;
    }
    public void setCategoryId(int categoryId) {
        this.categoryId = categoryId;
    }
    public int getSubCategoryId() {
        return subCategoryId;
    }

    public void setSubCategoryId(int subCategoryId) {
        this.subCategoryId = subCategoryId;
    }
    public int getStatusId() {
        return statusId;
    }
    public void setStatusId(int statusId) {
        this.statusId = statusId;
    }
    @Override
    public void setRevision(String s) {
        // downstream code does not like revision set to emtpy string, which Spring does when binding
        if (s != null && !s.isEmpty()) super.setRevision(s);
    }
    
    public boolean isNew() {
        return getId() == null;
    }
}

A Document Repository:

package com.nu.dms.couchdb.ektorp.dao;

import org.ektorp.CouchDbConnector;
import org.ektorp.support.CouchDbRepositorySupport;

public class CustomCouchDbRepositorySupport<T> extends CouchDbRepositorySupport<T> {


    protected CustomCouchDbRepositorySupport(Class<T> type, CouchDbConnector db) {
        super(type, db);
    }

    public CouchDbConnector getDb() {
        return super.db;
    }   
}

package com.nu.dms.couchdb.ektorp.dao;

import java.io.InputStream;

@Component
public class DocumentRepository  extends CustomCouchDbRepositorySupport<Document> {
    
    private static final Logger log = LoggerFactory.getLogger(DocumentRepository.class);
    
    @Autowired
    public DocumentRepository(@Qualifier("dmsDatabase") CouchDbConnector db) {
        super(Document.class, db);
        initStandardDesignDocument();
    }

    @GenerateView @Override
    public List<Document> getAll() {
        ViewQuery q = createQuery("all")
                        .includeDocs(true);
        return db.queryView(q, Document.class);
    }
    
    public Page<Document> getAll(PageRequest pr) {
        ViewQuery q = createQuery("all")
                        .includeDocs(true);
        return db.queryForPage(q, pr, Document.class);
    }
    
    @View( name = "tree", map = "classpath:/couchdb/tree_map.js", reduce = "classpath:/couchdb/tree_reduce.js")
    public InputStream getTree(String startKey, String endKey, int groupLevel) {
        ViewQuery q = createQuery("tree")
        .startKey(startKey)
        .endKey(endKey)
        .groupLevel(groupLevel)
        .group(true);
        InputStream is = db.queryForStream(q);
        return is;
    }

}

Map and Reduce javascript functions in src/main/resources/couchdb in other words in the classpath:

//by_categoryId_map.js
function(doc) { 
 if(doc.title && doc.categoryId) {
  emit(doc.categoryId, doc._id)
 } 
}

//by_categoryId_reduce.js
_count

//tree_map.js
function(doc) {
  var tokens = doc.dateEffective.split("-");
  var year = null;
  var month = null;
  if(tokens.length == 3) {
    year = tokens[0];
    month = tokens[1];
  }
  var key = [doc.clientId, doc.categoryId, doc.subCategoryId, year, month].concat(doc.title);
  var value = null;
  emit(key, value);
}

//tree_reduce.js
_count

In web.xml we need to allow flash because we will use the swfUpload to upload multiple files:

<url-pattern>*.swf</url-pattern>
</servlet-mapping>

Keeping things short I am not using a Service layer so I provide a unique Controller that allows creating a document including the attachment and metadata, uploading multiple documents (using swfupload flash) which follow a strong naming convention to produce all necessary metadata our of their names, viewing the documents as explained in part two in a tree view (using jquery.treeview.async.js)

package com.nu.web.controller.dms;

import java.io.ByteArrayOutputStream;
import java.io.InputStream;
import java.net.URLConnection;
import java.text.SimpleDateFormat;
import java.util.Date;
import java.util.Iterator;
import java.util.Locale;
import java.util.Map;
import java.util.Scanner;

import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;

import org.apache.commons.codec.binary.Base64;
import org.apache.commons.fileupload.FileItemFactory;
import org.apache.commons.fileupload.FileUploadException;
import org.apache.commons.fileupload.disk.DiskFileItemFactory;
import org.apache.commons.fileupload.servlet.ServletFileUpload;
import org.apache.commons.lang.StringUtils;
import org.apache.poi.util.IOUtils;
import org.codehaus.jackson.JsonNode;
import org.codehaus.jackson.map.ObjectMapper;
import org.codehaus.jackson.node.ObjectNode;
import org.ektorp.Attachment;
import org.ektorp.AttachmentInputStream;
import org.ektorp.PageRequest;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Controller;
import org.springframework.ui.Model;
import org.springframework.validation.BindingResult;
import org.springframework.validation.ObjectError;
import org.springframework.web.bind.WebDataBinder;
import org.springframework.web.bind.annotation.InitBinder;
import org.springframework.web.bind.annotation.ModelAttribute;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.multipart.MultipartFile;
import org.springframework.web.multipart.support.ByteArrayMultipartFileEditor;
import org.springframework.web.servlet.ModelAndView;

import com.nu.dms.couchdb.ektorp.dao.DocumentRepository;
import com.nu.dms.couchdb.ektorp.model.Document;
import com.nu.web.ControllerContext;
import com.nu.web.RootController;
import com.nu.web.WebConstants;
import com.nu.web.validator.BeanValidator;
import com.windriver.gson.extension.GeneralObjectDeserializer;

@Controller
@RequestMapping("/dms/document/*")
public class DocumentController extends RootController {
    private static final Logger log = LoggerFactory.getLogger(DocumentController.class);
    
    @Autowired
    DocumentRepository documentRepository;
    
    @Autowired
    private BeanValidator validator;
    
    private static final String LIST_PATH = "/dms/document/list";
    private static final String FORM_PATH = "/dms/document/form";
    private static final String TREE_PATH = "/dms/document/tree";
    public static final long UPLOAD_MAX_FILE_SIZE = 20 * 1024 * 1024; //10 MB
    public static final long UPLOAD_MAX_TOTAL_FILES_SIZE = 1 * 1024 * 1024 * 1024; //1 GB
    private static final int DOCUMENT_LEVEL = 5;
    
    @RequestMapping("/list")
    public ModelAndView list(   HttpServletRequest request, 
                                HttpServletResponse response, 
                                Model m, 
                                @RequestParam(value = "p", required = false) String pageLink) {
        ControllerContext ctx = new ControllerContext(request, response);
        init(ctx);

        if (!isValidCsrfToken(ctx)) {
            return getModelAndView(ctx, LIST_PATH);
        }
        
        PageRequest pr = pageLink != null ? PageRequest.fromLink(pageLink) : PageRequest.firstPage(5);
        m.addAttribute(documentRepository.getAll(pr));
        return getModelAndView(ctx, LIST_PATH);
    }
    
    @RequestMapping("/add")
    public ModelAndView add(HttpServletRequest request,
            HttpServletResponse response,
            @RequestParam(value = "attachment", required = false) MultipartFile multipartFile,
            @ModelAttribute("document") Document document,
            BindingResult result) {

        //Store constants for JSP
        request.setAttribute("UPLOAD_MAX_FILE_SIZE", UPLOAD_MAX_FILE_SIZE); 
        
        ControllerContext ctx = new ControllerContext(request, response);
        init(ctx);

        if (!isValidCsrfToken(ctx)) {
            return getModelAndView(ctx, FORM_PATH);
        }

        if (!isSubmission(ctx)) {
            return getModelAndView(ctx, FORM_PATH);
        } else {
            
            validator.validate(document, result);
            
            if (result.hasErrors()) {
                return getModelAndView(ctx, FORM_PATH);
            } else {
                if(multipartFile == null) {
                    result.addError(new ObjectError("document", getMessage("error.add", new String[] {"document"})));
                    return getModelAndView(ctx, FORM_PATH);
                }
                
                try {
                    String title = document.getTitle();
                    if(StringUtils.isEmpty(title)) throw new Exception("Empty title");
                    document.setId(title);
                    String contentType = multipartFile.getContentType();
                    String base64 = new String (Base64.encodeBase64(multipartFile.getBytes()));
                    if(StringUtils.isEmpty(base64)) throw new Exception("Empty attachment");
                    Attachment a = new Attachment(title, base64, contentType);
                    document.addInlineAttachment(a);
                } catch (Exception ex) {
                    result.addError(new ObjectError("attachmentError", getMessage("error.attachingDocument")));
                    log.error(null, ex);
                    return getModelAndView(ctx, FORM_PATH);
                }   
                
                try {
                    document.setDateCreated(new Date());
                    documentRepository.add(document);
                } catch (Exception ex) {
                    result.addError(new ObjectError("document", getMessage("error.add", new String[] {"document"})));
                    log.error(null, ex);
                    return getModelAndView(ctx, FORM_PATH);
                }
                return getModelAndView(ctx, LIST_PATH, true, true);
            }
        }
    }
    
    
    @RequestMapping("/{id}/edit")
    public ModelAndView edit(HttpServletRequest request,
            HttpServletResponse response,
            @ModelAttribute("document") Document document,
            BindingResult result,
            @PathVariable("id") String id,
            Model model) {

        ControllerContext ctx = new ControllerContext(request, response);
        init(ctx);

        if (!isValidCsrfToken(ctx)) {
            return getModelAndView(ctx, FORM_PATH);
        }

        try {
            Document storedDocument = documentRepository.get(id);
            if(storedDocument == null) {
                throw new Exception("No document found with id '" + id + "'");
            }
           
            if (!isSubmission(ctx)) {
                model.addAttribute("document", storedDocument);
                return getModelAndView(ctx, FORM_PATH);
            } else {
                validator.validate(document, result);
                
                if (result.hasErrors()) {
                    return getModelAndView(ctx, FORM_PATH);
                } else {
                    String title = document.getTitle();
                    if(StringUtils.isEmpty(title)) throw new Exception("Empty title");
                    document.setId(title);
                    document.setDateCreated(storedDocument.getDateCreated());
                    document.setRevision(storedDocument.getRevision());
                    documentRepository.update(document);
                    return getModelAndView(ctx, LIST_PATH, true, true);
                }
            }
        } catch (Exception ex) {
            result.addError(new ObjectError("document", getMessage("error.edit", new String[] {"document"}) + "." + ex.getMessage()));
            log.error(null, ex);
            return getModelAndView(ctx, FORM_PATH);
        }
    }
    
    
    @RequestMapping("/{id}/delete")
    public ModelAndView delete(HttpServletRequest request,
            HttpServletResponse response,
            @ModelAttribute("document") Document document,
            BindingResult result,
            @PathVariable("id") String id,
            Model model) {
        ControllerContext ctx = new ControllerContext(request, response);
        init(ctx);

        if (!isValidCsrfToken(ctx)) {
            return getModelAndView(ctx, LIST_PATH);
        }

        try {
            document = documentRepository.get(id);
            if(document == null) {
                throw new Exception("No document found with id '" + id + "'");
            }
            documentRepository.remove(document);
        } catch (Exception ex) {
            result.addError(new ObjectError("document", getMessage("error.add", new String[] {"document"})));
            log.error(null, ex);
            return getModelAndView(ctx, LIST_PATH);
        }
        return getModelAndView(ctx, LIST_PATH, true, true);
    }
    
    @RequestMapping("/{id}/show")
    public ModelAndView show(HttpServletRequest request,
            HttpServletResponse response,
            @ModelAttribute("document") Document document,
            BindingResult result,
            @PathVariable("id") String id,
            Model model) {
        ControllerContext ctx = new ControllerContext(request, response);
        init(ctx);

        if (!isValidCsrfToken(ctx)) {
            return getModelAndView(ctx, LIST_PATH);
        }

        try {
            document = documentRepository.get(id);
            if(document == null) {
                throw new Exception("No document found with id '" + id + "'");
            }
            Map<String, Attachment> attachments = document.getAttachments();
            if(attachments == null || attachments.size() == 0) {
                throw new Exception("No attachment found for id '" + id + "'");
            }
            for(Map.Entry<String, Attachment> entry : attachments.entrySet()) {
                String attachmentId = entry.getKey();
                Attachment attachment = entry.getValue();
                //long contentLength = attachment.getContentLength();
                String contentType = attachment.getContentType();
                AttachmentInputStream ais = documentRepository.getDb().getAttachment(id, attachmentId);
                response.setHeader("Content-Disposition", "attachment; filename=\"" + document.getTitle() + "\"");
                response.setContentType(contentType);
                final ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
                IOUtils.copy(ais, outputStream);
                render(response, outputStream);
            }
            return getModelAndView(ctx, LIST_PATH, true, true);
        } catch (Exception ex) {
            result.addError(new ObjectError("document", getMessage("error.internal", new String[] {"document"})));
            log.error(null, ex);
            return getModelAndView(ctx, LIST_PATH);
        }
        
    }
    
    @RequestMapping("/tree")
    public ModelAndView tree(HttpServletRequest request,
            HttpServletResponse response,
            @RequestParam(value = "root", required = false) String root) {
        ControllerContext ctx = new ControllerContext(request, response);
        init(ctx);

        if (!isValidCsrfToken(ctx)) {
            return getModelAndView(ctx, TREE_PATH);
        }
        if(root == null) {
            return getModelAndView(ctx, TREE_PATH);
        }
        try {
            Object objTree = getTreeObject(root);
            if(objTree == null) {
                ctx.setRequestAttribute("treeInfo", getMessage("noItemFound", new String[] {"record"}));
            }
            ctx.setRequestAttribute("tree", objTree);
            return getModelAndView(ctx, TREE_PATH);
        } catch (Exception ex) {
            ctx.setRequestViewAttribute("treeError", ex.getMessage());
            log.error(null, ex);
            return getModelAndView(ctx, TREE_PATH);
        }
        
    }
    
    /**
     * The needs for the current treeview plugin makes mandatory certain json structure so parsing the /document/tree service is not an option at the moment
     * @param request
     * @param response
     * @param root
     * @return
     */
    @RequestMapping("/ajaxTree")
    public ModelAndView ajaxTree(HttpServletRequest request,
            HttpServletResponse response,
            @RequestParam(value = "root", required = true) String root) {
        ControllerContext ctx = new ControllerContext(request, response);
        init(ctx);

        if (!isValidCsrfToken(ctx)) {
            return getModelAndView(ctx, LIST_PATH);
        }

        try {
            InputStream is = getTreeInputStream(root);
            ObjectMapper mapper = new ObjectMapper();
            JsonNode rootNode = mapper.readValue(is, JsonNode.class);
            JsonNode rowsNode = rootNode.get("rows");
            Iterator<JsonNode> iter = rowsNode.getElements();
            while (iter.hasNext()) {
                JsonNode row = iter.next();
                JsonNode keyNode = row.get("key");
                String[] key = mapper.readValue(keyNode, String[].class);
                if(key == null) {
                    continue;
                }
                String name = key[key.length - 1];
                String classes = null;
                if(key.length == DOCUMENT_LEVEL + 1) {
                    //Listing files
                    String extension = "unknownExtension";
                    String fileName = key[DOCUMENT_LEVEL];
                    String[] tokens = fileName.split("\\.");
                    if(tokens.length >= 2) {
                        extension = tokens[tokens.length - 1];
                    }
                    classes = "file " + extension;
                } else {
                    //Listing folders
                    classes = "folder";
                }
                
                ((ObjectNode)row).put("classes", classes);
                ((ObjectNode)row).put("name", name);
                boolean hasChildren = false;
                //To use the key as id for next request
                if(key.length != DOCUMENT_LEVEL + 1) {
                    int value = mapper.readValue(row.get("value"), Integer.class);
                    if(value > 0) {
                        hasChildren = true;
                    }
                } else {
                    ((ObjectNode)row).put("url", request.getContextPath() + "/dms/document/" + name + "/show?ctoken=" + ctx.getSessionAttribute(WebConstants.CSRF_TOKEN, ""));
                }
                ((ObjectNode)row).put("hasChildren", hasChildren);
                ((ObjectNode)row).put("id", keyNode.toString().replaceAll("[\\[\\]]", ""));
                
            }
            
            response.setContentType("application/json");
            final ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
            mapper.writeValue(outputStream, rowsNode);
            render(response, outputStream);
            return null;
        } catch (Exception ex) {
            log.error(null, ex);
            return null;
        }
        
    }
    
    private Object getTreeObject(String root) {
        InputStream is = getTreeInputStream(root);
        String tree = new Scanner(is).useDelimiter("\\A").next();
        Object objTree = GeneralObjectDeserializer.fromJson(tree);
        return objTree;
    }
    
    private InputStream getTreeInputStream(String root) {
        //Making it compatible with the jquery tree view in use by Portal in Liferay
        String endKey = null;
        if("0".equals(root)) {
            endKey = "[{}]";
        } else {
            endKey = "[" + root.substring(0,root.length()) + ",{}]";
        }
         
        String[] tokens = endKey.replaceAll("[\\[\\]]", "").split(",");
        int groupLevel = tokens.length;
        String startKey = "[1]";
        if(!"{}".equals(tokens[0])) {
            startKey = "[" + root + "]";
        }
        InputStream is = documentRepository.getTree(startKey, endKey, groupLevel);
        return is;
    }
    
    @InitBinder
    public void initBinder(WebDataBinder binder) {
        binder.registerCustomEditor(byte[].class, new ByteArrayMultipartFileEditor());
    }

    
    /**
     * To be consumed by swfupload.swf which is in charge of uploading multiple files
     * 
     * @param request
     * @param response
     * @return
     */
    @RequestMapping("/addBatch")
    public void addBatch(HttpServletRequest request,
            HttpServletResponse response,
            @RequestParam(value = "Filedata", required = false) MultipartFile multipartFile,
            @ModelAttribute("document") Document document,
            BindingResult result) {

        String message = "Completed";
        ControllerContext ctx = new ControllerContext(request, response);
        init(ctx);
        
        try {
            if (!isValidCsrfToken(ctx)) {
                message = error("Invalid session token");
            } else {
                uploadFile(request, multipartFile);
            }
            
        } catch (FileUploadException e) {
            log.error("FileUploadException:", e);
            message = error(e.getMessage());
        } catch (Exception e) {
            String errorMessage = e.toString();
            log.error("FileUploadException:", e);
            if (errorMessage == null) {
                errorMessage = "Internal Error. Please look at server logs for more detail";
            }
            message = error(errorMessage);
        }
        
        try {
            render(response, message.getBytes());
        } catch (Exception ex) {
            log.error(null, ex);
        }
    }
    
    /**
     * File must follow this convention: clientId_categoryId_subCategoryId_year_month_investorId.extension
     * @param req
     * @param multipartFile
     * @throws Exception
     */
    private void uploadFile(HttpServletRequest req, MultipartFile multipartFile) throws Exception {

            // Create a new file upload handler
            FileItemFactory factory = new DiskFileItemFactory();
            ServletFileUpload upload = new ServletFileUpload(factory);
            upload.setFileSizeMax(UPLOAD_MAX_FILE_SIZE);
            upload.setSizeMax(UPLOAD_MAX_TOTAL_FILES_SIZE);

            if(multipartFile == null) {
                throw new FileUploadException(getMessage("error.add", new String[] {"document"}));
            }
            
            String fileName = multipartFile.getOriginalFilename();
            String[] tokens = fileName.split("_");
            if(tokens.length != 6) {
                throw new Exception("Filename must have 6 tokens");
            }
            int clientId = Integer.parseInt(tokens[0]);
            int categoryId = Integer.parseInt(tokens[1]);
            int subCategoryId = Integer.parseInt(tokens[2]);
            String year = tokens[3];
            String month = tokens[4];
            //Use whole filename as title
            int investorId = Integer.parseInt(tokens[5].split("\\.")[0]);
            String title = fileName;
            //Using swfupload we almost always get "application/octet-stream"
            String contentType = multipartFile.getContentType();
            String guessedContentType = URLConnection.guessContentTypeFromName(fileName);
            if (guessedContentType != null) {
                contentType = guessedContentType;
            }
            String base64 = new String (Base64.encodeBase64(multipartFile.getBytes()));
            if(StringUtils.isEmpty(base64)) throw new Exception("Empty attachment");
            Attachment a = new Attachment(fileName, base64, contentType);
            Document document = new Document(title);
            document.setId(fileName);
            document.addInlineAttachment(a);
            document.setDateCreated(new Date());
            document.setClientId(clientId);
            document.setCategoryId(categoryId);
            document.setSubCategoryId(subCategoryId);
            document.setDateEffective(new SimpleDateFormat("yyyy-MM-dd", Locale.ENGLISH).parse(year + "-" + month + "-" + 1));
            document.setInvestorId(investorId);
            documentRepository.add(document);
            
            //Within Spring the below does not work
            /*
            List<FileItem> items = (List<FileItem>) upload.parseRequest(req);
            Iterator<FileItem> iter = items.iterator();
            while (iter.hasNext()) {
                FileItem item = iter.next();
                if (item.isFormField()) {
                    String name = item.getFieldName();
                    String value = item.getString();
                    log.debug("Form field " + name + " with value " + value);
                } else {
                    String fileName = item.getName();
                    String contentType = item.getContentType();
                    Document document = new Document(fileName);
                    String base64 = new String (Base64.encodeBase64(item.get()));
                    if(StringUtils.isEmpty(base64)) throw new Exception("Empty attachment");
                    Attachment a = new Attachment(fileName, base64, contentType);
                    document.addInlineAttachment(a);
                    document.setDateCreated(new Date());
                    documentRepository.add(document);
                }
            }
            */
    }
    /*
     * To format the message so sfwupload understands it
     */
    private String error(String error) {
        return "ERROR: " + error;
    }
    
    
}

Below are some screenshots of our simple user interface:

Why CouchDB? and not a SQL database?

Relational Database Management Systems (RDBMS) are good to store tabular data, enforce relationship, remove duplicated information, ensure data consistency and the list goes on and on. There is one thing though that makes relational databases not ideal for distributed computing and that is locking. The need for an alternative comes from impediments related to replication but also from storing hierarchical structures in RDBMS which is not natural. Finally it is difficult to manage inheritance and schema changes can easily become a big problem when the system grows and new simple necessities emerge from business requirements. RDBMS are also slow if you want an index for all fields in a table or multiple complex indexes. If your content management system has a need for distributed computing, fast storage and you can afford the compromise about losing the ability for easy normalization (for example your documents once created are stamped with metadata available at that time and which does not change over time)

CouchDB is a noSql type database which stores data structures as documents (JSON Strings). Schema less, based on b-tree and with no locking (but Multi Version Concurrency Control) design it is a really fast alternative when you look for a solution to store hierarchical non-strict schema data like for example storing web pages or binary files. All this robustness is exposed with a HTTP REST API where JSON is used to send messages to the server as well as receive messages from it. This makes it really attractive for those looking for lightweight solutions. One of the "trade-offs" in couchDB is that the only way to query CouchDB without creating indexes is a temporary View and that is not an option for production as the mapped result will not be stored in a B-Tree hitting performance in your server.

I have no other option than considering CouchDB the logical pick for my BHUB Document Management functionality. CouchDB provides fast access to data specified by stored keys. Using Map functions in your Views there is no limit on the efficiency you can get out of the fact that you can create at any point a key composed of several document fields.

CouchDB has been engineered with replication in mind and that means you get distributed computing on top of the advantages I already discussed above. You just run a command specifying URLs for the source and the destination server and the replication is done. By default latest changes will be favored and if there are conflicts you will get the differences so you can update with changes that resolve the changes. Think about any versioning system like subversion for a comparison on how it works. You can replicate in both directions of course.

Document Management System with CouchDB - Second Part

2011-10-16T18:19:00.000-07:00

In the first part we installed CouchDB and explain how to create databases, documents, update them, create attachments and run queries.

Now we will design a DMS and plan for the different functionality we will need with one example.

For simplicity we will say our documents will be imported in batch for which it makes sense to have a convention for the file names client_cat_subcat_year_month_investor.ext. After importing the below 14 documents we can start navigating the tree. Note that in this example the importer code will replace the month by two digits so "3" becomes "03"

1_1_1_2003_1_1.pdf
1_1_1_2003_1_2.pdf
1_1_1_2003_2_3.pdf
1_1_1_2003_2_4.pdf
1_1_1_2004_3_5.pdf
1_1_2_2005_4_6.pdf
1_1_3_2006_5_7.pdf
1_1_3_2006_6_8.pdf
1_2_4_2007_7_9.pdf
2_3_5_2008_8_10.pdf
2_3_5_2009_9_11.pdf
2_3_6_2010_10_12.pdf
2_3_6_2010_11_13.pdf
2_3_7_2011_12_14.pdf

We want to offer a tree view of our documents. First we show the available clients. When the user clicks one of them we show the categories available. Clicking one of the categories will render all subcategories and so on. Let us go by the example of the first document (1_1_1_2003_1_1.pdf)

Here is how to pull all clients. Note the use of {} which means "any":

nestor-nu:~ nestor$ curl -X GET "http://127.0.0.1:5984/dms4/_design/Document/_view/tree?group=true&group_level=1" -G --data-urlencode startkey='[1]' --data-urlencode endkey='[{}]'
{"rows":[
{"key":[1],"value":9},
{"key":[2],"value":5}
]}

The categories for a client (1)

nestor-nu:~ nestor$ curl -X GET "http://127.0.0.1:5984/dms4/_design/Document/_view/tree?group=true&group_level=2" -G --data-urlencode startkey='[1]' --data-urlencode endkey='[1, {}]'
{"rows":[
{"key":[1,1],"value":8},
{"key":[1,2],"value":1}
]}

The subcategories for a client category (1,1)

nestor-nu:~ nestor$ curl -X GET "http://127.0.0.1:5984/dms4/_design/Document/_view/tree?group=true&group_level=3" -G --data-urlencode startkey='[1,1]' --data-urlencode endkey='[1, 1, {}]'
{"rows":[
{"key":[1,1,1],"value":5},
{"key":[1,1,2],"value":1},
{"key":[1,1,3],"value":2}
]}

The effective years for the client category subcategory (1,1,1)

nestor-nu:~ nestor$ curl -X GET "http://127.0.0.1:5984/dms4/_design/Document/_view/tree?group=true&group_level=4" -G --data-urlencode startkey='[1,1,1]' --data-urlencode endkey='[1,1,1,{}]'
{"rows":[
{"key":[1,1,1,"2003"],"value":4},
{"key":[1,1,1,"2004"],"value":1}
]}

The effective months for the client category subcategory year (1,1,1,"2003"). Note the quotes for 2003 as it is a String obtained from a token.

nestor-nu:~ nestor$ curl -X GET "http://127.0.0.1:5984/dms4/_design/Document/_view/tree?group=true&group_level=5" -G --data-urlencode startkey='[1,1,1,"2003"]' --data-urlencode endkey='[1,1,1,"2003",{}]'
{"rows":[
{"key":[1,1,1,"2003","01"],"value":2},
{"key":[1,1,1,"2003","02"],"value":2}
]}

The documents for the client category subcategory year month(1,1,1,"2003","01"). Note "01" instead "1" just because our importer is treating months as 2 digits values.

nestor-nu:~ nestor$ curl -X GET "http://127.0.0.1:5984/dms4/_design/Document/_view/tree?group=true&group_level=6" -G --data-urlencode startkey='[1,1,1,"2003","01"]' --data-urlencode endkey='[1,1,1,"2003","01",{}]'
{"rows":[
{"key":[1,1,1,"2003","01","1_1_1_2003_1_1.pdf"],"value":1},
{"key":[1,1,1,"2003","01","1_1_1_2003_1_2.pdf"],"value":1}
]}

So you have figured if we want to navigate to document 2_3_5_2009_9_11.pdf we just have to pass startkey='[2]' and endkey=[2,3,5,"2009","09",{}] and the group_level 6:

nestor-nu:~ nestor$ curl -X GET "http://127.0.0.1:5984/dms4/_design/Document/_view/tree?group=true&group_level=6" -G --data-urlencode startkey='[2,3,5,"2009","09"]' --data-urlencode endkey='[2,3,5,"2009","09",{}]'
{"rows":[
{"key":[2,3,5,"2009","09","2_3_5_2009_9_11.pdf"],"value":1}
]}

If we were to build a JSON web service we just need to accept the startkey. The endkey is always an array containing startkey with a new last element: {}.

Specifically if I use BHUB (which is just a concept around Spring Framework) a typical request and response will look like:

http://localhost:8080/nu-app/dms/document/tree?root=2,3,5,"2009","09"&ert=json
{"rows":[
{"key":[2,3,5,"2009","09","2_3_5_2009_9_11.pdf"],"value":1}
]}

In the final part I show how to use Java Erktop library to implement the DMS we have been covering so far.

Document Management System with CouchDB - First Part

2011-10-16T18:10:00.000-07:00

I will start documenting about my experience using CouchDB to build a Document Management System (DMS), an important component of any Content Management System (CMS).

The first part concentrates on installing and using CouchDB in OSX and Ubuntu.

OSX

Alternatively you could install from sources which I prefer to get later and greatest.

Download any pending updates for OSX. Then latest version of XCode
Install homebrew if you still not have it. It is the best package manager for OSX.

/usr/bin/ruby -e "$(curl -fsSL https://raw.github.com/gist/323731)"

Install couchDB. Following instructions from http://wiki.apache.org/couchdb/Installation with just one command. It could take a while, if it hangs then restart again, it will continue from where it broke.

brew install couchdb

Start the server.

$ couchdb
$ curl -X GET http://127.0.0.1:5984/

If you need to change the default port:

sudo vi  /usr/local/etc/couchdb/default.ini

Ubuntu

In Ubuntu I decided to build from sources to get latest available version for my 10.10 Maverick. For 11.4 I found this also works but you have to issue 'sudo apt-get remove libmozjs185-dev' in order to build.

Using CouchDB

Let us start interacting with CouchDB to create a database, a document, attach a file to it, update it etc. We use curl to be sure we can issue different HTTP request method (GET, POST, PUT, DELETE).

We create our Document Management System database and we confirm it was created:

$ curl -X PUT http://127.0.0.1:5984/dms
{"ok":true}
$ curl -X GET http://127.0.0.1:5984/_all_dbs
["_replicator","_users","dms"]

Let us create a first document. In this example we use POST instead of PUT so we get a UUID generated by CouchDB:

$ curl -d '{
    "name":"Investor Document 11",
    "clientId": "1001",
    "createdByEmployeeId": "2",
    "reviewedByEmployeeId": "1",
    "approvedByManagerId": "21",
    "created": "2/2/2011",
    "reviewed": "2/3/2011",
    "approved": "2/4/2011",
    "investorId": "32",
    "categoryId": "2",
    "statusId": "2"
}' -H "Content-Type: application/json" -X POST http://127.0.0.1:5984/dms

Result:

{"ok":true,"id":"296ef7cde8fe533efe0c7dded873505b","rev":"1-d5dd0fa82df07553f3a2b82947864fc6"}

Let us attach a file to the above document. Note we need the document is and rev:

$ curl -X PUT -H "Content-Type: application/pdf"  --data-binary @DailyReport.pdf  $DMS/296ef7cde8fe533efe0c7dded873505b/DailyReport.pdf?rev=1-d5dd0fa82df07553f3a2b82947864fc6

You can visually manage your CouchDB server via Futon user interface. Just hit http://localhost:5984/_utils/ and start playing with it.
Create some documents for different combinations of categoryId, clientId and investorId either from curl or from Futon.
Let us start querying our DB.

You query a View in CouchDB. Views are a combination of two functions that are applied to the original data: Map and Reduce (MapReduce style: Map functions generate indexes and Reduce queries are requests against them). Map function as its name suggests specifies the mapping between the document structure and the structure of the View. Reduce function as its name suggests specifies how to group the resulting data to reduce the results. The View is consequently just a transformation of the document where an index is usually defined. If you need to group a Reduce step will be applied as well.

You must become familiar with how to write the map and reduce functions for Views. This is done using the javascript language. From Futon select "Temporary View ..." option from the View dropdown. You have two panes now, the left is for your Map function and the right is for the Reduce. By default you see CouchDB proposes the below code which is equivalent to "Do not use any custom key and show all values from the document". There is no transformation nor custom index at all in this case, however if no key is specified couchDB uses the document id as unique identifier). Remember the Map function generates rows containing the id, an optional key and an optional value.
```
function(doc) {
  emit(null, doc);
}
```
Let us edit the function to "Use the name as key and show only clientId and investorId". When you run the view using both functions you will realize the difference. By now you should be aware that emit() just accepts two parameters, the key for an index and the value that will be returned. Of course the results come ordered by the Key if provided. Both key and value are json expressions as well.
```
function(doc) {
  //emit(doc.name, doc);
  if(doc.name && doc.clientId) {
    var key = doc.name;
    var value = {name: doc.name, clientId: doc.clientId, investorId: doc.investorId}
    emit(key, value);
  }
}
```

Here we use a composite key out of the clientId and the investorId so we can find the documents for that combination. Again the results are ordered first by clientId and later by investorId:

function(doc) {
  if(doc.clientId && doc.investorId) {
    var key = [doc.clientId, doc.investorId];
    var value = {name: doc.name, clientId: doc.clientId, investorId: doc.investorId}
    emit(key, value);
  }
}

Save the view. The options you pick will be used in the URL to retrieve the View results. I have decided to use "common" for the design document name and "by_client_investor" for the name of the view:
```
Design Document: _design/common
View Name: by_client_investor
```
Now the View is saved so we can query it at any time. The View is now "Permanent" and not longer "Temporary". Let us query it for just one key. Note that as we decided to use an array as key we will need to look for something like: ["1000","30"]. As you might have notice the key contains characters that must be URL encoded, in this case %5B%221000%22%2C%2230%22%5D. Here is how the command will look like:
```
$ curl -X GET http://127.0.0.1:5984/dms/_design/common/_view/by_client_investor?key=%5B%221000%22%2C%2230%22%5D
```
Alternatively you can use a more clear approach using some other curl flags:
```
curl -X GET http://127.0.0.1:5984/dms/_design/common/_view/by_client_investor -G --data-urlencode key='["1000","30"]'
```

Here is how you use curl to create and execute a temporary View from the command line. Here we are using categoryId as a key and getting the whole document as a result of the "non existent" transformation.

curl -X POST http://127.0.0.1:5984/dms/_temp_view -H "Content-Type: application/json" -d \
'{
  "map": "function(doc) {
            if (doc.categoryId) {
              emit(doc.categoryId, doc);
            }
          }"
}'

Let us explore the results of the below temporary View. Here we are insterested in the total documents by category. As we are grouping we need to use a Reduce function where we take advantage of the provided _count. Note the key is null because it counts all of the existing documents.

curl -X POST http://127.0.0.1:5984/dms/_temp_view -H "Content-Type: application/json" -d \
'{
  "map": "function(doc) {
            if (doc.categoryId) {
              emit(doc.categoryId, doc);
            }
          }",
  "reduce": "_count"
}'

Here is how we generate the counting by key which translates to use "Grouping=exact" from Futon or as shown below "group=true" from the HTTP request:

curl -X POST http://127.0.0.1:5984/dms/_temp_view?group=true -H "Content-Type: application/json" -d \
'{
  "map": "function(doc) {
            if (doc.categoryId) {
              emit(doc.categoryId, doc);
            }
          }",
  "reduce": "_count"
}'

We already saw how to make a View permanent while saving it from Futon. Here is how from an HTTP request you do the same. This time we are adding a View called category_count to a Design Document called category:

curl -X PUT http://127.0.0.1:5984/dms/_design/category -d \
'{
   "_id": "_design/category",
   "language": "javascript",
   "views": {
     "count": {
       "map":
         "function(doc) {
           if (doc.categoryId) {
             emit(doc.categoryId, doc);
           }
         }",
       "reduce": "_count"
      }
   }
}'

As we already saw we can query this view like this:

curl -X GET http://127.0.0.1:5984/dms/_design/category/_view/count
{"rows":[
{"key":"1","value":17},
{"key":"2","value":1}
]}

At this point you can interact with CouchDB from any language using plain REST commands. You might want to use some abstractions with an API that allows you to go through CRUD operations with CouchDB without being concern about the details of sending and parsing JSON.

In the next part we start the design of the DMS for which we will not use any specific language other than plain HTTP with the help of curl.

Tomcat 7 scans all jars for TLDs

2011-10-03T12:21:00.000-07:00

Tomcat 7 scans all jars for TLDs. I am unsure if tomcat 6 does the same:

INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.

Once the log level was increased to FINE in conf/logging.properties:

org.apache.catalina.core.ContainerBase.[Catalina].[localhost].level = FINE

We got:

...
FINE: No TLD files were found in [file:/opt/tomcat/webapps/nestorurquiza-app/WEB-INF/lib/org.springframework.transaction-3.0.5.RELEASE.jar]. Consider adding the JAR to the tomcat.util.scan.DefaultJarScanner.jarsToSkip property in CATALINA_BASE/conf/catalina.properties file.
Oct 3, 2011 2:05:58 PM org.apache.jasper.compiler.TldLocationsCache tldScanJar
FINE: No TLD files were found in [file:/opt/tomcat/webapps/nestorurquiza-app/WEB-INF/lib/jsr250-api-1.0.jar]. Consider adding the JAR to the tomcat.util.scan.DefaultJarScanner.jarsToSkip property in CATALINA_BASE/conf/catalina.properties file.
Oct 3, 2011 2:05:58 PM org.apache.jasper.compiler.TldLocationsCache tldScanJar
FINE: No TLD files were found in [file:/opt/tomcat/webapps/nestorurquiza-app/WEB-INF/lib/org.springframework.security.ldap-3.0.5.RELEASE.jar]. Consider adding the JAR to the tomcat.util.scan.DefaultJarScanner.jarsToSkip property in CATALINA_BASE/conf/catalina.properties file.
...

Solution

Add all the project jars to the list in catalina.properties. I will need to see a real impact in performance because of this issue before spending time filling this list out in all of our Tomcat servers.

Tomcat 7 reveals log4j memory leak

2011-10-03T12:17:00.000-07:00

We use MDC to log certain information in all traces

Tomcat 7 is notifying the following (As a difference with tomcat 6 which was silent):

SEVERE: The web application [/the-app] created a ThreadLocal with key of type [org.apache.log4j.helpers.ThreadLocalMap] (value [org.apache.log4j.helpers.ThreadLocalMap@757e5533]) and a value of type [java.util.Hashtable] (value [{sessionId=5366DB999B9EA1AC4CF30BED024BA44C, remoteAddress=127.0.0.1}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

Solution

Still waiting for a resolution on a Log4j memory leak: https://issues.apache.org/bugzilla/show_bug.cgi?id=50486

Tomcat 7 JSTL Failed to parse the expression

2011-10-03T12:14:00.001-07:00

In Tomcat 7 (v7.0.22) method like isNew() cannot be referred as ${myObject.new} as before:

org.apache.jasper.JasperException: /WEB-INF/jsp/client/form.jsp (line: 5, column: 4) "${client.new}" contains invalid expression(s): javax.el.ELException: Failed to parse the expr
ession [${client.new}

This problem was documented a year ago and someone might be tempted to change the code for something like ${myObject.isNew()} after realizing that does work. However latest version of jasper-el breaks for this case which makes me think I will need to change my code again in future versions of Tomcat.

Solution

Change the code from ${client.new} to ${client['new']}

In mailing lists I understood Apache 7 is less permissive and since 'new' is not a valid Java identifier it cannot be part of the EL expression like in ${client.new}. There is flag to make Tomcat 7 more permissive albeit rewriting the code should be preferred:

-Dorg.apache.el.parser.SKIP_IDENTIFIER_CHECK=false

I came up with an alternative and temporary (and again not recommended) solution to avoid changing the code which is downloading latest jasper-el http://repo1.maven.org/maven2/org/apache/tomcat/jasper-el/6.0.33/jasper-el-6.0.33.jar or even copying the jar from a previous tomcat installation. Just remember to remove the old jar file:

$ cd ~
$ curl http://repo1.maven.org/maven2/org/apache/tomcat/jasper-el/6.0.33/jasper-el-6.0.33.jar > jasper-el-6.0.33.jar
$ mv /opt/apache-tomcat-7.0.22/lib/jasper-el.jar .
$ cp jasper-el-6.0.33.jar /opt/apache-tomcat-7.0.22/lib/

Tomcat 7 TLD skipped ... is already defined

2011-10-03T11:52:00.000-07:00

After deployment Tomcat 7 would log the below messages. Tomcat 6 did not:

INFO: TLD skipped. URI: http://java.sun.com/jstl/core_rt is already defined
Oct 3, 2011 11:45:44 AM org.apache.catalina.startup.TaglibUriRule body
INFO: TLD skipped. URI: http://java.sun.com/jstl/core is already defined
Oct 3, 2011 11:45:44 AM org.apache.catalina.startup.TaglibUriRule body
INFO: TLD skipped. URI: http://java.sun.com/jsp/jstl/core is already defined
Oct 3, 2011 11:45:44 AM org.apache.catalina.startup.TaglibUriRule body
INFO: TLD skipped. URI: http://java.sun.com/jstl/fmt_rt is already defined
Oct 3, 2011 11:45:44 AM org.apache.catalina.startup.TaglibUriRule body
INFO: TLD skipped. URI: http://java.sun.com/jstl/fmt is already defined
Oct 3, 2011 11:45:44 AM org.apache.catalina.startup.TaglibUriRule body
INFO: TLD skipped. URI: http://java.sun.com/jsp/jstl/fmt is already defined
Oct 3, 2011 11:45:44 AM org.apache.catalina.startup.TaglibUriRule body
INFO: TLD skipped. URI: http://java.sun.com/jsp/jstl/functions is already defined
Oct 3, 2011 11:45:44 AM org.apache.catalina.startup.TaglibUriRule body
INFO: TLD skipped. URI: http://jakarta.apache.org/taglibs/standard/permittedTaglibs is already defined
Oct 3, 2011 11:45:44 AM org.apache.catalina.startup.TaglibUriRule body
INFO: TLD skipped. URI: http://jakarta.apache.org/taglibs/standard/scriptfree is already defined
Oct 3, 2011 11:45:44 AM org.apache.catalina.startup.TaglibUriRule body
INFO: TLD skipped. URI: http://java.sun.com/jstl/sql_rt is already defined
Oct 3, 2011 11:45:44 AM org.apache.catalina.startup.TaglibUriRule body
INFO: TLD skipped. URI: http://java.sun.com/jstl/sql is already defined
Oct 3, 2011 11:45:44 AM org.apache.catalina.startup.TaglibUriRule body
INFO: TLD skipped. URI: http://java.sun.com/jsp/jstl/sql is already defined
Oct 3, 2011 11:45:44 AM org.apache.catalina.startup.TaglibUriRule body
INFO: TLD skipped. URI: http://java.sun.com/jstl/xml_rt is already defined
Oct 3, 2011 11:45:44 AM org.apache.catalina.startup.TaglibUriRule body
INFO: TLD skipped. URI: http://java.sun.com/jstl/xml is already defined
Oct 3, 2011 11:45:44 AM org.apache.catalina.startup.TaglibUriRule body
INFO: TLD skipped. URI: http://java.sun.com/jsp/jstl/xml is already defined

Solution

Look for duplicates in the server/project jars. In my case spring JSTL has a dependency of Spring standard and eliminating the second solves the problem (The second includes the same TLDs again)

<dependency>
            <groupId>javax.servlet</groupId>
            <artifactId>com.springsource.javax.servlet.jsp.jstl</artifactId>
            <version>1.2.0</version>
            <exclusions>
             <exclusion>
              <artifactId>com.springsource.org.apache.taglibs.standard</artifactId>
              <groupId>org.apache.taglibs</groupId>
             </exclusion>
            </exclusions>
</dependency>

Upgrade Ubuntu Apache to latest version in available repositories

2011-10-01T08:09:00.000-07:00

We are using in some server Ubuntu 10.10 (maverick). It ships with Apache 2.2.14 and there is no repository with an upgrade for this highly compromised apache version.

The latest version of Ubuntu still in beta is 11.10 (Oneiric). It ships Apache 2.2.20 which includes important vulnerabilities fixes.

When you are in a situation like this you need to look for available debian repositories. A good place to search for them is http://repogen.simplylinux.ch/

From the site you will be able to obtain the sources.list file for any Ubuntu distro. Once you have the entries you need to add them locally and then run some commands.

So here is what you can do to upgrade Apache to 2.2.20 in Maverick (and probably other Ubuntu versions)

$ sudo vi /etc/apt/sources.list
...
deb http://us.archive.ubuntu.com/ubuntu/ oneiric main
...
$ sudo apt-get update
$ sudo apt-get install apache2
$ apache2 -v
Server version: Apache/2.2.20 (Ubuntu)
Server built:   Sep  6 2011 18:40:05
$ sudo vi /etc/apt/sources.list
...
#comment it out or delete it completely
#deb http://us.archive.ubuntu.com/ubuntu/ oneiric main
...
$ sudo apt-get update

Reusing Talend ETL Jobs

2011-09-30T09:39:00.001-07:00

The question about how to reuse Talend jobs is always in the forums. I will demonstrate here with a proof of concept how this can be achieved using lightweight JSON HTTP requests.

You can find the source code in here.

If you prefer not to hack into the code then just keep on reading for an explanation.

Build a job (retrieve_stock) like the below which basically uses a Yahoo API to retrieve data about a security symbol:

The job uses a tFileInputJson:

//Basic Settings
//./URL
context.url + "%22" + context.symbol + "%22"
//./Mapping
json="$[query.results.quote]"

... and a tJavaFlex:
//Advanced Settings/Import
import org.json.simple.JSONObject;
import org.json.simple.JSONArray;
import org.json.simple.parser.JSONParser;
//Basic Settings
//./Start Code
JSONParser parser = new JSONParser(); 
//./Main Code
JSONObject json = (JSONObject) parser.parse(row1.json);
System.out.println(json.toString());

As you already noticed you need to declare a context with two variables like below (showing default values):

url: "http://query.yahooapis.com/v1/public/yql?format=json&diagnostics=true&env=http%3A%2F%2Fdatatables.org%2Falltables.env&callback=&q=select%20*%20from%20yahoo.finance.quotes%20where%20symbol%20%3D%20"
symbol: "AAPL"

For those who like screenshots ;-)

All we are doing is getting the json from the remote URL (built out of context parameters) and returning it to the stdout. The URL will remain fixed in this example but I use a context variable to show it could change in the future.

Export the job as "Autonomous", uncompress the zip and run the wrapper shell/batch. I am working on Windows this time so we will need to modify the batch file adding @ECHO ON at the beginning of the file.

We can run now from command line our Talend job for a different symbol like Google (GOOG)

C:\etl\releases\retrieve_stock_0.1\retrieve_stock>retrieve_stock_run.bat --context_param symbol=GOOG

Here is a typical response

{"Open":"536.45","PriceEPSEstimateCurrentYear":"14.91","BookValue":"161.128","Bid":"485.00","DaysHigh":"537.30","OrderBookRealtime":null,"ChangeFromFiftydayMovingAverage":"-6.235","ErrorIndi
cationreturnedforsymbolchangedinvalid":null,"DaysRange":"519.41 - 537.30","MoreInfo":"cnprmiIed","AnnualizedGain":null,"Change_PercentChange":"-1.34 - -0.25%","DaysRangeRealtime":"N\/A - N\/
A","LowLimit":null,"PercentChangeFromTwoHundreddayMovingAverage":"-2.90%","ChangeFromTwoHundreddayMovingAverage":"-15.747","DividendYield":null,"EPSEstimateCurrentYear":"35.47","LastTradeDat
e":"9\/29\/2011","TwoHundreddayMovingAverage":"543.247","AskRealtime":"620.00","DividendPayDate":null,"PercentChange":"-0.25%","YearRange":"473.02 - 642.96","symbol":"GOOG","Change":"-1.34",
"PercentChangeFromFiftydayMovingAverage":"-1.17%","HoldingsGainPercent":"- - -","Notes":null,"HoldingsGain":null,"YearHigh":"642.96","Symbol":"GOOG","AfterHoursChangeRealtime":"N\/A - N\/A",
"HoldingsGainPercentRealtime":"N\/A - N\/A","MarketCapitalization":"170.3B","BidRealtime":"485.00","LastTradePriceOnly":"527.50","PERatio":"19.08","EPSEstimateNextQuarter":"10.04","MarketCap
Realtime":null,"AverageDailyVolume":"3820260","PercentChangeFromYearLow":"+11.52%","TickerTrend":"&nbsp;======&nbsp;","LastTradeWithTime":"4:00pm - <b>527.50<\/b>","ChangeFromYearHigh":"-115
.46","PERatioRealtime":null,"PreviousClose":"528.84","StockExchange":"NasdaqNM","FiftydayMovingAverage":"533.735","LastTradeTime":"4:00pm","DaysLow":"519.41","PriceEPSEstimateNextYear":"12.6
0","DaysValueChange":"- - -0.25%","HighLimit":null,"TradeDate":null,"OneyrTargetPrice":"719.68","ChangeRealtime":"-1.34","YearLow":"473.02","ExDividendDate":null,"EPSEstimateNextYear":"41.98
","PricePaid":null,"Volume":"2907381","Ask":"620.00","HoldingsValue":null,"ChangeFromYearLow":"+54.48","PercebtChangeFromYearHigh":"-17.96%","DividendShare":"0.00","EBITDA":"12.785B","Holdin
gsGainRealtime":null,"PEGRatio":"0.79","Name":"Google Inc.","Commission":null,"ChangePercentRealtime":"N\/A - -0.25%","DaysValueChangeRealtime":"N\/A - N\/A","LastTradeRealtimeWithTime":"N\/
A - <b>527.50<\/b>","HoldingsValueRealtime":null,"EarningsShare":"27.719","PriceBook":"3.28","ChangeinPercent":"-0.25%","SharesOwned":null,"ShortRatio":"1.60","PriceSales":"5.12"}

This job is supposed to be reused from others. If at any time the way the symbol is retrieved changes we will change the logic from just one place. After that we just need to export retrieve_stock job and deploy it in our server.

We could reuse retrieve_stock job using a tSystem component to invoke the command but I am going to go a step forward and propose something else.

It is well known that java will use fork() to invoke an external command which means the whole JVM heap memory will be duplicated when the process run. This is of course not efficient. I have posted a solution around this issue before.

So the proposal here is to run a local server that executes local talend command line scripts. The output will be JSON. As you have already figured I am advocating here to use JSON as a lightweight data structure that can be used to maintain the communication channel between different jobs.

Using a nodejs server is better than using Jetty or any other java container for just running shell commands. In my tests the memory footprint for nodejs is really low and the performance is similar to a java container.

After you setup your server you should get the same response after hitting a url like the below:

http://127.0.0.1:8088/?cmd=C%3A%5Cetl%5Creleases%5Cretrieve_stock_0.1%5Cretrieve_stock%5Cretrieve_stock_run.bat%20--context_param%20symbol%3DAAPL

What happens if the amount of data generated by one job is too big? Well in that case I recommend to use a shared resource like a file or a DB (or both like the case of sqlite ;-) If a job works generating a file or DB tables of course the interface will need to be documented and still you can invoke it via a REST call like we have explained here returning back to the caller (Parent job) the results of its execution.

The job we will use to invoke retrieve_stock actually uses the same pattern as you see below:

You will be able to hit a url like the below and your job will return the AAPL price:

http://127.0.0.1:8088/?cmd=C%3A%5Cetl%5Creleases%5Capple_status_0.1%5Capple_status%5Capple_status_run.bat%20

As you can see at the end a job will call external jobs using the same JSON response strategy.

I am not including any transformations in these examples because the whole purpose of this post is to discuss alternatives when it comes to inter-job communication in Talend. Here is what this architecture is allowing me to do:

Projects can be isolated and so better maintained in version control systems using the (free and open source) TOS version
There is a good separation of concern when it comes to building an ETL (You do not need to be a Java developer, Talend also supports Perl but even if you build Java projects the language is only used in a scripting fashion so there is no overhead of unneeded OOP for ETL. Many developers are tempted to push data logic in their java code once they have the luxury of working from java and using Talend jobs written in java ) The ETL developer can test absolutely everything without the need of merging jars, troubleshooting class loading issues etc.
Reusability.
Deployment is easy, just uncompressing a zip file.
Release is not hard if you keep the versioned zip file in a repository. Of course this should be so much better but unfortunately the Open Source version is still not providing maven integration for example. It would be great to be able to run 'mvn release' and get the project tagged.
Interoperability: For example a job written in version 4 could interact with others written in version 5 as the only interface between them is an HTTP JSON Service call.

Security

You are working with shell commands which can be pretty destructive if you do not ensure the json server just run locally attached to a loopback IP (in our case 127.0.0.1). You will need a more robust server implementation if you violate that rule or if you cannot guarantee your server will not be hosting any other service where a different user could run malicious code.

Encoding

In case you did not notice the below two lines are equivalent. The second is URL encoded. We need that to pass the whole command as a parameter from a query string or a POST request.

C:\etl\releases\retrieve_stock_0.1\retrieve_stock\retrieve_stock_run.bat --context_param symbol=AAPL
C%3A%5Cetl%5Creleases%5Cretrieve_stock_0.1%5Cretrieve_stock%5Cretrieve_stock_run.bat%20--context_param%20symbol%3DAAPL

A nodejs server to run shell commands

2011-09-30T03:51:00.000-07:00

I was tempted to use jetty as a lightweight server to resolve the JVM-fork() memory problem as I already posted however nodejs provides a lighter and so from my simplistic design poit of view better alternative.

Below is the code for such a server:

#!/usr/local/bin/node
/*
** shell-server.js returns json response with the stdout and stderr of a shell command
**
**
** @Author: Nestor Urquiza
** @Date: 09/29/2011
**
*/

/*
* Dependencies
*/
var http = require('http'),
    url = require('url'),
    exec = require('child_process').exec;

/*
* Server Config
*/
var host = "127.0.0.1",
    port = "8088",
    thisServerUrl = "http://" + host + ":" + port;

/*
* Main
*/
http.createServer(function (req, res) {
  req.addListener('end', function () {
        
  });
  var parsedUrl = url.parse(req.url, true);
  var cmd = parsedUrl.query['cmd'];
 
  res.writeHead(200, {'Content-Type': 'text/plain'});

  if( cmd ) {
    var child = exec(cmd, function (error, stdout, stderr) {
   var result = '{"stdout":' + stdout + ',"stderr":"' + stderr + '","cmd":"' + cmd + '"}';
   res.end(result + '\n');
    });
  } else {
 var result = '{"stdout":"' + '' + '","stderr":"' + 'cmd is mandatory' + '","cmd":"' + cmd + '"}';
 res.end(result + '\n');
  }  
  
}).listen(port, host);
console.log('Server running at ' + thisServerUrl );

Once the server is running you can hit the below URL to get a list of the users loged in a OSX/linux/unix box:

http://localhost:8088/?cmd=who

Development environment

If you are running Windows you should download the node executable and run the server as:

node c:\shell-server.js

If you are running OSX/Linux/Unix you have to install node make the script executable and run the below:

/path/to/shell-server.sh

Production deployment

You need to be sure the server runs as a daemon and that it restarts if it fails to serve. In debian/Ubuntu + monit you can follow these steps:

$ sudo vi /opt/nodejs/shell-server.js
$ sudo vi /etc/init/shell-server.conf
#!/sbin/upstart
description "shell server runs a command specified by HTTP GET param 'cmd'"
author      "admin"

start on startup
stop on shutdown

script
    export HOME="/root"
    exec sudo -u admin /usr/local/bin/node /opt/nodejs/shell-server.js 2>&1 >> /var/log/shell-server.log
end script
$ sudo vi /etc/monit/monitrc 
...
#################################################################
# shell-server
################################################################

check host shell-server with address 127.0.0.1
start = "/sbin/start shell-server"
stop = "/sbin/stop shell-server"
if failed port 8088 protocol HTTP
  request /
  with timeout 10 seconds
then restart
group server
...
$ sudo monit reload
$ sudo vi /opt/nodejs/shell-server.js
$ wget http://localhost:8088 -O -

Mediawiki email notifications for any changes

2011-09-27T11:11:00.000-07:00

I have to admit I love simplicity of MoinMoin wiki but more than anything I think it is really developer oriented.

Mediawiki on the other hand is more oriented to a general audience. It is a great product which for developers and other technical teams (my area of expertise) lacks of an important concept: People should be able to get alerts for any change even if they did not visit the page after a previous change was notified. If you want to do something like that in Mediawiki you need to add the email addresses in a configuration file.

The first thing you will try in Mediawiki to get notifications about a page update is to watch that page. Every time you get an email notification you must click on the page other wise you will stop getting new updates. Updates will resume once you have visited the page. The steps to get email notifications are:

Go to My Prefernces | Email. You might have a link asking for you to activate your email. Click it or you will never get an email.
Once you have done it you will receive the first email and after clicking on the link if you come back to "My Peferences | Profile | Email Options" you will see something like "Your e-mail address was authenticated on 12 August 2010 at 00:37."
Go to "My Peferences | Watchlist" to set extra options like automatically getting subscribed to pages you create.
Now you can monitor any page clicking on the "watch" link. In the monitored page the link should change from "watch" to "unwatch"

Your e-mail address was authenticated on 12 August 2010 at 00:37.

While you can subscribe to an RSS feed I prefer email for one good reason: There is no excuse from developers like "I did not know". Yes you do know because you received an email when your partner updated the documentation.

Fortunately there is a way around this using a freely available python application called rss2email.

This will send emails with the typical diff format but using colors to highlight conflicts, what is new and what has been changed/removed. See below an example:

Below are the steps to install and configure such application:

$ sudo apt-get remove rss2email
$ cd
$ tar xvf rss2email-2.70.tar.gz 
$ sudo cp -R rss2email-2.70 /opt/
$ sudo mv config.py.example config.py
$ sudo chown -R nestorurquizaadmin:nestorurquizaadmin /opt/rss2email-2.70/
$ cd /opt/rss2email-2.70/
$ sudo vi config.py 
...
DEFAULT_FROM = "donotreply@nestorurquiza.com"
...
SMTP_SERVER = "krms.krco.com:25"
...
$ r2e new developers@nestorurquiza.com
$ r2e add http://wiki.nestorurquiza.com/index.php/Special:RecentChanges?feed=rss
$ r2e run --no-send #remove the flag in case you want to receive emails right away. Keep the flag to receive new changes.
$ crontab -e
# Send wiki changes to developers every 10 minutes
*/10 * * * * cd /opt/rss2email-2.70;./r2e run

Securing your Apache SSL site

2011-09-22T18:46:00.000-07:00

The default apache SSL configuration accepts weak cipher and SSL v2 both of which are vulnerable

SSLCipherSuite ALL:!ADH:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP:+eNULL

Here is what you have to do to make it secure.

SSLProtocol all -SSLv2
SSLCipherSuite RC4-SHA:AES128-SHA:HIGH:!aNULL:!EXP:!MD5:!NULL
SSLHonorCipherOrder on

If you are in doubts you can use ssllabs free service to find out if your SSL server is secure enough.

You will be amazed how many websites are vulnerable to MIM attacks just because of the fact that some people still think it is enough to buy a signed certificate. What is perhaps even more sad is that some people were surprised about the recent Diginotar hack but if you actually run the test for www.diginotar.com you will see it rated as "D" because it accepts weak ciphers and still supports insecure SSL 2.0. At the time of this writing that is still the case (https://www.ssllabs.com/ssldb/analyze.html?d=www.diginotar.com). Below are the results I just got:

Please do yourself a favor and make sure your website is hosted in an "A" rated SSL host.

Thinking in Security after OWASP AppSec USA 2011

2011-09-22T18:41:00.000-07:00

Just came back from Minneapolis after two days of application security training where we went through several tools that can be used to find out vulnerabilities in Web Applications.

OWASP-WTE is a Ubuntu distribution packaging several open source utilities used to perform what is called Penetration Testing (PenTest). The training focused on basic concepts about HTTP specification that any security tester should know, it presented the most common attacks and the available tools and manual procedures the tester is supposed to master.

Here are some reflections that I have come up with after these two days.

In my current project I have gone through the top ten application security risks I have used skipfish and websecurity and I have documented at least one of my experiences using this product to PenTest Liferay. We have done the same for our BHUB based application however in terms of automated tools it is never enough. What a tool can find others will miss and vice versa.

Dealing with false positives is really annoying but in a world where the number of threats only increases we have no other option than going through this practice in a regular basis.

If you are hosting a web application go through the OWASP Testing Guide. Web apps should be prepared to live in the wild and that is as important as hardening the OS.

Perhaps the most forgotten point related to security is monitoring. Trying your best with dozens of tools and manual hacking attempts is a must do however that is not enough. Controls must be put in place and your server logs are full of useful information you should analyze looking for patterns, eliminating false positives and hopefully automating blocking when a threat is identified.

The PenTest individual is someone that must be willing to script, to be a hacker, a programmer, a human being who knows there is a big responsibility on the job s(he) does. S(he) might be saving the company from failure after all.

The skills for such a person go beyond being a tech savvy. Discipline and persistence are a must have.

Any additional effort you can put in protecting your web application will be worth it but application security is just the tip of the Iceberg because ultimately it will always rely on some credentials for a user to gain access to certain resources and the credentials can be obtained even in applications for which an exploit has not yet been spotted.

Social engineering is one example, take just the real life example of an employee from a security related company who dared to transmit a password via email. Ultimately your company is as secure as the most careless of the company employees.

Phishing is another good example. Look here and here for a couple of posts I have made in the past related to twitter hacking attempts.

I have left Minneapolis convinced that like in any other aspect of our mortal life there is no silver bullet. For sure there is none when you try to implement security.

Shell processes from Java and the infamous OutOfMemory

2011-09-21T16:40:00.000-07:00

If you run shell processes from a Java Application Server you can really easy run out of heap memory because Java will invoke a fork() system call which will duplicate the parent memory (current JVM memory in use) to be able to run the child (the command you are trying to run).

Here is a workaround for this issue. Basically it relies on a war file containing a servlet that accepts regular get parameters like "cmd" containing the complete command to run. It returns a JSON response with the stderr and stdout content. The war file will need to be deployed in an application server with really small footprint to prevent the child process from originating again an OutOfMemory.

Note that I on purpose do not use any special jar files because that way I keep memory usage to a minimum. Do not use a JSON parser for this, just build the response string yourself. Simpler is better ;-)

import java.io.*;
import javax.servlet.*;
import javax.servlet.http.*;

public class ShellServlet extends HttpServlet {
    public void doGet(HttpServletRequest request, HttpServletResponse response)
            throws ServletException, IOException {
        String cmd = request.getParameter("cmd");
        StringBuilder stdout = new StringBuilder();
        StringBuilder stderr = new StringBuilder();
        
        
        
        if( cmd != null && cmd.length() > 0 ) {
            try {
                Process process = new ProcessBuilder(cmd.split(" ")).start();
                
                InputStream is = process.getInputStream();
                InputStreamReader isr = new InputStreamReader(is);
                BufferedReader br = new BufferedReader(isr);
                String line;
                while ((line = br.readLine()) != null) {
                    stdout.append(line);
                }
                
                InputStream errorInputStream = process.getErrorStream();
                isr = new InputStreamReader(errorInputStream);
                br = new BufferedReader(isr);
                while ((line = br.readLine()) != null) {
                    stderr.append(line);
                }
            } catch (Exception e) {
                stderr.append(e.getMessage());
            }
            
        }
        
        StringBuilder sb = new StringBuilder();
        sb.append("{");
        sb.append("'stdout':'" + stdout + "'");
        sb.append(",");
        sb.append("'stderr':'" + stderr + "'");
        sb.append(",");
        sb.append("'cmd':'" + cmd + "'");
        sb.append("}");
        PrintWriter out = response.getWriter();
        out.println(sb);
    }
}

For a request like:

http://localhost:8088/shell-service/?cmd=ls%20-al%20notifier.png

You will get (provided notifier.png file exist in the working directory) something like:

{'stdout':'-rw-r--r--  1 nestor  staff  886 Oct 18  2010 notifier.png','stderr':'','cmd':'ls -al notifier.png'}

Probably your best bet in Java would be Jetty because it is really lightweight. Simply download the zip file, uncompress it and follow the below steps:

$ cd $JETTY_HOME
$ vi etc/jetty.xml #change port to 8088
$ java -jar start.jar &

Edit: A better alternative for a server to run local commands would be to build a lighter nodejs service like I explain here.

Mediawiki Error creating thumbnail: sh: convert: No such file or directory

2011-09-21T07:10:00.000-07:00

The error below should be referring to a non existent ImageMagic executable:

Error creating thumbnail: sh: /usr/local/bin/convert: No such file or directory

After installing the executable my MediaWiki installation was still complaining. I fixed the problem using action=purge on the page, for example:

http://wiki.nestorurquiza.com/index.php/Nestor?action=purge

error client ip File does not exist: /etc/apache2/htdocs after cloning

2011-09-17T06:09:00.000-07:00

We cloned an ESX VM today to take advantage of all configurations in there. I went ahead and removed all unnecessary services and just left apache however something was not working:

error] [client 192.168.9.3] File does not exist: /etc/apache2/htdocs

I knew this was not a configuration issue as in the original VM apache does run without complaining.

It ended up being something about relative and absolute paths. For some reason Apache was failing to find the sites-enabled directory when using relative paths. A correction in /etc/apache2/apache2.conf made the trick:

$ sudo vi /etc/apache2/apache2.conf 
#Include sites-enabled/
Include /etc/apache2/sites-enabled/

monit: fatal: /usr/sfw/lib/libfl-2.5.4.so.0: wrong ELF class: ELFCLASS32

2011-09-16T09:37:00.000-07:00

Monit Solaris 10 5/9 64 bits installation went good but it was not working:

monit: fatal: /usr/sfw/lib/libfl-2.5.4.so.0: wrong ELF class: ELFCLASS32

A simple ldd showed the library was not 64bits:

# ldd /usr/local/bin/monit
        libfl-2.5.4.so.0 =>      /usr/sfw/lib/libfl-2.5.4.so.0  - wrong ELF class: ELFCLASS32
        libpam.so.1 =>   /lib/64/libpam.so.1
        libpthread.so.1 =>       /lib/64/libpthread.so.1
        libresolv.so.2 =>        /lib/64/libresolv.so.2
        libnsl.so.1 =>   /lib/64/libnsl.so.1
        libsocket.so.1 =>        /lib/64/libsocket.so.1
        libkstat.so.1 =>         /lib/64/libkstat.so.1
        libssl.so.0.9.7 =>       /usr/sfw/lib/libssl.so.0.9.7  - wrong ELF class: ELFCLASS32
        libcrypto.so.0.9.7 =>    /usr/sfw/lib/libcrypto.so.0.9.7  - wrong ELF class: ELFCLASS32
        libc.so.1 =>     /lib/64/libc.so.1
        libcmd.so.1 =>   /lib/64/libcmd.so.1
        libmp.so.2 =>    /lib/64/libmp.so.2
        libmd.so.1 =>    /lib/64/libmd.so.1
        libscf.so.1 =>   /lib/64/libscf.so.1
        libdoor.so.1 =>  /lib/64/libdoor.so.1
        libuutil.so.1 =>         /lib/64/libuutil.so.1
        libgen.so.1 =>   /lib/64/libgen.so.1
        libm.so.2 =>     /lib/64/libm.so.2

Here is how you solve these kind of issues in your 64 bits Solaris box:

export LD_LIBRARY_PATH=/usr/sfw/lib/64:$LD_LIBRARY_PATH

However you better use crle in this case as you want the change to affect your whole system, after all this is a 64 machine ins't it?

crle -64 -u -l /usr/sfw/lib/64

Allowing changes in subversion log messages or comments

2011-09-12T10:20:00.000-07:00

When you commit your change to a subversion repository you have an option to change the comment you use:

svn propset --revprop -r 15125 svn:log "Editing this comment with more detail, blah, blah ..."

There is a hook script template that you could use to allow this but there is a reason why the default template should not be used as is. It is dangerous to be able to affect good comments with garbage if you just make a mistake with the version number but the most dangerous of all is that you could change comments done by a different user!!! From a GUI like Eclipse or Netbeans it is harder to make a mistake like the first one of course, however you should protect subversion so log changes are restricted to the owner of the commit.

Use the below to achieve it (Linux, if Windows translate to DOS)

$ sudo vi /var/local/svn/subversion.nestorurquiza.com/hooks/pre-revprop-change
REPOS="$1"
REV="$2"
USER="$3"
PROPNAME="$4"
ACTION="$5"

owner=`svnlook author -r "$REV" "$REPOS"`
if [ "$ACTION" = "M" -a "$PROPNAME" = "svn:log" -a "$owner" = "$USER" ]; then exit 0; fi

echo "Changing revision properties other than svn:log is prohibited and you must be the owner($owner) to change $REPOS@$REV" >&2
exit 1 
$sudo chmod +x  /var/local/svn/subversion.nestorurquiza.com/hooks/pre-revprop-change

Phishing Attack: Using redirection

2011-09-12T08:44:00.000-07:00

It has come to my attention the escalation in phishing attempts coming to my gmail account. In 4 days I got 3 emails that managed to pass the spam protection. They all claimed they were my twitter friends and that they found someone faking my account, my twitter picture, twitter avatar and what not.

I always inspect the url before clicking because I want to explore the vulnerabilities (of course the safest to do is just to report as spam anything looking suspicious) So I took a look at them and they were all referring to well known websites that are "offerring free redirection services". I hope this is just a bug in CNN.com and pepsi.com. Do not hit the urls below before reading the rest of this post. Here are the URLs:

http://pepsi.com/pepsi_redirect.php?theurl=jurism%2Ecom%2Foldweb%2Ftraffic168
http://mexico.cnn.com/redirectComplete.php?url=%2F%2Fjurism%2Ecom%2Foldweb%2Ftraffic168
http://mexico.cnn.com/redirectComplete.php?url=%2F%2Ftwitmytweets%2Eit%2Etc

Open Firefox and delete all your cookies. Failure to do that will probably compromise things like your google/gmail account.

If you take a look at the traces you will notice there were attempts to get some stuff from google.com. If you are logged into gmail or other google services your cookies for google.com will be compromised and the intruder could fake your session resulting in identity theft.

svnadmin: Couldn't perform atomic initialization database is locked

2011-09-08T11:48:00.000-07:00

I was trying to use a CIFS (Windows) path as the svn repository but I was having locking issues:

$ sudo svnadmin load /repo/path < ~/svn_dump
<<< Started new transaction, based on original revision 1
    * adding path : projects ... done.
svnadmin: Couldn't perform atomic initialization
svnadmin: database is locked

Taking a look at man pages I gave option "nobrl" a try and that seemed to solve the problem which apparently is that our NetApp SAN does not support byte range locks.

Using the below did the trick then:

mount -t cifs //windows.box/path /mnt/local/path -o credentials=/root/cifs/cifs_credentials.txt,domain=COMPANYX,file_mode=0600,dir_mode=0700,uid=admin,gid=admin,nobrl

CIFS VFS: cifs_mount failed return code -13 0xc000006d NT_STATUS_LOGON_FAILURE

2011-09-08T07:53:00.000-07:00

Using the following command to mount a CIFS (Windows) path:

mount -t cifs //windows.box/path /mnt/local/path -o credentials=/root/cifs/cifs_credentials.txt,domain=COMPANYX,file_mode=0600,dir_mode=0700,uid=admin,gid=admin

I was getting this error:

[307906.131366] Status code returned 0xc000006d NT_STATUS_LOGON_FAILURE
[307906.131374] CIFS VFS: Send error in SessSetup = -13
[307906.131575] CIFS VFS: cifs_mount failed w/return code = -13

While this can be caused by any permission problems sometimes the lack of more verbose log traces makes it harder to find the exact issue.

In my case this was related to the credentials file having spaces:

username = user 
password = password

Fixing it was a matter of trimming spaces out:

username=user 
password=password

FCKEditor for Mediawiki 1.17: WYSIWYG based on CKEditor

2011-09-03T11:14:00.000-07:00

Mediawiki is not longer supporting the deprecated FCKEditor. The new CKEditor is supported through the WYSIWYG extension. I tried WYSIWYG 1.5.6 in my a new installation of Mediawiki 1.17.0 but I would get nothing everytime I would try to switch to the Rich Editor.

Debugging with Firebug I found a variable not defined error: "CKEDITOR is not defined". This was a result of addType directives in wiki/extensions/WYSIWYG/ckeditor/.htaccess. You either need to comment the lines or move the file (rename it, delete it or move it to somewhere else)

So below are the commands to get your WYSIWYG editor working in Mediawiki:

$ unzip wysiwyg-1.5.6_0.zip

$ sudo cp -R extensions/WYSIWYG /var/www/wiki/extensions/

$ sudo chown -R www-data:www-data /var/www/wiki/

$ sudo mv /var/www/wiki/extensions/WYSIWYG/ckeditor/.htaccess /var/www/wiki/extensions/WYSIWYG/ckeditor/.htaccess.old

$ sudo vi /var/www/wiki/LocalSettings.php

  require_once("$IP/extensions/WYSIWYG/WYSIWYG.php");

  $wgGroupPermissions['*']['wysiwyg']=true;

Upgrading subversion

2011-09-02T09:17:00.000-07:00

We are using WebDAV with Apache for subversion. Below are the steps I followed to migrate an old subversion repository to a brand new Ubuntu server with latest subversion.

Note that "admin" is a user which can make administer subversion.

$ sudo apt-get install subversion libapache2-svn

$ sudo mkdir -p /var/local/svn/subversion.nestorurquiza.com

$ sudo addgroup svn

$ sudo usermod -a -G svn www-data

$ sudo usermod -a -G svn admin

$ sudo chmod 2770 /var/local/svn/subversion.nestorurquiza.com

$ sudo svnadmin create /var/local/svn/subversion.nestorurquiza.com

$ sudo vi /var/local/svn/subversion.nestorurquiza.com/conf/authz #ACL

$ sudo mkdir /var/log/apache2/subversion.nestorurquiza.com

$ sudo vi /etc/apache2/sites-available/subversion

<VirtualHost *>



 ServerName svn.nestorurquiza.com

 ServerAlias subversion.nestorurquiza.com

 DocumentRoot /var/local/svn/subversion.nestorurquiza.com



 <Location /repos/reporting>

   DAV svn

   SVNListParentPath off

   AuthType Basic

   AuthName "Subversion repository"

   SVNPath /var/local/svn/subversion.nestorurquiza.com

   AuthzSVNAccessFile /var/local/svn/subversion.nestorurquiza.com/conf/authz

   AuthUserFile /var/local/svn/subversion.nestorurquiza.com/conf/passwd

   Require valid-user

   <LimitExcept GET PROPFIND OPTIONS REPORT>

        Require valid-user

   </LimitExcept>

 </Location>



 <Directory "/var/local/svn/subversion.nestorurquiza.com">

   Options -Indexes

 </Directory>

</VirtualHost>

$ sudo cp authz /var/local/svn/subversion.nestorurquiza.com/conf/authz #assuming there is an existing svn access file. Better keep it on SVN ;-)

$ sudo cp passwd /var/local/svn/subversion.nestorurquiza.com/conf #assuming there is an existing password file. Better keep it on SVN ;-)

$ sudo htpasswd /var/local/svn/subversion.nestorurquiza.com/conf/passwd "new username here" #to create individual users

$ sudo ln -s /etc/apache2/sites-available/subversion /etc/apache2/sites-enabled/004-subversion

$ sudo svnadmin load /var/local/svn/subversion.nestorurquiza.com < ~/file_from_command_svnadmin_dump_originalRepoPath

$ sudo chown -R www-data:svn /var/local/svn/subversion.nestorurquiza.com

$ sudo chmod -R g+w /var/local/svn/subversion.nestorurquiza.com

$ sudo /etc/init.d/apache2 restart

Upgrading Bugzilla

2011-09-01T22:05:00.000-07:00

This one was pretty straightforward (from version 2 to 4 actually)

If this is a new server restore your DB from current bugzilla installation
Uncompress the distro in your document root, commonly /var/www/bugzilla-4.0.2
Update apache virtual host to point to that directory
Replace file 'localconfig' and directory 'local' from the previous installation
If needed change db user and password if needed in 'localconfig'
If needed update urlbase 'data/params' file
Run the below commands from the bugzilla document root directory and make sure you get no warnings nor errors. Correct them all before considering your migration completed
```
$ sudo ./runtests.pl 

$ sudo ./checksetup.pl 
```
Restart apache and hit the bugzilla url

Upgrading Mediawiki

2011-09-01T20:00:00.000-07:00

This one took a good chunk of time. More than anything the order was the important part:

Install a fresh copy
Modify LocalSettings.php to meet your goals
Restore the previous db backup
Run the below to update the tables so they work with the newest code
```
$ cd /var/www/mediawiki/maintenance/

$ php update.php
```

Transform XML with XSLT in Talend

2011-08-29T14:45:00.000-07:00

Talend Open Studio is an excellent ETL tool that can be used beyond the typical database and CSV manipulation. XML processing for example is today all over the places in the Enterprise.

As a consequence XML transformations are a key skill for those folks doing data transformations.

Even though there are more efficient tools XSLT is a standard which is supported in Talend through the tXSLT component. You just need to provide your XML, XSL and output files and Talend will apply the transformation for you. Talend uses Saxon at the moment so you get the benefit of clear error messages when trying to build your XSL.

Of course XSLT might be a skill that even scares some people, however XSLT is not difficult at all and the more you work with it the better you get as with any other human skill. Do not try to avoid it, if you have XML to process and your ETL tool is Talend then do your homework and learn some XSL.

As there is no better way to teach than providing an example I decided to write this quick showcase that will pivot the data resulting from running an Advent Geneva RSL report (A SOAP service) which comes in the form of key value pairs into a tabular output. I will provide two responses: XML and HTML. The first is probably what you need in Talend while the second is probably what you need if you want to provide a quick add hoc HTML report page.

Here is the XML:

<?xml version='1.0' encoding='UTF-8'?>

<SOAP-ENV:Envelope xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/" xmlns:SOAP-ENC="http://schemas.xmlsoap.org/soap/encoding/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:nsg="http://geneva.advent.com">

	<SOAP-ENV:Body id="_0">

		<reportResults xmlns="http://geneva.advent.com">

			<return xsi:type="nsg:reportResultsPortfolioStruct">

				<results xsi:type="nsg:reportResultsStruct">

					<portfolioName xsi:type="xsd:string">Fund1</portfolioName>

					<header xsi:type="nsg:reportResultsRecordStruct">

						<field xsi:type="nsg:reportResultsVectorElement">

							<name xsi:type="xsd:string">Head</name>

							<value xsi:type="xsd:string">Fund 1 Example</value>

						</field>

						<field xsi:type="nsg:reportResultsVectorElement">

							<name xsi:type="xsd:string">Head</name>

							<value xsi:type="xsd:string">DIVIDENDS RECEIVABLE AND PAYABLE</value>

						</field>

						<field xsi:type="nsg:reportResultsVectorElement">

							<name xsi:type="xsd:string">Head</name>

							<value xsi:type="xsd:string">FOR THE PERIOD INCEPTION TO July 31, 2011</value>

						</field>

						<field xsi:type="nsg:reportResultsVectorElement">

							<name xsi:type="xsd:string">Head</name>

							<value xsi:type="xsd:string"/>

						</field>

					</header>

					<record xsi:type="nsg:reportResultsRecordStruct">

						<field xsi:type="nsg:reportResultsVectorElement">

							<name xsi:type="xsd:string">CcyCode</name>

							<value xsi:type="xsd:string">BRL</value>

						</field>

						<field xsi:type="nsg:reportResultsVectorElement">

							<name xsi:type="xsd:string">Custodian</name>

							<value xsi:type="xsd:string">My Custodian</value>

						</field>

						<field xsi:type="nsg:reportResultsVectorElement">

							<name xsi:type="xsd:string">IDesc</name>

							<value xsi:type="xsd:string">My IDesc</value>

						</field>

						<field xsi:type="nsg:reportResultsVectorElement">

							<name xsi:type="xsd:string">TransID</name>

							<value xsi:type="xsd:string">10145834</value>

						</field>

						<field xsi:type="nsg:reportResultsVectorElement">

							<name xsi:type="xsd:string">TDate</name>

							<value xsi:type="xsd:string">March 15, 2011 12:00:00 am</value>

						</field>

						<field xsi:type="nsg:reportResultsVectorElement">

							<name xsi:type="xsd:string">SDate</name>

							<value xsi:type="xsd:string">December 31, 9999 11:59:59 pm</value>

						</field>

						<field xsi:type="nsg:reportResultsVectorElement">

							<name xsi:type="xsd:string">Qty</name>

							<value xsi:type="xsd:string">13528.013</value>

						</field>

						<field xsi:type="nsg:reportResultsVectorElement">

							<name xsi:type="xsd:string">PerShare</name>

							<value xsi:type="xsd:string">0.15100000</value>

						</field>

						<field xsi:type="nsg:reportResultsVectorElement">

							<name xsi:type="xsd:string">TaxRate</name>

							<value xsi:type="xsd:string"/>

						</field>

						<field xsi:type="nsg:reportResultsVectorElement">

							<name xsi:type="xsd:string">DivLocal</name>

							<value xsi:type="xsd:string">2042.73</value>

						</field>

						<field xsi:type="nsg:reportResultsVectorElement">

							<name xsi:type="xsd:string">TaxLocal</name>

							<value xsi:type="xsd:string">0.00</value>

						</field>

						<field xsi:type="nsg:reportResultsVectorElement">

							<name xsi:type="xsd:string">DivLocalNet</name>

							<value xsi:type="xsd:string">2042.73</value>

						</field>

						<field xsi:type="nsg:reportResultsVectorElement">

							<name xsi:type="xsd:string">DivBook</name>

							<value xsi:type="xsd:string">1225.76</value>

						</field>

						<field xsi:type="nsg:reportResultsVectorElement">

							<name xsi:type="xsd:string">TaxBook</name>

							<value xsi:type="xsd:string">0.00</value>

						</field>

						<field xsi:type="nsg:reportResultsVectorElement">

							<name xsi:type="xsd:string">DivBookNet</name>

							<value xsi:type="xsd:string">1225.76</value>

						</field>

						<field xsi:type="nsg:reportResultsVectorElement">

							<name xsi:type="xsd:string">UnrealFXGL</name>

							<value xsi:type="xsd:string">0.00</value>

						</field>

						<field xsi:type="nsg:reportResultsVectorElement">

							<name xsi:type="xsd:string">BookBal</name>

							<value xsi:type="xsd:string">1225.76</value>

						</field>

					</record>

					<record xsi:type="nsg:reportResultsRecordStruct">

						<field xsi:type="nsg:reportResultsVectorElement">

							<name xsi:type="xsd:string">CcyCode</name>

							<value xsi:type="xsd:string">USD</value>

						</field>

						<field xsi:type="nsg:reportResultsVectorElement">

							<name xsi:type="xsd:string">Custodian</name>

							<value xsi:type="xsd:string">My Custodian</value>

						</field>

						<field xsi:type="nsg:reportResultsVectorElement">

							<name xsi:type="xsd:string">IDesc</name>

							<value xsi:type="xsd:string">My IDesc</value>

						</field>

						<field xsi:type="nsg:reportResultsVectorElement">

							<name xsi:type="xsd:string">TransID</name>

							<value xsi:type="xsd:string">10756740</value>

						</field>

						<field xsi:type="nsg:reportResultsVectorElement">

							<name xsi:type="xsd:string">TDate</name>

							<value xsi:type="xsd:string">April 27, 2011 12:00:00 am</value>

						</field>

						<field xsi:type="nsg:reportResultsVectorElement">

							<name xsi:type="xsd:string">SDate</name>

							<value xsi:type="xsd:string">July 1, 2011 12:00:00 am</value>

						</field>

						<field xsi:type="nsg:reportResultsVectorElement">

							<name xsi:type="xsd:string">Qty</name>

							<value xsi:type="xsd:string">205212.046</value>

						</field>

						<field xsi:type="nsg:reportResultsVectorElement">

							<name xsi:type="xsd:string">PerShare</name>

							<value xsi:type="xsd:string">0.16918500</value>

						</field>

						<field xsi:type="nsg:reportResultsVectorElement">

							<name xsi:type="xsd:string">TaxRate</name>

							<value xsi:type="xsd:string"/>

						</field>

						<field xsi:type="nsg:reportResultsVectorElement">

							<name xsi:type="xsd:string">DivLocal</name>

							<value xsi:type="xsd:string">34718.80</value>

						</field>

						<field xsi:type="nsg:reportResultsVectorElement">

							<name xsi:type="xsd:string">TaxLocal</name>

							<value xsi:type="xsd:string">0.00</value>

						</field>

						<field xsi:type="nsg:reportResultsVectorElement">

							<name xsi:type="xsd:string">DivLocalNet</name>

							<value xsi:type="xsd:string">34718.80</value>

						</field>

						<field xsi:type="nsg:reportResultsVectorElement">

							<name xsi:type="xsd:string">DivBook</name>

							<value xsi:type="xsd:string">22153.39</value>

						</field>

						<field xsi:type="nsg:reportResultsVectorElement">

							<name xsi:type="xsd:string">TaxBook</name>

							<value xsi:type="xsd:string">0.00</value>

						</field>

						<field xsi:type="nsg:reportResultsVectorElement">

							<name xsi:type="xsd:string">DivBookNet</name>

							<value xsi:type="xsd:string">22153.39</value>

						</field>

						<field xsi:type="nsg:reportResultsVectorElement">

							<name xsi:type="xsd:string">UnrealFXGL</name>

							<value xsi:type="xsd:string">0.00</value>

						</field>

						<field xsi:type="nsg:reportResultsVectorElement">

							<name xsi:type="xsd:string">BookBal</name>

							<value xsi:type="xsd:string">23379.15</value>

						</field>

					</record>

					<addendumErrors xsi:type="nsg:reportResultsRecordStruct">

						<field xsi:type="nsg:reportResultsVectorElement">

							<name xsi:type="xsd:string">BegDesc</name>

							<value xsi:type="xsd:string"/>

						</field>

						<field xsi:type="nsg:reportResultsVectorElement">

							<name xsi:type="xsd:string">EndDesc</name>

							<value xsi:type="xsd:string">DIVIDENDS RECEIVABLE - CLOSING BALANCE</value>

						</field>

						<field xsi:type="nsg:reportResultsVectorElement">

							<name xsi:type="xsd:string">BegBal</name>

							<value xsi:type="xsd:string"/>

						</field>

						<field xsi:type="nsg:reportResultsVectorElement">

							<name xsi:type="xsd:string">EndBal</name>

							<value xsi:type="xsd:string">71643.79</value>

						</field>

					</addendumErrors>

					<addendumErrors xsi:type="nsg:reportResultsRecordStruct">

						<field xsi:type="nsg:reportResultsVectorElement">

							<name xsi:type="xsd:string">BegDesc</name>

							<value xsi:type="xsd:string"/>

						</field>

						<field xsi:type="nsg:reportResultsVectorElement">

							<name xsi:type="xsd:string">EndDesc</name>

							<value xsi:type="xsd:string">DIVIDENDS PAYABLE - CLOSING BALANCE</value>

						</field>

						<field xsi:type="nsg:reportResultsVectorElement">

							<name xsi:type="xsd:string">BegBal</name>

							<value xsi:type="xsd:string"/>

						</field>

						<field xsi:type="nsg:reportResultsVectorElement">

							<name xsi:type="xsd:string">EndBal</name>

							<value xsi:type="xsd:string">-20200.35</value>

						</field>

					</addendumErrors>

				</results>

			</return>

		</reportResults>

	</SOAP-ENV:Body>

</SOAP-ENV:Envelope>

If you open it with Firefox and you choose "View XPath" from a right click on the body of the page (contextual menu) you could try several XPATH expresions. Together with XSLT skill it comes XPATH which is just a way to address a node, an attribute or textual content in the XML. See below how I tested one of the XPATH using this tool. Pay attention to the namespace definition, I use simple letters to abbreviate more verbose prefixes.

Now that you have a quick tool for finding nodes in the XML document let us see the desired document structure. Here is a screenshot of what we would like to see in HTML:

Here is in the XML we would like to obtain for further processing in Talend:

Here is the XSL that will output HTML:

<?xml version="1.0"?>

<xsl:stylesheet version="1.0"

  xmlns:xsl="http://www.w3.org/1999/XSL/Transform"

  xmlns:e="http://schemas.xmlsoap.org/soap/envelope/"

  xmlns:g="http://geneva.advent.com">

    <xsl:output omit-xml-declaration="yes" indent="yes"/>



    <xsl:strip-space elements="*"/>



    <xsl:key name="kFieldNameByValue" match="/e:Envelope/e:Body/g:reportResults/g:return/g:results/g:record/g:field/g:name"

         use="."/>



    <xsl:variable name="vCols" select=

       "/e:Envelope/e:Body/g:reportResults/g:return/g:results/g:record/g:field/g:name

                [generate-id()

                =          

                 generate-id(key('kFieldNameByValue',.)[1])

                 ]"/>



    <xsl:template match="/">

             <table>

               <tr>

                 <xsl:apply-templates select="$vCols"/>

               </tr>



               <xsl:for-each select=

                 "/e:Envelope/e:Body/g:reportResults/g:return/g:results/g:record">                   

                 <tr>



                   <xsl:variable name="vPos" select="position()"/>



                   <xsl:for-each select="$vCols">

                     <td>

                       <xsl:value-of select=

                           "../../../g:record[$vPos]/g:field[g:name = current()]/g:value"/>

                     </td>

                   </xsl:for-each>



                 </tr>

              </xsl:for-each>

            </table>



    </xsl:template>



    <xsl:template match="/e:Envelope/e:Body/g:reportResults/g:return/g:results/g:record/g:field/g:name">

            <td>

              <xsl:value-of select="." />

            </td>   

    </xsl:template>

</xsl:stylesheet>

Here is the XSL that will output XML:

<?xml version="1.0"?>

<xsl:stylesheet version="1.0"

  xmlns:xsl="http://www.w3.org/1999/XSL/Transform"

  xmlns:e="http://schemas.xmlsoap.org/soap/envelope/"

  xmlns:g="http://geneva.advent.com" 

>

    <xsl:output omit-xml-declaration="no" indent="yes"/>



    <xsl:strip-space elements="*"/>



    <xsl:key name="kFieldNameByValue" match="/e:Envelope/e:Body/g:reportResults/g:return/g:results/g:record/g:field/g:name"

         use="."/>



    <xsl:variable name="vCols" select=

       "/e:Envelope/e:Body/g:reportResults/g:return/g:results/g:record/g:field/g:name

                [generate-id()

                =          

                 generate-id(key('kFieldNameByValue',.)[1])

                 ]"/>



    <xsl:template match="/">

	  	<xsl:element name="root">

               	<xsl:element name="header">

				  <xsl:apply-templates select="$vCols"/>

				</xsl:element>  



               <xsl:for-each select="/e:Envelope/e:Body/g:reportResults/g:return/g:results/g:record">                   

                 <xsl:element name="record">

                   <xsl:variable name="vPos" select="position()"/>

                   <xsl:for-each select="$vCols">

                       <xsl:element name="value"><xsl:value-of select="../../../g:record[$vPos]/g:field[g:name = current()]/g:value"/></xsl:element>

                   </xsl:for-each>

                 </xsl:element>

              </xsl:for-each>



        </xsl:element>

    </xsl:template>



    <xsl:template match="/e:Envelope/e:Body/g:reportResults/g:return/g:results/g:record/g:field/g:name">

            <xsl:element name="name"><xsl:value-of select="."/></xsl:element>

    </xsl:template>



</xsl:stylesheet>

A quick explanation of what the XSL code does:
1-6 Is just stating it is an XML document which uses a namespace prefixed "xsl" for the transaformation instructions, a soap namespace with prefix "e" and an Advent Geneva namespace with prefix "g". Note that I abreviated the original namespace suffixes for the last two.

7-9 How the final output should look like.

11-19 We use the Muenchian grouping that allows us to have a list of all possible column names.

21-37 XSLT is a functional language which works matching nodes and applying transformations to them. The logic can be affected using xsl:apply-template. We use xsl:element to create our custom nodes: First the headers which come out of the Muenchian key and later the records for which we again use the keys while addressing the correct record through an xsl:foreach nested loop.

39-41 The template responsible for generating the name nodes.

Twitter RSS: Follow silently

2011-08-27T06:30:00.000-07:00

I blog about the work that I do just because "the palest ink is better than the best memory"

I tweet about most of the stuff that I do or I am interested in just because I think for some people is just easier to get an SMS alike message with a title that makes you decide if the article might be of your interest.

I consume most of the information I am able to digest via RSS.

These three statements probably make me not a good social network user. I would love to have time to follow 2000 people, interact with them, engage in interesting conversations and so on but I don't believe that is possible for a first generation immigrant, father of three kids and with the hope to remain competitive in the fast paced World of IT.

The way I live and work push me for automation even when consuming information. RSS, Atom or whatever format for syndicated news is then the answer to my needs.

Syndicated feeds should be a mandatory option for any website or service providing information. If you are like me you will love this trick which allows you to silently consolidate in your News reader (I use Google Reader BTW) tweets from people you would like to follow.

With an example here is how you can consume my tweets from your favorite reader:

http://api.twitter.com/1/statuses/user_timeline.rss?screen_name=nestor_urquiza

I make my statements public because I do care about what others think. If you consume them via RSS, email, native program or any other means that is secondary for me.

TSQL Stored Procedure faster from SSMS than the application

2011-08-25T11:39:00.000-07:00

No, I have no quick fix for this because SQL Server is not about magic and there is a reason why there are guys that work 100% of the time as SQL developers.

You have to ensure your queries are optimized and you know what query plan will be used when they are run. This is not about Java or .NET problems because your SQL Server Management Studio (SSMS) is "apparently" running the query 10 or 20 times faster than the application. It is indeed about your SQL code optimizations.

So if you have this problem then run the stored procedure using something like:

SET ARITHABORT OFF

EXEC sp_custom_procedure 'param1', 2, 'param3'

GO

Then run it again like:

SET ARITHABORT ON

EXEC sp_custom_procedure 'param1', 2, 'param3'

GO

You should get a slower response for the first and that will only tell you that your stored procedure is using a non efficient query plan. Now it is time to do your homework reading the previous link and make your stored procedure run with a custom well predefined fast and efficient query plan (of course as fast and efficient as you can)

Error creating bean with name 'liferayTransactionManager' defined in class path resource [META-INF/hibernate-spring.xml]

2011-08-25T06:21:00.000-07:00

Every once in a while when I install Liferay in different servers I get something like the below:

15:39:20,176 ERROR [ContextLoader:215] Context initialization failed

org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'transactionAdvice' defined in class path resource [META-INF/base-spring.xml]: Cannot resolve reference to bean 'liferayTransactionManager' while setting bean property 'transactionManager'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'liferayTransactionManager' defined in class path resource [META-INF/hibernate-spring.xml]: Cannot resolve reference to bean 'liferayHibernateSessionFactory' while setting bean property 'sessionFactory'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'liferayHibernateSessionFactory' defined in class path resource [META-INF/hibernate-spring.xml]: Invocation of init method failed; nested exception is org.hibernate.HibernateException: Hibernate Dialect must be explicitly set

This is an error that relates to a connectivity problem with your database. Be sure you see the *whole* stacktrace and look for a clue about connection/permission problems. Commonly you should be able to get it right if you follow these directions (Using MySQL here):

Check what IP MySQL is listening to. Below it shows as listening in 192.168.0.6

$ netstat -an|grep 3306

tcp        0      0 192.168.0.6:3306        0.0.0.0:*               LISTEN

Try connecting with the same host and credentials Liferay is using from command line mysql client:

$ mysql -u liferay -p -h 192.168.0.6

Enter password: 

ERROR 1130 (HY000): Host '192.168.0.6' is not allowed to connect to this MySQL server

As the example above shows there is a permision problem so go ahead and correct it. In MySQL password must be specified per each granted user@host combination:
```
$ mysql -u root -p

...

mysql>  GRANT ALL ON lportal.* TO 'liferay'@'192.168.0.6';

Query OK, 0 rows affected (0.00 sec)
```
Review your /etc/hosts in the client to make sure if you are using a host it maps to the specified IP:
```
$ vi /etc/hosts

192.168.0.6  sqlHost
```

nullmailer smtp Failed 550 5.7.1 Unable to relay for user@myhost.myhost big log files

2011-08-23T20:19:00.002-07:00

One of our servers was reporting low HDD resources and as usual I took a look at log files. Yes, again /var/log/mail.* were above 1GB, in some cases even 5GB.

This is the error that I saw in the traces:

Aug 23 21:56:33 myhost nullmailer[782]: Starting delivery: protocol: smtp host: mail.nestorurquiza.com file: 1314082502.18221

Aug 23 21:56:33 myhost nullmailer[27145]: smtp: Failed: 550 5.7.1 Unable to relay for nestorurquizaadmin@myhost.myhost

Aug 23 21:56:33 myhost nullmailer[782]: Sending failed:  Permanent error in sending the message

This was originated because in etc/hosts I had a line like:

192.168.1.23 myhost

And the SMTP server was not configured to relay to such a thing like myhost.myhost. So just changing it to an authorized domain should fix it right?

192.168.1.23 myhost.nestorurquiza.com

Well I kept getting the same error!!!

The reason was queued emails. I was getting from logs also a line like below:

Aug 23 22:26:47 myhost nullmailer[828]: Delivery complete, 42489 message(s) remain.

So deleting the queue made finally the trick:

sudo rm -fR  /var/spool/nullmailer/queue/

sudo mkdir  /var/spool/nullmailer/queue

nullmailer smtp Failed 550 5.7.1 Unable to relay for user@myhost.myhost big log files

2011-08-23T20:19:00.000-07:00

Aug 23 21:56:33 myhost nullmailer[782]: Starting delivery: protocol: smtp host: mail.nestorurquiza.com file: 1314082502.18221

Aug 23 21:56:33 myhost nullmailer[27145]: smtp: Failed: 550 5.7.1 Unable to relay for nestorurquizaadmin@myhost.myhost

Aug 23 21:56:33 myhost nullmailer[782]: Sending failed:  Permanent error in sending the message

This was originated because in etc/hosts I had a line like:

192.168.1.23 myhost

And the SMTP server was not configured to relay to such a thing like myhost.myhost. So just changing it to an authorized domain should fix it right?

192.168.1.23 myhost.nestorurquiza.com

Well I kept getting the same error!!!

The reason was queued emails. I was getting from logs also a line like below:

Aug 23 22:26:47 myhost nullmailer[828]: Delivery complete, 42489 message(s) remain.

So deleting the queue made finally the trick:

sudo rm -fR  /var/spool/nullmailer/queue/

sudo mkdir  /var/spool/nullmailer/queue

apache2: Could not reliably determine the server's fully qualified domain name

2011-08-12T07:52:00.000-07:00

This error is caused by an apache miss configuration. Below are those cases where I have seen it and how I have corrected them:

Sometimes it is just about duplicated directive ServerName. If you use Virtual hosts do not include a global directive but instead just define it in the virtual hosts files, for example:

...
<VirtualHost  nestorurquiza.com:443>
 SSLEngine on
 DocumentRoot "/var/nestorurquiza-app"
 ServerName nestorurquiza.com
...

Your server IP must be mapped to the server domain in /etc/hosts. It cannot be a loopback IP. Provide a separated line for that entry. Do not include any other mapping for the IP. If you need more just use new lines like below:

...
192.168.0.137   nestorurquiza.com
192.168.0.137   nestorurquiza
...

Install openssl in Ubuntu

2011-08-11T13:01:00.001-07:00

sudo apt-get install libcurl4-openssl-dev

Using more than one certificate from the JVM for SSL Web Services LDAPS and more

2011-08-11T12:33:00.000-07:00

There is one time when your JVM hosting your servlets will demand connecting to more than one SSL Web Service, an LDAPS, passing in some cases private keys, using in some cases different keystore files and more.

In those cases the typical System properties to set up the keystore and password will apparently not work. You can of course pass the properties as part of the java command line that starts the JVM but you can do the same programatically of course.

System.setProperty("javax.net.ssl.keyStore", "keystore path");

System.setProperty("", "keystore password");

System.setProperty("javax.net.ssl.keyStoreType", keystore type);

You will be tempted to get into a really complicated solution.

However there is a simple approach around this problem. If you are contraint about having just one keystore to host multiple certificates that should not be a big issue. After all it is a key "store". All you need to do is import all your certificates and keys in just one keystore following a procedure like the one I described before.

Again what you need to have clear is how you deal with your keys which might be encrypted with a password and that password must be the same you use for your keystore. At that point if you have let us say just one case for a key using a password you can just use the same for your keystore. If not then negotiate a new common password with your service providers. This is after all your client key, yes it is private but it is private to you. A simple phone call and an agreement on the new password will make the trick. Do not use an email for that though!

We recently spent several days troubleshooting issues like this just because I was relying on "that was already tested and it did not work". It demanded a new developer to take over the task so he could start fresh and try the most basic stuff. Simpler is after all better.

As always log traces are your friend when Google is not. Be sure to activate them for SSL when you are dealing with certificate issues:

-Djavax.net.debug=ssl,handshake

Premature Optimization is not Agile or should I say it is Evil

2011-08-10T09:45:00.000-07:00

The team was having an issue after introducing JPASecurity in the project. Some feature suddenly broke so it must be JPASecurity which is buggy, still in its early phase, ...

Come on, this was working before so ...

Well the fact that your code is working does not mean it is correct. An Abstract class cannot be instantiated but with so much Dependency Injection and Inversion Of Control sometimes we lose control LOL.

An Abstract class is not to be used by any other than implementing classes. But what happens if the developer declares the class abstract and after that uses it as a JPA Entity? Of course this is a mistake, a common mistake I would say where the engineer thinks he is comming up with a great design that will save our lifes in the future, some kind of generalization up front that will address all details in the future, the silver bullet or a synonym (not about performance but about feature implementation) of what the Industry knows as Premature Optimization.

The class is declared Abstract and there is not a single implementation of it but somehow JPA Hibernate EntityManager.merge(Entity) will not complain (Probably because after all it uses a proxy and not the actual class). All goes "good" until one day when you actually try to use reflection on the entity for example if you introduce JPASecurity to provide ACL in your JPA entities.

At this point you will end up with exceptions like:

java.lang.SecurityException: java.lang.InstantiationException
 at net.sf.jpasecurity.util.ReflectionUtils.throwThrowable(ReflectionUtils.java:109)
 at net.sf.jpasecurity.util.ReflectionUtils.newInstance(ReflectionUtils.java:38)
 at net.sf.jpasecurity.mapping.DefaultClassMappingInformation.newInstance(DefaultClassMappingInformation.java:216)
 at net.sf.jpasecurity.entity.EntityPersister.createUnsecureObject(EntityPersister.java:250)
 at net.sf.jpasecurity.entity.AbstractSecureObjectManager.getUnsecureObject(AbstractSecureObjectManager.java:140)
 at net.sf.jpasecurity.entity.EntityPersister.getUnsecureObject(EntityPersister.java:240)
 at net.sf.jpasecurity.entity.EntityPersister.merge(EntityPersister.java:73)
 at net.sf.jpasecurity.persistence.DefaultSecureEntityManager.merge(DefaultSecureEntityManager.java:130)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:597)
 at org.springframework.orm.jpa.ExtendedEntityManagerCreator$ExtendedEntityManagerInvocationHandler.invoke(ExtendedEntityManagerCreator.java:365)
 at $Proxy1122.merge(Unknown Source)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:597)
 at org.springframework.orm.jpa.SharedEntityManagerCreator$SharedEntityManagerInvocationHandler.invoke(SharedEntityManagerCreator.java:240)
 at $Proxy1013.merge(Unknown Source)
 at com.nestorurquiza.dao.CrudDaoImpl.update(CrudDaoImpl.java:105)
 at com.nestorurquiza.service.impl.CrudServiceImpl.update(CrudServiceImpl.java:45)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:597)
 at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:309)
 at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
 at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
 at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:110)
 at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
 at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
 at $Proxy1037.update(Unknown Source)

This is most likely because the proxy does not implement the default contructor in which case the Abstract class is the one attempted to be instantiated originating the Exception.

But the bottom line of this post is to serve as a little bit of education to those young fellas who are catching up with programming and those experienced ones that love over bloated Enterprise Design up front.

Please guys, try to keep it simple and do not fear refactoring. Embrace change and hard work. You will find yourself applying patterns as a result of pragmatism rather than as a result of that new cool design you learned from GOF alike book on Patterns.

BHUB: Allow user and password from any URL

2011-08-09T14:28:00.000-07:00

The BHUB philosophy is a unique entry point for all the business logic. A Controller makes that happen and there is no question on the big savings. The logic can be used from a CLI script (even pure curl can do the job), it can be of course croned, it can be used from any device (handhelds, desktops, appliances). You get for free every single security you applied in your web tier and I better stop right here because I have written about this several times.

The common way to interact with BHUB would be to get a session, security token from an entry point let us say a login service and from there on keep requesting using that information.

It would be ideal though that the entry point could be any page. In other words allow the first request to be actually not just login but a service request where you provide login information.

In order to accomplish this we will need to hook into Spring once again. Just a custom filter will do the trick. Declare it and assign some permissions for the URLs. In this case I want to allow direct access to certain role (the API role) and I will be opening that functionality to just the pure CLI Controllers which of course does not mean that we cannot allow this functionality for the complete BHUB Service Suite:

<beans:bean id="customLoginFilter" class="com.nestorurquiza.web.filter.CustomLoginFilter"/>

<custom-filter  before="FORM_LOGIN_FILTER" ref="customLoginFilter" /> 

<http auto-config="true" use-expressions="true" access-decision-manager-ref="accessDecisionManager" disable-url-rewriting="true">

  <intercept-url pattern="/cli/**" access="hasRole('ROLE_API')" />

Here is the Filter code.

package com.nestorurquiza.web.filter;



import java.io.IOException;



import javax.servlet.Filter;

import javax.servlet.FilterChain;

import javax.servlet.FilterConfig;

import javax.servlet.ServletException;

import javax.servlet.ServletRequest;

import javax.servlet.ServletResponse;

import javax.servlet.http.Cookie;

import javax.servlet.http.HttpServletRequest;

import javax.servlet.http.HttpServletResponse;



import org.slf4j.Logger;

import org.slf4j.LoggerFactory;

import org.springframework.beans.factory.annotation.Autowired;

import org.springframework.security.core.Authentication;

import org.springframework.security.core.context.SecurityContext;

import org.springframework.security.core.context.SecurityContextHolder;

import org.springframework.security.web.authentication.UsernamePasswordAuthenticationFilter;

import org.springframework.security.web.authentication.rememberme.AbstractRememberMeServices;

import org.springframework.security.web.util.TextEscapeUtils;



import com.nestorurquiza.utils.CsrfUtil;

import com.nestorurquiza.web.WebConstants;



/**

 * Allowing the creation of a session on the fly when requesting any URL

 * We authenticate the user if not authenticated in the case a user and password is provided 

 * @author nestor

 *

 */

public class CustomLoginFilter implements Filter 

{

    private static final Logger log = LoggerFactory.getLogger(CustomLoginFilter.class);

    

    @Autowired

    UsernamePasswordAuthenticationFilter usernamePasswordAuthenticationFilter;

    

    @Override

    public void doFilter(ServletRequest request, ServletResponse response, FilterChain chain) throws IOException, ServletException 

    {

        HttpServletRequest httpRequest = ((HttpServletRequest) request);

        String uri = httpRequest.getRequestURI();

        if(! uri.contains(".")) {

            HttpServletResponse httpResponse = ((HttpServletResponse) response);

            SecurityContext securityContext = SecurityContextHolder.getContext();

            String method = httpRequest.getMethod();

            String userName = request.getParameter("j_username");

            if(userName != null && "POST".equals(method) && (securityContext == null || securityContext.getAuthentication() == null || !securityContext.getAuthentication().isAuthenticated())) {

                try {

                    Authentication auth = usernamePasswordAuthenticationFilter.attemptAuthentication(httpRequest, httpResponse);

                    securityContext.setAuthentication(auth);

                    SecurityContextHolder.setContext(securityContext);

                    httpRequest.getSession().setAttribute("SPRING_SECURITY_CONTEXT", securityContext);

                    httpRequest.getSession().setAttribute(UsernamePasswordAuthenticationFilter.SPRING_SECURITY_LAST_USERNAME_KEY, TextEscapeUtils.escapeEntities(userName));

                    if(auth.isAuthenticated()) {

                        //Initialize CSRF token

                        CsrfUtil.initializeCsrfToken(httpRequest);

                        //Set the token as an attribute in the request to make it pass the security check

                        httpRequest.setAttribute(WebConstants.CSRF_TOKEN, httpRequest.getSession().getAttribute(WebConstants.CSRF_TOKEN));

                    }

                } catch (Exception e){

                    log.info("User could not be authenticated. We go on ...");

                }

                

            }

        }

        chain.doFilter(request, response);

    }



    @Override

    public void destroy() 

    {   



    }



    @Override

    public void init(FilterConfig config) throws ServletException 

    {

    

    }

    

    private Cookie getRememberMeCookie(Cookie[] cookies) {

        if (cookies != null)

          for (Cookie cookie : cookies) {

            if (AbstractRememberMeServices.SPRING_SECURITY_REMEMBER_ME_COOKIE_KEY.equals(cookie.getName())) {

              return cookie;

            }

          }

        return null;

    }

}

Tainting LDAP data with Talend

2011-08-09T14:04:00.000-07:00

Even though I solved this through the use of a nodeJS script I wanted to get the problem solved from Talend.

Here I discuss how to do it from Talend. There are still some outstanding issues.

As you can see in the picture I am reusing the list of white listed emails from a hashmap (tHashOutput_9). Only one connection is done to the input LDAP server which needs normalization (tNormalize) for group/uniquemember. A tJavaRow is responsible for adding a column containing just the uniquemember email for filtering purposes executed by tMap_1. tLDAPOutput_3 inserts the root elements while tLDAPOutput_1 and tLDAPOutput_2 insert users and groups respectively. Note that the latest two are in a separate subjob to be able to create the root entries before. Again the use of hashes allow to communicate both subjobs.

Install gcc in Ubuntu

2011-08-09T09:56:00.000-07:00

The Linux sysadmin must build from sources from time to time. It makes sense then to have the C environment ready. In Ubuntu all you need to do is to follow the below commands:

sudo apt-get update

sudo apt-get upgrade

sudo apt-get install build-essential

gcc --version

Phishing Attack: Fake Twitter Email not marked as Spam

2011-08-06T04:38:00.001-07:00

I got an email in my Gmail account from "Twitter Support" "with subject "Your account has been suspended" with no text content but an image (that I have disabled of course for security reasons). The image content was something like "We detected unusual activity ..."

This phishing email is nothing new but what came to my attention was that Gmail was not able to detect the spam even though the full headers from the message are showing how Google identified it as a candidate for Spam "Authentication-Results: mx.google.com; spf=hardfail (google.com: domain of support@twitter.com does not designate 203.115.131.123 as permitted sender) smtp.mail=support@twitter.com"

Any software is plenty of bugs. Even with the best developers on board you are still vulnerable. Good that "Report phishing" option is available albeit a little bit hidden behind an arrow close to the Reply link. User experience should be helping better here I would say but regardless the important lesson to learn is to be always suspicious up front. Do not trust any bad news (account hacked or compromised) or too good news (You just won a million dollar) you receive.

See below for the full headers of the message:

Delivered-To: nestor.urquiza@gmail.com
Received: by 10.236.179.100 with SMTP id g64cs68259yhm;
        Fri, 5 Aug 2011 22:37:44 -0700 (PDT)
Received: from mr.google.com ([10.142.187.15])
        by 10.142.187.15 with SMTP id k15mr3461337wff.111.1312609063904 (num_hops = 1);
        Fri, 05 Aug 2011 22:37:43 -0700 (PDT)
Received: by 10.142.187.15 with SMTP id k15mr2917056wff.111.1312609063502;
        Fri, 05 Aug 2011 22:37:43 -0700 (PDT)
Return-Path: <support@twitter.com>
Received: from vsfilter2.roc.bti.net.ph (vsf-mx4.bti.net.ph [203.115.131.123])
        by mx.google.com with ESMTP id w1si282094wfw.62.2011.08.05.22.37.42;
        Fri, 05 Aug 2011 22:37:43 -0700 (PDT)
Received-SPF: fail (google.com: domain of support@twitter.com does not designate 203.115.131.123 as permitted sender) client-ip=203.115.131.123;
Authentication-Results: mx.google.com; spf=hardfail (google.com: domain of support@twitter.com does not designate 203.115.131.123 as permitted sender) smtp.mail=support@twitter.com
X-IronPort-Anti-Spam-Filtered: true
X-IronPort-Anti-Spam-Result: AqA2AOHRPE7Lc4NugWdsb2JhbAAoEwcXgjgBD4NgjV+EQwGOLRNcAQEWJiVxSxISGQELCk0BAQECDQ4MJAJQh3oKIgGeN5I1jSaDLQyCLl8Eh1qYFoMBgQaCYTA
Received: from smtp4-roc.bti.net.ph (HELO smtp1.skyinet.net) ([203.115.131.110])
  by vsfilter2.roc.bti.net.ph with ESMTP; 06 Aug 2011 13:37:41 +0800
Received: from 110.55.232.159.BTI.NET.PH (unknown [110.55.236.20])
 by smtp4-roc.bti.net.ph (Postfix) with ESMTP id 55F8793DB3A
 for <nestor.urquiza@gmail.com>; Sat,  6 Aug 2011 13:37:41 +0800 (PHT)
From: "Twitter Support" <support@twitter.com>
Subject: Your account has been suspended
To: "nestor.urquiza" <nestor.urquiza@gmail.com>
Content-Type: multipart/alternative; charset="iso-8859-10"; boundary="LMRJGCZhTXlUeMlXLirvgZD=_SMWAE68zR"
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
Date: Sat, 6 Aug 2011 13:37:37 +0800
Message-Id: <20110806053741.55F8793DB3A@smtp4-roc.bti.net.ph>

This is a multi-part message in MIME format

--LMRJGCZhTXlUeMlXLirvgZD=_SMWAE68zR
Content-Type: text/plain ; charset="iso-8859-10"
Content-Transfer-Encoding: quoted-printable




--LMRJGCZhTXlUeMlXLirvgZD=_SMWAE68zR
Content-Type: text/html ; charset="iso-8859-10"
Content-Transfer-Encoding: quoted-printable

<HTML><HEAD>
<META name=3DGENERATOR content=3D"MSHTML 8.00.6001.23019"></HEAD>
<BODY>
<P><A href=3D"mexico.cnn.com/redirectComplete.php?url=3D//emailus%2Eit=
%2Etc/2ule3B"><IMG border=3D0 src=3D"http://3.bp.blogspot.com/-u_sWLHS=
Yjes/TjyqVZ73vrI/AAAAAAAAAEQ/hX5mKS-R7-g/s1600?2ule3B"></A> </P>
<P>&nbsp;</P></BODY></HTML>


--LMRJGCZhTXlUeMlXLirvgZD=_SMWAE68zR--

Server scripting with NodeJs: Tainting LDAP data

2011-08-04T13:21:00.000-07:00

This is a real world example of using nodejs to do server side scripting. It shows how to deal with command line arguments, invoke shell commands and parse responses, interact with mysql and how to use the library (when possible and makes sense) in a synchronous way.

Node (also knows and nodejs) is a javascript library that allows to run javascript code out of the browser. It favors event programming which can be difficult for backend developers used to languages like Java. At the same time it is straightforward for front end developers which are already used to javascript callback functions necessary for UI programming.

If you havent install it yet go ahead and do it now:

I have tested all this with node-v0.4.10 as you see below.

cd ~/Downloads
wget http://nodejs.org/dist/node-v0.4.10.tar.gz
tar -zxvf node-v0.4.10.tar.gz
cd node-v0.4.10
./configure
make
sudo make install
node --version

Install npm (package manager for nodejs) in case you need extra modules (and you will when you get serious about nodejs development)
```
curl http://npmjs.org/install.sh | sudo sh
```
If you use MySQL here is how to install one library that just works (whole API: https://github.com/felixge/node-mysql)
```
npm install mysql
```
You should of course test with the simplest possible 'Hello World' script
```
$ vi hello.js
console.log('Hello World');
$ node hello.js
```

The Node project is making a big effort to enforce the use of non blocking code (code that will be triggered but will not block the current thread). That is the reason why most of the code you will see is written with nested callback functions.

Event programming is good paradigm for certain problems. Responsiveness of a program can be improved a lot without having to deal with blocking in multithreading programming. Those of us that have worked with threads know how hard it can be. I am not nodejs is better than java or any other language to deal with multiprocessing. It is just a different paradigm and I would appreciate if you do not start ranting against or in favor of any programming language in this post.

Here is one extract from the process api: http://nodejs.org/docs/v0.3.1/api/process.html

process.argv.forEach(function (val, index, array) {
  console.log(index + ': ' + val);
});
console.log

The previous code is non blocking, so if you need to work with the arguments you will need to work inside the anonymous function. Look at the code below. After running it you will see the asynchronous code does not run before the console logs the message:

var asyncArg = 'Async Arguments:';
var syncArg = 'Sync Arguments:';

var argv = process.argv;
argv.forEach(function (val, index, array) {
 setTimeout(function() {asyncArg += ' ' + val; console.log( asyncArg ); }, 0);
 
});
console.log( asyncArg );

for( argIndex in argv ) {
 syncArg += ' ' + argv[argIndex];
}
console.log( syncArg );

So probably to work with command line arguments you are better off the asynchronous flow. One might expect that there is always a way to go synchronous with nodejs but that is not the case. As I said the nodejs library pushes for non blocking === asynchronous code. Look at the below code which comments should be self explanatory:

var sys = require('sys')
var exec = require('child_process').exec;

var child = exec("ldapsearch -x -v -H 'ldap://nestorurquiza:10389' -D 'uid=admin,ou=system' -w 'secret' -b 'o=nestorurquiza'", function (error, stdout, stderr) {
  console.log(stdout); //Will print the output of the command
  if (error !== null) {
    console.log('exec error: ' + error);
  }
});
console.log(child.stdout); //Will not print the output of the command but rather the object prototype

Here is a script that clones the data from one LDAP Server (tested with ApacheDS) and imports it into a second LDAP server. The script taints the emails to ensure we do not send messages to real production users while testing. It excludes certain domains and it shows how to interact with mysql to pull a white list of emails for which no tainting should be done. It also changes the passwords to a well known test password so the team can use it to debug what happens when different users interact with the application. BTW this task is better to be done from Talend but I needed a real problem to demonstrate nodejs is ready to work as server side scripting while I was figuring out how to make it happen from Talend:

#!/usr/local/bin/node
/*
** WARNING: This program will wipe out the specified BASE_DN from the target LDAP Server
**
** taintLdap.js A nodejs script to taint data from ldap. Use it to change passwords for all users to a known value and to change their emails to avoid sending messages from testing environment to real users
**
** Example: node taintLdap.js 'ldap://jnestorurquiza:10389' 'uid=admin,ou=system' 'secret' 'ldap://localhost:10389' 'uid=admin,ou=system' 'secret'
**
** @Author: Nestor Urquiza
** @Date: 08/02/2011
**
*/

/*
** Imports
*/
var sys = require('sys')
var exec = require('child_process').exec;
var Client = require('mysql').Client;
var client = new Client();

/*
** Constants
*/
var BASE_DN = "o=nestorurquiza";
var EXCLUSION_DOMAINS = ['nestorurquiza.com','nestoru.com'];
var APPEND_DOMAIN = 'nestorurquiza.com'
var COMMON_PASSWORD = 'e1NIQX1JcnFMUVdMT1o3ZXF0WHRBdUlFSFRlUnZkRFk9' //Testtest1 after SHA1. To generate a different password use: echo -n "mypassword" | openssl dgst -sha1

/*
** Arguments
*/
var argv = process.argv;
if( argv.length != 8 ) {
 usage();
 process.exit(1);
}
var fromUrl = argv[2];
var fromUser = argv[3];
var fromPasword = argv[4];
var toUrl = argv[5];
var toUser = argv[6];
var toPasword = argv[7];

client.user = 'root';
client.password = 'root';
client.host = 'localhost';
client.port = '3306';
client.database = 'nestorurquiza'

/*
** Main
*/
//console.log('Cloning and tainting from ' + fromUrl + ' to ' + toUrl);
var ldif;
var authorizedEmails = new Array();
var pattern = /[^\s=,]*@[^\s=,]*/g
var child = exec("ldapsearch -x -v -H '" + fromUrl + "' -D '" + fromUser + "' -w '" + fromPasword + "' -b '" + BASE_DN + "'", function (error, stdout, stderr) {
  if (error) {
    throw error;
  }
  ldif = stdout;
  client.connect();
  var authorizedEmailQuery = client.query(
  'SELECT name FROM authorized_test_email',
  function (error, results, fields) {
    if (error) {
      throw error;
    }
    for (var resultIndex in results){
      var result = results[resultIndex];
      authorizedEmails[resultIndex] = result.name;
    }
    client.end();
    //console.log('****************'  + authorizedEmails);
    ldif = taintLdifEmail();
 
 child = exec("ldapdelete -r -x -H '" + toUrl + "' -D '" + toUser + "' -w '" + toPasword + "' '" + BASE_DN + "'", function (error, stdout, stderr) {
   if (error) {
        //throw error;
        console.log("WARNING: Could not delete " + BASE_DN);
      }
      var command = "echo '" + ldif + "' | ldapmodify -x -c -a -H '" + toUrl + "' -D '" + toUser + "' -w '" + toPasword + "'";
      //console.log(command);
      child = exec(command, function (error, stdout, stderr) {
     
  if (error) {
   throw error;
  }
   });
    });   
 //ldapmodify -x -c -a -H ldap://localhost:10389 -D "uid=admin,ou=system" -w 'secret' < ~/Downloads/taintedLdap.ldif
  }
);
});

/*
** Functions
*/
function taintLdifEmail() {
 var matches = ldif.match(pattern);

 for ( var matchIndex in matches ) {
  var taint = true;
  var match = matches[matchIndex];
  for( var exclusionDomainIndex in EXCLUSION_DOMAINS ) {
   if( match.indexOf(EXCLUSION_DOMAINS[exclusionDomainIndex]) >= 0 ) {
    taint = false;
    break;
   }
  }
  if( !taint ) {
   continue;
  }
  for ( var authorizedEmailIndex in authorizedEmails ) {
   var authorizedEmail = authorizedEmails[authorizedEmailIndex];
   if( match == authorizedEmail ) {
    taint = false;
    continue;
   }
  }
  if( taint ) {
   var replacement = match.replace('.', '') + APPEND_DOMAIN;
   //console.log( match + " >>> " + replacement );
   ldif = ldif.replace(match, replacement);
  }
 }
 ldif = ldif.replace(/userPassword.*/g, 'userPassword:: ' + COMMON_PASSWORD);
 return ldif;
}

function usage() {
 console.log("Usage: " + "./taintLdap.js <fromUrl> <fromUser> <fromPasword> <toUrl> <toUser> <toPasword>");
}

Why I am trying to use nodejs if I already have bash, awk, perl, python, ruby and what not? I am building a team with strong separation of concerns. While we know we cannot be as good as the guy that is spending 100% of the time in just writing SQL stored procedures the whole team could cover for some days to be able to compensate vacation time for example, so yes SQL is a mandatory skill. Javascript is a mandatory skill as well and if I can script with it I could have some of those scripting needs done by anybody in the team as well. I am just trying to keep really low the amount of technologies and languages we use.

Isn't it better to use RhinoJs? Probably yes, but I am tempted to use something out of the JVM that runs faster and consume less resources. I have recently decomisioned a whole CLI project based in Java just because it was really resource intensive. I have favored the use of Controllers in our Business Hub which are called from simple CURL or WGET statements. I am not claiming RhinoJs is unnacceptable slow nor that NodeJs is better. I see value in both of them.

Why I am considering scripting after all if I promote the idea of a Business Hub? There are cases in which definitely using Unix Power Tools, existing CLIs etc do the job quickly and more reliable.

Do I think NodeJs is a better answer for Server side logic than Java? At the moment I am happy with plain java for my backend. The amount of existing code available for free is amazing. If that will be the case in the future it will depend on the open source community. At least for my current project I stick to Java.

PKCS12 to JKS keystore

2011-08-04T09:04:00.000-07:00

Java uses a proprietary to Sun format (JKS) to store certificates in what is called a Keystore (A file containing entries of those certificates you trust)

When you get certificates included in a different keystore type like it is the case of PKCS12 (commonly using the *.p12 extension) you need to extract and add them to the JKS keystore. Failure to do so will make your program depending on individual keystore files rather than just one keystore where all certificates and private keys are kept.

Here is how you can do it (sample using OSX paths but applicable to other OS as well).
First find the key for the certificate to export. Basically the left first word for the specific entry from this command:

keytool -list -keystore /Users/nestor/Downloads/cert.p12 -storetype pkcs12

Then run something like the below. Note that alias.from.cert.p12 comes from the previous command.

keytool -importkeystore -srckeystore /Users/nestor/Downloads/cert.p12 -destkeystore /Library/Java/Home/lib/security/cacerts -srcstoretype PKCS12 -deststoretype JKS -srcstorepass cert.p12.password -deststorepass changeit.is.the.default.password -srcalias alias.from.cert.p12 -destalias alias.for.cacerts.new.certificate

Note that the private key inside PKCS12 might need a password and that password must be the same as the JKS where you are importing. Failure to do this will end up in Access Denied, Forbidden or any other error comming from the Server where the key is attempted to be used.

ldapsearch SSL with ApacheDS

2011-08-03T14:01:00.000-07:00

Self signed certificates are treated different by the ldap cli tools.

The task was to connect with ldapsearch to a remote ApacheDS server serving SSL. Long story short the certificate is self signed and only certain IP range can access the server via LDAP over SSL (TLS).

Here are the steps showing how to configure ldapsearch and the rest of ldap tools to work with SSL (both signed and self signed):

If not using self signed certificate then get the server certificate
```
$ openssl s_client -connect ldap.nestorurquiza.com:636
```
Non self signed certificate: Create a file with the contents from "-----BEGIN CERTIFICATE-----" up to "-----END CERTIFICATE-----") from the previous command
```
$ sudo mkdir /etc/openldap/certs/
$ vi /etc/openldap/certs/ldap.nestorurquiza.com.cert
```
Be sure your certificate is not self signed. Basically check for a return code=0, not someting like "Verify return code: 18 (self signed certificate)"
```
$ openssl s_client -connect ldap.nestorurquiza.com:636 -CAfile ldap.nestorurquiza.com.cert
```

Edit the ldap configuration

$ vi /etc/openldap/ldap.conf
...
#Use the below if you want ldapsearch to work with self signed certificate. Probably a better option security wise is to buy a certificate right ;-) Note that the path is for OSX. For Ubuntu it is /etc/ldap/certs...
#TLS_REQCERT    demand
TLS_REQCERT     never
#Use the below for non self signed certificates
#TLS_CERT    /etc/openldap/certs/ldap.nestorurquiza.com.cert

Run an ldapsearch command to be usre you get the ldif result

ldapsearch -x -v -H ldaps://ldap.nestorurquiza.com:10636 -D "uid=admin,ou=system" -w 'secretPassword' -b "o=nestorurquiza"

Redirect after login to requested page with Spring after CSRF protection

2011-07-18T08:46:00.000-07:00

Bookmarks, typing URLs directly in the address bar and getting to the requested page after login are functionalities you should not break as they impact user experience.

Once you have protected your Spring website against CSRF using the Synchronizer Token Pattern you will find that the redirection to the requested page after login functionality (You request a page, the login form shows up and after that you are taken to the page you originally requested) will be broken.

Basically the user might access a bookmark or just type a URL without the security token and the redirection will use the provided (and expired) security token or no security token at all. Of course you need to hook into Spring in order to change the default functionality.

First you use "authentication-success-handler-ref" form-login property in the Spring security context:

<beans:bean id="customAuthenticationHandler" class="com.nestorurquiza.web.handler.CustomAuthenticationHandler" />
    
<form-login login-page="/login"
            authentication-success-handler-ref="customAuthenticationHandler"
            authentication-failure-url="/login?error=authorizationFailed" />

Then implement the custom handler. Code should speak for itself.

package com.nestorurquiza.web.handler;

import java.io.IOException;

import javax.servlet.ServletException;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;

import org.springframework.security.core.Authentication;
import org.springframework.security.web.authentication.SavedRequestAwareAuthenticationSuccessHandler;
import org.springframework.security.web.savedrequest.DefaultSavedRequest;

import com.nestorurquiza.utils.UrlTool;
import com.nestorurquiza.web.WebConstants;

public class CustomAuthenticationHandler extends SavedRequestAwareAuthenticationSuccessHandler {

    @Override
    public void onAuthenticationSuccess(HttpServletRequest request,
            HttpServletResponse response, Authentication authentication)
            throws ServletException, IOException {
        // TODO Auto-generated method stub
        String ctoken = (String) request.getSession().getAttribute(WebConstants.CSRF_TOKEN);
        DefaultSavedRequest defaultSavedRequest = (DefaultSavedRequest) request.getSession().getAttribute("SPRING_SECURITY_SAVED_REQUEST_KEY");
        if( defaultSavedRequest != null && ctoken != null ) {
            String requestUrl = defaultSavedRequest.getRequestURL();
            requestUrl = UrlTool.addParamToURL(requestUrl, WebConstants.CSRF_TOKEN, ctoken, true);
            getRedirectStrategy().sendRedirect(request, response, requestUrl);
        } else {
            super.onAuthenticationSuccess(request, response, authentication);
        }
    }
}

Here is the little useful class that allows to override the ctoken parameter (or any other url parameter)

package com.nestorurquiza.utils;

public class UrlTool {
    public static String addParamToURL(String url, String param, String value,
            boolean replace) {
        if (replace == true)
            url = removeParamFromURL(url, param);
        return url + ((url.indexOf("?") == -1) ? "?" : "&") + param + "="
                + value;
    }

    public static String removeParamFromURL(String url, String param) {
        String sep = "&";
        int startIndex = url.indexOf(sep + param + "=");

        if (startIndex == -1) {
            startIndex = url.indexOf("?" + param + "=");
            sep = "?";
        }

        if (startIndex != -1) {
            int endIndex = url.indexOf(sep, startIndex + 1);
            return url.substring(0, startIndex)
                    + (endIndex > 0 ? (sep == "?" ? "?"
                            + url.substring(endIndex + 1) : url
                            .substring(endIndex)) : "");
        }

        return url;
    }

}

Caching with Spring ehcache and annotations

2011-07-06T18:13:00.000-07:00

Caching is trivial up to the moment you start wanting to cache too many entities. At that point you realize caching is actually a cross cutting concern which basically should be done the easy way, read using Inversion Of Control. But caching is also about where you cache: memory, file system?

If you are using Java then Spring in combination with ehCache can be used to provide caching while abstracting the developer from the details. To be able to achieve caching with minimum effort I recommend using ehcache spring annotations project. Note there is no need to add ehcache dependency as it is added by ehcache-spring-annotations project.

Here are the dependencies I used for Spring 3.0.4:

<!-- ehcache -->
        <dependency>
            <groupId>com.googlecode.ehcache-spring-annotations</groupId>
            <artifactId>ehcache-spring-annotations</artifactId>
            <version>1.1.2</version>
            <type>jar</type>
        </dependency>

Create WEB-INF/ehcache.xml with the below content:

<?xml version="1.0" encoding="UTF-8"?>
  <ehcache xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="http://ehcache.org/ehcache.xsd">
      <defaultCache eternal="true" maxElementsInMemory="100" overflowToDisk="false" />
      <cache name="findAllClients" maxElementsInMemory="10000" eternal="true" overflowToDisk="false" />
  </ehcache>

In application context add the below lines:

<beans ...xmlns:ehcache="http://ehcache-spring-annotations.googlecode.com/svn/schema/ehcache-spring"...
...
xsi:schemaLocation="
...
http://ehcache-spring-annotations.googlecode.com/svn/schema/ehcache-spring http://ehcache-spring-annotations.googlecode.com/svn/schema/ehcache-spring/ehcache-spring-1.1.xsd
...
<!-- ehcache -->
    <ehcache:annotation-driven />
 
    <ehcache:config cache-manager="cacheManager">
        <ehcache:evict-expired-elements interval="60" />
    </ehcache:config>
 
    <bean id="cacheManager" class="org.springframework.cache.ehcache.EhCacheManagerFactoryBean">
        <property name="configLocation"  value="/WEB-INF/ehcache.xml"/>
    </bean>
...

Look for a method that returns a collection of entities and annotate it like:

@Cacheable(cacheName = "findAllClients")
    public List<Client> findAll() {

If there is a method that inserts an entity related to the created cache (with name "findAllClients" in this case) then we annotate it so it cleans the cache after insertion:

@TriggersRemove(cacheName = "findAllClients", when = When.AFTER_METHOD_INVOCATION, removeAll = true)
    public void addClient(Client client) {

Of course there are cases where we are just consuming a collection let us say from a web service. We would like to force the cache to be cleaned even though we are never inserting an entity. Here is where you need to use a Controller that can be invoked to clean all or specific caches with a URL like:

http://localhost:8080/ehCache/remove?cacheName=findAllClients&cacheName=anotherCacheToRemove

Here is the Controller that will make this possible:

import java.util.ArrayList;
import java.util.List;

import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;

import net.sf.ehcache.Cache;
import net.sf.ehcache.CacheManager;

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Controller;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.servlet.ModelAndView;

@Controller
@RequestMapping("/ehCache/*")
public class EhCacheController extends RootController {
    private static final String EHCACHE_SHOW_PATH = "/ehCache/show";
    
    @Autowired
    private CacheManager cacheManager;

    /**
     * Removes all caches if no parameter is passed
     * Removes all caches for the specified "cacheName" parameters
     * 
     * @param request
     * @param response
     * @return
     */
    
    @RequestMapping("/remove")
    public ModelAndView home(HttpServletRequest request, HttpServletResponse response) {
        ControllerContext ctx = new ControllerContext(request, response);
        init(ctx);
        
        String[] storedCacheNames = cacheManager.getCacheNames();
        String[] cacheNames = ctx.getParameterValues("cacheName");
        
        List<Cache> caches = new ArrayList<Cache> (storedCacheNames.length);
        
        for( String storedCacheName : storedCacheNames ){
            Cache storedCache = cacheManager.getCache(storedCacheName);
            if( cacheNames == null ) {
                storedCache.removeAll();
            } else {
                for( String cacheName : cacheNames ) {
                    if( cacheName.equalsIgnoreCase(storedCacheName) ) {
                        storedCache.removeAll();
                    }
                }
            }
            caches.add(storedCache);
        }
        
        ctx.setRequestAttribute("caches", caches);

        return getModelAndView(ctx, EHCACHE_SHOW_PATH);
    }
}

The show.jsp would be something like:

<%@ include file="/WEB-INF/jsp/includes.jsp" %>
<%@ include file="/WEB-INF/jsp/header.jsp" %>
<div class="global_error"><c:out value="${csrfError}" /></div>
<div><c:out value="${caches}" /></div>
<%@ include file="/WEB-INF/jsp/footer.jsp" %>

To add more caching you will need to add the cache entry in the ehcache.xml (thing that I do not like to be honest as I think the annotation should be parsed and if there is no declaration for it in XML then use just the default values or better allow customization as part of the annotation itself) and you need to annotate the method which return value will be cached. To clean that cache you can have either one or a combination of the @TriggersRemove annotation and a URL for cache cleaning like explained before.

The palest ink is better than the best memory

2011-06-29T18:29:00.000-07:00

To document or not to document, that's the question.

Documentation saves hours of research and frustration and that translates into money. Yet so many IT managers tolerate systems without documentation while others go crazy after formalizations that end up in unmaintained material.

I still have to see documentation that is kept current. I am doing my best with my current team trying to enforce that practice. My opinion is documenting what you do is just about showing respect to your partners.

I am usually asked for the best anti-spyware or anti-virus for Windows and I always say: Rebuild your system every six months. Just have it documented: Basically keep a backup of your applications and related metadata and clear instructions as to how to recreate your system from scratch. Some years ago when I was using Windows as my main OS I found myself rebuilding my Windows box so often that I developed a procedure (part of it scripted I have to admit) that allowed me in just 2 hours to recreate my whole environment. That included: Apache, PHP, Java, Tomcat, JBoss, Eclipse, MS Office just to name few of them.

Nowadays I use VirtualBox and recovering Windows is just a matter of going to a safe Snapshot so the only reason for me to rebuild from scratch would be a company change (because of licensing)

The same happens in the world of OS virtualization. If you find that your P2V is difficult, go for a rebuild. With good documentation this could take anytime from 2 to 8 hours. Way less of what you will spend troubleshooting incompatibilities with your tools, different source and destination hardware etc. And if you do not have documentation you better start it.

This is exactly what happened to me when I tried to convert my Ubuntu 10 Server box into a VMware Server VM.

VMware converter will not work for Ubuntu 10. It is simply not supported http://www.vmware.com/support/converter/doc/releasenotes_conv40.html#platf

I tried anyway to install it (you know we always try just in case :-) but as you see below it is totally incompatible:

[several lines like below]
...
The script you are attempting to invoke has been converted to an Upstart
job, but lsb-header is not supported for Upstart jobs.
insserv: warning: script 'atd' missing LSB tags and overrides
insserv: Default-Start undefined, assuming empty start runlevel(s) for script `atd'
insserv: Default-Stop  undefined, assuming empty stop  runlevel(s) for script `atd'
The script you are attempting to invoke has been converted to an Upstart
job, but lsb-header is not supported for Upstart jobs.
insserv: warning: script 'udevmonitor' missing LSB tags and overrides
insserv: Default-Start undefined, assuming empty start runlevel(s) for script `udevmonitor'
insserv: Default-Stop  undefined, assuming empty stop  runlevel(s) for script `udevmonitor'
...
[more lines like above]

While I could have spent more time trying some other converters I decided to spend 8 hours building the box from scratch. But of course I had good old school documentation for building that box. WIKI is your fiend, don't forget that team member!

As the Chinese proverb says "The palest ink is better than the best memory"

Installing Configuring and Using Monit in Ubuntu

2011-06-22T09:48:00.000-07:00

This is so straight forward and well documented in the Monit project that I wouldn't post about this if not just to allow those users that are starting with Linux and monitoring is their next concern.

The steps here should allow the user to install monit and monitor Google home page in no more than 5 minutes. It is always gratifying to get things running while following simple steps. Dealing with environment issues is the most difficult and frustrating part when you are trying to prove your concept.

Install monit:
```
sudo apt-get install monit
```
Using sudo edit /etc/default/monit to setup automatic monit startup
```
startup=1
```

Using sudo edit /etc/monit/monitrc as per the content below. Customize your web interface password, an IP allowed to connect via WEB (if you are a GUI guy), email addressee for alerts and mailserver. The rest of the settings should be OK for you. Note that the only check I am providing is checking Google availability just as a proof of concept. The web is full of samples to monitor anything in your servers

##################################################################

# CUSTOM MONIT SETTINGS

##################################################################

set mail-format {

  subject: [ $SERVICE ] $EVENT - $DATE

  message: Action: $ACTION, Description: $DESCRIPTION, Service: $SERVICE, Tested From Host: $HOST }

# number of seconds between monit checks

set daemon 60

# Location of the monit log file

# /var/log/messages is just ok

# set logfile /var/log/monit.log

set logfile syslog facility log_daemon

# Change to the email address you want alerts to be sent to

set alert nestorurquizaappmon@nestorurquiza.com

# Port that the monit status page can be viewed

set httpd port 2812

# Allow localhost to connect

allow localhost

# The next line must be included to allow access from your computer

# change the number to what your local ip is

allow 0.0.0.0

# The next line assigns a username:password combination to login.

# Please change the password to something random.

allow admin:myAdminPassword

# Set the mailserver to send notification email.

set mailserver mail.nestorurquiza.com

################################################################# 
#REMOTE GOOGLE CHECKS
################################################################
check host google-test with address google.com
  if failed port 80 proto http then alert
group server

Start, stop, restart, force reload as needed

sudo /etc/init.d/monit start
sudo /etc/init.d/monit stop
sudo /etc/init.d/monit restart
sudo /etc/init.d/monit force-reload

See monit status, summary, unmonitor or monitor all as needed

sudo monit status
sudo monit summary
sudo monit monitor all
sudo monit unmonitor all

Get extra help. Learn how to start or stop processes. Master monit!
```
sudo monit -h
```
If you prefer a GUI then get it with the below URL (Use the password you set)
```
http://monit.server.address:2812/
```

Updating monit

Updating monit is easy. Just follow this steps after downloading latest version to your server home directory:

$ tar xvfz monit-5.3.tar.gz 
$ ./configure --sysconfdir=/etc/monit/
$ make
$ sudo make install
$ sudo monit quit
$ sudo monit

ACL based security in JPA with jpasecurity: the next step after spring security

2011-06-21T21:48:00.000-07:00

I had a clear need for ACL in my current project. Just protecting URLs is not enough and protecting method by method smells spaghetti code. Furthermore Spring solutions demand several hooks and still in my opinion they are still not addressing the real issue which is access control was removed from the database layer once ORM got mature but at the same time ORM did not provide a clean ACL solution.

If you are using JPA then you are in luck because jpasecurity project promises to resolve this limitation.

Rules are expressed in XML (I tested this so far) or Annotations (I had no luck with them so far). It allows granular access to CREATE, READ, UPDATE, DELETE operations based on roles and the current logged in user. It does that while wrapping all your JPA queries with the rules you specify. Basically you define access on let us say Client entity and every time Client entity appears in a JPA statement it wraps that statement adding the constraint. No need to say how powerful this is.

I used the trunk just because I was following up on some bugs that got corrected as I posted my questions. Probably you will get lucky and a stable 0.4.x release will be available by the time you decide to try it.

The examples here are based on a spring project using JPA + Hibernate + JTA + LDAP

Here are the main steps I followed:

Edit your pom.xml

<properties>
...
<jpasecurity.version>0.4.0-SNAPSHOT</jpasecurity.version>
...
</properties>
...
<dependency>
          <groupId>net.sf.jpasecurity</groupId>
          <artifactId>jpasecurity-spring</artifactId>
          <version>${jpasecurity.version}</version>
          <exclusions>
           <exclusion>
            <artifactId>geronimo-ejb_3.1_spec</artifactId>
            <groupId>org.apache.geronimo.specs</groupId>
           </exclusion>
           <exclusion>
            <artifactId>geronimo-jpa_2.0_spec</artifactId>
            <groupId>org.apache.geronimo.specs</groupId>
           </exclusion>
          </exclusions>
</dependency>
...

If you need to debug some jpasecurity issues it is always useful to increase log level for the package
```
log4j.logger.net.sf.jpasecurity=DEBUG
```

I tried to use rules from annotations but the feature is still not well supported. I ended up putting my rules in XML. So here some samples. One of them as you can see quite complex:

<security xmlns="http://jpasecurity.sf.net/xml/ns/security"
          xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
          xsi:schemaLocation="http://jpasecurity.sf.net/xml/ns/security
                              http://jpasecurity.sf.net/xml/ns/security/security_1_0.xsd">

  <persistence-unit name="nestorurquizaPersistenceUnit">
<access-rule>GRANT CREATE READ        ACCESS TO Client c</access-rule>
<access-rule>GRANT                    ACCESS TO Client c WHERE 'ROLE_ADMIN' IN (CURRENT_ROLES)</access-rule>
<access-rule>GRANT ACCESS TO Client c WHERE c.id IN (SELECT cs.client.id FROM ClientStaffing cs, ClientStatus cst, Employee e WHERE e.email=CURRENT_PRINCIPAL AND cs.employee=e AND cs.client=c AND cs.endDate IS NULL AND ( cst.name &lt;&gt; 'Closed' OR cst.name IS NULL) )</access-rule>
  </persistence-unit>
</security>

In persistence.xml for your container:

<!-- Comment the below if using jpasecurity -->
<!--    <provider>net.sf.jpasecurity.persistence.SecurePersistenceProvider</provider>-->

    <!-- Uncomment the below if not using jpasecurity -->
    <provider>org.hibernate.ejb.HibernatePersistence</provider>
...

    <properties>
            <!-- Comment the below if not using jpasecurity -->
<!--            <property name="net.sf.jpasecurity.persistence.provider" value="org.hibernate.ejb.HibernatePersistence" />-->
<!--            <property name="net.sf.jpasecurity.security.authentication.provider" value="com.nestorurquiza.security.JpasecuritySpringAuthenticationProvider"/>-->

Note the need for a custom SpringAuthenticationProvider. This is just a hook to guarantee that CURRENT_PRINCIPAL maps to the user email which is the username in LDAP.

package com.nestorurquiza.security;

import net.sf.jpasecurity.spring.authentication.SpringAuthenticationProvider;

import org.springframework.security.authentication.AnonymousAuthenticationToken;
import org.springframework.security.core.Authentication;
import org.springframework.security.core.context.SecurityContextHolder;
import org.springframework.security.core.userdetails.UserDetails;

public class JpasecuritySpringAuthenticationProvider extends SpringAuthenticationProvider{
    @Override
    public Object getPrincipal() {
        Authentication authentication = SecurityContextHolder.getContext().getAuthentication();
        if (authentication == null || (authentication instanceof AnonymousAuthenticationToken)) {
            return null;
        }
        UserDetails userDetails = (UserDetails) authentication.getPrincipal();
        return userDetails.getUsername();
    }
}

I had to create a Custom Converter so Spring Binding works for forms. There is a bug I reported to Spring on this regard but anyway here is the workaround:

package com.nestorurquiza.converter;

import net.sf.jpasecurity.SecureObject;

import org.springframework.core.convert.converter.Converter;

public class SecureObjectToStringConverter implements Converter {

    @Override
    public String convert(SecureObject source) {
        return (source != null ? source.toString() : null);
    }
    
}

Then we need a custom ConversionServiceFactoryBean

package com.nestorurquiza.converter;

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.core.convert.support.GenericConversionService;
import org.springframework.format.FormatterRegistry;
import org.springframework.format.support.FormattingConversionServiceFactoryBean;

/**
 * Not being used.
 * @author jia
 */
public class CustomConversionServiceFactoryBean extends FormattingConversionServiceFactoryBean {

    @Autowired
    private GenericConversionService genericConversionService;
    
    @Override
    protected void installFormatters(FormatterRegistry registry) {
        super.installFormatters(registry);
        //registry.addConverter(new BooleanToStringConverter());
        
        //Using org.springframework.format.support.FormattingConversionServiceFactoryBean from the xml declaration will not work
        //registry.addConverter(new SecureObjectToStringConverter());
        genericConversionService.addConverter(new SecureObjectToStringConverter());
    }
}

Then configure spring servlet with the necessary bean

<!-- Registering custom ConversionService --> 
    <bean id="conversionService" class="com.nestorurquiza.converter.CustomConversionServiceFactoryBean" />
    <mvc:annotation-driven conversion-service="conversionService" />
    <!-- The below will not work at least for Binding --> 
    <!-- <bean id="conversionService" class="org.springframework.format.support.FormattingConversionServiceFactoryBean"> 
        <property name="converters"> 
            <list> 
                <bean class="com.nestorurquiza.converter.SecureObjectToStringConverter"/> 
            </list> 
        </property> 
    </bean> 
    -->

Include the jpasecurity taglib for access rules in JSP

<%-- Comment the below if not using jpasecurity --%>
<%--@ taglib prefix="access" uri="http://jpasecurity.sf.net/access" --%>

Use rules as needed in JSP

<access:updating entity="client">
       <security:authorize url="/client/${client.id}/edit"><a href="<spring:url value="/client/${client.id}/edit?ctoken=${sessionScope.ctoken}"/>"><spring:message code="edit" /></a></security:authorize>
    </access:updating>

Typical response when security is violated:

java.lang.SecurityException: The current user is not permitted to update the specified object of type com.nestorurquiza.model.Client

Some Jpasecurity limitations so far:
- Does not accept CONCAT function so we must pass the percentages for the LIKE clauses within the parameters (which is best practice anyway)
- count(*) is not supported. Use the entity alias instead like "SELECT count(c) FROM Client c"
- LOWER and CONCAT functions are not supported but probably you can live without them and the less functions the best performance.

Existing JUnit tests will fail if you enable jpasecurity and do not use a user with valid roles in the current context. The way you correct them is presented below:

...
String adminEmail = "Admin.User@nestorurquiza.com";
injectCurrentUser(adminEmail, Roles.ROLE_ADMIN);
…
private void injectCurrentUser(String email, String role) {
        TestingAuthenticationToken token = new TestingAuthenticationToken(
                email, email, new GrantedAuthority[]{
                    new GrantedAuthorityImpl(role)});       
        SecurityContextHolder.getContext().setAuthentication(token);
}

Raw data and format in Jasper Reports

2011-06-19T08:42:00.000-07:00

Raw data let us say a Float or a BigDecimal can be formatted in Excel following some patterns in a way that it is clearer for printing. However the raw data remains intact in a way that formulas can be applied later on those raw numbers. In accounting and financing this is a simple yet important concept to understand: separation between data and format.

JasperReports JRXML format uses its own formatting which is different from Excel and so when you want to export to Excel directly from iReport for example two things are going to happen:

Probably you will have no formatting rule available and you will end up with an approximation of the real formatting you are after
Your raw data could disappear and you get in the Excel output just exacty what the formatting in JRXML is specifying

When my attention was brought to this problem by a member of my current team I saw a post with some solutions I did not like.

The reason why I did not like the solutions (besides the fact they are not addressing the two points I described above) is that they basically violate separation of concerns.

The solution for this problem is indeed addressing the problem from the right angle.

It allows to format anything for non excel output from iReport (jrxml). It then relies on mapping to translate the custom jrxml format to the proprietary to Excel syntax when that format is needed.

I only came to that post after some research and the more I research the more convince I am people get easily lost nowadays when searching for solutions on the web. This is of course a result of a non semantic web markup and consequently a poor search engine filtering. Still there is something that can be done if you stick to concepts and you refine your terms following them.

Here is my personal story when I researched this issue. This could probably help others. Remember do never get the first response you get from Google as the final response to your problem:

I found a similar issue within the JasperReports forums, not from Google. Then I asked if a solution was found
I realized using JRXlsAbstractExporter was apparently the way to go.
And so I finally got it.
As this is a solution addressable only from the JR Excel API Exporter, if you are really stuck with POI Exporter then what to do? I could not find anything but as I knew from simple concept this was a task to be done by the exporter and the exporter uses the POI API then I searched for the necessary formatting and I got into this post

I feel like posting how you research is as useful as posting a solution. A good researcher cannot give up. A path must be follow till the end which is a Proof Of Concept (POC). There is no other way.

If you are the architect you must do that job. If you are the CTO be sure you do it yourself or have an architect to delegate to. If you are a developer and you do this you are on your way to become a good architect and a good hands on CTO.