Wednesday, February 29, 2012

Patching Talend Components

We were having problems today with the Talend tFileInputJSON component which basically sends to the stderr an error message when it cannot find a key specified in a JSONPath, The error for which there is at least a reported talend bug looks like:
Can't find any data with JSONPath ...
To solve this issue you can edit yourself a simple file in my case /Users/nestor/Downloads/TOS-All-r67267-V4.2.3//plugins/org.talend.designer.components.localprovider_4.2.3.r67267/components/tFileInputJSON/tFileInputJSON_begin.javajet. Just comment out the System.out.println():
  //System.err.println("Can't find any data with JSONPath " + query);
  //add null to take a place in List(buffer)

Then you will need to restart Talend GUI.

IMO a better patch would be of course to just ignore the null if the schema is defined as "Allow Null".

Saturday, February 25, 2012

start: Job is already running: mysql

This error happens in some servers because one of two reasons as far as I can tell. You either are still using old commands like:
$ sudo /etc/init.d/mysql start
You are actually using the new command but you are not running it as root:
$ sudo service mysql start

Friday, February 24, 2012

Agile content deployment with Spring

Perhaps because I am not a native English speaker I have been always so concern about localized content (Internationalization). I have seen many applications that are not thought for other than English audiences from the beginning and later down the road have a painful path to get translated to other languages. With Spring all you need to do is to keep your messages in properties files and then configure a ReloadableResourceBundleMessageSource bean that will take care of loading the files following certain schedule. Below is an example that will load the properties every minute.
    <bean id="messageSource"
        <!--  <property name="basename" value="classpath:messages" />  -->
        <property name="basenames">
        <property name="cacheSeconds">
        <property name="fallbackToSystemLocale" value="false" />
Of course you use then taglibs for your content. You never hardcode a string in your application. That will allow you to ship to operations that concern.
 <spring:message code="welcome.loggedIn" />
Your properties file will need to live for example in a file like WEB-INF/i18n/ for the default language or WEB-INF/i18n/ for German in case that is not the default for you. The content is simple key values as we all know:
welcome.loggedIn=You are logged in as
A word of advice about Unicode: It can get complicated. For one java Property does not accept unicode. You can find a workaround for that issue as well as several hints to effectively handle Unicode in your app from a previous post I made three years ago.

Now your deployment script just needs to be able to drop the file from your repository into the exploded webapp and your new content will show up in the website. Easy, simple, agile.

Wednesday, February 15, 2012

Jasper export to Excel Integer Cell is blank

Excel was rendering a blank column that otherwise iReport was showing correctly with numbers.

An inspection of "Format Cells" option would show two semicolons ";;" which means "do not show". A click on the cell was revealing the values though.

In this case I solved the issue setting the pattern attribute of the textField node like below. Note that the code generates an auto-increment number BTW:

<textField pattern="###0;-###0" isBlankWhenNull="false">
  <reportElement mode="Transparent" x="0" y="0" width="34" height="13" forecolor="#000000" backcolor="#FFFFFF"/>
  <textElement textAlignment="Left" verticalAlignment="Top" rotation="None" markup="none">
  <font fontName="DejaVu Sans" isBold="false" isItalic="false" isUnderline="false" isStrikeThrough="false" pdfEncoding="Cp1252" isPdfEmbedded="false"/>
  <paragraph lineSpacing="Single"/>

ETL Encrypt and Decrypt from Talend

Let us say that you already developed your Java application using Jasypt to encrypt sensitive (private) data like I posted before.

Now you are trying to use the data as part of your ETL process in Talend.

This is not any different than using any other library, so here are the steps:

Use a tLibraryLoad to import a jar into the project
Configure the tLibraryLoad
Then declare the import in tJavaFlex

Now use the library to encrypt/decrypt

If you run the job you should see it encrypting and decrypting

Simpler software is better software

There is no silver bullet in Software Development but simplicity must be what the software developer strives for.

If the software engineer spends more time thinking about the simplest possible way to provide a solution then a great product will result.

 The word "Engineer" comes from the latin word "ingenium" which means "cleverness".

As an Electronics Engineer graduated from a poor Country I had to learn how to develop stuff out of the materials that were available and that I believe pushed me to think as a profesional about cost (every time).

I strongly believe that Software Development cannot be effective with a non Engineering mindset. The waste of resources causes not only a complicated system that is difficult to build but in the long run too expensive to be maintained.

Refactoring is a needed tool and it should be used every time there is place to make the existing software simpler. It should be used to eliminate garbage and not to add more.

Simpler is better, and that statement should drive our efforts as Software Developers.

“Perfection is achieved, not when there is nothing more to add, but when there is nothing left to take away.” ― Antoine de Saint-ExupĂ©ry

Tuesday, February 14, 2012

Running Advent Geneva RSL remotely through runrep

We have used for a while Advent Geneva SOAP Services to consume RSL reports as I posted before.

We noticed a problem with some reports running fast from runrep (a command prompt interactive tool supplied in the Advent Geneva distribution) but running really slow from SOAP so I decided to provide a remote tool to access runrep.

This is just Plain Old Bash* script that basically runs the command remotely using our old friend ssh.

Using this technique you should be able to run remote commands synchronously or asynchronously which comes handy for processes taking long time (like some long period accounting reports)

I have posted the script here. In order to run it you have to authorize your the client public keys in the geneva host. If you have no key then generate one:
if [ ! -f ~/.ssh/ ]; then ssh-keygen -t rsa; fi
You can then push the key to the geneva server using a handy command called ssh-copy-id:
ssh-copy-id -i ~/.ssh/
Here are some examples as to how to use it:

Use ASYNC_RUN to run the report and get a remote file in the server. Note the line starting with "$ 2012/02/14:16:50:09". The prompt means the command was started but in the background. As you can see just before the prompt the client script says it "Finished in 1 seconds" so the client program can now continue while the report is running in the server.
$ ./ geneva ASYNC_RUN my_report.xml myuser mypass my_report.rsl "-p MYPORTFOLIO -at Dynamic -ps 01/31/2012 -pe 01/31/2012 --SeparateLegalEntities 1 --FundLegalEntity \\\"MYPORTFOLIO MF\\\" -f xml"
Tue Feb 14 16:49:05 EST 2012
Finished in 1 seconds
$ 2012/02/14:16:50:09  INFO: runrep - Reading in RSL files
2012/02/14:16:50:11  INFO: runrep - Read in all RSL files
2012/02/14:16:50:11  INFO: runrep - Reading in GLM files
2012/02/14:16:50:11  INFO: runrep - Read in all GLM files
2012/02/14:16:50:11  INFO: Performing a(n) Dynamic accounting run for portfolio MYPORTFOLIO (MYPORTFOLIO MF) using a lockdown.
2012/02/14:16:50:11  INFO: ARM:00202: Generating new BIS, as either GIVEN or KBMS has changed.
2012/02/14:16:50:11  INFO: Using LockdownPool MYPORTFOLIO for Portfolio MYPORTFOLIO.
2012/02/14:16:50:11  INFO:       Loading Summary Positions from Fixed LockDown 12-31-2011.
runrep: Report run successful.

OK. Quitting.
Use ASYNC mode to move the file to the local drive
$ ./ geneva ASYNC my_report.xml myuser mypass my_report.rsl "-p MYPORTFOLIO -at Dynamic -ps 01/31/2012 -pe 01/31/2012 --SeparateLegalEntities 1 --FundLegalEntity \\\"MYPORTFOLIO MF\\\" -f xml"
ASYNC mode
receiving file list ... done

sent 42 bytes  received 465311 bytes  103411.78 bytes/sec
total size is 22068250  speedup is 47.42
Tue Feb 14 16:50:34 EST 2012
Finished in 4 seconds
Use SYNC to wait for the execution of the report and get the resulting file locally
$ ./ geneva SYNC my_report.xml myuser mypass my_report.rsl "-p MYPORTFOLIO -at Dynamic -ps 01/31/2012 -pe 01/31/2012 --SeparateLegalEntities 1 --FundLegalEntity \\\"MYPORTFOLIO MF\\\" -f xml"
SYNC mode
2012/02/15:03:29:38  INFO: runrep - Reading in RSL files
2012/02/15:03:29:40  INFO: runrep - Read in all RSL files
2012/02/15:03:29:40  INFO: runrep - Reading in GLM files
2012/02/15:03:29:40  INFO: runrep - Read in all GLM files
2012/02/15:03:29:40  INFO: Performing a(n) Dynamic accounting run for portfolio MYPORTFOLIO (MYPORTFOLIO MF) using a lockdown.
2012/02/15:03:29:40  INFO: ARM:00202: Generating new BIS, as either GIVEN or KBMS has changed.
2012/02/15:03:29:40  INFO: Using LockdownPool MYPORTFOLIO for Portfolio MYPORTFOLIO.
2012/02/15:03:29:40  INFO:       loading Summary Positions from Fixed LockDown 12-31-2011.
runrep: Report run successful.

OK. Quitting.
Connection to closed.
receiving file list ... done

sent 42 bytes  received 465311 bytes  186141.20 bytes/sec
total size is 22068250  speedup is 47.42
Wed Feb 15 03:29:39 EST 2012
Finished in 68 seconds
Test that indeed the resulting report is being moved to the client machine
$ ls -lh /tmp/my_report.xml
-rw-r--r--  1 nestor  wheel    21M Feb 14  2012 /tmp/my_report.xml
$ head -10 /tmp/my_report.xml

MYPORTFOLIO FUND L.P. nestorurquiza MASTER POSITIONS FILE 'January 31, 2012
While running all this from command prompt you will lose your prompt echo. Just run the below to recover it:
$ stty sane
It will make probably sense to implement a callback mode (ASYNC_CALLBACK) which basically will run the remote command asynchronously but on finalization it runs a callback service. That of course is left to the specific implementation.

Other useful bash geneva scripts

Here is another script that can be used to delete trades for a period of time for a give portfolio. It depends on an existing 6.2 RSL called newtransmaker.rsl which by default is missing support for deletion. The below diff should speak for itself so you can get the new version of the report available.
$ diff  /usr/advent/geneva-6.2.0/share/rslspecs/newtransmaker.rsl /usr/advent/geneva-6.2.0/share/rslspecs/newtransmaker_rev.rsl 
<  DATE KnowledgeDate = EndToday, STRING ManagementFirm, STRING  Uptransmaker = "No", STRING IDToUse ="ID", STRING StyleName = "d",
>  DATE KnowledgeDate = EndToday, STRING ManagementFirm, STRING  Uptransmaker = "No", STRING Deletetransmaker = "No", STRING IDToUse ="ID", STRING StyleName = "d",
<  DECODE(Lower(:Uptransmaker), "yes", "Update", "New") Action, //2
>  DECODE(Lower(:Uptransmaker), "yes", "Update", 
>  DECODE (Lower(:Deletetransmaker), "yes", "Delete",
>  "New")) Action, //2
>     if (Lower(:Deletetransmaker) == "yes") { :__reportName = "Delete Transaction Loader File"; }
You would the script as:
./ geneva anyRandomAndTemporaryFile.bcp genevaLoaderUser genevaLoaderPassword \\\"portfolioName\\\" 1/11/2012 1/11/2012
* I think I have coined POB for the first time BTW but of course many others have talked about Plain Old Bash before so I should not take credit for just an abbreviation. In fact in many cases like in the example I have brought today you have to be aware of other shells like the default CShell in a Solaris box. So it makes sense to actually talk about Plain Old Shell scripting or POS. I get exited when I see increased productivity just by using "old" concepts. It reminds me of what Plain Old Java Objects or POJO concepts meant to me by the time they showed up on the net. Simple is better.

Sunday, February 12, 2012

Recovering from unexpected JVM OutOfMemory with -XX:OnOutOfMemoryError

You should never have an OutOfMemoryError in your project but usually problems happen in production and it takes a while to solve them. There is where the -XX:OnOutOfMemoryError JVM Option comes handy. It just works supplying a script/command you want to run in the case the JVM reports a java.lang.OnOutOfMemoryError. Here is how to provide the flag, how to build the script that will put Tomcat down and how to test it.
  1. Generate a scaffold war project. I named it oom-app (The OutOfMemory App):
    mvn archetype:generate -DarchetypeGroupId=org.apache.maven.archetypes -DarchetypeArtifactId=maven-archetype-webapp -DgroupId=com.nestorurquiza.test -DartifactId=oom-app -Dversion=1.0.0-SNAPSHOT -Dpackage=com.nestorurquiza.test
  2. Edit web.xml
    $ cd oom-app/
    $ vi src/main/webapp/WEB-INF/web.xml 
      Archetype Created Web Application
  3. Create the listener that will cause the OutOfmemory error:
    $ mkdir -p src/main/java/com/nestorurquiza/test
    $ vi src/main/java/com/nestorurquiza/test/
    package com.nestorurquiza.test;
    import javax.servlet.ServletContextEvent;
    import javax.servlet.ServletContextListener;
    public class OomListener implements
            ServletContextListener {
        public void contextInitialized(ServletContextEvent event) {
            System.out.println("******************Originating an OutOfMemoryError Exception*************************");
      StringBuilder sb = new StringBuilder();
      while(true) {
        public void contextDestroyed(ServletContextEvent event) {
            System.out.println("Will I ever run?");
  4. Configure the JVM to run a script once the OutOfMemoryerror is gotten. For tomcat you do that in
    $ vi /opt/tomcat/bin/
    JAVA_OPTS="$JAVA_OPTS -XX:OnOutOfMemoryError=/usr/sbin/killTomcat"
  5. Make sure you have the script ready. Note we are just killing tomcat. In production you are supposed to automatically restart any service when it dies. There are multiple solutions for that but allow me to recommend "Monit" for Unix/Linux servers.
    $ sudo vi /usr/sbin/killTomcat
    ps -ef|grep tomcat|grep -v grep|awk '{print $2}'|xargs kill -9
    $ sudo chmod +x /usr/sbin/killTomcat
  6. Restart tomcat.
  7. Build and deploy (Using a simple tomcat webapps deployment):
    $ mvn clean install
    $ cp target/oom-app.war /opt/tomcat/webapps/
  8. Once the war is deployed you will get an output like:
    ******************Originating an OutOfMemoryError Exception*************************
    java.lang.OutOfMemoryError: Java heap space
    # java.lang.OutOfMemoryError: Java heap space
    # -XX:OnOutOfMemoryError="/usr/sbin/killTomcat"
    #   Executing "/usr/sbin/killTomcat"...

Saturday, February 11, 2012

Xalan install recipe: Perform XSLT transformations from command line

Doing ad-hoc XSLT transformations is a quick and agile way to test your XSL output. Every time I have to show a quick transformation to a developer I have to first download xalan and dependencies locally, type a long java command, build a script so the developer is abstracted from java command line stuff like having to specify each jar file etc.

I keep on finding myself writing Plain Old Bash (POB) recipes for this kind of stuff (automate the installation of tools)

Here is a recipe to get xalan installed and ready to be used just calling a command and passing the XML and XSL from the command line:

# 1. Copy this recipe to a directory where you wish to install xalan. 
# 2. Run it and it will install XALAN_JAVA executable inside INSTAL_DIR
# @author: Nestor Urquiza
# @date: 20110211

echo "--- Install Xalan XSLT Standalone processor ---"
set -e

mkdir -p $INSTAL_DIR
curl -O
curl -O
curl -O
echo "#!/bin/bash" > $XALAN_JAVA
echo "set -e" >> $XALAN_JAVA
echo "USAGE=\"Usage: $XALAN_JAVA <path to xml> <path to xsl>\"" >> $XALAN_JAVA
echo "if [ \$# -ne \"2\" ]; then echo \$USAGE; exit 1; fi" >> $XALAN_JAVA
echo "CP=\"`find \".\" -name '*.jar' | xargs echo | tr ' ' ':'`\"" >> $XALAN_JAVA
echo "export CLASSPATH" >> $XALAN_JAVA
echo "java org.apache.xalan.xslt.Process -IN \$1 -XSL \$2" >> $XALAN_JAVA
chmod +x $XALAN_JAVA
echo "$XALAN_JAVA was installed in $INSTAL_DIR directory. Run this to start using xalan: cd $INSTAL_DIR; ./$XALAN_JAVA"
It should be straightforward all you need to do to start using xalan is to navigate to the installation folder and run your transformation for example:
$ cd xalan-processor
$ ./xalan ~/Downloads/portfolio-struct.xml ~/Downloads/portfolio-struct.xsl

Not sure if I will need to change this script down the road but most likely latest version should be available from my svn repo.

Friday, February 10, 2012

Javadocs including UML Class Diagrams with Maven and Jenkins

Project documentation is important but it is difficult to maintain it. I still have to see a team where the project documentation is maintained accurate. Did I mention that before?

It makes sense then to perform some kind of reverse engineering every so often. Javadocs (in Java projects of course) and Class diagrams can help. The first is difficult to ensure gets updated but the latter is the real implementation and so it makes sense to have it at hand when needed

A Continuous Integration server is already building your project so it makes sense for it to build the documentation as well.

  1. Install graphviz. Ubuntu for example:
    $ apt-get install graphviz
    $ brew install graphviz
  2. Add the below to your maven pom.xml
                  <additionalparam>-all -constructors</additionalparam>
  3. Run the below to generate the local site
    $ mvn site
  4. Navigate the javadocs from target/site/apidocs/index.html and confirm a class shows up like below meaning a UML class diagram is added to the javadoc page
  5. Now of course you should create a project in your CI (for example Jenkins) so the javadocs get created everytime the project gets built. You better run this as a separated project just because it does take time and resources you do not want to spent when building the real project. In Jenkins you would access the site from a url like: http://jenkinsserver/myproject-documentation/site

Monday, February 06, 2012

Jasper Reports Release and Deployment

SDLC requires processes that are agile for development and ideally agile as well for releasing and deploying.

There is no silver bullet though so a strict release and deployment cycle will not hurt. Ultimately if you achieve extreme agility in these areas it should be completely automated but hey you must start from something: Outlining what a safe release and deployment is.

A project might consist of different packages and technologies which at the same time have different release/deployment cycles.

Here is what I mean when it comes to Jasper Reports.

Respecting separation of concerns there is a section of the team that is responsible for developing Jasper Reports. They are agile as they can develop a new report or change an existing and drop the changes in the server, they can see the end result of the work the have done. Even others can see the result as well of course.

However the fact that the code there in that environment you probably called test/integration is not ready for production. Why? It has not been released.

A release carries a version number and dependencies on other released artifacts. It is common to have sub-reports, images and in general dependencies all across the project.

In those cases where you have thousands of reports you will need to start thinking about how to define dependencies but I will not get that deep this time. Most of the projects out there could be releasing the whole Jasper Code with a simple bash command.

How can we release Jasper then?
  1. Compile jrxml files into jasper
  2. Find current version number for the to-be-released artifact from to the artifacts repository (Jenkins for example)
  3. Pack all resources in a zip file using the next version number
  4. Upload the zip versioned file to the artifacts repository

Of course all this must be scripted. Here is a proposal on how to address the Jasper Compilation. This ant script accepts a folder to look for jasper files and a destination zip file path where to store the packed folder.

 Jasper Reports Builder

Now you could call this script to have a versioned zip file that you can push into an artifact repository:
$ ant -f /opt/jasper-reports-build/jasper-reports-builder.xml clean compile -Dsrcdir=/Users/nestor/jasperreports -Ddestfile=/Users/nestor/

If you want to try this yourself here is a Plain Old Bash (POB) recipe to properly install the above script:
echo "--- Setting up the Jasper Reports Build script ---"

set -e

mkdir -p /opt/jasper-reports-build/lib
cd /opt/jasper-reports-build/lib
curl -O
curl -O
curl -O
curl -O
curl -O
curl -O
cd ..
curl -O