Wednesday, November 30, 2011

svn dump stderr output causing false positives

We were getting these annoying alerts every time the svn backup was running. Reason was 'svn dump' by default sends feedback like the below to the stderr:
* Dumped revision 0.
* Dumped revision 1.
* Dumped revision 2.
...

From the help command the reason and the workaround came clear:
# svnadmin dump --help
dump: usage: svnadmin dump REPOS_PATH [-r LOWER[:UPPER] [--incremental]]

Dump the contents of filesystem to stdout in a 'dumpfile'
portable format, sending feedback to stderr.  Dump revisions
LOWER rev through UPPER rev.  If no revisions are given, dump all
revision trees.  If only LOWER is given, dump that one revision tree.
If --incremental is passed, the first revision dumped will describe
only the paths changed in that revision; otherwise it will describe
every path present in the repository as of that revision.  (In either
case, the second and subsequent revisions, if any, describe only paths
changed in those revisions.)

Valid options:
  -r [--revision] ARG      : specify revision number ARG (or X:Y range)
  --incremental            : dump incrementally
  --deltas                 : use deltas in dump output
  -q [--quiet]             : no progress (only errors) to stderr

Just using the --quiet option does the trick:
svnadmin dump -q $MASTER_REPO >  $dump_file_path/dumpfile

Monday, November 28, 2011

Jasper Reports Excel Cross Sheet Formulas

A member of the Data team brought my attention to an iReports bug affecting Excel Output containing formulas which reference a following sheet. If the formula is let us say referring Sheet1 from Sheet2 there is no problem but if it refers Sheet2 from Sheet1 Jasper will fail to generate a correct XLS file. This is not the case when we try to generate an XLSX file. In that case it does work as expected.

When running with "Excel 2007 (XLSX) Preview" the cell does get what we expect:
=mySheet2!A1

However when we pick "XLS Preview" we get:
=#REF!A1

Here is the screen shot showing a successful rendering when using XLSX:
Here is the failure when using XLS:

This is clearly a problem related to different exporters. I still need to test this from Java with different exporter options but it would be ideal if a simple cross sheets formula like this could work from either XLS or XLSX files.

Below is the source code for this report. I have picked one jrxm posted in forums and I have added a couple of lines just to make sure I generate two different spreadsheets.

The output contains two sheets which reference each other through a simple formula that brings the content of the A1 cell from one sheet to the other in both directions. As stated at the beginning the formula in Sheet1 will fail when exporting to XLS but will succeed when exporting to XLSX. The formula in Sheet 2 will always succeed.

<?xml version="1.0" encoding="UTF-8"?>
<jasperReport xmlns="http://jasperreports.sourceforge.net/jasperreports" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://jasperreports.sourceforge.net/jasperreports http://jasperreports.sourceforge.net/xsd/jasperreport.xsd" name="formulaSample" pageWidth="595" pageHeight="842" whenNoDataType="AllSectionsNoDetail" columnWidth="535" leftMargin="0" rightMargin="0" topMargin="0" bottomMargin="0">
 <property name="net.sf.jasperreports.export.xls.detect.cell.type" value="true"/>
 <property name="net.sf.jasperreports.export.xls.one.page.per.sheet" value="true"/>
 <property name="net.sf.jasperreports.export.xls.sheet.names.all values" value="mySheet1/mySheet2"/>
 <property name="ireport.zoom" value="1.0"/>
 <property name="ireport.x" value="0"/>
 <property name="ireport.y" value="0"/>
 <import value="net.sf.jasperreports.engine.*"/>
 <import value="java.util.*"/>
 <import value="net.sf.jasperreports.engine.data.*"/>
 <title>
  <band height="262" splitType="Stretch">
   <textField>
    <reportElement key="textField-1" x="0" y="0" width="68" height="23"/>
    <textElement/>
    <textFieldExpression><![CDATA[new Integer(2)]]></textFieldExpression>
   </textField>
   <textField>
    <reportElement key="textField-2" x="0" y="23" width="68" height="23"/>
    <textElement/>
    <textFieldExpression><![CDATA[new Integer(1)]]></textFieldExpression>
   </textField>
   <textField>
    <reportElement key="textField-3" x="0" y="46" width="68" height="23">
     <propertyExpression name="net.sf.jasperreports.export.xls.formula"><![CDATA["mySheet2!A1"]]></propertyExpression>
    </reportElement>
    <textElement/>
    <textFieldExpression><![CDATA[]]></textFieldExpression>
   </textField>
   <break>
    <reportElement x="0" y="116" width="1" height="1"/>
   </break>
   <textField>
    <reportElement key="textField-3" x="0" y="163" width="68" height="23">
     <propertyExpression name="net.sf.jasperreports.export.xls.formula"><![CDATA["mySheet1!A1"]]></propertyExpression>
    </reportElement>
    <textElement/>
    <textFieldExpression><![CDATA[]]></textFieldExpression>
   </textField>
   <textField>
    <reportElement key="textField-2" x="0" y="140" width="68" height="23"/>
    <textElement/>
    <textFieldExpression><![CDATA[new Integer(9)]]></textFieldExpression>
   </textField>
   <textField>
    <reportElement key="textField-1" x="0" y="117" width="68" height="23"/>
    <textElement/>
    <textFieldExpression><![CDATA[new Integer(7)]]></textFieldExpression>
   </textField>
  </band>
 </title>
</jasperReport>

Note that this example contains a useful hint that allows to quickly come up with a test case that can be shared with the community. This is just using java.lang.* package to include hard coded values in text fields (that later become Excel cells) without the need to having a datasource defined (Use "Empty Datasource" to run this jrxml)

Last but not least we set the property "whenNoDataType="AllSectionsNoDetail"" because we are not using any datasource. If you are rendering the jrxml with no datasource from Java you will need it otherwise you will just get blank pages or corrupted XML files depending on the options you use.

Wednesday, November 23, 2011

The plugin 'org.apache.maven.plugins:maven-jetty-plugin' does not exist or no valid version could be found

This issue happens when the repository does reach the Mortbay release repo http://jetty.mortbay.org/maven2/release so to be safe I always add the below to the build/plugins node in the pom.xml.

<plugin>
        <groupId>org.mortbay.jetty</groupId>
        <artifactId>maven-jetty-plugin</artifactId>
        <version>6.0.1</version>
      </plugin>

Tuesday, November 22, 2011

SWFUpload plugin in Self Signed Certificate websites

When you are developing you will not buy an SSL certificate but instead you will use a self signed one. Adobe Flash is used to provide cool multiple file upload capabilities through the use of the SWFPlugin but it will fail if the Certificate is self signed.

This was affecting our QA team today and thanks to this post I figured a solution for the problem:

  1. In IE double click the "Certificate Error" icon
  2. Pick Install Certificate. Pick the defaults and finish

For some people this was not working still so I suggested to restart browsers and even Windows, you know that kind of stuff you need to do when suspicious right? At least one of my team members reported cleaning cookies resolved the problem for him (of course after following the above steps)

So SSL support based on self-signed certificates is buggy in flash and you can expect the same for SWFUpload. Look for SSL in the Known issues page for a similar statement.

Cannot determine realm for numeric host address

I could not connect today to one of our servers using password less public key ssh login. Using -vvv ssh option would show:
debug1: An invalid name was supplied 
Cannot determine realm for numeric host address 

So from the Red Hat target I inspected the secure log:
$ tail -f /var/log/secure
...
Nov 22 11:37:03 t2215179 sshd[23123]: Authentication tried for admin with correct key but not from a permitted host (host=tools, ip=xxx.xxx.xxx.xxx).
...

This was resolved by adding the server IP to /etc/hosts in the client
$ vi /etc/hosts
...
xxx.xxx.xxx.xxx tools
...

Friday, November 18, 2011

Android on the HP Touchpad

Update 20140610: The links for the original steps were not longer working so I followed http://liliputing.com/2014/06/use-touchpad-toolbox-install-android-erase-webos-hp-touchpad.html and I was able to get kitkat in the touchpad! The only thing that took me sometime was to find the right combination of files (gapps-kk-20140606-signed.zip, update-TWRP-jcs-dm-tenderloin-20140512.zip, cm-11-20140409-SNAPSHOT-jcsullins-tenderloin.zip). MAX OSX does not support MTP so you will need a program like http://www.android.com/filetransfer/ to transfer files to the android kitkat.

Update 20131002: I have been happy now for some time running Android on the HP Touchpad. Still running the old CyanigenMod7 which just crashed and reinstalled following the steps below.

Just follow instructions from cyanogenmod touchpad port team, Download section. Be sure you install everything including ClockworkMod.

Then install GoogleApps so you get Market application:
  1. Boot into Android mode
  2. Connect TouchPad to Computer
  3. Touch the notification icon so you are able to turn on USB
  4. From the computer drop the GoogleApps zip file (for example gapps-gb-20110828-signed.zip) into the TouchPad
  5. Reboot the TouchPad (Power + Home buttons for 15-20sec will do it in case just Power does not give you the reboot option)
  6. In the boot menu quickly move using the volume keys to option "ClockworkMod". Press Home button to accept
  7. choose "install zip from sdcard", then "choose zip from sdcard" and pick the GoogleApps zip file.
  8. Reboot

Thursday, November 17, 2011

Hide link or anchor after click with jquery

This is a common need when you provide what I call a naked user interface meaning, well I am waiting for my front end engineer and I provide just the bare minimums so the whole functionality can be used. But this is also a need for complicated user interfaces where sometimes a link is just supposed to trigger something, you want to stay in the page and you do not want the link to be available again (avoiding double submission for example).

So here is the JQuery code that will keep the link text but will change it by just a spanned text:
    //
    // Hide link after click
    //
    $('.hideAfterClickLink').click(function(e) {
     $(this).replaceWith('' + $(this).text() + '')
    });

Friday, November 11, 2011

Tomcat Servlet Context initialized twice or multiple times

Tomcat Servlet Context initialized twice or multiple times
I have seen this issue both in JBoss and Tomcat and in both cases this is about a miss configuration. It is worth to be said this issue affects the agility of the developer because extra time needs to be spent just waiting for tomcat to finish its initialization. This is also an issue that hits production systems many of which have very slow startup just because of miss configurations.

The first thing that you need to do is to include a log INFO level message when the servlet context is initialized like below:
public class ApplicationServletContextListener implements
        ServletContextListener {
...
    Logger log = LoggerFactory.getLogger(ApplicationServletContextListener.class.getName());
    @Override
    public void contextInitialized(ServletContextEvent event) {
       ...
       log.info("Servlet Context initialized");
...

Now whenever you restart your server you should get only one line:
2011-11-11 16:29:38,027 INFO [com.nestorurquiza.utils.web.ApplicationServletContextListener] -   Servlet Context initialized

If you get more than one look for errors in server.xml. Just to show an example of something that actually hit me Today this was the configuration in one of my servers:
...
<Host name="bhubdev.nestorurquiza.com"">
  <Context path="" docBase="/opt/tomcat/webapps/nestorurquiza-app"/>
...

Just changing for the below fixed the issue:
...
<Host name="bhubdev.nestorurquiza.com" appBase="webapps/nestorurquiza-app"/>
...

However it also screwed the root context you get with the previous configuration. If you still need the application to respond to the root context you will need to deployed as ROOT.war as explained in the official documentation.

Thursday, November 10, 2011

Excel Jasper Report with default Zoom using a Macro

I could not find a way to specify a default zoom for an Excel Report generated from iReport/Jasper Reports so I had to apply the Macro hack I recently documented.

Here is the Visual Basic for Applications code that make this possible. It zoom out the whole workbook to 75%:
Private Sub Workbook_Open()
    If Application.Sheets.Count > 1 Then
        Worksheets("Macro").Visible = False
    End If
    Application.ScreenUpdating = False

    For Each ws In Worksheets
        ws.Activate
        ActiveWindow.Zoom = 75
    Next

    Application.ScreenUpdating = True
End Sub

Wednesday, November 09, 2011

Excel Jasper Reports With Macros

I am a big fan of separation of concerns. Did I say this before? :-)

When it comes to View technologies there is a miss conception to think about View part as "passive" when in reality we have seen "active" Views for years with Javascript in the center of our HTML pages. Well, the same applies for PDF and Excel.

So I don't understand why providing Macro support in a Jasper Report should be considered wrong and since especially in the Financial Industry Excel is a core technology where Macros are simply needed I decided to present here a solution to provide Excel Jasper Reports containing Macros.

Some times the macro is needed by business while other times the macro can save time with formatting features that are unavailable in the iReport/Jasper last version. For the second case I do recommend to fill a feature request because it is a hack to force a formatting capability using a Macro and your users might not be OK with code running in their Excel which imposes a big security concern. But for sure this hack allows to move while keeping separation of concerns, in other words liberating Java developers from the need to touch code, recompile and redeploy (The Controller) just to fulfill a need in formatting that is not available where it should be available (The View).


Create an Excel Workbook containing the Macro to be applied to the Excel Jasper Report

  1. Create a brand new Excel Workbook
  2. Rename Sheet1 as 'Macro' and delete the rest of them
  3. Select from the Menu: Tools | Macro | Visual Basic Editor
  4. Right click on "This Workbook" and select "View Code". Paste the below in the code pane:
    Private Sub Workbook_Open()
        If Application.Sheets.Count > 1 Then
            Worksheets("Macro").Visible = False
        End If
        MsgBox "Hello Excel Jasper Report!"
    End Sub
    
  5. Save the Workbook as "myTemplate.xls", close and open. A pop up should come up automatically. This code as you already noticed has a hack which basically allows to hide the unique sheet we are forced to leave in the workbook because of an Excel imposed limitation.
  6. Now we are ready to use myTemplate.xls as the template out of which our Jasper Report will be built.

Use the Excel Template from jasper API

First, you will need to extend the exporter. I am showing here how to do it with JRXlsExporter which uses POI API but the same idea should be applicable to other exporters.
package com.nestorurquiza.utils;

import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;

import net.sf.jasperreports.engine.export.JRXlsExporter;

import org.apache.poi.hssf.usermodel.HSSFWorkbook;
import org.apache.poi.hssf.util.HSSFColor;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

public class CustomJRXlsExporter extends JRXlsExporter {
    private static final Logger log = LoggerFactory.getLogger(CustomJRXlsExporter.class);
    
    private InputStream is;
    
    public InputStream getIs() {
        return is;
    }

    public void setIs(InputStream is) {
        this.is = is;
    }

    @Override
    protected void openWorkbook(OutputStream os) {
        if(is != null) {
            try {
                workbook = new HSSFWorkbook(is);
                emptyCellStyle = workbook.createCellStyle();
                emptyCellStyle.setFillForegroundColor((new HSSFColor.WHITE()).getIndex());
                emptyCellStyle.setFillPattern(backgroundMode);
                dataFormat = workbook.createDataFormat();
                createHelper = workbook.getCreationHelper();
                return;
            } catch (IOException e) {
                log.error("Creating a new Workbook when I was supposed to use an existing one.", e);
            }
        }
        super.openWorkbook(os);
    }
}

Second just use the Custom Exporter passing the input stream to your Excel Template file.
...
CustomJRXlsExporter exporter = new CustomJRXlsExporter();
String xlsTemplate = ctx.getParameter("xlsTemplate");
if( xlsTemplate != null ) {
    String xlsTemplateFilePath = getReportsPath() + "/xlsTemplate/" + ctx.getParameter("xlsTemplate");
    exporter.setIs(new FileInputStream(xlsTemplateFilePath));
}
...

You end up with your report and a message box popping up from the macro code "Hello Excel Jasper Report!".

A more realistic example? I will probably start documenting some of them in the near future so search this blog for jasper excel macro.

This technique is simple but powerful. I hope it is integrated somehow in the different exporters as it was proposed 5 years ago.

As I have said there I think iReport should be able to use a hint so the front end report designer can get the formatting they want using an Excel Template without the need to wait for an external API implementation.

Monday, November 07, 2011

error running shared postrotate script for /var/log/mysql

I am sure the error below can be related to many different circumstances:
/etc/cron.daily/logrotate:
error: error running shared postrotate script for '/var/log/mysql.log /var/log/mysql/mysql.log /var/log/mysql/mysql-slow.log '
run-parts: /etc/cron.daily/logrotate exited with return code 1

In my case after cloning an ESX Ubuntu VM where MySQL was installed into some other servers where we did not need mysql and after uninstalling mysql I could still see mysqld running.

Just running the below command was enough to stop the alert:
$ sudo apt-get autoremove
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages will be REMOVED:
  mysql-server-5.1 mysql-server-core-5.1
...

Sunday, November 06, 2011

User impersonation with Spring and LDAP

Impersonating a user (also known as "switch a user" or "do as user") in a web application is an important feature for doing Quality Assurance (QA) or any Manual User Acceptance Test (UAT).

With Spring you can achieve it using the SwitchUserFilter. If you are using LDAP here are the relevant pieces.

...
<beans:bean id="ldapUserDetailsService" class="org.springframework.security.ldap.userdetails.LdapUserDetailsService">
  <beans:constructor-arg><beans:ref bean="ldapUserSearch"/></beans:constructor-arg>
  <beans:constructor-arg><beans:ref bean="ldapAuthoritiesPopulator"/></beans:constructor-arg>
  <beans:property name="userDetailsMapper" ref="customUserDetailsContextMapper" />
</beans:bean>
...
<beans:bean id="ldapUserSearch" class="org.springframework.security.ldap.search.FilterBasedLdapUserSearch">
    <beans:constructor-arg type="String"><beans:value>ou=people,o=nestorurquiza</beans:value></beans:constructor-arg>
    <beans:constructor-arg type="String"><beans:value>mail={0}</beans:value></beans:constructor-arg>
    <beans:constructor-arg><beans:ref bean="ldapContextSource"/></beans:constructor-arg>
</beans:bean>
 ...                            
<beans:bean id="switchUserProcessingFilter" class="org.springframework.security.web.authentication.switchuser.SwitchUserFilter">
        <beans:property name="userDetailsService" ref="ldapUserDetailsService" />
        <beans:property name="switchUserUrl" value="/admin/switchUser" />
        <beans:property name="exitUserUrl" value="/admin/switchUserExit" />
        <beans:property name="targetUrl" value="/" />
</beans:bean>
...
<http auto-config="true" use-expressions="true" access-decision-manager-ref="accessDecisionManager" disable-url-rewriting="true">
    ...
 <custom-filter  after="FILTER_SECURITY_INTERCEPTOR" ref="switchUserProcessingFilter" />
    ...
    <intercept-url pattern="/admin/**" access="hasAnyRole('ROLE_ADMIN','ROLE_PREVIOUS_ADMINISTRATOR')" />
... 
<beans:bean id="ldapContextSource" class="org.springframework.ldap.core.support.LdapContextSource">

    <beans:property name="url" value="${ldap.url}" />
    <beans:property name="userDn" value="${ldap.userDn}" />
    <beans:property name="password" value="${ldap.password}" />
</beans:bean>
...
<beans:bean id="ldapAuthProvider"
        class="org.springframework.security.ldap.authentication.LdapAuthenticationProvider">
  <beans:constructor-arg>
    <beans:bean class="org.springframework.security.ldap.authentication.BindAuthenticator">
      <beans:constructor-arg ref="ldapContextSource"/>
      <beans:property name="userDnPatterns">
        <beans:list><beans:value>mail={0},ou=people,o=nestorurquiza</beans:value></beans:list>
      </beans:property>
    </beans:bean>
  </beans:constructor-arg>
  <beans:constructor-arg ref="ldapAuthoritiesPopulator"/>
</beans:bean>
...
<beans:bean id="ldapAuthoritiesPopulator" class="org.springframework.security.ldap.userdetails.DefaultLdapAuthoritiesPopulator">
      <beans:constructor-arg ref="ldapContextSource"/>
      <beans:constructor-arg value="ou=groups,o=nestorurquiza"/>
      <beans:property name="groupSearchFilter" value="uniquemember={0}" />
</beans:bean>
...    
<beans:bean id="ldapTemplate" class="org.springframework.ldap.core.LdapTemplate">
    <beans:constructor-arg ref="ldapContextSource" />
</beans:bean>
...
I18n messages:
switchUser=Impersonate
switchUserExit=Be yourself
A link to impersonate in the users listing page should be available only for admins:
<security:authorize access="hasRole('ROLE_ADMIN')">
                        | <a href="<spring:url value="/admin/switchUser?j_username=${employee.email}" htmlEscape="true" />">
                            <spring:message code="switchUser"/></a>
                </security:authorize>
A link to switch back to admin user should be available only if the user is being impersonated:
<security:authorize access="hasRole('ROLE_PREVIOUS_ADMINISTRATOR')">
                        | <a href="<spring:url value="/admin/switchUserExit" htmlEscape="true" />">
                            <spring:message code="switchUserExit"/></a>
                    </security:authorize>
The way you use it is hitting the below URLs to switch the user and come back to the original admin user context:
http://localhost:8080/nestorurquiza-app/admin/switchUser?j_username=nurquiza@nestorurquiza.com
http://localhost:8080/nestorurquiza-app/admin/switchUserExit

Followers