Friday, March 25, 2011

Old gSOAP web services from Java

I spent a considerable amount of time trying to get an old web service (gSOAP/2.6) consumed from Java. This is the Web Services exposed by an old version of Advent Geneva.

Dynamic Axis2

While Axis2 using axiom seemed to do the trick it was failing to parse some error results.

Static JAX-WS stubs

Someone in the team proposed to go with stubs and as that service does not changes every so often I agreed to give it a try. After some time spent on the task it came back to my plate as JAX-WS stubs were not working as expected. The stubs were generated only after some edition of the WSDL but I could not get the responses from any available object even though TcpMon showed correct request and responses going over the wire. As there were warnings from the stubs generation I suspected there was further changes to be made to the wsdl. For example one of the stubs was returning empty string while from TcpMon a value could be seen.
GenevaService service = new GenevaService( wsdlLocation, new QName(targetNamespace, serviceName));
GenevaServicePortType port = service.getGenevaServicePort();
ObjectFactory objectFactory = new ObjectFactory();   , "genevaServer", "user", "password", ""));

Apache CXF

I tried Apache CXF but I got an error basically saying "Forget it, this is too old for me":
WSDLToJava Error: Rpc/encoded wsdls are not supported with CXF

Dynamic JAX-WS

Advent Geneva uses gSOAP to expose services that are used from clients to pull report data and so there are no complex objects being returned, but more than anything tabular data. That gives the possibility to work without deserializing. While that approach as stated before was failing for some error responses for dynamic Axis2 approach (using Axiom) the problem got corrected when the service client was migrated to use dynamic JAX-WS. Below is the relevant code for the method failing before which just sends credentials and returns a session id:
            Service service = GenevaService.create(wsdlLocation, new QName(targetNamespace, serviceName));
            Dispatch dispatch = service.createDispatch(
                    new QName(targetNamespace, portName), SOAPMessage.class, GenevaService.Mode.MESSAGE);
            MessageFactory factory = MessageFactory.newInstance();
            SOAPMessage soapRequest = factory.createMessage();
            SOAPEnvelope envelope = soapRequest.getSOAPPart()
            SOAPElement soapElement = soapRequest.getSOAPBody().addBodyElement(
                envelope.createName("StartCallableRunrep", "gen", targetNamespace));
            SOAPMessage soapResponse = dispatch.invoke(soapRequest);
            Element startCallableRunrepResponse = (Element) soapResponse.getSOAPBody().getFirstChild();
            NodeList nodeList = startCallableRunrepResponse.getChildNodes();
            String runrepSessionId = null;
            for( int i= 0; i < nodeList.getLength(); i++ ) {
                Node node = nodeList.item(i);
                if( "runrepSessionId".equals(node.getLocalName()) ) {
                    runrepSessionId = node.getTextContent();


Once again I corroborated that there is no one technology better than other in terms of building web service clients. The specific needs of a project can really make you spend hundreds of hours if you want to stick to Axis2 or any other preferred API. It is wise to try from SOAPUI which internally uses apache http commons directly when consuming web services. If the web service works from there you still can have surprises. You can try stubs, dynamic services and DOM, SAX, XPATH or dynamic services and deserialization with any binding technology.

I personally prefer dynamic clients to be honest, they adapt to changes and while they can demand more low level plumbing they almost 100% of the time will work even when consuming really old web services.

Friday, March 18, 2011

Auditing entities with Hibernate JPA

In a real world project you will need to audit certain database changes keeping track of the author of the change. Hibernate includes nowadays the Envers API and here I am showing you how to achieve auditing when using Hibernate as JPA persistence provider.

  1. Include dependencies. I can confirm this works with the below version. 3.5.1 for example fails to update auditing tables. About other versions you will need to try yourself.
    <!-- Hibernate Audit -->
  2. In persistence.xml use"create" (I do not like update as it is dangerous to update your schema, I prefer creating it from scratch or failing if it already exists) to ensure tables are created from Java.
                <property name="" value="create" />
  3. And then add also the properties needed for audit
    <!-- Audit -->
                <property name=""
                value="org.hibernate.ejb.event.EJB3PostInsertEventListener,org.hibernate.envers.event.AuditEventListener" />
                <property name=""
                value="org.hibernate.ejb.event.EJB3PostUpdateEventListener,org.hibernate.envers.event.AuditEventListener" />
                <property name=""
                value="org.hibernate.ejb.event.EJB3PostDeleteEventListener,org.hibernate.envers.event.AuditEventListener" />
                <property name="hibernate.ejb.event.pre-collection-update"
                value="org.hibernate.envers.event.AuditEventListener" />
                <property name="hibernate.ejb.event.pre-collection-remove"
                value="org.hibernate.envers.event.AuditEventListener" />
                <property name=""
                value="org.hibernate.envers.event.AuditEventListener" />
  4. Annotate your entities or fields with @Audited. Let us start with a simple example where you annotate specific fields
        private String home_email;
  5. Now you will end up with two tables.
    CREATE TABLE `revinfo` (
      `rev` int(11) NOT NULL AUTO_INCREMENT,
      `revtstmp` bigint(20) DEFAULT NULL,
      PRIMARY KEY (`rev`)
    CREATE TABLE `employee_aud` (
      `id` int(11) NOT NULL,
      `rev` int(11) NOT NULL,
      `revtype` tinyint(4) DEFAULT NULL,
      `home_email` varchar(50) DEFAULT NULL,
      PRIMARY KEY (`id`,`rev`),
      KEY `FK9900899F8B9EFD24` (`rev`),
      CONSTRAINT `FK9900899F8B9EFD24` FOREIGN KEY (`rev`) REFERENCES `custom_revision_entity` (`id`)
    ) ENGINE=InnoDB;
  6. Update the entity and you will get records in both tables. For example:
    select * from employee_aud;
    | id | rev | revtype | home_email     |
    | 79 |   1 |       1 | |
    1 row in set (0.00 sec)
    select * from revinfo;
    | rev | revtstmp      |
    |   1 | 1300395363157 |
    1 row in set (0.02 sec)
  7. You might want to use the envers API to retrieve historical data but if you have to provide such data from reports you probably just want to use plain old SQL. For example see the below a query results after two updates for employee Paul:
    SELECT e.first_name, ea.home_email, FROM_UNIXTIME(r.revtstmp / 1000) FROM employee e INNER JOIN employee_aud ea ON = INNER JOIN revinfo r ON r.rev = ea.rev;
    | first_name | home_email      | FROM_UNIXTIME(r.revtstmp / 1000) |
    | Paula      |  | 2011-03-17 16:56:03              |
    | Paula      | | 2011-03-17 17:04:49              |
    2 rows in set (0.00 sec)
  8. We are now in the middle of the task resolution as we need to provide the user that actually made the change. The first thing to be done is to create a JPA entity that will create a third table
    import javax.persistence.Entity;
    import org.hibernate.envers.DefaultRevisionEntity;
    import org.hibernate.envers.RevisionEntity;
    public class CustomRevisionEntity extends DefaultRevisionEntity {
        private static final long serialVersionUID = 3775550420286576001L;
        private String username;
        public String getUsername() {
            return username;
        public void setUsername(String username) {
            this.username = username;
  9. Here is the CustomRevisionListener
    import org.hibernate.envers.RevisionListener;
    public class CustomRevisionListener implements RevisionListener {
        public void newRevision(Object revisionEntity) {
            CustomRevisionEntity revision = (CustomRevisionEntity) revisionEntity;
            UserDetails userDetails = (UserDetails) SecurityContextHolder.getContext().getAuthentication().getPrincipal();
  10. Here is the new table (In production I include this as part of the deployment plan). Note that table revinfo will not longer exist. This is the new table containing revision details.
    CREATE TABLE `custom_revision_entity` (
      `id` int(11) NOT NULL AUTO_INCREMENT,
      `timestamp` bigint(20) NOT NULL,
      `username` varchar(255) DEFAULT NULL,
      PRIMARY KEY (`id`)
    ) ENGINE=InnoDB;
  11. Finally after some editions you can pull some report for your manager directly from SQL
    SELECT e.first_name, ea.home_email, FROM_UNIXTIME(cre.timestamp / 1000), cre.username FROM employee e INNER JOIN employee_aud ea ON = INNER JOIN custom_revision_entity cre ON = ea.rev;
    | first_name | home_email      | FROM_UNIXTIME(cre.timestamp / 1000) | username          |
    | Paula      |  | 2011-03-18 11:44:47                 | |
    | Paula      | | 2011-03-18 11:47:19                 | |
    2 rows in set (0.00 sec)
  12. You can annotate whole entities with the exceptions of certain fields. For example let's say we want to audit the whole employee class with the exception of some ManyToOne and OneToMany fields but including at least a ManyToOne relationship
    public class Employee{
        @ManyToOne(fetch = FetchType.LAZY)
        private Office office;
        private List<EmployeePromotionHistory> promotionHistory;
        @ManyToOne(fetch = FetchType.LAZY)
        private Department department;
  13. For the above to work you will need to audit the department entity or at least one of its fields:
    public class Department {
        private String name;
  14. Note that when you assign a different department to the employee a new record will appear in employee_aud table, however no records will show up in department_aud. When a department name is changed then the change is registered in department_aud. Here is where the AuditQuery comes handy in terms of queries.

From now on the audited entity will keep not only the history but also who changed it. Note that you can add other columns creating the possibility to achieve literally anything you want from an entity auditing perspective.

If you want to audit the user iterations you can take a look at my recent post about login requests. Instead of just login to a file you can and use a table where you keep by sessionId all request details. This has proven to be extremely useful for a company I worked for in the past. Thanks to this "flow logging" they basically can do BI on their clients, dispute client billing and so much more.

Wednesday, March 16, 2011

Agile troubleshooting: LoggingFilter to log all requests

I have posted before about logMonitor and its role in an agile development team.

Being able to use the logs in a proactive way requires though something else: As a developer you must understand as soon as possible the specific test case to be able to recreate the issue in your development environment.

You need to make sure you can separate from the log traces those pieces related to the session where the error was found and that within the traces you can actually see the received request.

Here is how in a typical J2EE container you will do it:

Log the session

The session id is that parameter you cannot forget to log. For example with log4j you would use something like %X{sessionId}
log4j.appender.logfile.layout.ConversionPattern=%d %p [%c] - %X{sessionId} %m%n

For this to work you will need to use the below from your Java code:
MDC.put("sessionId", httpRequest.getSession().getId());

Log the request

The below class is a J2EE filter that will log the complete URL/URI including parameters. Both POST and GET parameters are logged which means the URL(s) could be used to recreate the whole test case.
package com.nestorurquiza.web.filter;

import java.util.Arrays;
import java.util.Map;

import javax.servlet.Filter;
import javax.servlet.FilterChain;
import javax.servlet.FilterConfig;
import javax.servlet.ServletException;
import javax.servlet.ServletRequest;
import javax.servlet.ServletResponse;
import javax.servlet.http.HttpServletRequest;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.slf4j.MDC;

public final class LoggingFilter implements Filter {

    private static final Logger log = LoggerFactory
    //Must be ordered!
    private static final String[] ORDERED_EXCLUDED_PARAMETERS = new String[]{"password", "card_number"};

    public void doFilter(ServletRequest request, ServletResponse response,
            FilterChain chain) throws IOException, ServletException {
        HttpServletRequest httpRequest = (HttpServletRequest) request;
        MDC.put("sessionId", httpRequest.getSession().getId());
        String uri = httpRequest.getRequestURI();
        if(! uri.contains(".")) {
            String remoteAddress = request.getRemoteAddr();
            String requestedResourceAsHttpGet = getRequestedResourceAsHttpGet(httpRequest);
   + " " + remoteAddress);
        chain.doFilter(request, response);

    public void destroy() {

    public void init(FilterConfig filterConfig) {

    private String getRequestedResourceAsHttpGet(HttpServletRequest httpRequest) {
        StringBuffer sb = new StringBuffer();
        String uri = httpRequest.getRequestURI();

        Map<String, String[]> parameters = httpRequest.getParameterMap();

        for (String key : parameters.keySet()) {
            String[] values = parameters.get(key);
            if (values != null) {
                for (String token : values) {
                    if (sb.indexOf("?") < 0) {
                    sb.append('&' + key + '=');
                    if(!isExcluded(key)) {
        return sb.toString();
    private boolean isExcluded(String s) {
        int index = Arrays.binarySearch(ORDERED_EXCLUDED_PARAMETERS, s);
        return index > 0;

Obtaining the test case from logs

Just a simple grep by sessionId and LoggingFilter will do the trick
grep 4AB145AC791D2711119A6FFCCC6A85BC my-app.log | grep LoggingFilter

Monday, March 14, 2011

Using tcpmon to sniff HTTP traffic

This tool can be used as I have posted before to debug SOAP but it is not limited to that of course.

I was asked the question about step by step instructions on how to install it. Here they are for OSX. If using windows translate bash to DOS batch and for Linux it should be pretty straightforward after reading the below.

  1. Download tcpmon
  2. Uncompress in a folder of your choice, let us say in your home directory
  3. Create a bash script for example /usr/sbin/tcpmon with the below content:
    cd /Users/nestor/tcpmon-1.0-bin/build
  4. Run tcpmon from command line:
    tcpmon &

Liferay User Interface Development book review

Liferay User Interface Development is the new Pakt Publishing book focusing on the front end side of Liferay development.

The book offer good guidelines for a migration from previous versions to Liferay 6.

An experienced architect would expect the book to be targeting Front end developers but the reality is in the words of the authors "This book is for any Java developers"

I will not jump again into separation of concerns discussions as I have covered that in deep already. Rathert I will go through the book highlighting my impressions. Hopefully it will make the reader aware of how much can be expected from this publishing work.

The first chapter shows Liferay from inside with information about the myriad of technologies it uses. Not a useful chapter for front end development but perhaps a good chapter to educate yourself on terminology and internal Liferay architecture.

The second chapter addresses Theme development. It goes through steps about configuring Eclipse to host the plugins SDK folder, the bundle folder, the source folder etc. I personally prefer Maven projects for any plugin development and that combined with agile deployment gives a faster environment but the book goes through the traditional ant method to build and deploy plugins.

Chapter 3 introduces Layout development and goes deep in the details about how actually the rendering mechanism works in Liferay (which is based on Struts framework)

Chapter 4 combines an explanation about permissions, navigation, internationalization (prepare the portal to support different locales) and localization (Adapting Liferay to respond to specific locale) and how to actually set default values for theming and layout. Velocity tips are given for common operations like removing the Dockbar from the default theme, showing a different logo and so on. It also covers configuring Liferay through XML changes to achieve things like grouping portlets in categories or disabling certain portliest. Finally it goes over customization of Portlets and Control Panel. The name of the chapter in my opinion is some kind misleading "Styling pages" but other than that the information provided is very practical and of great use especially for Portal Admins in charge of Page/Portlet configuration.

Chapter 5 is called "Advanced Theme" and it guides the user through features like color schemes. Again more usable real life hints are provided like for example embedding the SignIn portlet in the header area. The majority of the tricks are about tweaking the velocity template portal_normal.vm from your selected theme. Guidance is provided to migrate a theme from previous version of Liferay to the new version. Be prepare for a lot of diff/copy/paste. The newly supported FreeMarker is introduced as an alternative to the existing Velocity templates. It explains conventions followed on existing CSS, Javascript, folder structure for images and browser compatibility hints. Especially for Java developers that are not that aware of front end difficulties this chapter offers several hints that would help on skinning Liferay. An introduction to Liferay Eclipse plugin follows with some recommendations to use ViewDesigner Dreamweaver plugin for Designers (I have met several designers and they all use OSX and not much of Dreamweaver but if you are a Designer that works on Windows and use Dreamweaver then the plugin might be good for you) Finally it presents several plugins for Firefox and Chrome as recommendations for your toolset.

Chapter 6 is about the Portlet User Interface. It explains the different types of Portlets, how to deploy them using the Liferay Plugins SDK, portlet tags, permission tags and starts a discussion that will be enhanced in following chapters about Alloy UI and UI Taglibs.

Chapter 7 is about Velocity templates, an alternative for JSP widely supported by Liferay.

Chapter 8 is devoted to the newly introduced Alloy taglibs which is based on YUI. This chapter is a good introduction to both YUI and Alloy UI.

Chapter 9 brings a UI Taglib introduction. With UI Taglibs the front end developer has access to rapid error reporting, i18n support, searching, paginator, breadcrumb, navigation, and more.

Chapter 10 finalizes the book with an explanation about migrating from a previous version of Liferay (like 5.2.3) to the newest version 6. While the chapter is titled "User Interface in Production" it does explain how to migrate other components that are not directly related to the front end, after all a web project is divided by concerns but it needs all of its parts as a whole to really work.

Final Thoughts

Even though a pure Front End Developer is not the target of the book (as someone might expect from the title) the book does provide valuable information for Front end developers trying to put their hands on customizing Liferay UI.

To jump into Liferay development the architect will need to make sure the hired front end developer will be fine using Eclipse at least for JSP validations to ensure structural changes in JSP files do not break JSP compilation. But that is half of the path, hopefully the front end developer will be fine to run the whole Liferay bundle locally to develop/customize themes, layouts and portlets.

A Front End developer could use for plugin development just a combination of TextMate and maven to get fast theme development but at this point JSP support in TextMate is very limited.

Saturday, March 12, 2011

EWS API Exchange Online or in Premises

Business Productivity Online Standard (BPOS) includes Exchange Online product. Today I am sharing my experiences with that service and its API.

If regulations allow you to put email services in the cloud that could save a lot of money. Google has made this a dream come true but if you really enjoy Exchange services and you simply want a less invasive to your existing Exchange/Outlook based infrastructure (and still inexpensive) move then it makes sense to use Microsoft Exchange Online.

If your are still running an Exchange version prior to Exchange 2007 there is an added value in terms of how much you can actually achieve as part of this move (besides a really cheap cost per user account=$5.00/month) and it is the developer API that comes with it thanks to the fact that you will enjoy Microsoft Exchange 2007 in the cloud. Just search for "Exchange Online Developer Guide" to get a feeling of everything you can do through the EWS (SOAP API).

Once you have signed up for a 30 days trial or whenever you engage in the service you must use the corresponding to your zone URL from the online OWA URLs list to access your mailbox from a browser. In my case such URL was You get a very close to Outlook interface but online. Certainly the advanced GUI only works good enough for IE so this is definitely a complete solution only for companies where Windows and Outlook are present in the Enterprise. Support for other browsers is basic.

Of course you can configure your Outlook to work with the Online Exchange Server as well. It worked for me with Outlook 2003 SP3.

I decided to test Exchange Online after realizing how our Java development team could leverage on this API to assist business in a seamless integration of existing Outlook/Exchange users and email automations.

Let us consider the following task as an example: Send an email with this account and be sure it is saved in a Sent Items subfolder. This can be achieved through the EWS API without the need to handle distributed transactions or even worst implementing a solution without them. A SOAP call will carry all necessary information to make all that happen in just one request. So if you are still sending an email and using IMAP as a second step to put that email in a different folder, consider using EWS.

I signed for the trial and I have to say Microsoft Exchange Online customer support is really great. You open a Service Request and you get help normally within 15 minutes. People that assist you really know Exchange Online and their help goes beyond the service itself, they provide guidance to integrating third party applications like RightFax or coexistence with your Exchange on premises if any. They are guys with a real name, they share their contact information with the customer, they talk over the phone besides emails and they work no rest to resolve your ticket. I did not spend time talking to machines this time.

Exchange Online might not be perfect for you. For example if you have strict requirements about legal hold not even the upcoming to the cloud Exchange 2010 version might work for you. Still you can engage in a second service from Microsoft (more pricy) called EHS to provide archiving but of course that will put your bill about the $5.00 per user per month. Being in a public (shared) cloud might be also a problem if you are in the Financial business for example. Even though the service is SAS-70 certified the risk must be assessed and your legal department might not feel good with the move. In fact there is no robust DR solution in the standard Exchange Online service so currently you need to purchase EHS if you want to ensure 8 hours worth of emails won't go away after a disaster situation.

In any case EWS will work with Exchange Online or with Exchange on premises. That is after all the good news.

smtp: Failed: Connect failed-Ubuntu big log files

I got an alert (busy weekend ;-) about disk growing too much. There were zillions of the below lines in /var/log/syslog :
Mar 12 06:26:18 server-name nullmailer[23913]: Starting delivery: protocol: smtp host: file: 1297459801.9713
Mar 12 06:26:18 server-name nullmailer[31436]: smtp: Failed: Connect failed

This is because of a miss configuration. By the time the server was installed a wrong smarthost was configured.

All I had to do to correct this was to edit nullmailer remotes file:
sudo vi /etc/nullmailer/remotes.

Handling slow SOAP responses in the client

Real world is different from the developer machine and testing tools must be used if you want to avoid code failures in real life situations.

Regardless the programming language you use a TCP proxy with delay capability is the tool you need to look for. Axis (not Axis2) comes with such a tool named TcpMon. The tool is maintained our of the Axis project nowadays. You can download it from here.

As the tool is developed with Java you can run it in any OS. Below is a picture of it configured to wait 30 seconds after each chunk of 10KB transmitted, listening in localhost port 4640 and intercepting request and responses going to and from internal IP.

Below is a sample screenshot showing the request and the response of a SOAP call done to Advent Geneva SOAP Server:

The Problem

This post came as a result of some hours debugging slow responses that my SOAP was experimenting.

I am using Axis2 ServiceClient for that project and I thought slow responses could be handled by the timeout configurations:
//options.setProperty(HTTPConstants.SO_TIMEOUT, new Integer(timeOutInMilliSeconds));
//options.setProperty(HTTPConstants.CONNECTION_TIMEOUT, new Integer(timeOutInMilliSeconds));



The above does not address the problem of slow packets or servers just performing to many operations before sending back the whole SOAP response. Ideally there would be a way to tell the client just to timeout if the whole operation takes too much time.

I thought this could be achieved accessing HttpURLConnection.setReadTimeout() and that made me post the question in the Axis2 Forum.

After seeing some people visiting the post but actually not replying I felt like I had to hack deeper in the code and so I did.

Axis2 uses Commons Http Client library and that one just has support for socket and connection timeouts. This is probably right approach if we talk about TCP but I think when it comes to HTTP, handling timeouts when data takes too long over the wire is a must have.

This as a common problem to be solved: You do not want your client side to hang waiting for slow server responses. Ideally the client should timeout in circumstances like slow network throughput, high latency and so on.

A Solution

I ended up using concurrency to solve this issue. A Callable task does the job and if the task does not finish within certain amount of time the Callable task is killed. This is easily done through the use of an ExecutorService.

Here is the client code before:
OMElement resultOfOperation = setupServiceClient().sendReceive(operation.get(null));

Here is the client code after:
SendReceiveTask sendReceiveTask = getSendReceiveTask( operation, null );
OMElement resultOfOperation = sendReceiveTask.resultOfOperation;

Below are the snippets you need for the code above to run. They are an extract of real code and so there are special components related to the specifics of the particular SOAP Service like for example the session but you should get the idea on how to achive the same in your code:
    class SendReceiveTask implements Callable {
        SoapOperation soapOperation;
        OMElement resultOfOperation;
        Exception exception;
        Session session;
        SendReceiveTask(SoapOperation soapOperation, Session session) {
            this.soapOperation = soapOperation;
            this.session = session;
        public String call() {
            try {
                resultOfOperation = setupServiceClient().sendReceive(soapOperation.get(session));
            } catch (Exception e) {
                exception = e;
            return null;
    private SendReceiveTask getSendReceiveTask( SoapOperation operation, Session session ) throws InterruptedException {
        SendReceiveTask sendReceiveTask = new SendReceiveTask(operation, session);
        ExecutorService executor = Executors.newSingleThreadExecutor();
        int timeOutInMilliSeconds = genevaSettings.getTimeoutMilliseconds();
        executor.invokeAll(Arrays.asList(sendReceiveTask), timeOutInMilliSeconds, TimeUnit.MILLISECONDS);
        boolean taskFinished = sendReceiveTask.exception != null || sendReceiveTask.resultOfOperation != null;
        if( !taskFinished ) {
            throw new RuntimeException("Response Timeout. It took more than " + timeOutInMilliSeconds + " milliseconds.");

        if(sendReceiveTask.exception != null) {
            throw new RuntimeException(sendReceiveTask.exception);
        return sendReceiveTask;

Friday, March 11, 2011

org.apache.axis2.AxisFault: Unable to find host

Someone might think this error happens when the host name is not resolved and that is all:
org.apache.axis2.AxisFault: Unable to find host

However in axis 2 version 1.5.4 at least this error shows up also when there is a problem with the SOAP port.