Saturday, February 26, 2011

Exchange Web Services EWS from Java

Update 20120629: The managed API looks a little bit better after a year. I have updated the project to reflect so. Read more here.

I have tested the EWS SOAP API using a trial account of Microsoft Exchange Online. I opened a trial account for 30 days and I got availability for 20 users with 25GB per user. Of course you can setup Exchange 2007 and go through this journey yourself. It should work there too.

For the impatient here is the whole source code. Be sure you read README file.

The first thing you want to do is to get the Services.WSDL. Using a valid account you will be able to get it from one of the URLs provided to you, in my case https://red001.mail.microsoftonline.com/ews/Services.wsdl. As a side note https://red001.mail.microsoftonline.com/ews/Exchange.asmx redirects to the first URL.

The WSDL by default cannot be consumed as it lacks the wsdl:service node. Here is a corrected version of the wsdl file and the xsd files as well. You need to download it locally as well as types.xsd and messages.xsd from the same path. Once downloaded just add the below node to the wsdl file. Note that the Exchange Server URL might be different for you:
<wsdl:service name="ExchangeService">
  <wsdl:port name="ExchangeServicePort" binding="tns:ExchangeServiceBinding">
    <soap:address location="https://red001.mail.microsoftonline.com/EWS/Exchange.asmx"/>
  </wsdl:port>
  </wsdl:service>
</wsdl:definitions>

Now open SoapUi and point to the local modified wsdl copy.

Go to GetFolder node on the left. Double click "Request 1" and click on the Aut button, then provide Username and Password for HTTP Authentication. Replace the request content by the below:
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:typ="http://schemas.microsoft.com/exchange/services/2006/types" xmlns:mes="http://schemas.microsoft.com/exchange/services/2006/messages">
   <soapenv:Body>
      <mes:GetFolder>
         <mes:FolderShape>
            <typ:BaseShape>AllProperties</typ:BaseShape>
         </mes:FolderShape>
         <mes:FolderIds>
            <typ:DistinguishedFolderId Id="inbox"/>
         </mes:FolderIds>
      </mes:GetFolder>
   </soapenv:Body>
</soapenv:Envelope>

Hit the run button (green arrow) and you will get a response like the below. Look how you get the number of unread items in your inbox together with other useful information:
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
   <soap:Header>
      <t:ServerVersionInfo MajorVersion="8" MinorVersion="3" MajorBuildNumber="83" MinorBuildNumber="4" xmlns:t="http://schemas.microsoft.com/exchange/services/2006/types"/>
   </soap:Header>
   <soap:Body>
      <m:GetFolderResponse xmlns:t="http://schemas.microsoft.com/exchange/services/2006/types" xmlns:m="http://schemas.microsoft.com/exchange/services/2006/messages">
         <m:ResponseMessages>
            <m:GetFolderResponseMessage ResponseClass="Success">
               <m:ResponseCode>NoError</m:ResponseCode>
               <m:Folders>
                  <t:Folder>
                     <t:FolderId Id="AAAhAG51cnF1aXphQGtyZnMubWljcm9zb2Z0b25saW5lLmNvbQAuAAAAAAD++u3ArxgdQrBBhvckACnkAQB8H0u8rlXuQomN7tXZc6ZqAAATKH0iAAA=" ChangeKey="AQAAABQAAADy++j7kEQqS5TFatdF3GSPAAAIEA=="/>
                     <t:ParentFolderId Id="AAAhAG51cnF1aXphQGtyZnMubWljcm9zb2Z0b25saW5lLmNvbQAuAAAAAAD++u3ArxgdQrBBhvckACnkAQB8H0u8rlXuQomN7tXZc6ZqAAATKH0fAAA=" ChangeKey="AQAAAA=="/>
                     <t:FolderClass>IPF.Note</t:FolderClass>
                     <t:DisplayName>Inbox</t:DisplayName>
                     <t:TotalCount>5</t:TotalCount>
                     <t:ChildFolderCount>0</t:ChildFolderCount>
                     <t:UnreadCount>1</t:UnreadCount>
                  </t:Folder>
               </m:Folders>
            </m:GetFolderResponseMessage>
         </m:ResponseMessages>
      </m:GetFolderResponse>
   </soap:Body>
</soap:Envelope>

This is really fun and in fact you can develop plain XML over HTTP applications but there must be something there for the OO fans right?

It is easier if you actually use Stubs. Of course Objects are easier to manipulate than plain XML responses. Be sure though you understand the overhead inherent to serialization/deserialization: using Objects simplifies the work at expenses of a performance penalty which for most cases is not big deal but I have been through projects in which basically I needed to create XML and send it over HTTP directly and then parse XML responses coming back over the wire as well because of unacceptable performance overhead.

Before we jump into coding you will notice at least one schema error in types.xsd file.

Replace the commented line by the next one:
<!-- <xs:import namespace="http://www.w3.org/XML/1998/namespace"/> -->
  <xs:import namespace="http://www.w3.org/XML/1998/namespace" schemaLocation="http://www.w3.org/2001/xml.xsd"/>


Microsoft Managed EWS Java API

It would be really nice if you would have an EWS Java (Managed) API like .NET developers enjoy. Microsoft did some effort in the past but I tried hard to get to the team involved in that effort through Microsoft Online Services Customer Support and got not positive feedback. It looks like that API is abandoned. Anyway that was my first attempt as I explain below. If you want to jump directly into something that really works start at JAX-WS section. I have put everything together on Google code SVN. Checkout the project and look for "Overloaded method to work around the infamous xsl:nill attribute error" to see the code I changed from ExchangeServicePortType class. Take a look at README file from there for instructions how to get the project working.

Even though it did not work at the end as I got "Server redirected too many times" error when trying https://red001.mail.microsoftonline.com/ews I am documenting here the steps to follow in case you want to give this a try with any instance of Exchange 2007.
  1. Download EWS Java API http://archive.msdn.microsoft.com/ewsjavaapi
  2. Build the jar
    cd EWSJavaAPI1.1 
    ant
    
  3. Include file EWSJavaAPI1.1/EWSAPI1.1.jar in your project. Below is how I installed in my local maven repo
    mvn install:install-file -DgroupId=ews -DartifactId=ews-api -Dversion=1.1 -Dpackaging=jar -Dfile=/Users/nestor/Downloads/EWSJavaAPI1.1/EWSAPI1.1.jar
    
  4. Generate a jar scaffold project
    mvn archetype:generate -DarchetypeGroupId=org.apache.maven.archetypes -DarchetypeArtifactId=maven-archetype-quickstart -DarchetypeVersion=5-SNAPSHOT -DgroupId=com.nestorurquiza.microsoft -DartifactId=ews -Dversion=1.0.0-SNAPSHOT
    
  5. As I will use Eclipse as IDE I add the eclipse plugin to the pom.xml
    <build>
        <plugins>
          <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-eclipse-plugin</artifactId>
            <version>2.8</version>
          </plugin>
        </plugins>
      </build>
    </project>
    
  6. Create the necessary eclipse files
    cd ews
    mvn eclipse:eclipse
    
  7. Open Eclipse an give it a try with this commented out code

JAXB

I could not get it working with JAXB. If you were successful please share. Below is a little bit of history.

You will need to create a new file named for example jaxb-custom-bindings.xml with the below content:
<jaxb:bindings xmlns:jaxb="http://java.sun.com/xml/ns/jaxb"
               xmlns:xsd="http://www.w3.org/2001/XMLSchema"
               jaxb:version="2.0">
  <jaxb:bindings >
      <jaxb:globalBindings typesafeEnumMaxMembers="1000"/>
  </jaxb:bindings>
</jaxb:bindings>


You can use a command to generate all necessary Java classes, for example:
java -jar jaxb-ri-20110115/lib/jaxb-xjc.jar -p com.nestorurquiza.axis2.client.ews -wsdl "file:///Users/nestor/Downloads/Services.wsdl" -d /Users/nestor/Downloads/ews/jaxb/ -b jaxb-custom-bindings.xml

Somehow this was not generating the main Stub class for me.

Axis2

Again I could not get it working but if you could then share your experience please.

You can use axis2 to generate the stubs. Here is an example in commands:
./axis2.sh org.apache.axis2.wsdl.WSDL2Java -uri ~/Downloads/Services.wsdl  -o ~/Downloads/ews/axis2
cd ~/Downloads/ews/axis2
ant

This generated an ant project but I got several unresolved symbols.

JAX-WS

One could be tempted to build a jar with the stubs but you will be disappointed because of the infamous "xsl:nill attribute error". Just Google it!

Download JAX-WS http://jax-ws.java.net/2.2.1/ and run similar to the below commands (change paths to adjust to your environment) to get your sources files and include them in your standalone (jar) project.
mkdir -p /Users/nestor/Downloads/ews/jaxws/src
mkdir /Users/nestor/Downloads/ews/jaxws/classes
cd /Users/nestor/Downloads/jaxws-ri/bin
wsimport /Users/nestor/Downloads/Services.wsdl -p com.nestorurquiza.ews.jaxws -d /Users/nestor/Downloads/ews/jaxws/classes -s /Users/nestor/Downloads/ews/jaxws/src
cp -R /Users/nestor/Downloads/ews/jaxws/src/* ~/eclipse-workspace-test/ews/src/main/java/

You have now the API ready to be used. I have to tell you this is half of the path. Hopefully in the web you will find snippets that will help you consuming the services as they are not that straightforward I have to say. Any single step needs several commands. For a test of it if you did not do it already checkout sample working classes I have shared with the hope more snippets will come and perhaps even a developer willing to take this to the extra mile and build a real abstraction around this API. Feel free to GitHub my code. I would appreciate if you give me some credit. All the work I make publicly available is published with a BSD license BTW. Happy coding!

Thursday, February 24, 2011

Windows NTLM Proxy Authentication from Linux and OSX

If you are working in a company that uses a proxy with NTLM Authentication to grant access to Internet you will find yourself using a tool to run a local proxy to get around an issue: NTLM is not supported neither in Linux nor (surprisingly) in OSX.

You can use Firefox though with no pain to get to Internet thanks to its embedded NTLM support.

  1. In Firefox address bar type "about:config" without quotes and hit return.
  2. In the filter type "ntlm" and make sure you have the keys and values below, if not add the one that might be missing:

  3. In Firefox Preferences go to Network | Connection | Settings and select "Manual Proxy Configuration", type the FQDN/IP as name of the proxy (in our example 192.168.1.7) and the Port (It will depend again on your particular network details, for the purpose of this example let us say it is 3131).
  4. Add the exceptions for those URLs that should not go through the proxy like intranet URLs
  5. Hit a proxy controlled URL and you should be prompted for the authentication. Use for user name "domain\credential".
  6. Firefox will offer you to remember the credentials with a statement like "Do you want Firefox to remember the password for "myDomain\myUseNname" on mox-proxy://192.168.1.7:3131?".

Wednesday, February 16, 2011

Keep track of web requests in log files

Especially in integration and staging environments it is useful to have the whole request so when you get notifications about current problems you get a better picture of what actually happened.

You must be careful what you log (exclude any sensitive information like passwords, SSN, even user names) but you must have a utility that allows you to dump the request.

In Servlet programming I have my own class iterating through headers and parameters, sometimes cookies and this time I told myself there must be something in Jakarta Commons or one of the existing frameworks (Spring/Struts). To my surprise this is still not standardized.

Here is a good class that could be used in current frameworks or extensions to them. This helps what I call proactive coding (You fix or at least are aware of problems before you get notified about them)

Friday, February 04, 2011

Handling Errors in Spring Framework

To handle errors in your Spring Framework application you will need to play with two files: web.xml and applicationContext.xml (or whatever the name is for your application context configuration file).

web.xml

<error-page>
        <exception-type>java.lang.Exception</exception-type>
        <location>/WEB-INF/jsp/uncaughtException.jsp</location>
    </error-page>
    <error-page>
        <error-code>500</error-code>
        <location>/WEB-INF/jsp/uncaughtException.jsp</location>
    </error-page>
    <error-page>
        <error-code>404</error-code>
        <location>/WEB-INF/jsp/notFound.jsp</location>
    </error-page>

Note how I use the same JSP for 500 errors and Exceptions. From that JSP I log the trace in the server logs (so I can monitor problems occurring in user browsers for example) Below is spring pet clinic uncaughtException.jsp modified to log the complete stacktrace:
<%@ page isErrorPage="true" %>
<%@ page  import="org.slf4j.Logger" %>
<%@ page  import="org.slf4j.LoggerFactory" %>
<%@ include file="/WEB-INF/jsp/includes.jsp" %>
<%@ include file="/WEB-INF/jsp/header.jsp" %>

<%
final Logger log = LoggerFactory.getLogger("com.nestorurquiza.web.internal-error");
log.error("Internal Server Error", exception);
%>

Internal error

<% try { // The Servlet spec guarantees this attribute will be available //Throwable exception = (Throwable) request.getAttribute("javax.servlet.error.exception"); if (exception != null) { if (exception instanceof ServletException) { // It's a ServletException: we should extract the root cause ServletException sex = (ServletException) exception; Throwable rootCause = sex.getRootCause(); if (rootCause == null) rootCause = sex; out.println("** Root cause is: "+ rootCause.getMessage()); rootCause.printStackTrace(new java.io.PrintWriter(out)); } else { // It's not a ServletException, so we'll just show it exception.printStackTrace(new java.io.PrintWriter(out)); } } else { out.println("No error information available"); } // Display cookies out.println("\nCookies:\n"); Cookie[] cookies = request.getCookies(); if (cookies != null) { for (int i = 0; i < cookies.length; i++) { out.println(cookies[i].getName() + "=[" + cookies[i].getValue() + "]"); } } } catch (Exception ex) { ex.printStackTrace(new java.io.PrintWriter(out)); } %>


<%@ include file="/WEB-INF/jsp/footer.jsp" %>



applicationContext.xml

<bean class="com.nestorurquiza.web.handler.CustomSimpleMappingExceptionResolver" >
        <property name="exceptionMappings">
            <props>
                <prop key="org.springframework.web.servlet.PageNotFound">notFound</prop>
                <prop key="java.lang.Exception">failure</prop>
            </props>
        </property>
    </bean>

Note I use a custom MappingExceptionResolver. I do this again to make sure I log the error. I strongly believe error monitoring must be done from outside the application.

Below is the code for CustomSimpleMappingExceptionResolver:
package com.nestorurquiza.web.handler;

import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.web.servlet.ModelAndView;
import org.springframework.web.servlet.handler.SimpleMappingExceptionResolver;

public class CustomSimpleMappingExceptionResolver extends SimpleMappingExceptionResolver{
    @Override
    public ModelAndView resolveException(HttpServletRequest request,
            HttpServletResponse response, Object handler, Exception ex) {
        final Logger log = LoggerFactory.getLogger(CustomSimpleMappingExceptionResolver.class);
        log.error(null,ex);
        return super.resolveException(request, response, handler, ex);
    }
}

Thursday, February 03, 2011

Hide context URL in Tomcat

Regardless if you use Apache plus modjk or any other way to separate the HTTP server from Tomcat (highly recommended in production environment) you will face the alternative between a nice looking URL like:
http://example.com/login
versus
http://example.com/my-app/login

For this to work you will need to configure the Host node in server.xml as shown below. Notice the full path which is required and makes sense as this is an environment specific setting.
...
      <Host name="bhub.nestorurquiza.com" debug="0">
                <Context path=""
                         docBase="/opt/tomcat/webapps/my-app"/>
      </Host>
    </Engine>
  </Service>
</Server>

Reloading log4j properties in Tomcat

I would love to see on demand log4j properties reloading in application servers.

If you want to reload log4j configuration it is because you have made a change let us say to debug a specific issue appearing right now in production. Is there a real need to be monitoring the file for changes? Wouldn't it be better to use resources just by the time we really want a reload?

Tomcat provides a mechanism to allow dynamic reloading and JBoss ships with the solution already integrated (meaning the standard installation will check the file to reload it every so often)

The point is they do not provide a simple admin link to force reloading which in my opinion goes against the natural process: change configuration file, save it and expect the changes. Just in order to avoid a final step "push the changes" is not enough in my opinion. Apache Server knows that very well, you can reload your configuration but on demand.

On demand reloading has a big advantage: You use resources just when you need them. With that approach we could avoid performance problems: read using "reloadable" for in production for tomcat 7 or memory leaks which I have to say have hit my teams both in JBoss and Tomcat environments.

I see no big deal on allowing an on demand reloading but while I wait for that day I prefer to reload the application context after changing the log4j properties file:
$ vi tomcat/webapps/my-app/WEB-INF/classes/log4j.properties 
$ touch tomcat/webapps/my-app/WEB-INF/web.xml 

And yes you must do this with caution in production as you kick out some of your users from that node in your cluster or even worst, you could be putting your service down for some seconds (Yes Tomcat is extremely fast especially when you avoid complicated processes at "Servlet Listener" time)

This is clearly far from ideal as the issue you are trying to debug might not be happening again after you restart your application. So think twice before deciding to increase log level in production and think more about creating load test scenarios in integration and staging environments.

Followers