Monday, January 31, 2011

Current page or resource in J2EE servlet application

Every so often I get the same question on my plate: How to find the current invoked resource or page in a J2EE servlet application. Hopefully the examples below go straight to what you need:
((HttpServletRequest)request).getRequestURL().toString()
http://sample.com/sample-app/css/main.css

((HttpServletRequest)request).getRequestURI()
/sample-app/css/main.css

((HttpServletRequest)request).getSession().getServletContext().getRealPath(((HttpServletRequest)request).getContextPath())
/Users/nestor/tomcat/webapps/sample-app/css/main.css

Friday, January 28, 2011

Remember Me with LDAP Spring Security

Remember Me functionality in Spring Security can be as easy as implement as just using a checkbox in your HTML ...
<input type='checkbox' name='_spring_security_remember_me' value="on"/>

... plus a declaration in the spring application context xml file:
<remember-me/>

However that is not true when using LDAP

For LDAP a persistent token will be needed which is actually good news as it is more secure. Here are the important bits to ensure the system remembers the user for 4 hours:
<http auto-config="true" use-expressions="true" access-decision-manager-ref="accessDecisionManager">
...
   <remember-me key="_spring_security_remember_me" token-validity-seconds="14400" token-repository-ref="tokenRepository"/>
A UserService is needed:
<ldap-user-service id="ldapUserService" group-search-base="ou=groups,o=MyCompany" group-role-attribute="cn" group-search-filter="(uniquemember={0})" user-search-base="ou=people,o=MyCompany" user-search-filter="mail={0}"/>
And a token Repository:
<beans:bean id="tokenRepository"
      class="org.springframework.security.web.authentication.rememberme.JdbcTokenRepositoryImpl">
      <beans:property name="createTableOnStartup" value="false" />
      <beans:property name="dataSource" ref="myDataSource"/>
   </beans:bean>
Note createTableOnStartup can be true the first time you start your application so it creates the "persistent_logins" table automatically. Below is the schema creation statement for MySQL. You can just run the statement and leave createTableOnStartup=false.
CREATE TABLE `persistent_logins` (
  `username` varchar(64) NOT NULL,
  `series` varchar(64) NOT NULL,
  `token` varchar(64) NOT NULL,
  `last_used` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
  PRIMARY KEY (`series`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1

Now spring will keep track of the last time the user logs in and will remember the user as long as the token-validity-seconds is not reached.

If you are protecting your site against CSRF attacks which you should of course then you will face an additional problem: The user will be remembered but the security token will be invalid.

To address that issue you could use a combination of before and after security filters or just a before filter. Basically you can put any code you want before (or after) the remember me functionality is triggered. Here the relevant xml bits:
<beans:bean id="customRememberMeFilter" class="com.nestorurquiza.web.filter.CustomRememberMeFilter" />
...
<http auto-config="true" use-expressions="true" access-decision-manager-ref="accessDecisionManager">
...
   <custom-filter  before="REMEMBER_ME_FILTER" ref="customRememberMeFilter" />

And here is the filter:
package com.nestorurquiza.web.filter;

import java.io.IOException;

import javax.servlet.Filter;
import javax.servlet.FilterChain;
import javax.servlet.FilterConfig;
import javax.servlet.ServletException;
import javax.servlet.ServletRequest;
import javax.servlet.ServletResponse;
import javax.servlet.http.Cookie;
import javax.servlet.http.HttpServletRequest;

import org.springframework.security.core.Authentication;
import org.springframework.security.core.context.SecurityContextHolder;
import org.springframework.security.web.authentication.rememberme.AbstractRememberMeServices;
import com.nestorurquiza.web.ControllerContext;

public class CustomRememberMeFilter implements Filter 
{
    public void doFilter(ServletRequest request, ServletResponse response, FilterChain chain) throws IOException, ServletException 
    {
        Authentication authentication = (Authentication) SecurityContextHolder.getContext().getAuthentication();
        String ctoken = request.getParameter(ControllerContext.CSRF_TOKEN);
        HttpServletRequest req = (HttpServletRequest)request;
        if(authentication == null && ctoken != null && getRememberMeCookie(req.getCookies()) != null) {
            req.getSession().setAttribute(ControllerContext.CSRF_TOKEN, ctoken);
        }
        chain.doFilter(request, response);
    }

    public void destroy() 
    {   

    }

    public void init(FilterConfig config) throws ServletException 
    {
    
    }
    
    private Cookie getRememberMeCookie(Cookie[] cookies) {
        if (cookies != null)
          for (Cookie cookie : cookies) {
            if (AbstractRememberMeServices.SPRING_SECURITY_REMEMBER_ME_COOKIE_KEY.equals(cookie.getName())) {
              return cookie;
            }
          }
        return null;
    }
}

While remember me functionality is very handy in some circumstances it is abused by some developers who ignore the vulnerabilities behind this feature.

AS you can see from the above code we are disabling CSRF protection for users opting to be remembered.

Here are some measures:
  1. Avoid if possible using Remember Me functionality
  2. Explain your users they will be more vulnerable
  3. Instruct your users to log-out if they opted to use remember-me functionality but have finished their job. Closing the browser is not enough to prevent a different user from gaining control over the original account

Sunday, January 23, 2011

Liferay Agile JSP development with Maven

If you use Maven for Liferay SDK Plugin development you can get your JSPs changes instantaneously in the browser. This involves two steps:
  1. Create an xml file named for example my-portlet.xml and copy it to the Liferay deploy folder. The content of the file is a single xml node with attribute "docBase" which value must be the target exploded application directory. For example:
    <Context
       docBase="/Users/nestor/projects/my-portlet/target/my-portlet"
       />
    
  2. You need to make sure when you change your JSPs they are automatically pushed into the target directory. This is the default behaviour in Netbeans. In Eclipse you can play with the Maven plugin for this however there is a fastest way: Use the fileSync eclipse plugin

You might be wondering why you need to use a context file while you don't need it when doing servlet programming.

The answer is that Liferay will manage web applications in specific folders below the temp directory. The folders have a portlet number followed by a dash and then the name of the portlet. Under those folders is where you need to touch JSP files in order for the changes to show up. However the number might change so the folder path is not a constant. Definitely the "/Context@docBase" directive comes in really handy.

Saturday, January 22, 2011

Load Tests with Jmeter: Liferay Example

If you build a WEB application regardless the programming language or framework you must test how it will perform under real world load:
  1. Concurrency problems will not be picked by code reviews, best practices, programming to interfaces, TDD (You could but I am wonder who would be able to afford it), agile project management techniques etc.
  2. You must be prepared for your expected traffic and in fact you should write an email to your supervisors making them aware of the software limitation: We currently can handle x concurrent users.
  3. Formulas and calculations are OK for estimatives but that is just the theory. Do your hoework and be sure your software will be able to handle the expected traffic.
Jakarta JMeter is your free open source friend.

I have decided to use Liferay to show an example of a load test. You just need to download Jmeter unzip it and run jmeter from the bin folder, open then the file I pasted below and edit the "Http Request Defaults" to point to your Liferay instance. Then verify how your server will handle 100 concurrent users using a ramp up period of 10 seconds.

<?xml version="1.0" encoding="UTF-8"?>
<jmeterTestPlan version="1.2" properties="2.1">
  <hashTree>
    <TestPlan guiclass="TestPlanGui" testclass="TestPlan" testname="Test Plan" enabled="true">
      <stringProp name="TestPlan.comments"></stringProp>
      <boolProp name="TestPlan.functional_mode">false</boolProp>
      <boolProp name="TestPlan.serialize_threadgroups">false</boolProp>
      <elementProp name="TestPlan.user_defined_variables" elementType="Arguments" guiclass="ArgumentsPanel" testclass="Arguments" testname="User Defined Variables" enabled="true">
        <collectionProp name="Arguments.arguments"/>
      </elementProp>
      <stringProp name="TestPlan.user_define_classpath"></stringProp>
    </TestPlan>
    <hashTree>
      <ThreadGroup guiclass="ThreadGroupGui" testclass="ThreadGroup" testname="Thread Group" enabled="true">
        <stringProp name="ThreadGroup.on_sample_error">continue</stringProp>
        <elementProp name="ThreadGroup.main_controller" elementType="LoopController" guiclass="LoopControlPanel" testclass="LoopController" testname="Loop Controller" enabled="true">
          <boolProp name="LoopController.continue_forever">false</boolProp>
          <stringProp name="LoopController.loops">1</stringProp>
        </elementProp>
        <stringProp name="ThreadGroup.num_threads">100</stringProp>
        <stringProp name="ThreadGroup.ramp_time">10</stringProp>
        <longProp name="ThreadGroup.start_time">1289581012000</longProp>
        <longProp name="ThreadGroup.end_time">1289581012000</longProp>
        <boolProp name="ThreadGroup.scheduler">false</boolProp>
        <stringProp name="ThreadGroup.duration"></stringProp>
        <stringProp name="ThreadGroup.delay"></stringProp>
      </ThreadGroup>
      <hashTree>
        <HTTPSampler guiclass="HttpTestSampleGui" testclass="HTTPSampler" testname="/web/guest" enabled="true">
          <elementProp name="HTTPsampler.Arguments" elementType="Arguments" guiclass="HTTPArgumentsPanel" testclass="Arguments" enabled="true">
            <collectionProp name="Arguments.arguments"/>
          </elementProp>
          <stringProp name="HTTPSampler.domain"></stringProp>
          <stringProp name="HTTPSampler.port"></stringProp>
          <stringProp name="HTTPSampler.connect_timeout"></stringProp>
          <stringProp name="HTTPSampler.response_timeout"></stringProp>
          <stringProp name="HTTPSampler.protocol"></stringProp>
          <stringProp name="HTTPSampler.contentEncoding"></stringProp>
          <stringProp name="HTTPSampler.path">/web/guest</stringProp>
          <stringProp name="HTTPSampler.method">GET</stringProp>
          <boolProp name="HTTPSampler.follow_redirects">true</boolProp>
          <boolProp name="HTTPSampler.auto_redirects">false</boolProp>
          <boolProp name="HTTPSampler.use_keepalive">true</boolProp>
          <boolProp name="HTTPSampler.DO_MULTIPART_POST">false</boolProp>
          <boolProp name="HTTPSampler.monitor">false</boolProp>
          <stringProp name="HTTPSampler.embedded_url_re"></stringProp>
        </HTTPSampler>
        <hashTree>
          <HeaderManager guiclass="HeaderPanel" testclass="HeaderManager" testname="HTTP Header Manager" enabled="true">
            <collectionProp name="HeaderManager.headers">
              <elementProp name="Accept-Language" elementType="Header">
                <stringProp name="Header.name">Accept-Language</stringProp>
                <stringProp name="Header.value">en-us,en;q=0.5</stringProp>
              </elementProp>
              <elementProp name="Accept" elementType="Header">
                <stringProp name="Header.name">Accept</stringProp>
                <stringProp name="Header.value">text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8</stringProp>
              </elementProp>
              <elementProp name="Keep-Alive" elementType="Header">
                <stringProp name="Header.name">Keep-Alive</stringProp>
                <stringProp name="Header.value">115</stringProp>
              </elementProp>
              <elementProp name="User-Agent" elementType="Header">
                <stringProp name="Header.name">User-Agent</stringProp>
                <stringProp name="Header.value">Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2.12) Gecko/20101026 Firefox/3.6.12 GTB7.1</stringProp>
              </elementProp>
              <elementProp name="Accept-Encoding" elementType="Header">
                <stringProp name="Header.name">Accept-Encoding</stringProp>
                <stringProp name="Header.value">gzip,deflate</stringProp>
              </elementProp>
              <elementProp name="Accept-Charset" elementType="Header">
                <stringProp name="Header.name">Accept-Charset</stringProp>
                <stringProp name="Header.value">ISO-8859-1,utf-8;q=0.7,*;q=0.7</stringProp>
              </elementProp>
            </collectionProp>
          </HeaderManager>
          <hashTree/>
        </hashTree>
        <HTTPSampler guiclass="HttpTestSampleGui" testclass="HTTPSampler" testname="/html/js/barebone.jsp" enabled="true">
          <elementProp name="HTTPsampler.Arguments" elementType="Arguments" guiclass="HTTPArgumentsPanel" testclass="Arguments" enabled="true">
            <collectionProp name="Arguments.arguments">
              <elementProp name="browserId" elementType="HTTPArgument">
                <boolProp name="HTTPArgument.always_encode">false</boolProp>
                <stringProp name="Argument.name">browserId</stringProp>
                <stringProp name="Argument.value">firefox</stringProp>
                <stringProp name="Argument.metadata">=</stringProp>
              </elementProp>
              <elementProp name="themeId" elementType="HTTPArgument">
                <boolProp name="HTTPArgument.always_encode">false</boolProp>
                <stringProp name="Argument.name">themeId</stringProp>
                <stringProp name="Argument.value">005E82_WAR_005E82theme</stringProp>
                <stringProp name="Argument.metadata">=</stringProp>
              </elementProp>
              <elementProp name="colorSchemeId" elementType="HTTPArgument">
                <boolProp name="HTTPArgument.always_encode">false</boolProp>
                <stringProp name="Argument.name">colorSchemeId</stringProp>
                <stringProp name="Argument.value">01</stringProp>
                <stringProp name="Argument.metadata">=</stringProp>
              </elementProp>
              <elementProp name="minifierType" elementType="HTTPArgument">
                <boolProp name="HTTPArgument.always_encode">false</boolProp>
                <stringProp name="Argument.name">minifierType</stringProp>
                <stringProp name="Argument.value">js</stringProp>
                <stringProp name="Argument.metadata">=</stringProp>
              </elementProp>
              <elementProp name="minifierBundleId" elementType="HTTPArgument">
                <boolProp name="HTTPArgument.always_encode">false</boolProp>
                <stringProp name="Argument.name">minifierBundleId</stringProp>
                <stringProp name="Argument.value">javascript.barebone.files</stringProp>
                <stringProp name="Argument.metadata">=</stringProp>
              </elementProp>
              <elementProp name="t" elementType="HTTPArgument">
                <boolProp name="HTTPArgument.always_encode">false</boolProp>
                <stringProp name="Argument.name">t</stringProp>
                <stringProp name="Argument.value">1274823175000</stringProp>
                <stringProp name="Argument.metadata">=</stringProp>
              </elementProp>
            </collectionProp>
          </elementProp>
          <stringProp name="HTTPSampler.domain"></stringProp>
          <stringProp name="HTTPSampler.port"></stringProp>
          <stringProp name="HTTPSampler.connect_timeout"></stringProp>
          <stringProp name="HTTPSampler.response_timeout"></stringProp>
          <stringProp name="HTTPSampler.protocol"></stringProp>
          <stringProp name="HTTPSampler.contentEncoding"></stringProp>
          <stringProp name="HTTPSampler.path">/html/js/barebone.jsp</stringProp>
          <stringProp name="HTTPSampler.method">GET</stringProp>
          <boolProp name="HTTPSampler.follow_redirects">true</boolProp>
          <boolProp name="HTTPSampler.auto_redirects">false</boolProp>
          <boolProp name="HTTPSampler.use_keepalive">true</boolProp>
          <boolProp name="HTTPSampler.DO_MULTIPART_POST">false</boolProp>
          <boolProp name="HTTPSampler.monitor">false</boolProp>
          <stringProp name="HTTPSampler.embedded_url_re"></stringProp>
        </HTTPSampler>
        <hashTree>
          <HeaderManager guiclass="HeaderPanel" testclass="HeaderManager" testname="HTTP Header Manager" enabled="true">
            <collectionProp name="HeaderManager.headers">
              <elementProp name="Accept-Language" elementType="Header">
                <stringProp name="Header.name">Accept-Language</stringProp>
                <stringProp name="Header.value">en-us,en;q=0.5</stringProp>
              </elementProp>
              <elementProp name="Accept" elementType="Header">
                <stringProp name="Header.name">Accept</stringProp>
                <stringProp name="Header.value">*/*</stringProp>
              </elementProp>
              <elementProp name="Keep-Alive" elementType="Header">
                <stringProp name="Header.name">Keep-Alive</stringProp>
                <stringProp name="Header.value">115</stringProp>
              </elementProp>
              <elementProp name="User-Agent" elementType="Header">
                <stringProp name="Header.name">User-Agent</stringProp>
                <stringProp name="Header.value">Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2.12) Gecko/20101026 Firefox/3.6.12 GTB7.1</stringProp>
              </elementProp>
              <elementProp name="Accept-Encoding" elementType="Header">
                <stringProp name="Header.name">Accept-Encoding</stringProp>
                <stringProp name="Header.value">gzip,deflate</stringProp>
              </elementProp>
              <elementProp name="Accept-Charset" elementType="Header">
                <stringProp name="Header.name">Accept-Charset</stringProp>
                <stringProp name="Header.value">ISO-8859-1,utf-8;q=0.7,*;q=0.7</stringProp>
              </elementProp>
            </collectionProp>
          </HeaderManager>
          <hashTree/>
        </hashTree>
        <HTTPSampler guiclass="HttpTestSampleGui" testclass="HTTPSampler" testname="/image/company_logo" enabled="true">
          <elementProp name="HTTPsampler.Arguments" elementType="Arguments" guiclass="HTTPArgumentsPanel" testclass="Arguments" enabled="true">
            <collectionProp name="Arguments.arguments">
              <elementProp name="img_id" elementType="HTTPArgument">
                <boolProp name="HTTPArgument.always_encode">false</boolProp>
                <stringProp name="Argument.name">img_id</stringProp>
                <stringProp name="Argument.value">10303</stringProp>
                <stringProp name="Argument.metadata">=</stringProp>
              </elementProp>
              <elementProp name="amp;t" elementType="HTTPArgument">
                <boolProp name="HTTPArgument.always_encode">false</boolProp>
                <stringProp name="Argument.name">amp;t</stringProp>
                <stringProp name="Argument.value">1289581656500</stringProp>
                <stringProp name="Argument.metadata">=</stringProp>
              </elementProp>
            </collectionProp>
          </elementProp>
          <stringProp name="HTTPSampler.domain"></stringProp>
          <stringProp name="HTTPSampler.port"></stringProp>
          <stringProp name="HTTPSampler.connect_timeout"></stringProp>
          <stringProp name="HTTPSampler.response_timeout"></stringProp>
          <stringProp name="HTTPSampler.protocol"></stringProp>
          <stringProp name="HTTPSampler.contentEncoding"></stringProp>
          <stringProp name="HTTPSampler.path">/image/company_logo</stringProp>
          <stringProp name="HTTPSampler.method">GET</stringProp>
          <boolProp name="HTTPSampler.follow_redirects">true</boolProp>
          <boolProp name="HTTPSampler.auto_redirects">false</boolProp>
          <boolProp name="HTTPSampler.use_keepalive">true</boolProp>
          <boolProp name="HTTPSampler.DO_MULTIPART_POST">false</boolProp>
          <boolProp name="HTTPSampler.monitor">false</boolProp>
          <stringProp name="HTTPSampler.embedded_url_re"></stringProp>
        </HTTPSampler>
        <hashTree>
          <HeaderManager guiclass="HeaderPanel" testclass="HeaderManager" testname="HTTP Header Manager" enabled="true">
            <collectionProp name="HeaderManager.headers">
              <elementProp name="Accept-Language" elementType="Header">
                <stringProp name="Header.name">Accept-Language</stringProp>
                <stringProp name="Header.value">en-us,en;q=0.5</stringProp>
              </elementProp>
              <elementProp name="Accept" elementType="Header">
                <stringProp name="Header.name">Accept</stringProp>
                <stringProp name="Header.value">image/png,image/*;q=0.8,*/*;q=0.5</stringProp>
              </elementProp>
              <elementProp name="Keep-Alive" elementType="Header">
                <stringProp name="Header.name">Keep-Alive</stringProp>
                <stringProp name="Header.value">115</stringProp>
              </elementProp>
              <elementProp name="User-Agent" elementType="Header">
                <stringProp name="Header.name">User-Agent</stringProp>
                <stringProp name="Header.value">Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2.12) Gecko/20101026 Firefox/3.6.12 GTB7.1</stringProp>
              </elementProp>
              <elementProp name="Accept-Encoding" elementType="Header">
                <stringProp name="Header.name">Accept-Encoding</stringProp>
                <stringProp name="Header.value">gzip,deflate</stringProp>
              </elementProp>
              <elementProp name="Accept-Charset" elementType="Header">
                <stringProp name="Header.name">Accept-Charset</stringProp>
                <stringProp name="Header.value">ISO-8859-1,utf-8;q=0.7,*;q=0.7</stringProp>
              </elementProp>
            </collectionProp>
          </HeaderManager>
          <hashTree/>
        </hashTree>
        <HTTPSampler guiclass="HttpTestSampleGui" testclass="HTTPSampler" testname="/web/guest/home?p_p_id=58&amp;p_p_lifecycle=1&amp;p_p_state=normal&amp;p_p_mode=view&amp;p_p_col_id=column-1&amp;p_p_col_count=1&amp;saveLastPath=0&amp;_58_struts_action=%2Flogin%2Flogin" enabled="true">
          <elementProp name="HTTPsampler.Arguments" elementType="Arguments" guiclass="HTTPArgumentsPanel" testclass="Arguments" enabled="true">
            <collectionProp name="Arguments.arguments">
              <elementProp name="_58_redirect" elementType="HTTPArgument">
                <boolProp name="HTTPArgument.always_encode">false</boolProp>
                <stringProp name="Argument.name">_58_redirect</stringProp>
                <stringProp name="Argument.value"></stringProp>
                <stringProp name="Argument.metadata">=</stringProp>
                <boolProp name="HTTPArgument.use_equals">true</boolProp>
              </elementProp>
              <elementProp name="_58_rememberMe" elementType="HTTPArgument">
                <boolProp name="HTTPArgument.always_encode">false</boolProp>
                <stringProp name="Argument.name">_58_rememberMe</stringProp>
                <stringProp name="Argument.value">on</stringProp>
                <stringProp name="Argument.metadata">=</stringProp>
                <boolProp name="HTTPArgument.use_equals">true</boolProp>
              </elementProp>
              <elementProp name="_58_login" elementType="HTTPArgument">
                <boolProp name="HTTPArgument.always_encode">false</boolProp>
                <stringProp name="Argument.name">_58_login</stringProp>
                <stringProp name="Argument.value">nurquiza@sample.com</stringProp>
                <stringProp name="Argument.metadata">=</stringProp>
                <boolProp name="HTTPArgument.use_equals">true</boolProp>
              </elementProp>
              <elementProp name="_58_password" elementType="HTTPArgument">
                <boolProp name="HTTPArgument.always_encode">false</boolProp>
                <stringProp name="Argument.name">_58_password</stringProp>
                <stringProp name="Argument.value">test</stringProp>
                <stringProp name="Argument.metadata">=</stringProp>
                <boolProp name="HTTPArgument.use_equals">true</boolProp>
              </elementProp>
            </collectionProp>
          </elementProp>
          <stringProp name="HTTPSampler.domain"></stringProp>
          <stringProp name="HTTPSampler.port"></stringProp>
          <stringProp name="HTTPSampler.connect_timeout"></stringProp>
          <stringProp name="HTTPSampler.response_timeout"></stringProp>
          <stringProp name="HTTPSampler.protocol"></stringProp>
          <stringProp name="HTTPSampler.contentEncoding"></stringProp>
          <stringProp name="HTTPSampler.path">/web/guest/home?p_p_id=58&amp;p_p_lifecycle=1&amp;p_p_state=normal&amp;p_p_mode=view&amp;p_p_col_id=column-1&amp;p_p_col_count=1&amp;saveLastPath=0&amp;_58_struts_action=%2Flogin%2Flogin</stringProp>
          <stringProp name="HTTPSampler.method">POST</stringProp>
          <boolProp name="HTTPSampler.follow_redirects">true</boolProp>
          <boolProp name="HTTPSampler.auto_redirects">false</boolProp>
          <boolProp name="HTTPSampler.use_keepalive">true</boolProp>
          <boolProp name="HTTPSampler.DO_MULTIPART_POST">false</boolProp>
          <boolProp name="HTTPSampler.monitor">false</boolProp>
          <stringProp name="HTTPSampler.embedded_url_re"></stringProp>
        </HTTPSampler>
        <hashTree>
          <HeaderManager guiclass="HeaderPanel" testclass="HeaderManager" testname="HTTP Header Manager" enabled="true">
            <collectionProp name="HeaderManager.headers">
              <elementProp name="Content-Type" elementType="Header">
                <stringProp name="Header.name">Content-Type</stringProp>
                <stringProp name="Header.value">application/x-www-form-urlencoded</stringProp>
              </elementProp>
              <elementProp name="Accept-Language" elementType="Header">
                <stringProp name="Header.name">Accept-Language</stringProp>
                <stringProp name="Header.value">en-us,en;q=0.5</stringProp>
              </elementProp>
              <elementProp name="Accept" elementType="Header">
                <stringProp name="Header.name">Accept</stringProp>
                <stringProp name="Header.value">text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8</stringProp>
              </elementProp>
              <elementProp name="Keep-Alive" elementType="Header">
                <stringProp name="Header.name">Keep-Alive</stringProp>
                <stringProp name="Header.value">115</stringProp>
              </elementProp>
              <elementProp name="User-Agent" elementType="Header">
                <stringProp name="Header.name">User-Agent</stringProp>
                <stringProp name="Header.value">Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2.12) Gecko/20101026 Firefox/3.6.12 GTB7.1</stringProp>
              </elementProp>
              <elementProp name="Accept-Encoding" elementType="Header">
                <stringProp name="Header.name">Accept-Encoding</stringProp>
                <stringProp name="Header.value">gzip,deflate</stringProp>
              </elementProp>
              <elementProp name="Accept-Charset" elementType="Header">
                <stringProp name="Header.name">Accept-Charset</stringProp>
                <stringProp name="Header.value">ISO-8859-1,utf-8;q=0.7,*;q=0.7</stringProp>
              </elementProp>
            </collectionProp>
          </HeaderManager>
          <hashTree/>
        </hashTree>
      </hashTree>
      <ResultCollector guiclass="ViewResultsFullVisualizer" testclass="ResultCollector" testname="View Results Tree" enabled="true">
        <boolProp name="ResultCollector.error_logging">false</boolProp>
        <objProp>
          <name>saveConfig</name>
          <value class="SampleSaveConfiguration">
            <time>true</time>
            <latency>true</latency>
            <timestamp>true</timestamp>
            <success>true</success>
            <label>true</label>
            <code>true</code>
            <message>true</message>
            <threadName>true</threadName>
            <dataType>true</dataType>
            <encoding>false</encoding>
            <assertions>true</assertions>
            <subresults>true</subresults>
            <responseData>false</responseData>
            <samplerData>false</samplerData>
            <xml>true</xml>
            <fieldNames>false</fieldNames>
            <responseHeaders>false</responseHeaders>
            <requestHeaders>false</requestHeaders>
            <responseDataOnError>false</responseDataOnError>
            <saveAssertionResultsFailureMessage>false</saveAssertionResultsFailureMessage>
            <assertionsResultsToSave>0</assertionsResultsToSave>
            <bytes>true</bytes>
          </value>
        </objProp>
        <stringProp name="filename"></stringProp>
      </ResultCollector>
      <hashTree/>
      <CookieManager guiclass="CookiePanel" testclass="CookieManager" testname="HTTP Cookie Manager" enabled="true">
        <collectionProp name="CookieManager.cookies"/>
        <boolProp name="CookieManager.clearEachIteration">false</boolProp>
        <stringProp name="CookieManager.policy">rfc2109</stringProp>
      </CookieManager>
      <hashTree/>
      <ConfigTestElement guiclass="HttpDefaultsGui" testclass="ConfigTestElement" testname="HTTP Request Defaults" enabled="true">
        <elementProp name="HTTPsampler.Arguments" elementType="Arguments" guiclass="HTTPArgumentsPanel" testclass="Arguments" testname="User Defined Variables" enabled="true">
          <collectionProp name="Arguments.arguments"/>
        </elementProp>
        <stringProp name="HTTPSampler.domain">portal.sample.com</stringProp>
        <stringProp name="HTTPSampler.port">443</stringProp>
        <stringProp name="HTTPSampler.connect_timeout"></stringProp>
        <stringProp name="HTTPSampler.response_timeout"></stringProp>
        <stringProp name="HTTPSampler.protocol">https</stringProp>
        <stringProp name="HTTPSampler.contentEncoding"></stringProp>
        <stringProp name="HTTPSampler.path"></stringProp>
      </ConfigTestElement>
      <hashTree/>
      <ResultCollector guiclass="TableVisualizer" testclass="ResultCollector" testname="View Results in Table" enabled="true">
        <boolProp name="ResultCollector.error_logging">false</boolProp>
        <objProp>
          <name>saveConfig</name>
          <value class="SampleSaveConfiguration">
            <time>true</time>
            <latency>true</latency>
            <timestamp>true</timestamp>
            <success>true</success>
            <label>true</label>
            <code>true</code>
            <message>true</message>
            <threadName>true</threadName>
            <dataType>true</dataType>
            <encoding>false</encoding>
            <assertions>true</assertions>
            <subresults>true</subresults>
            <responseData>false</responseData>
            <samplerData>false</samplerData>
            <xml>true</xml>
            <fieldNames>false</fieldNames>
            <responseHeaders>false</responseHeaders>
            <requestHeaders>false</requestHeaders>
            <responseDataOnError>false</responseDataOnError>
            <saveAssertionResultsFailureMessage>false</saveAssertionResultsFailureMessage>
            <assertionsResultsToSave>0</assertionsResultsToSave>
            <bytes>true</bytes>
          </value>
        </objProp>
        <stringProp name="filename"></stringProp>
      </ResultCollector>
      <hashTree/>
      <ResultCollector guiclass="GraphVisualizer" testclass="ResultCollector" testname="Graph Results" enabled="true">
        <boolProp name="ResultCollector.error_logging">false</boolProp>
        <objProp>
          <name>saveConfig</name>
          <value class="SampleSaveConfiguration">
            <time>true</time>
            <latency>true</latency>
            <timestamp>true</timestamp>
            <success>true</success>
            <label>true</label>
            <code>true</code>
            <message>true</message>
            <threadName>true</threadName>
            <dataType>true</dataType>
            <encoding>false</encoding>
            <assertions>true</assertions>
            <subresults>true</subresults>
            <responseData>false</responseData>
            <samplerData>false</samplerData>
            <xml>true</xml>
            <fieldNames>false</fieldNames>
            <responseHeaders>false</responseHeaders>
            <requestHeaders>false</requestHeaders>
            <responseDataOnError>false</responseDataOnError>
            <saveAssertionResultsFailureMessage>false</saveAssertionResultsFailureMessage>
            <assertionsResultsToSave>0</assertionsResultsToSave>
            <bytes>true</bytes>
          </value>
        </objProp>
        <stringProp name="filename"></stringProp>
      </ResultCollector>
      <hashTree/>
    </hashTree>
  </hashTree>
</jmeterTestPlan>

There are a few important things you need to know to keep a Jmeter test growing and usable for your needs.
  1. Thread Group: Besides simulating a specific number of users do use a ramp up period that makes sense. Any system even Google can be a victim of denial of service so do test real non-attack case scenarios. This means it is unlikely (if not practically impossible) the 100 users end up hitting your server at the same time. As you see I have distributed the load in 10 seconds.
  2. See how I include pulling resources like the logo besides hitting the home page and trying to login into the website. Resources are cached so it is up to you to decide if they should be included in a stress test. For example if you launched a marketing campaign and your users will be new all resources will be downloaded. Contrary to the common core developers believe front end code is usually the bottleneck in many web applications
  3. Always use a "Default Requests Defaults Config Element" so you can easily use the same test against your local environment and real servers (Be sure you do not hit real production during high traffic hours or you will end up affecting real users)
  4. Look into the server logs looking for exceptions
  5. Think about simulating cases where local sample data is affected by several different users. In this case I have added only one user but you can use expressions and build different users on the fly or harcode them in different threads. Analyze the data after running the tests looking for inconsistencies.
  6. Use a Cookie Manager so you maintain the session between requests
  7. Run the test first with one user and analyze all responses. There are several views, the "View Results Tree Listener" is your friend to see request and response of each request.
  8. Use a "View Results in Table Listener" for a quick inspection about response delays. It helps your identify botlenecks.
  9. Use Graph results to get statistics. Be sure you understand what average, median and deviation is. Use a larger user base for example 1000 users distributed through 2 minutes or a little more to be sure the statistics have a better meaning. The more samples the better as the server will be warmed up simulating better those enjoying a large uptime
  10. Use a random timer after each request. My example does not include them but you can easily add it from Menu | Edit | Add. That ensures a closer to reality scenario. A user will not be hitting every second your website.
  11. Simulate a denial of service attack and see if you are prepared for it in the case you are not handling that at firewall layer (recommended)
  12. There is so much more you can do including creating the test cases from a normal browser navigation using a Proxy

The SDLC is a long circle and testing is an indispensable part of it. Stress tests or load tests are important, way more than what many developers think. Take control and stress the application, don't let the application stress yourself.

Tuesday, January 18, 2011

Liferay MVC with JSPPortlet

JSPPortlet can be used to quickly develop a Liferay SDK plugin type portlet or even as I demonstrated before migrate one of the existing native portlets to the more agile plugin environment.

My example for the update password portlet was actually still using the original Struts action (UpdatePasswordAction.java).

When further customization is needed we face a problem: Parameters sent to one portlet cannot be retrieved from a second portlet. This is simply correct for a JSR-168 specification implementation. You could go around this issue using JSR-286 public render parameters but there is an alternative which in this case is actually cleaner: Use JSPPortlet MVC capabilities to migrate the extension environment struts portlet java code to a plugin environment implementation.

So let us change then the portlet to show a results page (view) when the passwordis changed instead of keeping the form (edit).

I have committed the code to SVN but for the sake of clarity here are the changes I have made to the Mavenized SDK Plugin Update Password Portlet:
  1. Use two JSPs (view and edit)
            <init-param>
                <name>view-jsp</name>
                <value>/update_password_view.jsp</value>
            </init-param>
            <init-param>
                <name>edit-jsp</name>
                <value>/update_password_edit.jsp</value>
            </init-param>
    
  2. Make UpdatePasswordJSPPortlet inherit from the typical com.sample.jsp.portlet.JSPPortlet. The code should be self-explanatory
    package com.sample.jsp.portlet;
    
    import java.io.IOException;
    
    import javax.portlet.ActionRequest;
    import javax.portlet.ActionResponse;
    import javax.portlet.PortletException;
    import javax.portlet.PortletSession;
    import javax.portlet.RenderRequest;
    import javax.portlet.RenderResponse;
    import javax.servlet.http.HttpServletRequest;
    
    import org.apache.commons.logging.Log;
    import org.apache.commons.logging.LogFactory;
    
    import com.liferay.portal.NoSuchUserException;
    import com.liferay.portal.UserPasswordException;
    import com.liferay.portal.kernel.servlet.SessionErrors;
    import com.liferay.portal.kernel.util.ParamUtil;
    import com.liferay.portal.kernel.util.Validator;
    import com.liferay.portal.security.auth.PrincipalException;
    import com.liferay.portal.service.UserServiceUtil;
    import com.liferay.portal.util.PortalUtil;
    
    public class UpdatePasswordJSPPortlet extends JSPPortlet {
        private static final String CMD = "cmd";
    
        @Override
        public void doDispatch(
                RenderRequest renderRequest, RenderResponse renderResponse)
                throws IOException, PortletException {
            HttpServletRequest request = PortalUtil.getHttpServletRequest(renderRequest);
            HttpServletRequest originalRequest = PortalUtil.getOriginalServletRequest(request);
    
            String cmd = originalRequest.getParameter(CMD);
            if(Validator.isNull(cmd)) {
                include(editJSP, renderRequest, renderResponse);
            } else {
                if(SessionErrors.isEmpty(request)) {
                    include(viewJSP, renderRequest, renderResponse);
                }else{
                    include(editJSP, renderRequest, renderResponse);
                }
            }
        }
    
        
        @Override
        public void processAction(ActionRequest actionRequest,
            ActionResponse actionResponse) throws IOException, PortletException {
            
            String cmd = ParamUtil.getString(actionRequest, CMD);
    
            if (Validator.isNull(cmd)) {
                    actionResponse.setRenderParameter("error", "cmd is null");
                    return;
            }
            
            HttpServletRequest request = PortalUtil.getHttpServletRequest(actionRequest);
            SessionErrors.clear(request);
            try {
                    updatePassword(actionRequest, actionResponse);
                    return;
            }
            catch (Exception e) {
                    if (e instanceof UserPasswordException) {
                            SessionErrors.add(request, e.getClass().getName(), e);
                    }
                    else if (e instanceof NoSuchUserException ||
                                     e instanceof PrincipalException) {
                            SessionErrors.add(request, e.getClass().getName());
                    }
                    else {
                            PortalUtil.sendError(e, actionRequest, actionResponse);
                    }
            }
        }
        
        private void updatePassword(ActionRequest request, ActionResponse response)
                throws Exception {
    
            PortletSession session = request.getPortletSession();
    
            long userId = PortalUtil.getUserId(request);
            String password1 = ParamUtil.getString(request, "password1");
            String password2 = ParamUtil.getString(request, "password2");
            boolean passwordReset = false;
    
            UserServiceUtil.updatePassword(userId, password1, password2,
                    passwordReset);
    
            session.setAttribute("USER_PASSWORD", password1);
        }
    
        private static final Log log = LogFactory.getLog(UpdatePasswordJSPPortlet.class);
    }
    
    

Monday, January 17, 2011

Eclipse and Netbeans Agile JSP Development

I have used rsync, maven and shell/ant scripts to deploy JSP files into the tomcat exploded application folder but if you are using Eclipse then you cannot miss the filesync plugin.

This is simply JSP development/deployment done the right way.

On one side you can hot deploy your JSPs to any application server supporting exploded deployment (who doesn't nowadays) and at the same time you just save the file in Eclipse and your changes are pushed into the server.

This is perfect for Servlet development, however for Liferay Portlet development for example it won't work as Liferay uses specific folders to store the JSPs with the application server. A similar plugin but running some kind of custom action "on save" could resolve that problem but Liferay has tools of its own nowadays to deal with hot deployment.

Netbeans currently does not have a plugin like Eclipse filesync, however "JSP deployment on save" is accomplished out of the box in a per supported server scenario. Let us see how you configure it for tomcat 6:
  1. Add a user and role for Tomcat Manager:
    $ vi conf/tomcat-users.xml
    <role rolename="manager"/>
    <user username="tomcat" password="tomcat" roles="manager"/>
    
  2. In Netbeans go to Services | Servers | Add Server | point to the tomcat root directory and supply tomcat/tomcat for username/password
  3. Right click on the Tomcat 6 Server entry and select Run. If the server was not running Netbeans will start it
  4. Any JSP changes will be deployed in the server. To see how to hot deploy java code without restarting the server refer to my previous post

Sunday, January 16, 2011

Spring thread safe injection

This is a problem that hits all teams working with Spring and in general Servlet programming.

The developer tests locally and even in an integration server with some users hitting the application. Everything looks great but when traffic gets a little bit serious all kind of problems arise.

To find out if the program is thread safe you can use smart load tests with Jakarta JMeter but it would be time consuming. It would be ideal if a Unit test could simply make the build fail in case a potential thread safe problem is detected.

One of the advantages of using Spring injected singleton beans is the application heap footprint, so light, so fast ...

But the developer must be aware of the fact that every @Service, @Repository and any injected resource must be thread safe. That is the developer responsibility. A simple instance (class) member could result in a total application disaster.

So here is a JUnit test you can use to find out thread safe problems in your Spring code. Just modify it to take into account the exceptions to the rule: If an instance field is needed it should be either thread-safe or it must be declared final and static.
package com.nestorurquiza;

import java.lang.annotation.Annotation;
import java.lang.reflect.Field;
import java.lang.reflect.Modifier;
import java.util.List;
import java.util.Set;

import net.sourceforge.stripes.util.ResolverUtil;

import org.junit.Test;
import org.junit.runner.RunWith;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.stereotype.Controller;
import org.springframework.stereotype.Repository;
import org.springframework.stereotype.Service;
import org.springframework.test.context.ContextConfiguration;
import org.springframework.test.context.junit4.SpringJUnit4ClassRunner;

@RunWith(SpringJUnit4ClassRunner.class)
@ContextConfiguration(locations = {"classpath:spring-config.xml"})
public class ThreadSafeTest {
    
    private static final Logger log = LoggerFactory.getLogger(ThreadSafeTest.class);
    
    @Test
    public void testInstanceVariables() {
        ResolverUtil<Object> resolver = new ResolverUtil<Object>();
        resolver.findAnnotated(Repository.class, "com.nestorurquiza.dao");
        Set<Class<? extends Object>> classes = resolver.getClasses();
        resolver.findAnnotated(Service.class, "com.nestorurquiza.service");
        classes.retainAll(resolver.getClasses());
        resolver.findAnnotated(Controller.class, "com.nestorurquiza.web");
        classes.retainAll(resolver.getClasses());
        List<String> errors;
        for (Class<? extends Object> controller : classes) {
            Field[] fields = controller.getDeclaredFields();
            for(Field field : fields) {
                Annotation[] annotations = field.getAnnotations();
                boolean isThreadSafe = false;
                for(Annotation annotation : annotations) {
                    String annotationName = annotation.annotationType().getName();
                    if(annotationName.equals("org.springframework.beans.factory.annotation.Autowired") || annotationName.equals("javax.persistence.PersistenceContext")) {
                        isThreadSafe = true;
                    }
                }
                int mod = field.getModifiers();
                boolean noError = isThreadSafe || Modifier.isFinal(mod) && Modifier.isStatic(mod);
                String errorDescription = controller.getName() + " is not thread-safe because of field " + field.getName();
                //TODO: Activate the assertion once all problems are corrected to ensure this problem does not hit the team again
                if(!noError) {
                    log.error(errorDescription);
                }
                //assertTrue(errorDescription, noError);
            }
        }
    }
}

Thursday, January 13, 2011

Importance of a Deployment plan

As part of the SDLC an application grows beyond the code changes. This includes DB, setting files, script files and so on.

Ideally if you have the luxury of a BA s(he) will ensure all the necessary areas work together to achieve a successful deployment but regardless as a software developer you are in charge of creating your deployment plan.

One efficient way to achieve this is a semi-automated way is to keep a file with the changes as part of a "misc" folder in your application.

The folder of course is excluded from the final application package. If you are using Java and Maven this is the default behavior.

The file should get tagged automatically with your release. Again Maven's default.

The rest is about a little discipline. Once you release you should delete the content of the file. I am unaware of a Maven plugin for this but definitely something that can be achieved through scripting.

Wednesday, January 12, 2011

Spring log not showing Runtime Exceptions

I had to increase log level to DEBUG to find out that Spring was swallowing an important SQLException.

Commonly you use com.nestorurquiza.web.handler.SimpleMappingExceptionResolver but the exception gets logged with DEBUG level instead.

To solve this problem I extended the class with the code shown below:
public class CustomSimpleMappingExceptionResolver extends SimpleMappingExceptionResolver{
    @Override
    public ModelAndView resolveException(HttpServletRequest request,
            HttpServletResponse response, Object handler, Exception ex) {
        final Logger log = LoggerFactory.getLogger(CustomSimpleMappingExceptionResolver.class);
        log.error(Utils.stack2string(ex));
        return super.resolveException(request, response, handler, ex);
    }
}

I have to say it again application monitoring should be done from outside and never from inside. Swallowing the exception to rely on things like an SMTP appender and so on break that statement. If email fails and you are dealing with external users most likely the user will complain about a problem you are simply unable to see on your back end. Be aware!

Tuesday, January 11, 2011

Eclipse or Netbeans HotSwap for the Agile Java Developer

HotSwap technology has been there for a while yet I still find many developers not taking advantage of it.

Let us see the typical non agile Java development:
  1. Change (Code is changed. Building can be configured to be automated)
  2. Deploy (Can be automated but at least the application will reload, sometimes even the server must be restarted)
  3. Setup (Deal with the specific test case probably navigating some application states before getting to the point)
  4. Test (the change, the feature, the bug fix etc)

I do not want to enter the discussion between compiled vs interpreted or static vs. dynamic and yet I want to be as fast as possible when fixing a bug, changing some logic or adding a quick feature. Java HotSwap technology (backed by JPDA) addresses the issue allowing you in many cases to just:
  1. Change
  2. Test

There are limitations (for example you cannot change a method signature) but there is work in progress to make HotSwap better. UPDATE: In fact as of June 2014 there is a free full solution for this problem which I have tested for Linux and jdk 1.7 but for sure supported in other OS and JVM versions as well: DCEVM + HotswapAgent

In any case the amount of time HotSwap saves can be huge especially if you get smart at debugging (thing that you will do in order to avoid deploy and setup phases).

I have used HotSwap now for years in Eclipse. My current team uses Netbeans and I was tempted to just do the same just to realize that was not straightforward at least on version 6.9.1. However not impossible.

While Eclipse will accept change and test (fix and continue) with just "CTRL+S" Netbeans needs "CTRL+S | Menu -> Debug -> Apply Code changes". However following the steps below you can get "fix and continue" or "change and test" working in Netbeans as well:
  1. From Project properties select "build | compile | Compile on Save | For both ... "
  2. From Netbeans preferences select "Miscelaneous | Java Debugger | General | Apply code changes after save"



It is worth to be said JRebel rocks. Not an open source project but not that expensive either. As far as I am concerned I tend to address those parts that cause 90% of the problem first. Following that logic I have found myself spending way more time troubleshooting than actually developing. So definitely if you are performing big changes in your code, you are probably better using junit/selenium for quick initial development in which case the impact of not having the luxury to change/add method signatures, class fields and so on would not be that big after all. But if you are serious about Plain Java Web Development then try JRebel or if you are low on budget and not working on an open source project I definitely recommend (again) DCEVM + HotSwapAagent.

Of course I am assuming you are connecting to the JVM debugging port. For example I commonly have the below in my tomcat setenv.sh (see differences for pre and post JDK 5):
#debugger < JDK 5
#JAVA_OPTS="$JAVA_OPTS -Xrunjdwp:transport=dt_socket,address=8787,server=y,suspend=n"
#debugger >= JDK 5
JAVA_OPTS="$JAVA_OPTS -agentlib:jdwp=transport=dt_socket,address=8000,server=y,suspend=n"

Then from Eclipse or Netbeans I connect to the remote port using JPDA with SocketAttach connector. For Netbeans this is done from "Menu | Debug | Attach Debugger"

Finally for those loving dynamic programming take a look at the cool features from the Da Vinci Machine coming up as part of the JDK 7 release.

Friday, January 07, 2011

Use LogMonitor to email log errors

Inspecting application server logs to act proactively is a must do for developers. In their local environments they should inspect their logs before committing to the VCS.

The question is what is happening in other environments, especially in production. How the developer can be promptly notified about an issue.

I am not sure if I am just so "lucky" to find so many tech guys proposing monitoring the logs of applications from within the applications. It is simply like saying I am my own Doctor, period.

In my opinion log monitoring should be done from the outside.

Below are the instructions to get any log and send an email when a line meets certain criteria.
  1. Install http://www.logix.cz/michal/devel/smtp-cli/ perl script to send email. I haven't seen so far a most powerful unix sendmail command than this script.
  2. sudo vi /usr/sbin/smtp-cli
    sudo chmod +x /usr/sbin/smtp-cli
    sudo apt-get install libio-socket-ssl-perl  libdigest-hmac-perl  libterm-readkey-perl libmime-lite-perl libfile-type-perl libio-socket-inet6-perl
    
  3. Be sure logtail is installed in your system.
    apt-get install logtail
    
  4. Before creating the logMonitor script let us be sure we do not get too much information with the first email sent. Run just the logtail command, for example:
    apt-get install logtail
    logtail -f /opt/tomcat-6.0.18/logs/catalina.out -o /opt/tomcat-6.0.18/catalina.out.offset
    





  5. Create a logMonitor script
    #!/bin/bash
    #!/bin/bash
    #
    # @fileName: logMonitor.sh:
    # @description: Sends an email every time a new PATTERN is found in a growing log file.
    # @author: Nestor Urquiza
    # @date: Jan 7, 2011
    #
    
    #
    # Constants
    #
    SMART_HOST=mail.nestorurquiza.com
    TEMP_FILE=`mktemp /tmp/logMonitor.XXXXXXXXXX`
    LINES_AFTER=5
    
    
    #
    # Functions
    #
    function usage {
      echo "Usage - $0 logFile  offsetFile includePattern excludePattern from to subject"
      echo "If you need no exclusions then simply put some string you know will not be matched"
      exit 1
    }
    
    #
    # Main program
    #
    if [ $# -lt 7 ]
    then
      usage
    fi
    
    logFile=$1
    offsetFile=$2
    includePattern=$3
    excludePattern=$4
    from=$5
    to=$6
    subject=$7
    
    command="/usr/sbin/logtail -f $logFile -o $offsetFile | egrep -v \"$excludePattern\"  | egrep \"$includePattern\" -A $LINES_AFTER > $TEMP_FILE"
    echo "Running: $command"
    eval $command
    echo "Result: `cat $TEMP_FILE`"
    
    hostname=`/bin/hostname`
    #if [ -n "$body" ]
    if [ -s $TEMP_FILE ]; then
      command="/usr/sbin/smtp-cli --host=$SMART_HOST --from $from --to $to --subject \"$hostname: $subject\" --body-plain \"$TEMP_FILE\""
      echo "Running: $command"
      eval $command
    fi
    rm $TEMP_FILE
    
    





  6. Cron your script to run let us say every five minutes
    */5 * * * * /usr/sbin/logMonitor /opt/tomcat-6.0.18/logs/catalina.out /opt/tomcat-6.0.18/catalina.out.offset "SEVERE|ERROR|WARN" "thingsIwantToExclude"  donotreply@sample.com nurquiza@sample.com "catalina.out problems"
    





Wednesday, January 05, 2011

Memory leaks: OutOfMemoryError: PermGen space

I have used JProfiler, Eclipse TPTP, a combination of jconsole and other command line tools in the past to resolve memory leaks problems. We are using JDK 6 and so Java Visual VM (jvisualvm command) can be used to troubleshoot what is going on.

I recall using JBoss as part of one of my assignments and concluding that a total refactoring of the code was needed after finding out memory leaks were most likely there to stay.

So I have seen people deciding to restart the servers every so often and even minimize the amount of deployments made to production just to avoid the inevitable error:
OutOfMemoryError: PermGen space

Regardless the fact that an application server itself might be buggy (and you can just Google about memory leaks for any Java application Server to see that actually all of them have suffered from this in the past) or even the JVM can be buggy there is a lot you can do before just going ahead and blaming others.

I decided to post the results of a task that involved 10 hours of work plus 12 hours of automated tests to get rid of memory leaks in an application deployed in tomcat. Basically after some deployments the server had to be restarted.

Below is a screenshot showing the initial situation. As you see PermGen consumption goes up with each deployment (steps in graphic) and it never goes down (Garbage Collection has no effect === memory leak)


Here are some of the errors I was getting from tomcat logs after each deployment:
Jan 3, 2011 5:24:07 PM org.apache.catalina.loader.WebappClassLoader clearReferencesJdbc
SEVERE: The web application [/] registered the JBDC driver [com.mysql.jdbc.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
Jan 3, 2011 5:24:07 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/] appears to have started a thread named [Thread-43] but has failed to stop it. This is very likely to create a memory leak.
Jan 3, 2011 5:24:07 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/] appears to have started a thread named [MultiThreadedHttpConnectionManager cleanup] but has failed to stop it. This is very likely to create a memory leak.
Jan 3, 2011 5:24:07 PM org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap
SEVERE: The web application [/] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@44d2963b]) and a value of type [org.springframework.security.core.context.SecurityContextImpl] (value [org.springframework.security.core.context.SecurityContextImpl@ffffffff: Null authentication]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak.
Jan 3, 2011 5:24:07 PM org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap
SEVERE: The web application [/] created a ThreadLocal with key of type [net.sf.json.AbstractJSON.CycleSet] (value [net.sf.json.AbstractJSON$CycleSet@5d851ec9]) and a value of type [java.lang.ref.SoftReference] (value [java.lang.ref.SoftReference@4ea84e16]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak.
Jan 3, 2011 5:24:07 PM org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap
SEVERE: The web application [/] created a ThreadLocal with key of type [org.apache.commons.lang.builder.HashCodeBuilder$1] (value [org.apache.commons.lang.builder.HashCodeBuilder$1@b9eaeb2]) and a value of type [java.util.HashSet] (value [[]]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak.
INFO: Illegal access: this web application instance has been stopped already.  Could not load org.apache.axis2.context.ConfigurationContext$1.  The eventual following stack trace is caused by an error thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access, and has no functional impact.
java.lang.IllegalStateException
 at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1531)
 at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1491)
 at org.apache.axis2.context.ConfigurationContext.cleanupTemp(ConfigurationContext.java:759)
 at org.apache.axis2.context.ConfigurationContext.terminate(ConfigurationContext.java:747)
 at org.apache.axis2.client.ServiceClient.cleanup(ServiceClient.java:816)
 at org.apache.axis2.client.ServiceClient.finalize(ServiceClient.java:795)
 at java.lang.ref.Finalizer.invokeFinalizeMethod(Native Method)
 at java.lang.ref.Finalizer.runFinalizer(Finalizer.java:83)
 at java.lang.ref.Finalizer.access$100(Finalizer.java:14)
 at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:160)
Exception in thread "MultiThreadedHttpConnectionManager cleanup" java.lang.NullPointerException
 at org.apache.commons.logging.impl.SLF4JLocationAwareLog.isDebugEnabled(SLF4JLocationAwareLog.java:59)
 at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ReferenceQueueThread.handleReference(MultiThreadedHttpConnectionManager.java:1105)
 at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ReferenceQueueThread.run(MultiThreadedHttpConnectionManager.java:1124)

 



Below are the results from jvisualvm thread dump
2011-01-04 14:45:40
Full thread dump Java HotSpot(TM) 64-Bit Server VM (17.1-b03-307 mixed mode):

"Thread-12" daemon prio=5 tid=102d68000 nid=0x1263b2000 in Object.wait() [1263b1000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
 at java.lang.Object.wait(Native Method)
 - waiting on <10d131898> (a org.enhydra.jdbc.pool.PoolKeeper)
 at org.enhydra.jdbc.pool.PoolKeeper.run(PoolKeeper.java:55)
 - locked <10d131898> (a org.enhydra.jdbc.pool.PoolKeeper)
 at java.lang.Thread.run(Thread.java:680)

   Locked ownable synchronizers:
 - None

 "Thread-15" daemon prio=5 tid=125539000 nid=0x129983000 runnable [129982000]
    java.lang.Thread.State: RUNNABLE
  at java.net.SocketInputStream.socketRead0(Native Method)
  at java.net.SocketInputStream.read(SocketInputStream.java:129)
  at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
  at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
  at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
  - locked <10bd35918> (a java.io.BufferedInputStream)
  at com.sun.jndi.ldap.Connection.run(Connection.java:808)
  at java.lang.Thread.run(Thread.java:680)

    Locked ownable synchronizers:
  - None
  
"JotmClock" daemon prio=5 tid=10190e000 nid=0x126db5000 waiting on condition [126db4000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
 at java.lang.Thread.sleep(Native Method)
 at org.objectweb.jotm.TimerManager.clock(TimerManager.java:171)
 at org.objectweb.jotm.Clock.run(TimerManager.java:65)

   Locked ownable synchronizers:
 - None

"JotmBatch" daemon prio=5 tid=101e9d000 nid=0x126cb2000 in Object.wait() [126cb1000]
   java.lang.Thread.State: WAITING (on object monitor)
 at java.lang.Object.wait(Native Method)
 - waiting on <10948f3c0> (a java.util.Vector)
 at java.lang.Object.wait(Object.java:485)
 at org.objectweb.jotm.TimerManager.batch(TimerManager.java:223)
 - locked <10948f3c0> (a java.util.Vector)
 at org.objectweb.jotm.Batch.run(TimerManager.java:87)

   Locked ownable synchronizers:
 - None

Clearly we have a combination of problems here.

  1. The mysql error is related to a DBCP memory leak bug which went away after switching to the newest jar version 1.4
  2. [Thread-15] for example could not be stopped because of a memory leak bug in apache commons lang which went away after using version 2.5
  3. JotmClock, JotmBatch and [Thread-12] are related to a combination of the DBCP leaking error
  4. Axis2 was upgraded to version 1.5.4 from version 1.4.1 which had a memory leak
  5. json-lib was upgraded to 2.4 because of memory leak in previous version
  6. The rest of the problems are related to an application bug. The spring context was not properly closed (webApplicationContext.close() must be called). Below is the code showing the correction:
    public class ApplicationServletContextListener implements
            ServletContextListener {
    ...
    public void contextInitialized(ServletContextEvent event) {
           ServletContext servletContext = event.getServletContext();
           webApplicationContext = (ConfigurableApplicationContext) WebApplicationContextUtils
                   .getWebApplicationContext(servletContext);
    ...
    public void contextDestroyed(ServletContextEvent event) {
            webApplicationContext.close();
    }
    ...
    

After I finished all the changes I scripted an atomatic redeployment to occur every minute and I left it running the whole night.

Below is a screenshot showing the final situation. As you see PermGen consumption goes up with each deployment (steps in graphic) but it goes down again as a result of Garbage Collection === no memory leak.

Followers