Friday, April 29, 2011

Enabling Resource caching in Tomcat

If you are running Tomcat use something like HTTP Headers Firefox plugin to confirm that your resources like javascript, CSS and images are cached in the browser. If you see they keep on being retrieved while navigating your website then you could do so much better in terms of performance for your website that you might want to enable caching.

This procedure is the one I am currently using but of course is not the only one. You could use Apache server to rewrite headers for example or like I always say use the right tool for the job: Apache to serve static content. This is though the only procedure I know will provide effective resource caching at tomcat level.

  1. First you will need a dependency if you are running Tomcat version 6 or below. Tomcat 7 includes the very same ExpiresFilter you are about to use here. This dependency includes the filter that will stamp the "Cache-Control" and "Expires" headers which are responsible to ensure browsers will cache the served resource.
    
            
                 fr.xebia.web.extras
                 xebia-servlet-extras
                 1.0.4
                 runtime
            
    
  2. You will need then a second custom NoEtagFilter because Tomcat will send back an ETag header that will force the browser to ask the server if the resource has been modified through the header "If-None-Match". This will cause unnecessary requests to Tomcat which will return back just a 304 when the resource has not been modified. While this still saves throughput is far from ideal as the Browser still needs to open a socket (read consume resources) to the server.
    import java.io.IOException;
    
    import javax.servlet.Filter;
    import javax.servlet.FilterChain;
    import javax.servlet.FilterConfig;
    import javax.servlet.ServletException;
    import javax.servlet.ServletRequest;
    import javax.servlet.ServletResponse;
    import javax.servlet.http.HttpServletResponse;
    import javax.servlet.http.HttpServletResponseWrapper;
    
    import org.slf4j.Logger;
    import org.slf4j.LoggerFactory;
    
    public final class NoEtagFilter implements Filter {
    
        private static final Logger log = LoggerFactory
                .getLogger(NoEtagFilter.class);
    
        public void doFilter(ServletRequest request, ServletResponse response,
                FilterChain chain) throws IOException, ServletException {
            chain.doFilter(request, new HttpServletResponseWrapper(
                    (HttpServletResponse) response) {
                public void setHeader(String name, String value) {
                    if (!"etag".equalsIgnoreCase(name)) {
                        super.setHeader(name, value);
                    } else {
                        log.debug("Ignoring etag header: " + name + " " + value);
                    }
                }
            });
        }
    
        public void destroy() {
        }
    
        public void init(FilterConfig filterConfig) {
        }
    }
    
  3. In web.xml include the filters:
    <!-- Resource caching : Tomcat 7 already includes this filter -->
    
         <filter>
            <filter-name>ExpiresFilter</filter-name>
            <filter-class>fr.xebia.servlet.filter.ExpiresFilter</filter-class>
            <init-param>
               <param-name>ExpiresByType image</param-name>
               <param-value>access plus 1 years</param-value>
            </init-param>
            <init-param>
               <param-name>ExpiresByType text/css</param-name>
               <param-value>access plus 1 years</param-value>
            </init-param>
            <init-param>
               <param-name>ExpiresByType text/javascript</param-name>
               <param-value>access plus 1 years</param-value>
            </init-param>
         </filter>
    
         <filter-mapping>
             <filter-name>ExpiresFilter</filter-name>
             <url-pattern>/*</url-pattern>
         </filter-mapping>
    
         <!-- Only needed for Tomcat which stamps the eTag header to all responses -->
         <filter>
             <filter-name>NoEtagFilter</filter-name>
             <filter-class>com.nestorurquiza.web.filter.NoEtagFilter</filter-class>
         </filter>
         <filter-mapping>
             <filter-name>NoEtagFilter</filter-name>
             <servlet-name>default</servlet-name>
             <dispatcher>REQUEST</dispatcher>
             <dispatcher>FORWARD</dispatcher>
         </filter-mapping>
    
  4. Finally do not forget to change the URL of your included resources every time you change them otherwise the new resource will never be served. I have seen so many erroneous posts on the Internet and actually more than one team blindly following them around how to solve this issue for example adding a timestamp that I think it is worth to mention: If you *always* add a random parameter to the URL so the resource is not cached then it will be never cached in which case you are still at the beginning of this post. You kept reading up to here because you do want caching, don't you? BTW you do not need to increment the number every time you do a change in the script while developing as just using the Refresh button in a browser like Firefox will force reloading all resources including those already cached in the browser.
    <!-- Increment version number for resources when they change for example -->
    <script src"/js/main.js?v=2" type="text/javascript"/>    
    

Wednesday, April 13, 2011

Liferay test with web security scanners

I have used WebSecurify and SkipFish to provide some kind of penetration testing in a Liferay 5.2.3 based Portal.

SkipFish found way more than WebSecurify but there were more false positives than real issues, however it allowed me to find out that actually 500 and 404 errors were not be handled and instead plain tomcat headers were coming back to the browser.

Of course the is a simple solution for this: Adding these entries to web.xml:
    <error-page>
        <error-code>500</error-code>
        <location>/serverError.html</location>
    </error-page>
    <error-page>
        <error-code>404</error-code>
        <location>/notFound.html</location>
    </error-page>

You could get more sophisticated but the above at least protects your system from exposing back end server type and version to potential intruders.

Monday, April 11, 2011

Log Inspector: production server log for developer

This is not a tool as you might have expected from the title. It is rather the implementation of a concept that allows developers to look at production log files without actually login into production systems.

Let us face it, it is on the best interest both for developers and IT people to make applications less buggy and more reliable. For that ecosystem to work both areas must be able to work without stepping of each other toes AKA Separation of Concerns.

So if a systems architect understands this well s(he) should look for a way to allow developers to access what they are supposed to read in order to troubleshoot real production problems and at the same time ensure the production system is stable and secured.

This is not a post about security so I assume you already took care of not logging user passwords and privacy related information.

The whole problem here can be easily addressed separating where the logs are created (production server) and where the logs are inspected (development server or if you can afford it just a logs server). Our old always handy RSYNC friend will allow us to push from any server any logs we want to a development/logs server.

  1. Setup the log/development server
    $ sudo useradd -d /home/logsviewer -m logsviewer -s /bin/bash
    $ sudo passwd logsviewer #set password for example to logs4all ;-)
    $ su -u logsviewer
    $ mkdir ~/remotelogs
    
  2. Setup the production server to send logs to the logs/development server
    $ ssh-keygen -t rsa -f  /home/admin/.logs_rsa #No passphrase
    $ scp /home/admin/.logs_rsa.pub logsviewer@logsMachine:~/
    
  3. In the logs/development server authorize the key
    $ test -d .ssh || mkdir .ssh
    $ cat ~/.logs_rsa.pub >> .ssh/authorized_keys #WARNING: If this is not just a dev box you will need to do extra work to increase security. This is authorizing the remote box to do anything the current user can do
    
  4. In the production server set a cron to update logs let us say every minute
    $ crontab -e
    #Update app logs in log server
    */1 * * * * rsync -avz -e "ssh -i /home/admin/.logs_rsa" /opt/tomcat/logs/app.log logsviewer@logsMachine:/home/logsviewer/remotelogs/app.log
    
  5. Now as a developer you can check any log file from your log/development server
    less /home/logsviewer/remotelogs/app.log 
    

Saturday, April 09, 2011

Monitor Event Log in Windows Servers

If you want an ad-hoc solution for this follow the below instructions for pre-Windows7/2008 systems. I realized the script needed some changes For Windows 7/2008 so if you are on those OS then check here.

  1. Create a directory to store scripts
    mkdir C:\Scripts\events
    
  2. Save this script in it. Be sure to update smarthost and to/from addresses
  3. Create the event trigger that will send the email if any error. It will ask for the password for the domain user the system will use to run the script.
    eventtriggers /create /ru domainuser /l "APPLICATION" /tr "All Errors" /t error /tk "C:\scripts\events\sendEventErrorByEmail.vbs\"
    
  4. You can either wait for an error to happen or use the below script to insert an error yourself
    const SUCCESS = 0
    const ERROR = 1
    const WARNING = 2
    const INFORMATION = 4
    const AUDIT_SUCCESS = 8
    const AUDIT_FAILURE = 16
    
    Dim WshShell
    Set WshShell = WScript.CreateObject("WScript.Shell")
    wshshell.Logevent ERROR, "Test ERROR Event Log"
    set wshshell=nothing
    
  5. Use a file (eventLogsExclusionsFilePath="eventLogsExclusions.txt") for those log messages you would like to exclude. Yes sometimes developer log as ERROR stuff that should be logged at different level
  6. From now on you will get notified when an error is stamped in the event logs. You can always query the tasks that you have configured to be triggered for events:
    c:\>eventtriggers /query
    Trigger ID Event Trigger Name        Task
    ========== ========================= ========================================
             1 Application Errors        C:\scripts\events\sendEventErrorByEmail.vbs"
    
Here is how to get SSL authentication or TLS Authentication support.

Monitor SOAP services with Python

Monitoring a server is more than just being notified of disk, memory or CPU usage. It is more than inspecting the logs and notify when something goes wrong. Each application should have a heartbeat interface meaning there must be a way to check every five minutes for example that the application is up and running, responding to requests.

Python is a great language for scripting and I am showing here how to use it to monitor SOAP services.

I tested this to monitor Advent Geneva report engine which is exposed through SOAP.

This script can be used to monitor a SOAP web service and send an alert in case it does not work as expected. Note that we send a first request, we parse the response to get a session ID and then we send a second request using that session ID to assert there is a proper response (We assert the presence of certain node in that final response) It works with responses containing namespaces, a typical scenario in real life SOAP Web Services.

#!/usr/bin/python
#
# soap-geneva-monitor.py <url>
#
# Author: Nestor Urquiza
# Date: 04/07/2011
#
# Sends an email alert whenever the Advent Geneva report fails
# Used to monitor Geneva Report Engine through its SOAP interface
#
# Preconditions:
# Install httplib2 from http://code.google.com/p/httplib2/wiki/Install
# Install ElementTree from http://effbot.org/zone/element-index.htm
#


import sys
import socket
from smtplib import SMTPException
import httplib2
import elementtree.ElementTree as etree
from urllib2 import HTTPError
import smtplib

########
# Config
#######

sender = 'donotreply@nestorurquiza.com'
receivers = ['nurquiza@nestorurquiza.com']
host = 'mail.nestorurquiza.com'
thisHostname = socket.gethostname() 



########
# Functions
#######

def sendRequest( xml_str ):
 headers = {'SOAPAction': ''}
 headers['User-Agent'] = "Monitoring"
 headers['Content-length'] = str(len(xml_str))

 


 h = httplib2.Http(".cache")

 response, content = h.request(url, "POST", body=xml_str, headers=headers)

 if response.status == 500 and not \
 (response["content-type"].startswith("text/xml") and \
 len(content) > 0):
   raise HTTPError(response.status, content, None, None, None)

 if response.status not in (200, 500):
   raise HTTPError(response.status, content, None, None, None)

 doc = etree.fromstring(content)
 

 if response.status == 500:
   faultstring = doc.findtext(".//faultstring")
   detail = doc.findtext(".//detail")
   raise HTTPError(response.status, faultstring, detail, None, None)
 
 return doc

########
# Main
#######
if len(sys.argv) != 3:  
  sys.exit("Usage: soap-geneva-monitor.py <url> <hostname>")

#url = "http://genevatest:4640"
url = sys.argv[1]
subject = url + " monitored from " + thisHostname 
hostname = sys.argv[2]
print hostname
#Define the namespace for responses
namespace = "http://geneva.advent.com"

xml_str = """<?xml version="1.0" encoding="UTF-8"?>
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
 <soapenv:Body>
  <gen:StartCallableRunrep xmlns:gen="http://geneva.advent.com">
   <gen:portNumber>9000</gen:portNumber>
   <gen:hostname>""" + str(hostname) + """</gen:hostname>
   <gen:username>user</gen:username>
   <gen:password>password</gen:password>
   <gen:extraFlags></gen:extraFlags>
  </gen:StartCallableRunrep>
 </soapenv:Body>
</soapenv:Envelope>
"""

try:
  doc = sendRequest( xml_str )
  
  print etree.tostring(doc)
  runrepSessionId = doc.findtext(".//{%s}runrepSessionId" % namespace)
  #print "runrepSessionId:" + str(runrepSessionId)

  xml_str = """<?xml version="1.0" encoding="UTF-8"?>
  <soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
  <soapenv:Body>
    <gen:RunCallableRunrepRunReport xmlns:gen="http://geneva.advent.com">
      <gen:runrepSessionId>""" + str(runrepSessionId) + """</gen:runrepSessionId>
        <gen:runrepFileName>taxlotappacc.rsl</gen:runrepFileName>
        <gen:extraFlags>-p TREE -at Dynamic -ap "TREE" -ps 04/06/2011 -pe 04/06/2011 --SeparateLegalEntities 1 --FundLegalEntity "TREE MF"</gen:extraFlags>
    </gen:RunCallableRunrepRunReport>
  </soapenv:Body>
  </soapenv:Envelope>"""

  doc = sendRequest( xml_str )
  portfolioName = doc.findtext(".//{%s}portfolioName" % namespace)
  #print portfolioName

  if portfolioName != 'TREE':
    raise Exception("Data Error", "Portfolio name '" + portfolioName + "' != TREE") 
except Exception, err:
 error = "I could not get a successful response for the SOAP Service. Details: " + str( err )
 message = "From: %s\r\nTo: %s\r\nSubject: %s\r\n\r\n" % (sender, receivers, subject) + error
 try:
    smtpObj = smtplib.SMTP(host)
    smtpObj.sendmail(sender, receivers, message)         
 except SMTPException:
    print "Error: unable to send email"

The script runs every 5 minutes from an external machine (Not from Geneva)
*/10 * * * * /usr/sbin/soap-geneva-monitor.py http://genevatest:4640

Installing wget in Solaris 10

In Solaris 10 there is no need to install wget as it is already included in /usr/sfw/bin/wget

To install wget from sources was such a pain in this fresh box that I decided to document it all.
  1. Get the binary from ftp://ftp.sunfreeware.com/pub/freeware/ for your platform in my case it was intel so I got it from ftp://ftp.sunfreeware.com/pub/freeware/intel/10/wget-1.12-sol10-x86-local.gz
  2. Upload the file to the server and run the below commands in there:
    $ gunzip wget-1.12-sol10-x86-local.gz
    $ pkgadd -d wget-1.12-sol10-x86-local
    
  3. In my case running wget after the installation was really painful. I had to install openssl-1.0.0d-sol10-x86-local, libiconv-1.13.1-sol10-x86-local, libgcc-3.4.6-sol10-x86-local, libidn-1.19-sol10-x86-local, libintl-3.4.0-sol10-x86-local. To save time I recommend you find out if there is any dependencies missing:
    $ ldd /usr/local/bin/wget 
    
  4. If any of the dependencies is missing then find it in your system like below but be prepared you might need to install some of them if you do not find them
    $find / -name libgcc_s.so.1
    
  5. Finally add it to your dynamic libraries load path:
    $ vi ~/.bash_profile
    export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/ssl/lib
    export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/gcc4/lib
    $ source ~/.bash_profile
    

Mounting windows shared drives in linux CIFS

This question comes again and again every time I ask for such a simple thing: Please use this windows network share to read/write files from a Linux server. Mounting CIFS in linux is easy.

First be sure you have smbfs installed. The below command should return a path identifying where the command is. This is needed so the mount command accepts a credentials file where Windows credentials are stored:
$ sudo find / -name  "mount.cifs"
/sbin/mount.cifs

Here is how to install it in Ubuntu:
$ sudo aptitude install cifs-utils

After that just follow the below steps, adapting them to your environment paths to get the Windows path mounted in Linux:
$ mkdir /mnt/linuxDirectory
$ mkdir -p /root/projects/cifs/
$ vi /root/projects/cifs/mount.txt
username=windowsUserName
password=windowsPassword
$ vi /etc/fstab
//windows.machine/volume/or/path/to/folder /mnt/linuxDirectory cifs auto,users,credentials=/root/projects/cifs/mount.txt,domain=windowsDomain,file_mode=0600,dir_mode=0700,uid=linuxUser,gid=linuxUser
$ mount /mnt/linuxDirectory
$ ls /mnt/linuxDirectory

Friday, April 08, 2011

Installing Monit in Solaris 10

Here are the instructions to get Monit installed in Solaris 10. Note that your system might need several packages that you might not have. As you might be aware you can get most of them from Sun Free Ware.

At least version 5/09 (which I was forced to use as the latest supported by the server I am hosting: Advent Geneva) comes with gcc out of the box so be sure you add to your path the location for it "export PATH=$PATH:/usr/sfw/bin"

I had all kind of issues when I tried to use gcc 4.5.1 especially the -m64 flag not working. Solaris 10 5/09 comes with gcc 3.4.3 and it after all does compile monit without any problems.

Pay special attention to your specific environment. In my case this was an AMD 64 bits machine and I had to set LD_LIBRARY_PATH to get 64 bits library:
export LD_LIBRARY_PATH=/usr/sfw/lib/64:$LD_LIBRARY_PATH

While the above is good for installation purposes it is not enough to run monit later on. So make the path mandatory for the whole system library loader:
crle -64 -u -l /usr/sfw/lib/64

Considering you have read the above then you should be OK following this steps:
  1. Copy sources to the server
    $ scp /Users/nestor/Downloads/monit-5.2.5.tar.gz admin@solaris.sample.com:/export/home/admin
    
  2. Install the binary in the server
    $ cd /export/home/admin
    $ digest -a sha256 monit-5.2.5.tar.gz #3c2496e9f653ff8a46b75b61126a86cb3861ad35e4eeb7379d64a0fc55c1fd8d
    $ gzip -cd monit-5.2.5.tar.gz | tar xfv -
    $ cd monit-5.2.5
    ./configure \
          --with-ssl-incl-dir=/usr/sfw/include \
          --with-ssl-lib-dir=/usr/sfw/lib/64 \
          CFLAGS='-m64 -mtune=opteron' \
          LDFLAGS='-m64 -mtune=opteron'
    $ gmake
    $ gmake install
    
  3. Configure monit
    $ mkdir /usr/local/etc #in case it does not exist
    $ vi /usr/local/etc/monitrc
    $ chmod 700 /usr/local/etc/monitrc
    $ ps -ef|grep monit|grep -v grep|awk '{print $2}'|xargs kill
    $ vi /etc/inittab
    m:2345:respawn:/usr/local/bin/monit -Ic /usr/local/etc/monitrc
    mon:2345:wait:/usr/local/bin/monit -Ic /usr/local/etc/monitrc start all
    moff:06:wait:/usr/local/bin/monit -Ic /usr/local/etc/monitrc stop all
    $ /etc/telinit q
    
  4. Now monit should be running and if it is killed it will be brought back. After restarting the server it should start automatically as well. You can check its status with:
    /usr/local/bin/monit status
    

IE select drop down border and size with jquery

IE uses OS native libraries to render the HTML SELECT element. This is why some styles will not be respected in Internet Explorer.

Thanks to a jquery plugin you can go around this issue. After downloading and installing the plugin:

  1. Include the specific CSS just for IE. Using the spring:url tag below so if you do not use Spring you can just include the plain url instead.
    
    
  2. Here is the code for ie_only.css. First CSS selector below is just for the border. The rest is the default for 13px as explained in the author URL. I am pasting the whole file so get the idea of what to customize if needed.
    .ie-hacked-select {
        border: 1px inset #000000;
    }
    
    /* Basic overlay CSS */
    .select-overlay { background:#fff }
    .ie-select-width-overlay span
    {
        display:block;
        float:left;
        clear:both;
        background:transparent url(../images/bg-ie-select-width-13px.png) no-repeat 0 0
    }
    
    
    /** Overlay CSS for Internet Explorer 6 and 7 */
    .ie6 .ie-select-width-overlay span,
    .ie7 .ie-select-width-overlay span
    {
        width:17px;
        height:20px;
        background-position:0 -18px 
    }
    .ie6 .ie-select-width-overlay-hover span,
    .ie7 .ie-select-width-overlay-hover span { background-position:-17px -18px }
    .ie6 .ie-select-width-overlay-mouseover span,
    .ie7 .ie-select-width-overlay-mouseover span { background-position:-18px -18px }
    
    
    /* Overlay CSS for Internet Explorer 8 */
    .ie8 .ie-select-width-overlay span
    {
        width:17px;
        height:18px;
        background-position:0 0
    }
    .ie8 .ie-select-width-overlay-hover span { background-position:-17px 0 }
    .ie8 .ie-select-width-overlay-mouseover span { background-position:-18px 0 }
    
  3. Include the javascript
    <script src="<spring:url value="/js/jquery.ie-select-style.pack.js" htmlEscape="true" />" type="text/javascript"></script>
    
  4. I use the below in one big form where all select boxes need to be corrected. In IE8 I have noticed it fails in some pages so it is always better to set the class explicitely per select box.
    $(document).ready(function() {
        //You better avoid this and set the class in the select element
        $('select').addClass('ie-hacked-select');
    
        //This is the real statement needed. It applies the style to certain select boxes
        $('.ie-hacked-select').ieSelectStyle({});
    });
    

Wednesday, April 06, 2011

Encrypt and decrypt files in your CMS

Every so often I read this question in forums: Does this repository API (for example a the Jackrabbit JCR implementation) support document encryption? And then the answer: This is not a concern of the repository but of the client.

The question though is if this is really so difficult task and actually at least in Java thanks to the Jasypt library the answer is: It is not difficult at all.

Below is a fully working program that will allow to encrypt a document and store it in a given path (or in the same folder path with different name by default). Few lines of code from Jasypt library are actually enough to accomplish this goal.

Note that this code uses the BasicBinaryEncryptor but you can also use the StrongBinaryEncryptor which is not any different.

package com.nestorurquiza.pdf.tools;

import java.io.File;
import java.io.FileInputStream;
import java.io.FileOutputStream;

import org.jasypt.util.binary.BasicBinaryEncryptor;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

public class CryptFile {
    
    private static final Logger log = LoggerFactory.getLogger(CryptFile.class.getName());
    
    private CryptFile(){};
    
    public static void main( String[] args ) throws Exception
    {
        if(args.length < 3) {
            usage();
        }
        String action = args[0];
        boolean encrypt = action.equalsIgnoreCase("encrypt");
        boolean decrypt = action.equals("decrypt");
        
        if( !encrypt && !decrypt  ) {
            usage();
        }
        
        String password = args[1];
        String inputFilePath = args[2];
        String ext = inputFilePath.substring( inputFilePath.length() - 4 );
        String outputFilePath = inputFilePath.substring( 0, inputFilePath.length() -4 ) + "_" + action + ext;
        if(args.length >3) {
            outputFilePath = args[3];
        }
        
        File inputFile = new File(inputFilePath);

        byte[] bytes = new byte[(int) inputFile.length()];
        try {
            FileInputStream fileInputStream = new FileInputStream(inputFile);
            fileInputStream.read(bytes);
            BasicBinaryEncryptor binaryEncryptor = new BasicBinaryEncryptor();
            binaryEncryptor.setPassword(password);
            File outputFile = new File(outputFilePath);
            FileOutputStream fileOutputStream = new FileOutputStream(outputFile);
            if(encrypt) {
                fileOutputStream.write(binaryEncryptor.encrypt(bytes));
            } else {
                fileOutputStream.write(binaryEncryptor.decrypt(bytes));
            }
            fileOutputStream.close();
            fileInputStream.close();
        } catch (Exception e) {
            log.error(e.getMessage());
        }
        

    }
    

    private static void usage()
    {
        System.err.println( "Usage: com.nestorurquiza.pdf.tools.EncryptFile <action(encrypt/decrypt)> <password> <input file> [output file]\n");
        System.exit( 1 );
    }
}

Followers