Thinking In Software

Tuesday, August 09, 2011

Install gcc in Ubuntu

The Linux sysadmin must build from sources from time to time. It makes sense then to have the C environment ready. In Ubuntu all you need to do is to follow the below commands:

sudo apt-get update
sudo apt-get upgrade
sudo apt-get install build-essential
gcc --version

Saturday, August 06, 2011

Phishing Attack: Fake Twitter Email not marked as Spam

I got an email in my Gmail account from "Twitter Support" "with subject "Your account has been suspended" with no text content but an image (that I have disabled of course for security reasons). The image content was something like "We detected unusual activity ..."

This phishing email is nothing new but what came to my attention was that Gmail was not able to detect the spam even though the full headers from the message are showing how Google identified it as a candidate for Spam "Authentication-Results: mx.google.com; spf=hardfail (google.com: domain of support@twitter.com does not designate 203.115.131.123 as permitted sender) smtp.mail=support@twitter.com"

Any software is plenty of bugs. Even with the best developers on board you are still vulnerable. Good that "Report phishing" option is available albeit a little bit hidden behind an arrow close to the Reply link. User experience should be helping better here I would say but regardless the important lesson to learn is to be always suspicious up front. Do not trust any bad news (account hacked or compromised) or too good news (You just won a million dollar) you receive.

See below for the full headers of the message:

Received-SPF: fail (google.com: domain of support@twitter.com does not designate 203.115.131.123 as permitted sender) client-ip=203.115.131.123;
From: "Twitter Support" <support@twitter.com>
Subject: Your account has been suspended
To: "nestor.urquiza" <nestor.urquiza@gmail.com>
Content-Type: multipart/alternative; charset="iso-8859-10"; boundary="LMRJGCZhTXlUeMlXLirvgZD=_SMWAE68zR"
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
Date: Sat, 6 Aug 2011 13:37:37 +0800
Message-Id: <20110806053741.55F8793DB3A@smtp4-roc.bti.net.ph>

This is a multi-part message in MIME format

--LMRJGCZhTXlUeMlXLirvgZD=_SMWAE68zR
Content-Type: text/plain ; charset="iso-8859-10"
Content-Transfer-Encoding: quoted-printable




--LMRJGCZhTXlUeMlXLirvgZD=_SMWAE68zR
Content-Type: text/html ; charset="iso-8859-10"
Content-Transfer-Encoding: quoted-printable

<HTML><HEAD>
<META name=3DGENERATOR content=3D"MSHTML 8.00.6001.23019"></HEAD>
<BODY>
<P><A href=3D"mexico.cnn.com/redirectComplete.php?url=3D//emailus%2Eit=
%2Etc/2ule3B"><IMG border=3D0 src=3D"http://3.bp.blogspot.com/-u_sWLHS=
Yjes/TjyqVZ73vrI/AAAAAAAAAEQ/hX5mKS-R7-g/s1600?2ule3B"></A> </P>
<P>&nbsp;</P></BODY></HTML>


--LMRJGCZhTXlUeMlXLirvgZD=_SMWAE68zR--

Thursday, August 04, 2011

Server scripting with NodeJs: Tainting LDAP data

This is a real world example of using nodejs to do server side scripting. It shows how to deal with command line arguments, invoke shell commands and parse responses, interact with mysql and how to use the library (when possible and makes sense) in a synchronous way.

Node (also knows and nodejs) is a javascript library that allows to run javascript code out of the browser. It favors event programming which can be difficult for backend developers used to languages like Java. At the same time it is straightforward for front end developers which are already used to javascript callback functions necessary for UI programming.

If you havent install it yet go ahead and do it now:

I have tested all this with node-v0.4.10 as you see below.

cd ~/Downloads
wget http://nodejs.org/dist/node-v0.4.10.tar.gz
tar -zxvf node-v0.4.10.tar.gz
cd node-v0.4.10
./configure
make
sudo make install
node --version

Install npm (package manager for nodejs) in case you need extra modules (and you will when you get serious about nodejs development)
```
curl http://npmjs.org/install.sh | sudo sh
```
If you use MySQL here is how to install one library that just works (whole API: https://github.com/felixge/node-mysql)
```
npm install mysql
```
You should of course test with the simplest possible 'Hello World' script
```
$ vi hello.js
console.log('Hello World');
$ node hello.js
```

The Node project is making a big effort to enforce the use of non blocking code (code that will be triggered but will not block the current thread). That is the reason why most of the code you will see is written with nested callback functions.

Event programming is good paradigm for certain problems. Responsiveness of a program can be improved a lot without having to deal with blocking in multithreading programming. Those of us that have worked with threads know how hard it can be. I am not nodejs is better than java or any other language to deal with multiprocessing. It is just a different paradigm and I would appreciate if you do not start ranting against or in favor of any programming language in this post.

Here is one extract from the process api: http://nodejs.org/docs/v0.3.1/api/process.html

process.argv.forEach(function (val, index, array) {
  console.log(index + ': ' + val);
});
console.log

The previous code is non blocking, so if you need to work with the arguments you will need to work inside the anonymous function. Look at the code below. After running it you will see the asynchronous code does not run before the console logs the message:

var asyncArg = 'Async Arguments:';
var syncArg = 'Sync Arguments:';

var argv = process.argv;
argv.forEach(function (val, index, array) {
 setTimeout(function() {asyncArg += ' ' + val; console.log( asyncArg ); }, 0);
 
});
console.log( asyncArg );

for( argIndex in argv ) {
 syncArg += ' ' + argv[argIndex];
}
console.log( syncArg );

So probably to work with command line arguments you are better off the asynchronous flow. One might expect that there is always a way to go synchronous with nodejs but that is not the case. As I said the nodejs library pushes for non blocking === asynchronous code. Look at the below code which comments should be self explanatory:

var sys = require('sys')
var exec = require('child_process').exec;

var child = exec("ldapsearch -x -v -H 'ldap://nestorurquiza:10389' -D 'uid=admin,ou=system' -w 'secret' -b 'o=nestorurquiza'", function (error, stdout, stderr) {
  console.log(stdout); //Will print the output of the command
  if (error !== null) {
    console.log('exec error: ' + error);
  }
});
console.log(child.stdout); //Will not print the output of the command but rather the object prototype

Here is a script that clones the data from one LDAP Server (tested with ApacheDS) and imports it into a second LDAP server. The script taints the emails to ensure we do not send messages to real production users while testing. It excludes certain domains and it shows how to interact with mysql to pull a white list of emails for which no tainting should be done. It also changes the passwords to a well known test password so the team can use it to debug what happens when different users interact with the application. BTW this task is better to be done from Talend but I needed a real problem to demonstrate nodejs is ready to work as server side scripting while I was figuring out how to make it happen from Talend:

#!/usr/local/bin/node
/*
** WARNING: This program will wipe out the specified BASE_DN from the target LDAP Server
**
** taintLdap.js A nodejs script to taint data from ldap. Use it to change passwords for all users to a known value and to change their emails to avoid sending messages from testing environment to real users
**
** Example: node taintLdap.js 'ldap://jnestorurquiza:10389' 'uid=admin,ou=system' 'secret' 'ldap://localhost:10389' 'uid=admin,ou=system' 'secret'
**
** @Author: Nestor Urquiza
** @Date: 08/02/2011
**
*/

/*
** Imports
*/
var sys = require('sys')
var exec = require('child_process').exec;
var Client = require('mysql').Client;
var client = new Client();

/*
** Constants
*/
var BASE_DN = "o=nestorurquiza";
var EXCLUSION_DOMAINS = ['nestorurquiza.com','nestoru.com'];
var APPEND_DOMAIN = 'nestorurquiza.com'
var COMMON_PASSWORD = 'e1NIQX1JcnFMUVdMT1o3ZXF0WHRBdUlFSFRlUnZkRFk9' //Testtest1 after SHA1. To generate a different password use: echo -n "mypassword" | openssl dgst -sha1

/*
** Arguments
*/
var argv = process.argv;
if( argv.length != 8 ) {
 usage();
 process.exit(1);
}
var fromUrl = argv[2];
var fromUser = argv[3];
var fromPasword = argv[4];
var toUrl = argv[5];
var toUser = argv[6];
var toPasword = argv[7];

client.user = 'root';
client.password = 'root';
client.host = 'localhost';
client.port = '3306';
client.database = 'nestorurquiza'

/*
** Main
*/
//console.log('Cloning and tainting from ' + fromUrl + ' to ' + toUrl);
var ldif;
var authorizedEmails = new Array();
var pattern = /[^\s=,]*@[^\s=,]*/g
var child = exec("ldapsearch -x -v -H '" + fromUrl + "' -D '" + fromUser + "' -w '" + fromPasword + "' -b '" + BASE_DN + "'", function (error, stdout, stderr) {
  if (error) {
    throw error;
  }
  ldif = stdout;
  client.connect();
  var authorizedEmailQuery = client.query(
  'SELECT name FROM authorized_test_email',
  function (error, results, fields) {
    if (error) {
      throw error;
    }
    for (var resultIndex in results){
      var result = results[resultIndex];
      authorizedEmails[resultIndex] = result.name;
    }
    client.end();
    //console.log('****************'  + authorizedEmails);
    ldif = taintLdifEmail();
 
 child = exec("ldapdelete -r -x -H '" + toUrl + "' -D '" + toUser + "' -w '" + toPasword + "' '" + BASE_DN + "'", function (error, stdout, stderr) {
   if (error) {
        //throw error;
        console.log("WARNING: Could not delete " + BASE_DN);
      }
      var command = "echo '" + ldif + "' | ldapmodify -x -c -a -H '" + toUrl + "' -D '" + toUser + "' -w '" + toPasword + "'";
      //console.log(command);
      child = exec(command, function (error, stdout, stderr) {
     
  if (error) {
   throw error;
  }
   });
    });   
 //ldapmodify -x -c -a -H ldap://localhost:10389 -D "uid=admin,ou=system" -w 'secret' < ~/Downloads/taintedLdap.ldif
  }
);
});

/*
** Functions
*/
function taintLdifEmail() {
 var matches = ldif.match(pattern);

 for ( var matchIndex in matches ) {
  var taint = true;
  var match = matches[matchIndex];
  for( var exclusionDomainIndex in EXCLUSION_DOMAINS ) {
   if( match.indexOf(EXCLUSION_DOMAINS[exclusionDomainIndex]) >= 0 ) {
    taint = false;
    break;
   }
  }
  if( !taint ) {
   continue;
  }
  for ( var authorizedEmailIndex in authorizedEmails ) {
   var authorizedEmail = authorizedEmails[authorizedEmailIndex];
   if( match == authorizedEmail ) {
    taint = false;
    continue;
   }
  }
  if( taint ) {
   var replacement = match.replace('.', '') + APPEND_DOMAIN;
   //console.log( match + " >>> " + replacement );
   ldif = ldif.replace(match, replacement);
  }
 }
 ldif = ldif.replace(/userPassword.*/g, 'userPassword:: ' + COMMON_PASSWORD);
 return ldif;
}

function usage() {
 console.log("Usage: " + "./taintLdap.js <fromUrl> <fromUser> <fromPasword> <toUrl> <toUser> <toPasword>");
}

Why I am trying to use nodejs if I already have bash, awk, perl, python, ruby and what not? I am building a team with strong separation of concerns. While we know we cannot be as good as the guy that is spending 100% of the time in just writing SQL stored procedures the whole team could cover for some days to be able to compensate vacation time for example, so yes SQL is a mandatory skill. Javascript is a mandatory skill as well and if I can script with it I could have some of those scripting needs done by anybody in the team as well. I am just trying to keep really low the amount of technologies and languages we use.

Isn't it better to use RhinoJs? Probably yes, but I am tempted to use something out of the JVM that runs faster and consume less resources. I have recently decomisioned a whole CLI project based in Java just because it was really resource intensive. I have favored the use of Controllers in our Business Hub which are called from simple CURL or WGET statements. I am not claiming RhinoJs is unnacceptable slow nor that NodeJs is better. I see value in both of them.

Why I am considering scripting after all if I promote the idea of a Business Hub? There are cases in which definitely using Unix Power Tools, existing CLIs etc do the job quickly and more reliable.

Do I think NodeJs is a better answer for Server side logic than Java? At the moment I am happy with plain java for my backend. The amount of existing code available for free is amazing. If that will be the case in the future it will depend on the open source community. At least for my current project I stick to Java.

PKCS12 to JKS keystore

Java uses a proprietary to Sun format (JKS) to store certificates in what is called a Keystore (A file containing entries of those certificates you trust)

When you get certificates included in a different keystore type like it is the case of PKCS12 (commonly using the *.p12 extension) you need to extract and add them to the JKS keystore. Failure to do so will make your program depending on individual keystore files rather than just one keystore where all certificates and private keys are kept.

Here is how you can do it (sample using OSX paths but applicable to other OS as well).
First find the key for the certificate to export. Basically the left first word for the specific entry from this command:

keytool -list -keystore /Users/nestor/Downloads/cert.p12 -storetype pkcs12

Then run something like the below. Note that alias.from.cert.p12 comes from the previous command.

keytool -importkeystore -srckeystore /Users/nestor/Downloads/cert.p12 -destkeystore /Library/Java/Home/lib/security/cacerts -srcstoretype PKCS12 -deststoretype JKS -srcstorepass cert.p12.password -deststorepass changeit.is.the.default.password -srcalias alias.from.cert.p12 -destalias alias.for.cacerts.new.certificate

Note that the private key inside PKCS12 might need a password and that password must be the same as the JKS where you are importing. Failure to do this will end up in Access Denied, Forbidden or any other error comming from the Server where the key is attempted to be used.

Wednesday, August 03, 2011

ldapsearch SSL with ApacheDS

Self signed certificates are treated different by the ldap cli tools.

The task was to connect with ldapsearch to a remote ApacheDS server serving SSL. Long story short the certificate is self signed and only certain IP range can access the server via LDAP over SSL (TLS).

Here are the steps showing how to configure ldapsearch and the rest of ldap tools to work with SSL (both signed and self signed):

If not using self signed certificate then get the server certificate
```
$ openssl s_client -connect ldap.nestorurquiza.com:636
```
Non self signed certificate: Create a file with the contents from "-----BEGIN CERTIFICATE-----" up to "-----END CERTIFICATE-----") from the previous command
```
$ sudo mkdir /etc/openldap/certs/
$ vi /etc/openldap/certs/ldap.nestorurquiza.com.cert
```
Be sure your certificate is not self signed. Basically check for a return code=0, not someting like "Verify return code: 18 (self signed certificate)"
```
$ openssl s_client -connect ldap.nestorurquiza.com:636 -CAfile ldap.nestorurquiza.com.cert
```

Edit the ldap configuration

$ vi /etc/openldap/ldap.conf
...
#Use the below if you want ldapsearch to work with self signed certificate. Probably a better option security wise is to buy a certificate right ;-) Note that the path is for OSX. For Ubuntu it is /etc/ldap/certs...
#TLS_REQCERT    demand
TLS_REQCERT     never
#Use the below for non self signed certificates
#TLS_CERT    /etc/openldap/certs/ldap.nestorurquiza.com.cert

Run an ldapsearch command to be usre you get the ldif result

ldapsearch -x -v -H ldaps://ldap.nestorurquiza.com:10636 -D "uid=admin,ou=system" -w 'secretPassword' -b "o=nestorurquiza"

Monday, July 18, 2011

Redirect after login to requested page with Spring after CSRF protection

Bookmarks, typing URLs directly in the address bar and getting to the requested page after login are functionalities you should not break as they impact user experience.

Once you have protected your Spring website against CSRF using the Synchronizer Token Pattern you will find that the redirection to the requested page after login functionality (You request a page, the login form shows up and after that you are taken to the page you originally requested) will be broken.

Basically the user might access a bookmark or just type a URL without the security token and the redirection will use the provided (and expired) security token or no security token at all. Of course you need to hook into Spring in order to change the default functionality.

First you use "authentication-success-handler-ref" form-login property in the Spring security context:

<beans:bean id="customAuthenticationHandler" class="com.nestorurquiza.web.handler.CustomAuthenticationHandler" />
    
<form-login login-page="/login"
            authentication-success-handler-ref="customAuthenticationHandler"
            authentication-failure-url="/login?error=authorizationFailed" />

Then implement the custom handler. Code should speak for itself.

package com.nestorurquiza.web.handler;

import java.io.IOException;

import javax.servlet.ServletException;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;

import org.springframework.security.core.Authentication;
import org.springframework.security.web.authentication.SavedRequestAwareAuthenticationSuccessHandler;
import org.springframework.security.web.savedrequest.DefaultSavedRequest;

import com.nestorurquiza.utils.UrlTool;
import com.nestorurquiza.web.WebConstants;

public class CustomAuthenticationHandler extends SavedRequestAwareAuthenticationSuccessHandler {

    @Override
    public void onAuthenticationSuccess(HttpServletRequest request,
            HttpServletResponse response, Authentication authentication)
            throws ServletException, IOException {
        // TODO Auto-generated method stub
        String ctoken = (String) request.getSession().getAttribute(WebConstants.CSRF_TOKEN);
        DefaultSavedRequest defaultSavedRequest = (DefaultSavedRequest) request.getSession().getAttribute("SPRING_SECURITY_SAVED_REQUEST_KEY");
        if( defaultSavedRequest != null && ctoken != null ) {
            String requestUrl = defaultSavedRequest.getRequestURL() + "?" + defaultSavedRequest.getQueryString();
            requestUrl = UrlTool.addParamToURL(requestUrl, WebConstants.CSRF_TOKEN, ctoken, true);
            getRedirectStrategy().sendRedirect(request, response, requestUrl);
        } else {
            super.onAuthenticationSuccess(request, response, authentication);
        }
    }
}

Here is the little useful class that allows to override the ctoken parameter (or any other url parameter)

package com.nestorurquiza.utils;

public class UrlTool {
    public static String addParamToURL(String url, String param, String value,
            boolean replace) {
        if (replace == true)
            url = removeParamFromURL(url, param);
        return url + ((url.indexOf("?") == -1) ? "?" : "&") + param + "="
                + value;
    }

    public static String removeParamFromURL(String url, String param) {
        String sep = "&";
        int startIndex = url.indexOf(sep + param + "=");
        boolean firstParam = false;
        if (startIndex == -1) {
            startIndex = url.indexOf("?" + param + "=");
            if (startIndex != -1) {
                startIndex++;
                firstParam = true;
            }
        }

        if (startIndex != -1) {
            String startUrl = url.substring(0, startIndex);
            String endUrl = "";
            int endIndex = url.indexOf(sep, startIndex + 1);
            if(firstParam && endIndex != 1) {
                //remove separator from remaining url
                endUrl = url.substring(endIndex + 1);
            }
            return startUrl + endUrl;
        }

        return url;
    }

}

Wednesday, July 06, 2011

Caching with Spring ehcache and annotations

Update Oct 2012: In fact Spring supports now (JSR-107 AKA JCache although partially as the spec is not still ready as of 2012

Caching is trivial up to the moment you start wanting to cache too many entities. At that point you realize caching is actually a cross cutting concern which basically should be done the easy way, read using Inversion Of Control. But caching is also about where you cache: memory, file system?

If you are using Java then Spring in combination with ehCache can be used to provide caching while abstracting the developer from the details. To be able to achieve caching with minimum effort I recommend using ehcache spring annotations project. Note there is no need to add ehcache dependency as it is added by ehcache-spring-annotations project. Note that Spring 3.1 provides native support so probably it is a better idea to go that route.

Here are the dependencies I used for Spring 3.0.4:

<!-- ehcache -->
        <dependency>
            <groupId>com.googlecode.ehcache-spring-annotations</groupId>
            <artifactId>ehcache-spring-annotations</artifactId>
            <version>1.1.2</version>
            <type>jar</type>
        </dependency>

Create WEB-INF/ehcache.xml with the below content:

<?xml version="1.0" encoding="UTF-8"?>
  <ehcache xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="http://ehcache.org/ehcache.xsd">
      <defaultCache eternal="true" maxElementsInMemory="100" overflowToDisk="false" />
      <cache name="findAllClients" maxElementsInMemory="10000" eternal="true" overflowToDisk="false" />
  </ehcache>

In application context add the below lines:

<beans ...xmlns:ehcache="http://ehcache-spring-annotations.googlecode.com/svn/schema/ehcache-spring"...
...
xsi:schemaLocation="
...
http://ehcache-spring-annotations.googlecode.com/svn/schema/ehcache-spring http://ehcache-spring-annotations.googlecode.com/svn/schema/ehcache-spring/ehcache-spring-1.1.xsd
...
<!-- ehcache -->
    <ehcache:annotation-driven />
 
    <ehcache:config cache-manager="cacheManager">
        <ehcache:evict-expired-elements interval="60" />
    </ehcache:config>
 
    <bean id="cacheManager" class="org.springframework.cache.ehcache.EhCacheManagerFactoryBean">
        <property name="configLocation"  value="/WEB-INF/ehcache.xml"/>
    </bean>
...

Look for a method that returns a collection of entities and annotate it like:

@Cacheable(cacheName = "findAllClients")
    public List<Client> findAll() {

If there is a method that inserts an entity related to the created cache (with name "findAllClients" in this case) then we annotate it so it cleans the cache after insertion:

@TriggersRemove(cacheName = "findAllClients", when = When.AFTER_METHOD_INVOCATION, removeAll = true)
    public void addClient(Client client) {

Of course there are cases where we are just consuming a collection let us say from a web service. We would like to force the cache to be cleaned even though we are never inserting an entity. Here is where you need to use a Controller that can be invoked to clean all or specific caches with a URL like:

http://localhost:8080/ehCache/remove?cacheName=findAllClients&cacheName=anotherCacheToRemove

Here is the Controller that will make this possible:

import java.util.ArrayList;
import java.util.List;

import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;

import net.sf.ehcache.Cache;
import net.sf.ehcache.CacheManager;

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Controller;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.servlet.ModelAndView;

@Controller
@RequestMapping("/ehCache/*")
public class EhCacheController extends RootController {
    private static final String EHCACHE_SHOW_PATH = "/ehCache/show";
    
    @Autowired
    private CacheManager cacheManager;

    /**
     * Removes all caches if no parameter is passed
     * Removes all caches for the specified "cacheName" parameters
     * 
     * @param request
     * @param response
     * @return
     */
    
    @RequestMapping("/remove")
    public ModelAndView home(HttpServletRequest request, HttpServletResponse response) {
        ControllerContext ctx = new ControllerContext(request, response);
        init(ctx);
        
        String[] storedCacheNames = cacheManager.getCacheNames();
        String[] cacheNames = ctx.getParameterValues("cacheName");
        
        List<Cache> caches = new ArrayList<Cache> (storedCacheNames.length);
        
        for( String storedCacheName : storedCacheNames ){
            Cache storedCache = cacheManager.getCache(storedCacheName);
            if( cacheNames == null ) {
                storedCache.removeAll();
            } else {
                for( String cacheName : cacheNames ) {
                    if( cacheName.equalsIgnoreCase(storedCacheName) ) {
                        storedCache.removeAll();
                    }
                }
            }
            caches.add(storedCache);
        }
        
        ctx.setRequestAttribute("caches", caches);

        return getModelAndView(ctx, EHCACHE_SHOW_PATH);
    }
}

The show.jsp would be something like:

<%@ include file="/WEB-INF/jsp/includes.jsp" %>
<%@ include file="/WEB-INF/jsp/header.jsp" %>
<div class="global_error"><c:out value="${csrfError}" /></div>
<div><c:out value="${caches}" /></div>
<%@ include file="/WEB-INF/jsp/footer.jsp" %>

To add more caching you will need to add the cache entry in the ehcache.xml (thing that I do not like to be honest as I think the annotation should be parsed and if there is no declaration for it in XML then use just the default values or better allow customization as part of the annotation itself) and you need to annotate the method which return value will be cached. To clean that cache you can have either one or a combination of the @TriggersRemove annotation and a URL for cache cleaning like explained before.