Automating Code Auditing and Security: Talend showcase

When it comes to Security people think about protecting networks, using firewalls, patching, following OWASP recommendations for web security and a lot more. Code mistakes have always been and will remain the main cause for exploits though.

Code review is in some companies a well established procedure but what about when you actually find an issue that goes beyond automated tools? Tooling cannot keep you distracted from the goal. You should put auditing as part of the SDLC with tooling or without it.

I cannot think about a software engineer (developer or not) not interested in scripting, seriously. Do you think you will always be in charge of just a couple of servers? Wait to see how your job evaporates and is shipped to an efficient team that handles farms of thousand of severs with a dozen individuals. So a word of advise: Do learn scripting.

I believe SQL injection account for the highest percentage of security holes exploitable nowadays in the wild. You better audit it.

Let's use as example Talend Open Source. All queries you use in your system are in a plain text file (wise approach) so just using few Unix Power Tools you can audit the code for sql injection:

find /path/to/talend/jobs/ -name "*.item"|xargs grep 'elementParameter.*name="QUERY'|grep '\+.*context\.'

Simple snippets like the above can be included as part of your release/build script (Did I mention you should have those automated?). If there are hits then the build must stop. Period.

Visual Land Games, an inexpensive game console?

Visual Land sells a device in Walmart called "Phantom Portable Media Player" for less than $60 bucks. In some stores you might get it for just $40. I would say this gadget has a decent quality for the price. You can play videos and music, look at pictures, and play some FLASH games. No WIFI neither camera.

With some luck you might be able to find some FLASH games that would play in this device (Only if they are built in flashlite 2.1 and actionscript 2.0 according to Walmart's Q&A).

I had the opportunity to play with this gadget today while helping a friend that bought it for a 6 year old.

Here are the steps to install games in this device:

1. Connect the microsd card to a computer (A USB to MicroSD adapter will be needed). Alternatively you can use the included USB cable, just be sure you eject the two folders that will be mounted before you disconnect the device.
2. Open the microsd card folder named "Games and Applications". The same is available if you use the USB cable under a drive that shows up as "untitled". There is a second drive that shows as "NO NAME" when connected via USB cable which contains I guess the internal storage data.
3. drag and drop flash games into it. Not all flash games work for it. So you will need to try one by one. I downloaded all games from http://vtouchpro.weebly.com/games.html. To download games search for games compatible with this device like the ones you can download from http://vtouchpro.weebly.com/games-downloads.html. You can also experiment with other flash games freely available: Right click on any page with flash games and select "view source". From the source page you see HTML code with some links. Look for links with extension "*.swf". Click on the link and the game alone should open in your browser. Go to "save" option from the browser to download them to your computer (local disk or the microsd directly)
4. Eject the card from your Computer
5. Retrieve the microsd and insert it back to the device
6. In the Device go to "Applications|Browser|Card|Games and Applications" and select a game from there.

Find Java packages or jars containing a class

"Find Class In Jars" or "Find package in Jars" is a common need for people using Java and Java related products especially after getting the infamous java.lang.NoClassDefFoundError

This POB recipe named findClassInJars.sh can be run locally or remotely to inspect all jars recursively starting at a parent directory, hopefully finding which jar contains the class that is not being loaded.

Here is an example of the script in action:

$ ~/eclipse-workspace-test/recipes/common/tools/findClassInJars.sh ~/Downloads/TOS-All-r67267-V4.2.3 javax/mail/internet/MimeUtility


---------------------------------------------
file: /Users/nestor/Downloads/TOS-All-r67267-V4.2.3//configuration/org.eclipse.osgi/bundles/2193/1/.cp/components/tESBConsumer/geronimo-javamail_1.4_spec-1.7.1.jar
---------------------------------------------
16881 Fri Jun 11 06:45:40 EDT 2010 javax/mail/internet/MimeUtility.class

---------------------------------------------
file: /Users/nestor/Downloads/TOS-All-r67267-V4.2.3//lib/java/geronimo-javamail_1.4_spec-1.7.1.jar
---------------------------------------------
16881 Fri Jun 11 06:45:40 EDT 2010 javax/mail/internet/MimeUtility.class

---------------------------------------------
file: /Users/nestor/Downloads/TOS-All-r67267-V4.2.3//lib/java/mail-1.4.jar
---------------------------------------------
455 Fri Apr 28 12:28:34 EDT 2006 javax/mail/internet/MimeUtility$1NullInputStream.class 18168 Fri Apr 28 12:28:34 EDT 2006 javax/mail/internet/MimeUtility.class

---------------------------------------------
file: /Users/nestor/Downloads/TOS-All-r67267-V4.2.3//lib/java/mail.jar
---------------------------------------------
18817 Fri Jan 14 15:37:10 EST 2011 javax/mail/internet/MimeUtility.class 455 Fri Jan 14 15:37:10 EST 2011 javax/mail/internet/MimeUtility$1NullInputStream.class

---------------------------------------------
file: /Users/nestor/Downloads/TOS-All-r67267-V4.2.3//plugins/org.talend.designer.components.localprovider_4.2.3.r67267/components/ext/tesb/tESBConsumer/geronimo-javamail_1.4_spec-1.7.1.jar
---------------------------------------------
16881 Fri Jun 11 06:45:40 EDT 2010 javax/mail/internet/MimeUtility.class

---------------------------------------------
file: /Users/nestor/Downloads/TOS-All-r67267-V4.2.3//plugins/org.talend.designer.components.localprovider_4.2.3.r67267/components/tMicrosoftCrmInput/mail-1.4.jar
---------------------------------------------
455 Fri Apr 28 12:28:34 EDT 2006 javax/mail/internet/MimeUtility$1NullInputStream.class 18168 Fri Apr 28 12:28:34 EDT 2006 javax/mail/internet/MimeUtility.class

---------------------------------------------
file: /Users/nestor/Downloads/TOS-All-r67267-V4.2.3//plugins/org.talend.designer.components.localprovider_4.2.3.r67267/components/tWebServiceInput/mail.jar
---------------------------------------------
18817 Fri Jan 14 15:37:10 EST 2011 javax/mail/internet/MimeUtility.class 455 Fri Jan 14 15:37:10 EST 2011 javax/mail/internet/MimeUtility$1NullInputStream.class

---------------------------------------------
file: /Users/nestor/Downloads/TOS-All-r67267-V4.2.3//plugins/org.talend.libraries.apache.axis2_4.2.3.r67267/lib/mail-1.4.jar
---------------------------------------------
455 Fri Apr 28 12:28:34 EDT 2006 javax/mail/internet/MimeUtility$1NullInputStream.class 18168 Fri Apr 28 12:28:34 EDT 2006 javax/mail/internet/MimeUtility.class

Put apache in maintenance mode

Sometimes you need to perform actions in your backend and at the same time inform your visitors about it without presenting a weird error message.

Here is a POB recipe to switch on and off "maintenance mode" in a remote apache server. Dod I mention I run this remotely in my servers using Remoto-IT?

It assumes you have a virtual hosts file with some commented out rewrite rules, for example:

#RewriteEngine on
#RewriteCond %{REQUEST_URI} !^(/html|/images|/js|/css).*$ [NC]
#RewriteRule ^(.*)$ /html/error/503.html [L,R=301]

It will then just comment the lines out or put them back based on the remote invocation of the script:

common/apache/site-in-maintenance.sh /etc/apache2/sites-available/bhub-ssl true

So of course rather than touching the files manually in every server you can just send the command over the wire which is both more predictable and faster.

I am "reusing" here the same page that will be rendered in the case the application is indeed down for any other reason but can definitely have two different pages for such different situations.

Serving static content from Apache instead of tomcat with just modjk

Is apache faster than tomcat serving static content? It was true some years ago, I am not that sure the statement is true nowadays.

However if you use apache as load balancer with modjk there will be overhead sending all those packets to each tomcat server and getting the response back. So actually with such environment static content should be served by Apache unless of course you have security constraints that are already implemented in tomcat which affect that static content.

On the other hand what happens when you go on maintenance mode for let us say a deployment that is taking beyond a couple of seconds (very common)? Would you leave the default HTTP 503 error to be sent back to the user? Bad User eXperience I would say.

All you need to do is play with two directives in your virtual host to make sure certain requests do not go to tomcat but stay in Apache. Here are the two statements which make sure a directory called html containing (you guessed it) static markup is rendered from Apache. We use it to make sure our customized HTTP 503 error page is correctly rendered.

<VirtualHost  sample.com:443>
...
 ErrorDocument 503 /html/error/503.html
 SetEnvIf Request_URI "/html/*" no-jk
 Alias /html /var/myapp/html
...

You might want a more complicated page respecting the look and feel of your website. You already have images and CSS and you of course will not repeat yourself so you won't copy and paste. You can share your current images and CSS adding more "SetEnvIf Request_URI" and "Alias" directives like the above:

<VirtualHost  sample.com:443>
...
 ErrorDocument 503 /html/error/503.html
 SetEnvIf Request_URI "/html/*" no-jk
 Alias /html /var/myapp/html
 SetEnvIf Request_URI "/css/*" no-jk
 Alias /css /var/myapp/css
 SetEnvIf Request_URI "/images/*" no-jk
 Alias /images /var/myapp/images
...

Your deployment script should of course account for deploying what is necessary to apache. Your resources stay in your project like usual and of course they can be used from plain tomcat in case you (for example) develop in a machine without Apache (which I do not recommend, did I say I believe the development environment should be as close as possible to production environment?) or from Apache as explained.

There are other important tasks the sysadmin will need to perform (or you still call it the webmaster? :) like for example communicating users about future hours of downterm for maintenance purpose. I can't picture someone doing all this manually really. At a minimum I would expect POB recipes ready to be run in remote apache servers to ensure configuration files and resources are properly changed and reloaded as needed. Remoto-IT can help with that.

Tomcat catalina.out rotation and truncation

Tomcat rotates catalina.out in a way that it logs for the current date everything in its own file ( catalina.yyyy-MM-DD.log ) but it also keeps the whole log without truncation in catalina.out. Most of the traces going to catalina.out can be avoided removing the Console Handler from the below line in conf/logging.properties:

.handlers = 1catalina.org.apache.juli.FileHandler, java.util.logging.ConsoleHandler

The reality is that still catalina.out will grow if any library not using logging decides to send its output to the stdout, for example (probably not the best example for production as you won't run a debugger there, will you?):

Listening for transport dt_socket at address: 8000

This might not a big deal after all in terms of file size but we will agree it is easier to read from the same file when monitoring your logs rather than having to inspect a different log every day. It makes sense IMO to go to rotated logs only for historical reasons.

I believe that setting a file size you are comfortable with and making sure any time the catalina.out goes beyond that size is truncated makes the most sense. However that is not easy to be achieved from outside tomcat itself.

Truncating catalina.out from the stop command in catalina.sh makes the most sense out of the very different ways you can stop catalina.out from growing. That means right before the stop section ends (before the else for configtest) we can resolve this issue just adding:

...
  if [ "`stat -c %s $CATALINA_OUT`" -gt "$CATALINA_OUT_MAX_SIZE" ]; then
    cp /dev/null "$CATALINA_OUT"
  fi
elif [ "$1" = "configtest" ] ; then
...

Of course you need to configure the value for $CATALINA_OUT_MAX_SIZE in setenv.sh.

Please understand that catalina.out will stop growing once the log file is greater than CATALINA_OUT_MAX_SIZE *AND* the stop command is used. This might mean you can have eventually a file bigger than the expected CATALINA_OUT_MAX_SIZE but hopefully it will not go out of control as it is the case when you do not take proper measures.

Ubuntu 12.04 Unity Launcher How To

Every time I have to come back to Ubuntu Desktop there is some major change I need to adapt to. I will make no comparisons with other Desktop OS but instead will keep here some notes I found useful for Ubuntu 12.04.

1. You run applications by searching for them via "Dash Board", the top icon from the Launcher (the by-default left bar)
2. You can add a shortcut in the Launcher to a particular application: Run the app, right click on its icon from the Launcher and select "Lock to Launcher"
3. The bar can autohide if you use the "Appearance" app, "Behavior" option
4. The Launcher is buggy at least in 12.04 so it might disappear, freeze, God knows what else. You just need to be able to connect to the console and run the below (which will restart the box and try to rebuild all compiz related settings)

   $ mv ~/.config/compiz-1 ~/.config/compiz-1.BACKUP
   $ shutdown -r now