openssl hanging CONNECTED(00000003) SSL_connect:unknown state

This one took me an hour to provide a workaround. Our POB recipes to import LDAP (apacheds) SSL certificates running from Remoto-IT were failing in a remote machine with just one message from openssl s_client: "CONNECTED(00000003)". Using the msg and the state flags we could see a little bit more:

$ openssl version
OpenSSL 1.0.1 14 Mar 2012
$ openssl s_client -msg -state -connect w.x.y.z:10636
CONNECTED(00000003)
SSL_connect:before/connect initialization
>>> TLS 1.1  [length 00dd]
    01 00 00 d9 03 02 51 0a c4 32 c9 50 79 93 ea 10
    7f d1 41 57 3f 52 27 c6 86 df 9b fa 20 09 0c 92
    b4 e8 ae 5f 6c fa 00 00 66 c0 14 c0 0a c0 22 c0
    21 00 39 00 38 00 88 00 87 c0 0f c0 05 00 35 00
    84 c0 12 c0 08 c0 1c c0 1b 00 16 00 13 c0 0d c0
    03 00 0a c0 13 c0 09 c0 1f c0 1e 00 33 00 32 00
    9a 00 99 00 45 00 44 c0 0e c0 04 00 2f 00 96 00
    41 c0 11 c0 07 c0 0c c0 02 00 05 00 04 00 15 00
    12 00 09 00 14 00 11 00 08 00 06 00 03 00 ff 02
    01 00 00 49 00 0b 00 04 03 00 01 02 00 0a 00 34
    00 32 00 0e 00 0d 00 19 00 0b 00 0c 00 18 00 09
    00 0a 00 16 00 17 00 08 00 06 00 07 00 14 00 15
    00 04 00 05 00 12 00 13 00 01 00 02 00 03 00 0f
    00 10 00 11 00 23 00 00 00 0f 00 01 01
SSL_connect:unknown state

Look at the negotiation trying to use TLS 1.1 for openssl 1.0.1. From my MAC I could negotiate though:

$ openssl version
OpenSSL 0.9.8r 8 Feb 2011
$ echo |  openssl s_client -msg -state -connect w.x.y.z:10636
CONNECTED(00000003)
SSL_connect:before/connect initialization
>>> SSL 2.0 [length 0080], CLIENT-HELLO
    01 03 01 00 57 00 00 00 20 00 00 39 00 00 38 00
    00 35 00 00 16 00 00 13 00 00 0a 07 00 c0 00 00
    33 00 00 32 00 00 2f 00 00 9a 00 00 99 00 00 96
    03 00 80 00 00 05 00 00 04 01 00 80 00 00 15 00
    00 12 00 00 09 06 00 40 00 00 14 00 00 11 00 00
    08 00 00 06 04 00 80 00 00 03 02 00 80 00 00 ff
    5d df 78 59 05 15 8f fc d4 df 62 0f b5 b7 e8 79
    af 6e 49 22 09 5b 1c 89 5a 96 49 fa b1 a2 41 91
SSL_connect:SSLv2/v3 write client hello A
<<< TLS 1.0 Handshake [length 0051], ServerHello
    02 00 00 4d 03 01 51 0a c5 e1 1c 6f 7f cc c2 92
    8c 3c 02 3c ad 42 04 59 35 35 be 24 72 b6 92 27
    a7 b9 a7 fc ae c2 20 51 0a c5 e1 a5 5e 9a 62 d8
    1f 5d c4 be 3e 7d ef 89 b1 67 6d 5a db 20 3c d2
    5b 3f 1f 19 e2 f1 83 00 39 00 00 05 ff 01 00 01
    00
SSL_connect:SSLv3 read server hello A
...

But then look at the negotiation trying to use SSL2.0 up front for the Lion client openssl. While the openssl project has been trying to solve issues like this (hangs) I believe there is still some code review to be made as previous versions of openssl were working correctly in terms of negotiations. For now we managed to get around this issue passing SSL3 as the preferred secure transport:

$ echo |  openssl s_client -ssl3 -msg -state -connect w.x.y.z:10636
CONNECTED(00000003)
SSL_connect:before/connect initialization
>>> SSL 3.0 Handshake [length 005a], ClientHello
...

It might not be an option for you in which case just try with other available protocol options. BTW this happened in Ubuntu 12.04.

Infrastructure as a Service demands high level of automation. Basically all best practices taught for software developers should be followed by Ops guys as well, especially when it comes to automating the building of environments and testing. These practices should be mandated and not overlooked to avoid surprises in production systems while still being as agile as possible on the Infrastructure side. Back to the lab ...

Caching and minifying with Apache = Faster websites

Static content should be cached and we already know how to do that in apache. Note the removal of the E-Tag header is necessary otherwise the browser will be forced to ask the server if the resource has been modified through the header "If-None-Match". This will cause unnecessary requests to Apache which will return back with just a 304 since the resource has not been modified. While this still saves throughput is far from ideal as the Browser still needs to open a socket (read consume resources) to the server:

$ sudo a2enmod expires
$ vi /etc/sites-available/mysite-ssl
...
 <Directory "/var/mysite">
    
    FileETag none
    ExpiresActive On
    ExpiresByType text/html "access plus 5 month"
    ExpiresByType text/plain "access plus 1 month"
    ExpiresByType image/png "access plus 1 month"
    ExpiresByType image/jpg "access plus 1 month"
    ExpiresByType text/css  "access plus 1 month"
    ExpiresByType application/javascript  "access plus 1 month"
    ExpiresDefault "access plus 1 month"

...

The above should result on something like the below (Note the date will always be something in the future):

...
Cache-Control:max-age=2592000
...
Expires:Sun, 08 Jun 2012 21:03:23 GMT
...

Needless to say the above should be enclosed in recipes. The first is just a command while the second needs to download from your repository the server vhost file and restart apache after.

So we are done now, we are caching and our site is really fast. Well, not that fast. Developers will not take care of minifying the code, they need it to be legible so they can change it. We need to make sure javascript, CSS and HTML contains just what is needed. Yes we already enabled gzip in apache but that is not enough. Formatting characters (only needed at development time) will need to be compressed adding to CPU time and after all the file will be bigger than a file not containing formatting characters. So yes you need to minify the code.

Here is a POB recipe which installs and configures Google apache mod-pagespeed with the bare minimums you need to make your site faster. Of course you can run that recipe in remote servers using Remoto-IT. Here is the output associated to that recipe which BTW needs an NFS share hosting the debian binary:

Installing Google pagespeed module
Selecting previously unselected package mod-pagespeed-beta.
(Reading database ... 157359 files and directories currently installed.)
Unpacking mod-pagespeed-beta (from mod-pagespeed-beta_current_amd64.deb) ...
Setting up mod-pagespeed-beta (1.2.24.1-r2300) ...
Enabling module pagespeed.
To activate the new configuration, you need to run:
  service apache2 restart
New Google pagespeed module configuration:
    # ModPagespeedDisableFilters and ModPagespeedEnableFilters
    ModPagespeedEnableFilters rewrite_javascript,rewrite_css
    ModPagespeedEnableFilters collapse_whitespace,elide_attributes
    ModPagespeedEnableFilters canonicalize_javascript_libraries
    # ModPagespeedEnableFilters add_instrumentation
 * Restarting web server apache2   

To test your site is caching static content:

1. Open the website from chrome
2. Right click on the page and select "Inspect element"
3. Go to network tab
4. Hit enter in the address bar. Repeating this action should show all static content like images taking zero second to load as they are loaded from the cache
5. As a reminder you can force a reloading just hitting reload (F5)

Automating Code Auditing and Security: Talend showcase

When it comes to Security people think about protecting networks, using firewalls, patching, following OWASP recommendations for web security and a lot more. Code mistakes have always been and will remain the main cause for exploits though.

Code review is in some companies a well established procedure but what about when you actually find an issue that goes beyond automated tools? Tooling cannot keep you distracted from the goal. You should put auditing as part of the SDLC with tooling or without it.

I cannot think about a software engineer (developer or not) not interested in scripting, seriously. Do you think you will always be in charge of just a couple of servers? Wait to see how your job evaporates and is shipped to an efficient team that handles farms of thousand of severs with a dozen individuals. So a word of advise: Do learn scripting.

I believe SQL injection account for the highest percentage of security holes exploitable nowadays in the wild. You better audit it.

Let's use as example Talend Open Source. All queries you use in your system are in a plain text file (wise approach) so just using few Unix Power Tools you can audit the code for sql injection:

find /path/to/talend/jobs/ -name "*.item"|xargs grep 'elementParameter.*name="QUERY'|grep '\+.*context\.'

Simple snippets like the above can be included as part of your release/build script (Did I mention you should have those automated?). If there are hits then the build must stop. Period.

Visual Land Games, an inexpensive game console?

Visual Land sells a device in Walmart called "Phantom Portable Media Player" for less than $60 bucks. In some stores you might get it for just $40. I would say this gadget has a decent quality for the price. You can play videos and music, look at pictures, and play some FLASH games. No WIFI neither camera.

With some luck you might be able to find some FLASH games that would play in this device (Only if they are built in flashlite 2.1 and actionscript 2.0 according to Walmart's Q&A).

I had the opportunity to play with this gadget today while helping a friend that bought it for a 6 year old.

Here are the steps to install games in this device:

1. Connect the microsd card to a computer (A USB to MicroSD adapter will be needed). Alternatively you can use the included USB cable, just be sure you eject the two folders that will be mounted before you disconnect the device.
2. Open the microsd card folder named "Games and Applications". The same is available if you use the USB cable under a drive that shows up as "untitled". There is a second drive that shows as "NO NAME" when connected via USB cable which contains I guess the internal storage data.
3. drag and drop flash games into it. Not all flash games work for it. So you will need to try one by one. I downloaded all games from http://vtouchpro.weebly.com/games.html. To download games search for games compatible with this device like the ones you can download from http://vtouchpro.weebly.com/games-downloads.html. You can also experiment with other flash games freely available: Right click on any page with flash games and select "view source". From the source page you see HTML code with some links. Look for links with extension "*.swf". Click on the link and the game alone should open in your browser. Go to "save" option from the browser to download them to your computer (local disk or the microsd directly)
4. Eject the card from your Computer
5. Retrieve the microsd and insert it back to the device
6. In the Device go to "Applications|Browser|Card|Games and Applications" and select a game from there.

Find Java packages or jars containing a class

"Find Class In Jars" or "Find package in Jars" is a common need for people using Java and Java related products especially after getting the infamous java.lang.NoClassDefFoundError

This POB recipe named findClassInJars.sh can be run locally or remotely to inspect all jars recursively starting at a parent directory, hopefully finding which jar contains the class that is not being loaded.

Here is an example of the script in action:

$ ~/eclipse-workspace-test/recipes/common/tools/findClassInJars.sh ~/Downloads/TOS-All-r67267-V4.2.3 javax/mail/internet/MimeUtility


---------------------------------------------
file: /Users/nestor/Downloads/TOS-All-r67267-V4.2.3//configuration/org.eclipse.osgi/bundles/2193/1/.cp/components/tESBConsumer/geronimo-javamail_1.4_spec-1.7.1.jar
---------------------------------------------
16881 Fri Jun 11 06:45:40 EDT 2010 javax/mail/internet/MimeUtility.class

---------------------------------------------
file: /Users/nestor/Downloads/TOS-All-r67267-V4.2.3//lib/java/geronimo-javamail_1.4_spec-1.7.1.jar
---------------------------------------------
16881 Fri Jun 11 06:45:40 EDT 2010 javax/mail/internet/MimeUtility.class

---------------------------------------------
file: /Users/nestor/Downloads/TOS-All-r67267-V4.2.3//lib/java/mail-1.4.jar
---------------------------------------------
455 Fri Apr 28 12:28:34 EDT 2006 javax/mail/internet/MimeUtility$1NullInputStream.class 18168 Fri Apr 28 12:28:34 EDT 2006 javax/mail/internet/MimeUtility.class

---------------------------------------------
file: /Users/nestor/Downloads/TOS-All-r67267-V4.2.3//lib/java/mail.jar
---------------------------------------------
18817 Fri Jan 14 15:37:10 EST 2011 javax/mail/internet/MimeUtility.class 455 Fri Jan 14 15:37:10 EST 2011 javax/mail/internet/MimeUtility$1NullInputStream.class

---------------------------------------------
file: /Users/nestor/Downloads/TOS-All-r67267-V4.2.3//plugins/org.talend.designer.components.localprovider_4.2.3.r67267/components/ext/tesb/tESBConsumer/geronimo-javamail_1.4_spec-1.7.1.jar
---------------------------------------------
16881 Fri Jun 11 06:45:40 EDT 2010 javax/mail/internet/MimeUtility.class

---------------------------------------------
file: /Users/nestor/Downloads/TOS-All-r67267-V4.2.3//plugins/org.talend.designer.components.localprovider_4.2.3.r67267/components/tMicrosoftCrmInput/mail-1.4.jar
---------------------------------------------
455 Fri Apr 28 12:28:34 EDT 2006 javax/mail/internet/MimeUtility$1NullInputStream.class 18168 Fri Apr 28 12:28:34 EDT 2006 javax/mail/internet/MimeUtility.class

---------------------------------------------
file: /Users/nestor/Downloads/TOS-All-r67267-V4.2.3//plugins/org.talend.designer.components.localprovider_4.2.3.r67267/components/tWebServiceInput/mail.jar
---------------------------------------------
18817 Fri Jan 14 15:37:10 EST 2011 javax/mail/internet/MimeUtility.class 455 Fri Jan 14 15:37:10 EST 2011 javax/mail/internet/MimeUtility$1NullInputStream.class

---------------------------------------------
file: /Users/nestor/Downloads/TOS-All-r67267-V4.2.3//plugins/org.talend.libraries.apache.axis2_4.2.3.r67267/lib/mail-1.4.jar
---------------------------------------------
455 Fri Apr 28 12:28:34 EDT 2006 javax/mail/internet/MimeUtility$1NullInputStream.class 18168 Fri Apr 28 12:28:34 EDT 2006 javax/mail/internet/MimeUtility.class

Put apache in maintenance mode

Sometimes you need to perform actions in your backend and at the same time inform your visitors about it without presenting a weird error message.

Here is a POB recipe to switch on and off "maintenance mode" in a remote apache server. Dod I mention I run this remotely in my servers using Remoto-IT?

It assumes you have a virtual hosts file with some commented out rewrite rules, for example:

#RewriteEngine on
#RewriteCond %{REQUEST_URI} !^(/html|/images|/js|/css).*$ [NC]
#RewriteRule ^(.*)$ /html/error/503.html [L,R=301]

It will then just comment the lines out or put them back based on the remote invocation of the script:

common/apache/site-in-maintenance.sh /etc/apache2/sites-available/bhub-ssl true

So of course rather than touching the files manually in every server you can just send the command over the wire which is both more predictable and faster.

I am "reusing" here the same page that will be rendered in the case the application is indeed down for any other reason but can definitely have two different pages for such different situations.

Serving static content from Apache instead of tomcat with just modjk

Is apache faster than tomcat serving static content? It was true some years ago, I am not that sure the statement is true nowadays.

However if you use apache as load balancer with modjk there will be overhead sending all those packets to each tomcat server and getting the response back. So actually with such environment static content should be served by Apache unless of course you have security constraints that are already implemented in tomcat which affect that static content.

On the other hand what happens when you go on maintenance mode for let us say a deployment that is taking beyond a couple of seconds (very common)? Would you leave the default HTTP 503 error to be sent back to the user? Bad User eXperience I would say.

All you need to do is play with two directives in your virtual host to make sure certain requests do not go to tomcat but stay in Apache. Here are the two statements which make sure a directory called html containing (you guessed it) static markup is rendered from Apache. We use it to make sure our customized HTTP 503 error page is correctly rendered.

<VirtualHost  sample.com:443>
...
 ErrorDocument 503 /html/error/503.html
 SetEnvIf Request_URI "/html/*" no-jk
 Alias /html /var/myapp/html
...

You might want a more complicated page respecting the look and feel of your website. You already have images and CSS and you of course will not repeat yourself so you won't copy and paste. You can share your current images and CSS adding more "SetEnvIf Request_URI" and "Alias" directives like the above:

<VirtualHost  sample.com:443>
...
 ErrorDocument 503 /html/error/503.html
 SetEnvIf Request_URI "/html/*" no-jk
 Alias /html /var/myapp/html
 SetEnvIf Request_URI "/css/*" no-jk
 Alias /css /var/myapp/css
 SetEnvIf Request_URI "/images/*" no-jk
 Alias /images /var/myapp/images
...

Your deployment script should of course account for deploying what is necessary to apache. Your resources stay in your project like usual and of course they can be used from plain tomcat in case you (for example) develop in a machine without Apache (which I do not recommend, did I say I believe the development environment should be as close as possible to production environment?) or from Apache as explained.

There are other important tasks the sysadmin will need to perform (or you still call it the webmaster? :) like for example communicating users about future hours of downterm for maintenance purpose. I can't picture someone doing all this manually really. At a minimum I would expect POB recipes ready to be run in remote apache servers to ensure configuration files and resources are properly changed and reloaded as needed. Remoto-IT can help with that.