Friday, December 20, 2013

On Kanban: Is there a need for Kanban Certification? My answer is Yes, there is.

David J. Anderson has posted this question in a couple of places. My answer in short is: Yes.

I would certify the theoretical understanding of the individual to act as a coach.

"Kanban Coaching Professional" can be a good title. As the title suggests the certification will support the fact that the professional can coach a team guiding them through continuous improvement, from following Deming's System of Profound Knowledge and the 14 Points for Management all the way up to your 5 steps in the recipe for success.

I personally like the idea of Project Coach better than Project Manager. And of course as any other certification, passing it does not mean that you are actually a seasoned coach but that is of course subject for a different discussion.

Maven and Jenkins for Continuous Delivery (release and deployment)

That is the agile lean goal, to deliver value at a constant pace with minimum manual intervention right?

I have written before about continuously releasing snapshots but in reality what you want to make sure is that once something is tested it can be deployed and that can only be achieved if what you have tested and verified is a release.

Here is how to use Maven and Jenkins to help the team with continuous releases of maven projects.

The proposal here is to use only 3 digits ( M.m.b meaning Major, minor and bug fixing ) for any release version number. We can continuously release increasing the minor version number keeping bug fixing number equal zero. Of course we want to leave the major version number unchangeable because most likely that would say something about the maturity of our platform. This is usually a version that has even commercial meaning so we better leave it as a configuration parameter which will be touched only rarely.


Developers work on a SNAPSHOT project. If the project has modules then you use Aggregation (or Multi-Module) How can we make sure such project and its modules are all released without human intervention? In other words how can use Maven to force a release version from a single command? The below command uses -DdryRun which you would remove in real life but it is handy to see what would the command do as we soon will learn:
mvn clean --batch-mode release:prepare -DdryRun=true -DautoVersionSubmodules=true -DreleaseVersion=2.3000.0 -DdevelopmentVersion=2.3001.0-SNAPSHOT
The above releases all releasable (meaning ending in -SNAPSHOT) modules tagging them with the same ${releaseVersion} and the root project as well. You can confirm that looking at all pom.xml.tag files that are locally generated for each project. It also (as expected) changes the version number for the projects so developers can continue working on new features for the provided ${developmentVersion}. You might wonder why I decided to increase by 1 the minor number for the next development SNAPSHOT version. The reason is I want to make sure the team understand the last number is reserved for patching an existing in production version. The next expected version will have the released minor version number plus one only if the CI server actually did no other release before as you soon will learn so most likely the next released version minor number will be higher than 3001 in reality.

You can try the above again and again with different combinations provided that you remove the tag, next, releaseBackup and release properties temporary files:
find ../ -name "pom.xml.*"|grep -v svn|xargs rm -f; \
find ../ -name ""|xargs rm -f


  1. Check "This build is parametrized". Define a String parameter called "MAJOR_VERSION_NUMBER" with default value equal your current release major version number. In our case this is just "2". Later on when building manually from Jenkins the parameter is pre-completed but definitely changeable.
  2. Configure the build to invoke maven with goals and options that apply to your specific case. Following our example:
    mvn clean --batch-mode release:prepare -DdryRun=true -DautoVersionSubmodules=true -DreleaseVersion=$MAJOR_VERSION_NUMBER.$BUILD_NUMBER.0 -DdevelopmentVersion=$MAJOR_VERSION_NUMBER.$(($BUILD_NUMBER+1)).0-SNAPSHOT
    The shebang here is important to make sure parameter expansion works. You can see how we add 1 as discussed before.
  3. I started configuring the build to invoke maven with goals and options that apply to your specific case. In our case go to "Perform Maven Release" and fill the boxes as in:
    Development version: $MAJOR_VERSION_NUMBER.$(($BUILD_NUMBER+1)).0-SNAPSHOT
    Dry run only?: check it to run just this as a POC
    However at the time of this writing the parameter expansion won't work in the spot above. You will need to use the literal maven command instead.

bash: extract tokens from a string using parameter expansion for example domain or host from url

There is literally no developer or sysadmin which will not understand what $var is but parameter expansion in bash is more than just a simple variable. When it comes to parsing strings most developers are familiar with high level functions that allow to tokenize them. Bash parameter expansion can help with this task using substring removal. Double hash-mark and double percentage-mark are your friends.

Below is a solution for the typical problem related to parsing urls:
echo $url_no_params 
echo $params
echo $host_and_path
echo $host
The above will result in:
$echo $url_no_params
$echo $params
$echo $host_and_path
$echo $host

Java, is the JVM responsible for: OS Unable to fork: Cannot allocate memory ?

CREDITS: I would like to thank my old friend and now again coworker Josu Feijoo for his help on getting to the conclusions below.

This error means that no more processes or threads can be open due to memory starvation. Most likely a hard reset will be needed so certainly it is not a nice problem to have.

Clearly its resolution depends on what you are running but most likely there are too many threads been run in your system so the first step for troubleshooting would be to get all threads and identify the culprit:
$ top -H -b -n 1 
After you identify the culprit (tomcat in our case) take a look at how threads are going up indicating a leak:
$ top -H -b -n 1 | grep tomcat -c
In Java the JVM memory requirements are the sum of maximum heap memory, the maximum "perm space" memory and the number of threads multiplied by the thread stack size:
Xmx + MaxPermSize + (Xss * number of threads)
That is the reason Virtual memory consumption is so important. I see a lot of miss-leading comments and posts on the web claiming that you should only worry about resident memory (top RES) and not much for virtual memory (top VIRT) and under those assumptions some engineers might think the below is simply OK. Note how this JVM is using 2.3GB physical memory but 18GB virtual memory:
$ ps -e -o rss,vsize,cmd | grep tomcat
2298588 18019720 /opt/jdk/bin/java -Xmx2048m -XX:MaxPermSize=512m
This should not be OK for most web applications. Unless on purpose you need to maintain a lot of threads you should be suspecting of thread leaking. At that point you need to use jstack, jvisualvm or any other tool that allows you to connect to the debug port of the JVM to further troubleshoot which is the responsible leaker in the Java code. Or perhaps you can quickly inspect your code for threads you open and simply do not stop. You might be using some library or middleware responsible for it like the Camel ProducerTemplate. Housekeeping is necessary, resources are not infinite and those of us ignoring that rule will pay with a resource starvation surprise. stack

Needless to say that if you really need to spawn so many threads then do not forget to run the JVM with enough memory that accounts also for the thread stack consumption.

Add network routes in MAC OSX

Here is a snippet that shows how to add routes to a network only available after a VPN connection has been established. Note the two commands used, one refers to the interface (the vpn interface ppp0) while the other refers to the gateway (
$ sudo route -n add -net  -interface ppp0
add net gateway ppp0
$ telnet 3389
Connected to
Escape character is '^]'
$ sudo route -n delete
delete host
$ telnet 3389
^C #timeout
$ sudo route -n add -net  -gateway
add net
$ telnet 3389
Connected to
Escape character is '^]'

Monday, December 16, 2013

Find which process owns the socket listener AKA open port

In one word, lsof. For example the below will list the process which holds port 8080 open:
lsof -i tcp:8088
This is a useful one which any Linux sysadmin should be aware of.

Saturday, December 14, 2013

top and cron - Log all commands being run every minute

To troubleshoot what is going on in Unix and Linux systems your first step is usually top. You can run it in two modes: interactive and batch mode. In batch mode the output can be piped to any other command making it ideal to leave it croned in a server to later on analyze what is going on there:
*/1 * * * *  COLUMNS=512 top -b -n 1 >> /tmp/top.log
Option b stands for batch mode. Finally "n 1" means top to end after a single iteration/refresh.

Friday, December 13, 2013

Java client unable to find valid certification path to requested target

The keytool command is used to manipulate the java keystore. Using this POB recipe you should be able to authorize any certificate including self signed certificates, even expired certificates to your keystore. Do not add a self signed or expired certificate to the keystore of production servers though! So today we had the below problem which I have seen before multiple times before:
com.sun.jersey.api.client.ClientHandlerException: PKIX path building failed: unable to find valid certification path to requested target
at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(
However adding the certificate to the keystore had no effect. The only explanation for a behavior like this is that the client java program is not using still the certificate meaning most likely is not pointing to the keystore we think it is. Path issues are the first to look for and in my case someone for some reason pointed the java binary to the default java installation in the servers:
$ ls -al /usr/bin/java
lrwxrwxrwx 1 root root 22 Dec 13 13:39 /usr/bin/java -> /etc/alternatives/java
Which was corrected manually as:
sudo rm /usr/bin/java
sudo ln -s /opt/jdk/bin/java  /usr/bin/java

Thursday, December 12, 2013

Kanban: Prioritization, Estimation, Classes of Service, Severity, Priority Number and Technology debt

The blue book is a gem for Software Project Managers. For many years after reading the Bible of Software Engineering I did not find a concise explanation that would give me hope on the possible existence after all of a Silver Bullet for Project Management but I have found in David J. Anderson's book a great must-read-first for those still looking for it.
Prioritization and estimation are important and at the same wasteful activities. At the core of a good decision is a proper triage system.
Every software issue should have a class of service (COS) to make sure its high level priority is understood and that the proper SLA is applied.
IMO they are not to be confused with severity which is an important classification for Standard and Intangible COS or priority number which allows to further qualify tickets within severity.
COS and teams are orthogonal. Let us analyze an example to clarify how technology support tickets can have any COS.
Technology tickets are *not* always intangible. Technology debt can be a huge problem if the tickets are not properly prioritized. As technology leaders we need to make sure we explain these important concepts to the stakeholders. Classes of Services are completely orthogonal to teams.

Expedite Class Of Service

Consider a bug in SSL like Perfect Forward Secrecy (PFS) which might result in a violation of Massachusetts and California Privacy Laws. Resolving this vulnerability in your web server must be considered probably an Expedite class issue.

Fixed Delivery Date Class Of Service

Let's supposed you have received an alert about the need to put another server in the cluster as capacity has gone over the imposed threshold and in less than a week the cluster will most likely become unresponsive. This case will be considered a Fixed Delivery Date class issue.

Standard Class Of Service

Your calendar has reminded you about the need to upgrade to the newest long term supported Operating System before the current one gets out of support in a year from now. This is to be considered a Standard class issue which could escalate to Fixed Delivery Date or even Expedite but at the moment we are plenty of time to get it done.

Intangible Class Of Service

There is a request for a devops script responsible for a server clone to go faster. This will be Intangible class unless demonstrated that it will save hours of development work in which case it might become Standard.

Tuesday, December 10, 2013

Add username in apache logs with form authentication

If you use basic authentication the Apache log option "%u" will log the current logged in user. However if you use form authentication you will need to play with response headers. Below a typical custom log for your apache + Java/JEE:
LogFormat "%h %l %{USER}o %t \"%r\" %>s %b %{JSESSIONID}C" custom
CustomLog /var/log/apache2/ custom
It will generate something like: - myUser - [28/Sep/2013:18:09:40 -0400] "GET /my/path HTTP/1.1" 200 24292 6FCC544E05F7F5D31691C5907F99CFAA.node1
The user will only be logged if "USER" is set as response header in the Java / JEE server.

Friday, December 06, 2013

When ping does not work use tcptraceroute

WARNING: Make sure you own the machine you are troubleshooting. Port probing might be considered illegal where you live.
tcptraceroute 80