strace and java - Talend JVM troubleshooting

While you could use jstack I always find myself preferring my good all friends strace or truss when things get really difficult. Ultimately system calls will reveal way more of what you need. You actually find interesting problems in your code as well like unnecessary disk and network access ;-)

Suppose you have a Talend java job which is after all a shell script wrapping a java command. Such job would look like:

jobname/jobname/jobname_run.sh --context_param 'p1=v1' --context_param 'p2=v2'

To trace that command you would create a test.sh, add the line above followed by an ampersand to run the command in the background and the command below after:

sudo strace -F -p`/opt/jdk/bin/jps|grep jobname|cut -f1 -d ' '`

All you are doing is making sure your "jobname" java command is traced so far. Now the final step would be to send to a file the output of that trace:

./test.sh 2>&1 | tee ~/strace-talend-ti.log

As an example just so you have an idea the talend job was not saying so much, just that it could not open a file:

Exception in component tFileTouch_2 java.io.IOException: No such file or directory at java.io.UnixFileSystem.createFileExclusively(Native Method) at java.io.File.createNewFile(File.java:947)

But the name of the file was missing. Of course inspecting the source code could help you getting an idea, but what about business rules? The file name might be a complicated string to be determined without realtime debugging. Now you start thinking, I just need to put a trace from my talend job so I log the exact file name. However this might be production and your release cycle is prohibiting at the moment any quick traces without compromising other deliverables. Tracing at system level will give you that information:

[pid 20946] open("/path/to/tmp.file", O_RDWR|O_CREAT|O_EXCL, 0666) = -1 ENOENT (No such file or directory)

SFTP with password from script

The first question I have is if you have access to SFTP client and server? If you do then do not use password, instead use public key authentication.

Then now back to real world. You need to connect to an external SFTP server, the owner does not want to provide public key authentication. A second scenario is a client who wants to connect to your SFTP and they do not want to use certificates either. These two scenarios are driven by bad decisions but those are decisions that are not for you to take nor to fight. You make it clear in a polite email "Please be advise that using passwords is less secure than using public key authentication since it is more vulnerable to brute force attacks". I do not want to engage into the eternal discussion about where to keep the password or the key, so let us move on.

You could use expect and bash to resolve this issue. Remoto-IT uses expect for example to command remote servers. It works perfectly and definitely you could use the same way expect to interact with SFTP.

But there is a great program called lftp which is available in most Unix/Linux distros and if not you can always compile from sources. Here is an example of how to install it and use it in ubuntu:

# Ubuntu installation
$ sudo apt-get install lftp
# Save the list directory command to a file
$ echo dir > commands.sftp
# Run all commands from a file
$ cat commands.sftp | lftp -u myUser,myPassword sftp://sftp.sample.com
#Run a command directly from #linuxoneliner
$ echo dir | lftp -u -u myUser,myPassword sftp://sftp.sample.com

Mac OSX brew MD5 mismatch

I was trying to install aacgain today when I got:

$ brew install aacgain
==> Downloading http://altosdesign.com/aacgain/alvarez/aacgain-1.8.tar.bz2
######################################################################## 100.0%
Error: MD5 mismatch
Expected: 61ce9e648fa1773adb3d4b3c84c6e4ca
Got: 18461da7c93ef44001051ea8aa07d34c
Archive: /Library/Caches/Homebrew/aacgain-1.8.tar.bz2
(To retry an incomplete download, remove the file above.)

The below commands allowed me to install aacgain without issue. Basically you just need to update brew and then delete the cached file.

$ brew update
$ rm /Library/Caches/Homebrew/aacgain-1.8.tar.bz2

Generate private keys and SSL certificate requests in batch or unattended way

When you have to generate multiple certificates for a lot of domains you better get smart at it ;-) Below is a bash script to generate a password-less key (needless to say you need this for servers but at the same time you must protect them from unauthorized access) and a certificate request:

#!/bin/bash -e
# gencert.sh
# @author: Nestor Urquiza

USAGE="Usage: `basename $0`       "

if [ $# -ne "7" ]
then
  echo $USAGE
  exit 1
fi

countryCode=$1
state=$2
city=$3
company=$4
organizationalUnitName=$5
domain=$6
email=$7

openssl req -nodes -newkey rsa:2048 -keyout ${domain}.key -out  ${domain}.csr -batch -subj "/C=$countryCode/ST=$state/L=$city/O=$company/OU=$organizationalUnitName/CN=$domain/emailAddress=$email"

Here is how you would use it:

export domain=domain.sample.com && ./gencert.sh "US" "CA" "San Francisco" "Domain Sample LLC" "Operations" "$domain" "@sample.com"

brew Warning: Your Xcode is outdated even though I had last version

Yup. After installing XCode and after installing the Command Line Tools (Following Downloads/Components/Command Line Tools/Install) I still got issues with brew:

$ brew install aacgain
Warning: Your Xcode (4.2) is outdated
Please install Xcode 4.6.2.
...

The issue was related to a miss configuration:

$ xcode-select -print-path
/Volumes/Xcode/Xcode.app/Contents/Developer

And corrected when pointing to the right path:

$ sudo xcode-select --switch /Applications/Xcode.app 

Create a zip file out of certain files in the file system

I had to generate several certificates today which I did from a script. To just pack them all I used the below which can be used to zip respecting any number of files respecting their directory structure.

find . -name "*.csr" -exec zip -g certs.zip {} +

Ubuntu The following packages have unmet dependencies: $package1 : Depends: $package2 but it is not going to be installed

I got today the below from recipes that are supposed to configure a brand new desktop:

$ sudo apt-get -q -y -f build-dep virtualbox-ose-guest-utils virtualbox-ose-guest-x11 virtualbox-ose-guest-dkms
...
The following packages have unmet dependencies:
 virtualbox-ose-guest-x11 : Depends: virtualbox-guest-x11 but it is not going to be installed
...

If you add virtualbox-guest-x11 to the installation then you get something else:

$ sudo apt-get -q -y -f install virtualbox-ose-guest-utils virtualbox-guest-x11 virtualbox-ose-guest-x11 virtualbox-ose-guest-dkms
...
The following packages have unmet dependencies:
 virtualbox-guest-x11 : Depends: xorg-video-abi-11
                        Depends: xserver-xorg-core (>= 2:1.10.99.901)
E: Unable to correct problems, you have held broken packages.
...

Just running the below will take care of installing the package and all its dependencies, however be prepared to wait a really long time before all packages are built and installed. You probably do not want to proceed this route and just find absolutely all dependencies by hand. It will depend on your needs:

$ sudo apt-get -q -y -f build-dep virtualbox-ose-guest-utils virtualbox-guest-x11 virtualbox-ose-guest-x11 virtualbox-ose-guest-dkms

The bottom line is that after using build-dep you can now proceed without errors with your install:

$ sudo apt-get -q -y -f install virtualbox-ose-guest-utils virtualbox-guest-x11 virtualbox-ose-guest-x11 virtualbox-ose-guest-dkms