Map hostname to interface IP from a POB recipe

Here is a POB recipe you can use to add an entry in /etc/hosts for the IP of a given interface. This is great for boxes like developer Desktops where for example you want to make sure apache is installed using SSL virtual hosts, just to name a simple example.

Here is how to call it with an example which creates two entries in the remote /etc/hosts (if used through Remoto-IT):

common/tools/map-hostname-to-interface-ip.sh bhub eth0
common/tools/map-hostname-to-interface-ip.sh bhub.sample.com eth0

Linux Server cries for Linux Desktop - bash ': invalid option'

This should be an FAQ for bash. If you edit a bash file in a Windows machine or with an editor that uses carriage return (CR) in addition to line feed (LF) to produce new line characters you end up with:

: invalid option

If you open that file in Linux with some editors like vi you see ^M characters (corresponding to CR+LF) whenever a new line happens. With vim you are less successful but using ":set" command it shows the file format is "dos". Another way to look at the file is just using the file command:

$ file myscript.sh
myscript.sh: Bourne-Again shell script, ASCII text executable, with CRLF line terminators

And we all know you can correct that file in various ways like:

$ dos2unix myFile

Running again the file command will not state it uses CRLF because now the file is converted to use just line feed or ^N as line terminator.

Even if you edit such a file in Linux you will still end up respecting the original ^M in most editors.

Some editors will do this trick as part of the encoding conversion. For example I have seen Eclipse editor converting the file correctly when switched to UTF-8 in file properties, text file encoding option. While I saw that working on linux before in OSX Mountain Lion at least you can't. Luckily Eclipse comes with "File, Convert Line Delimiters To, Unix" option that does work on OSX as well. In my MAC even the file command will not report the file having CRLF when in fact it does.

What everybody keeps on forgetting is the amount of time we lose when we develop in one OS and deploy in a different one. Nobody questions that different environments like Integration, Staging, Production should be similar and ideally clones just with different data state so why is it so difficult to understand that if you deploy in Linux your code should be produced in a Linux machine? Way easier for everybody and with powerful hardware like we have today having Windows and Linux running at the same time in your machine should not be that difficult (in case you need to code in both).

Specifically for bash programming you need to be careful with file permissions as well. Go figure how to handle that in Windows. Unless you work with cygwin on top of Windows and use an editor from inside, edition of bash files in Windows should be banned! Just kidding, but seriously your environment is the foundation of your SDLC, anything wrong there will have tremendous consequences in productivity.

org.quartz.JobPersistenceException: Couldn't retrieve job because a required class was not found

Quartz API will fail if you ever dare to change the name of any existing Job class:

org.quartz.JobPersistenceException: Couldn't retrieve job because a required class was not found

So if you make changes in a scheduled job class do remember to provide migration plans before your release. For sure you will need to rename in the database the class name in case it changes before you use the Quartz API. This is true even if you are trying to delete the job and triggers like I have posted before.

The agile alternative to Ubuntu release upgrade

We had some Ubuntu 10.10 machines we wanted to upgrade to 12.04.1 LTS so I said let us give it a try in a clone to see how it goes before we work on the real servers.

After updating and upgrading 10.10 we proceed to a release upgrade:

do-release-upgrade

After running this command, going through some questions and restarting the box the system was upgraded to natty (11.04), not precise (12.04). Right there the problems started, mysql won't start, couchdb service would not listen and more. Running the command again did some progress and as you can see below after repeating the procedure 3 times I finally got the version I wanted (albeit several services will need to be tweaked to correctly work):

$ cat /etc/lsb-release 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=10.10
DISTRIB_CODENAME=maverick
DISTRIB_DESCRIPTION="Ubuntu 10.10"

$ cat /etc/lsb-release 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=11.04
DISTRIB_CODENAME=natty
DISTRIB_DESCRIPTION="Ubuntu 11.04"

$ cat /etc/lsb-release 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=11.10
DISTRIB_CODENAME=oneiric
DISTRIB_DESCRIPTION="Ubuntu 11.10"

$ cat /etc/lsb-release 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=12.04
DISTRIB_CODENAME=precise
DISTRIB_DESCRIPTION="Ubuntu 12.04.1 LTS"

So this upgrade in phases, restarting again and again the machine, responding to questions etc demands too much human intervention for my taste, too much place for errors. Granted this would not be as bad if coming from 10.10LTS but yet ...

What is the alternative? Having your system scriptable, being able to reinstall the whole server from recipes. Plain Old Bash (POB) recipes would suffice and you can remotely deploy them using Remoto-IT.

Here are some advantages of rebuilding the box from scratch in comparison to upgrading the OS in the box:

1. Going beyond Disaster Recovery (DR): To recover even from total lost of snapshots (SQL data corruption in snapshots for example), or corrupted DR data. Just tape backups are enough
2. System Tests: As a real life exercise to test backup/restore procedure plus automated DevOps recipes for configuration and deployment management
3. Recipes update: Recipes might not actually work with newer versions of OS so with this process recipes are kept current or worst case scenario customized per OS release version
4. House keeping: From time to time IT guys will do manual stuff, keep big files where they should not, forget about cleaning up broken installations or simply can't afford the time it takes to correct broken dependencies. POB recipes on top of a brand new OS is a solution for that mess for sure
5. Performance: The bares minimum needed are guaranteed in the new fresh OS. That is not the situation for most of the servers that have seen several IT people tweaking them in many cases for several years
6. Reliability: Responding questions about the upgrade, manually hitting the button will always be less reliable than automated scripting. It is just the nature of manual intervention
7. Efficiency: Even today with virtualized environments you will spend more time upgrading the OS and dealing with potential problems than the time it will take to automatically deploy the necessary services and data in a brand new OS release
8. Security: Last but not least going with POB recipes makes sure there is no hidden malware (root-kit or not) deployed from an external or internal attacker. You simply get a cleaner, more reliable and secure system.

If you use an Ubuntu LTS distribution you need to do this every two years, not bad.

couchdb not listening

After latest release upgrade to Ubuntu 12.04.1 LTS couchdb server would not complain starting, all processes will show up and yet it won't be listening in any port. Log would say nothing.

In this case we need to look at the standard console after running the couchdb server manually:

$ sudo /usr/local/etc/init.d/couchdb stop
 * Stopping database server couchdb                                                                                                                                           [ OK ] 
$ sudo -u couchdb couchdb
Apache CouchDB 1.1.1a1187836 (LogLevel=info) is starting.

=CRASH REPORT==== 29-Nov-2012::16:35:18 ===
  crasher:
    initial call: application_master:init/4
    pid: <0.31.0>
    registered_name: []
    exception exit: {bad_return,
                        {{couch_app,start,
                             [normal,
                              ["/usr/local/etc/couchdb/default.ini",
                               "/usr/local/etc/couchdb/local.ini"]]},
                         {'EXIT',
                             "libicuuc.so.42: cannot open shared object file: No such file or directory"}}}
      in function  application_master:init/4
    ancestors: [<0.30.0>]
    messages: [{'EXIT',<0.32.0>,normal}]
    links: [<0.30.0>,<0.7.0>]
    dictionary: []
    trap_exit: true
    status: running
    heap_size: 987
    stack_size: 24
    reductions: 176
  neighbours:

=INFO REPORT==== 29-Nov-2012::16:35:18 ===
    application: couch
    exited: {bad_return,{{couch_app,start,
                                    [normal,
                                     ["/usr/local/etc/couchdb/default.ini",
                                      "/usr/local/etc/couchdb/local.ini"]]},
                         {'EXIT',"libicuuc.so.42: cannot open shared object file: No such file or directory"}}}
    type: temporary

At that point I bet you can correct a lot of stuff but I save time in this cases with a fresh installation using a POB recipe remotely through the help of remoto-IT. Here is such recipe.

Especially after a release upgrade expect to get package corruption. More on Ubuntu release upgrades soon.

Debugging JDBC based applications

You are a Java developer and you use JDBC because you use a RDBMS like MySQL. Clearly you will be blind if everything you do is at JPA or ORM level. You must understand JDBC and databases if you want to find out why your application is suffering from performance issues, deadlocks and more.

The p6spy driver helps you with that. It will give you all the complete queries including parameters that are run in your system, when they are run and how long it takes to execute them. We have built a POB recipe that easily installs p6spy in tomcat. Below is a copy of it:

#!/bin/bash -e
# common/tomcat/p6spy-install.sh

TOMCAT_SETENV_PATH=/opt/tomcat/bin/setenv.sh
ARTIFACTORY_P6SPY_JAR=http://artifactory-server/ext-releases-local/p6spy/p6spy/1.1/p6spy-1.1.jar
SVN_P6SPY_PROPERTIES=http:/my.path.to.svn.spy.properties/spy.properties 

curl -o /opt/tomcat/lib/psspy.jar $ARTIFACTORY_JAR
common/tools/export-from-svn.sh  /opt/tomcat/lib/ SVN_P6SPY_PROPERTIES
sed -i 's/\(jdbc.*driverClassName\).*/\1=com.p6spy.engine.spy.P6SpyDriver/g' /opt/tomcat/lib/sample-app.properties
sed -i "/JAVA_OPTS -Dp6.home/d" $TOMCAT_SETENV_PATH
echo "JAVA_OPTS=\"\$JAVA_OPTS -Dp6.home=/opt/tomcat/lib/\"" >> $TOMCAT_SETENV_PATH

Breakpoints do not help when troubleshooting concurrency issues. Real world goes beyond the development box and so only hitting your application with a stress test (read JMeter for example) can prove that you have no issues that will later come back to your plate. At a minimum run those stress tests in Integration environment every so often.

MySQL EXPLAIN Divide and Conquer

Especially when you are using sub-selects (AKA subqueries) things gets not that clear when analyzing the results of EXPLAIN. A good technique is to run your subqueries and try to tune them separately before trying to adjust the whole query.

I found myself wearing the DBA hat during this long weekend resulting in a speed improvement of more than four thousand times.

As MySQL manual suggests make sure you run EXPLAIN and look first at those entries with the "extra" field stating "Using temporary" and/or "Using filesort".

Then identify the query bits using those conflicting tables and try to isolate them (divide)

Continue running EXPLAIN again on those bits to come up with a solution that increase performance (conquer).

Contrary to the belief it is not always a lack of indexes, in fact too many indexes will slow the query or other queries down, for sure it will slow down inserts.

Many times is about a query that should be rewritten.

Look for things like not filtering in the subquery. The WHERE clause is your friend. In my case I found the developer used a WHERE clause in the wrapper dataset but not in the sub-selects where he could have done the same for the very same key.

Look for multiple keys or indexes that are just redundant. Remember the rule: A composite index/key is read from left to right, there is no need to add other keys or indexes with portions of the composite, if your fields are organized in that order you can use one, two or more fields and that unique composite key will be enough.

Any job can be done using multiple ways, a developer can use any but the engineer should always pick the faster. Let us strive to be better Engineers when we develop code! Rewrite your query for which of course you must have documentation of what you are trying to achieve. Yes documentation is on the right side in the Agile Manifesto but that actually does not mean it is not necessary.

Learn by challenge. You will not master MySQL optimization by reading the whole manual and ten books. You will master it as you face problems and wonder why the occur. In that process you will definitely learn *just* what you need to perform your job.

In extreme cases some table structures will have to be changed. Even de-normalization will be sometimes the only option but please be sure there is nothing else to do with indexes or rewriting your query. Changing table structure might be hiding a lot more time, complexity and probably no that much efficiency improvement.

Windows account locked when working from a MAC

Your Enterprise Domain Controller is for sure expiring passwords but your MAC is not integrated with it and yet you access several network resources like Exchange and the Shared File System (AKA Common Internet File System or CIFS)

I thought I had found all the entries related to my account from the KeyChain.app and so I told the sysadmin. However today I had a second thought, what about if the KeyChain search is actually not that smart? And indeed. The search will always look at key entries but never inside them.

In my case the sysadmin was seeing the account being locked from "\\workstation" which is not a name defined in any place after all.

Running the below command I was able to find several old entries for Remote Desktop Connections, File System, Exchange, iCal, Address Book and more.

$ security dump-keychain | grep $username

Then I realized I really had too many entries so I started going through them using:

$ security dump-keychain | more

I ended up then verifying I had a lot of old entries with expired passwords (I went back to KeyChain.app and searched for the specific key named "srvr" in the CLI output). Some of those key/attribute/srvr names I found were having interesting Access Control names like "localhost/Address Book", "https/exchange", "IISupport/Mail,Mail,iSync" (note the multiple access control values).

Deleting and later recreating keys on demand sounds like a good option especially if you have migrated from a previous major version of the Operating system. We all know software is ultimately buggy so you never know how the old garbage can actually impact in your newly installed OS. In my case I upgraded from Snow Leopard to Mountain Lion. BTW to avoid confusion I am not stating though the upgrade is responsible for the locking as this was happening when I was in Snow Leopard as well.

Other causes

I have found manually created entries in /etc/fstab to a be cause for this issue as well.

Talend Eclipse default new line and Control M characters

I noticed some code I wrote in Talend Component's Designer Perspective was inserting Control M characters as new lines. Apparently the default for "New text file line delimiter" is not being picked correctly in OSX (10.8.2) using Talend Version: 4.2.3 Build id: r67267-20110905-0421

OSX brew: Error: Cannot write to /usr/local/Cellar

sudo chown -RL `logname` /usr/local/Cellar

Filter the flow in Talend Components

When building a custom Talend component you might want to ignore certain input rows. I spent a lot of time trying to figure out why my tFileInputCSVFilter component was insisting on printing empty records for the filter flow and for the reject flow, the reason? Just an attribute from /COMPONENT/HEADER node :

HAS_CONDITIONAL_OUTPUTS="true"

In Talend by default when you use a flow from a main template and you don't assign anything to the output connector the result will be an empty record unless you set HAS_CONDITIONAL_OUTPUTS="true".

VDI Path: Access Linux Desktop remotely using RDP

Update 20220123: Works well for 20.04.

Update 20190511: In 18.04 clipboard does work again

Update 20161201: In 16.04 Clipboard is not longer working.

Update 20150731: Clipboard works great in Ubuntu 14.04 LTS, however Unity is not supported.

Original post for historical reasons: RDP has been for ages a great protocol giving Windows the big advantage on the Virtual Desktop Infrastructure (VDI) competition. The FreeRDP GitHub project is changing that little by little especially thanks to XRDP. Here is a POB Recipe to install XRDP. For the latest version of the script please pull the recipe from github pob-recipes project.

#!/bin/bash -e
# common/ubuntu/xrdp-reinstall.sh

apt-get -q -y remove xrdp
apt-get -q -y update
apt-get -q -y clean
apt-get -q -y purge xrdp
apt-get -q -y install xrdp

After you have done so the Ubuntu Desktop will then be available via RDP so you can use any RDC to connect to it on default port 8309. You must login with an existing linux user so make sure the user you are trying to connect with is in /etc/passwd

I have found a very good performance after setting the desktop background to a simple color. Also restarting the desktop is recommended to make sure after a fresh reboot RDP will be available.

You can configure the port and a lot of other settings from ini files in /etc/xrdp/

RDP Sessions

XRDP does a great job and by default it does what you would expect if you are disconnected from your session: Upon reconnection you will continue working using the very same session you were when you got disconnected. Just make sure you do not change the resolution of your client before you completely logout from an existing session because that will end up creating a brand new session.