Thursday, May 07, 2015

Talend OutOfMemoryError: Java heap space because of many files in a directory

I have blogged in the past about how to debug OutOfMemoryError in Talend jobs.

There is at least one official Talend component that would be generating these errors when we point to a specific directory containing a really large amount of files. The reason is that some code generates an array of strings containing the file names which clearly will not scale. The way I figured this out was following the steps in that previous post. From Eclipse Memory Analyzer I saw the cause for high memory consumption was an array of strings which matched file names.

Of course it is a bad practice to use a root directory to store all files, one should use a temporary directory per run. So the solution is actually simple. Nevertheless keeping such array of strings is just a waste of resources so that should be avoided as well.

The bottom line is that just automatically increasing memory when a JVM code throws OutOfMemoryError is not an option. Instead the engineer should investigate and get to the bottom of why processes are inefficient. Failure to do so will only postpone the inevitable because simply underperforming jobs won't scale. In the case of Talend as in any java application the JVM provides the tools to understand what happened when a memory leak originated a crash.

Saturday, May 02, 2015

Fastest idempotent way to install nodejs in linux or MAC OSX

Simply install the binary from a plain old bash (POB) recipe ;-)

Thursday, April 30, 2015

Fastest idempotent way to install nodejs in Ubuntu

Originally I created a gist tailored at Ubuntu however a fastest way is just to use a Plain Old Bash script to install the binaries as presented here. This will work in any linux and MAC OSX.

Friday, April 17, 2015

run cygwin sshd under SYSTEM user - The CYGWIN sshd service on Local Computer started and then stopped. Some services stop automatically if they are not in use by other services or programs

I got this error when trying to switch cygwin sshd to run as windows SYSTEM user. This is a need if you want to allow cygwin to interact with graphical applications.
The CYGWIN sshd service on Local Computer started and then stopped. Some services stop automatically if they are not in use by other services or programs
The first thing to do is to look into the sshd logs:
$ tail -10 /var/log/sshd.log
/var/empty must be owned by root and not group or world-writable.
The reason for this error is the fact that SYSTEM user (look upper case) is not the owner of /var/empty. The exact reason why this error happens is explained here. So the solution is simple:
$ chown SYSTEM /var/empty
Even though I went pass this issue I later realized there was no way to run desktop applications remotely using this method as explained here and here.

Monday, April 13, 2015

dockerized mediawiki 1.25 upgrade

The latest version of mediawiki comes with VisualEditor which marks the beginning of real wiki visual edition thanks to the use of node parsoid project. This is a radical change when we compare with the former CKEditor which would add a lot of html tags to your wiki markup. With Visual Editor we get plain wiki markup that can be edited by those who do enjoy the keyboard more than the mouse ;-)

Here is a partial docker guideline that can be used to install this latest version.

Friday, March 27, 2015

When my Apple MAC notes disappeared ...

Several posts on the web suggest to open the different files you see in ~/Library/Containers/ renaming them to use the ".html" extension, however I was able to recover them after deleting the ".storedata-wal" and ".storedata-shm" files.

Before proceeding please make a copy of the whole Notes directory. The file(s) with extension ".storedata" is/are what you want to keep and in fact these are sqlite files so you can open them, see the schema and inspect tables with the sqlite3 command.

BTW this happened to me after I started playing with iCloud. My iPhone and MAC notes were not synchronizing anymore and after finding that the iPhone notes were the ones in iCloud I started playing with iTunes and Notes accounts' settings to end up losing my MAC notes and getting those from the iPhone locally.