Plain Old Bash (POB) idempotent recipe to install redis

	# install redis
	cd /tmp
	rm -fr redis-stable
	wget http://download.redis.io/redis-stable.tar.gz
	tar xvzf redis-stable.tar.gz
	cd redis-stable
	make
	make install
	# prepare necessary directories
	rm -fr /etc/redis
	rm -fr /var/redis
	mkdir /etc/redis
	mkdir /var/redis
	mkdir -p /var/log/redis
	mkdir -p /var/run/redis
	# copy init script and change it to allow running and stopping (without asking for password) as current user
	redis_init=/etc/init.d/redis_6379
	cp /tmp/redis-stable/utils/redis_init_script $redis_init
	user=`logname`
	sed -i "s#\$EXEC \$CONF#su - $user -c \"\$EXEC \$CONF\"#" $redis_init
	sed -i "s#\$CLIEXEC -p \$REDISPORT shutdown#su - $user -c \"kill \$PID\"#" $redis_init
	sed -i "s#PIDFILE=/var/run#PIDFILE=/var/run/redis#" $redis_init
	# copy configuration
	cp redis.conf /etc/redis/6379.conf
	# create database directory
	mkdir /var/redis/6379
	chown -R $user:$user /etc/redis
	chown -R $user:$user /var/redis
	chown -R $user:$user /var/log/redis
	chown -R $user:$user /var/run/redis
	# kill any redis-server process
	set +e
	killall -9 redis-server
	set -e
	# remove any pid file left behind
	rm -fr /var/run/redis/*
	# for non production environments we use a simple common password
	requirepass=secret
	# for production requirepass is supposed to be defined in redis-vars.properties which you could host in a subversion directory for example
	if [ "$environment" == "production" ]; then
	svn export https://svn.sample.com/protected/production/path/redis-vars.properties /tmp
	. /tmp/redis-vars.properties
	rm /tmp/redis-vars.properties
	fi
	# modify configuration file to use proper password, pid path, log file and database path. Start it as a daemon
	redis_conf_file=/etc/redis/6379.conf
	sed -i "s/.*requirepass.*/requirepass $requirepass/" $redis_conf_file
	sed -i "s#^pidfile.*#pidfile /var/run/redis/redis_6379.pid#" $redis_conf_file
	sed -i "s/^daemonize.*/daemonize yes/" $redis_conf_file
	sed -i "s#.*logfile.*#logfile /var/log/redis/redis_6379.log#" $redis_conf_file
	sed -i "s#^dir .*#dir /var/redis/6379#" $redis_conf_file
	# make sure the service starts after system startup
	update-rc.d -f redis_6379 defaults
	# start the service
	/etc/init.d/redis_6379 start
	# test the service
	echo "Testing redis replies with pong after authentication ..."
	echo -e "auth $requirepass\nping" | redis-cli

view raw Plain Old Bash (POB) idempotent recipe to install redis hosted with ❤ by GitHub

Talend OutOfMemoryError: Java heap space because of many files in a directory

I have blogged in the past about how to debug OutOfMemoryError in Talend jobs.

There is at least one official Talend component that would be generating these errors when we point to a specific directory containing a really large amount of files. The reason is that some code generates an array of strings containing the file names which clearly will not scale. The way I figured this out was following the steps in that previous post. From Eclipse Memory Analyzer I saw the cause for high memory consumption was an array of strings which matched file names.

Of course it is a bad practice to use a root directory to store all files, one should use a temporary directory per run. So the solution is actually simple. Nevertheless keeping such array of strings is just a waste of resources so that should be avoided as well.

The bottom line is that just automatically increasing memory when a JVM code throws OutOfMemoryError is not an option. Instead the engineer should investigate and get to the bottom of why processes are inefficient. Failure to do so will only postpone the inevitable because simply underperforming jobs won't scale. In the case of Talend as in any java application the JVM provides the tools to understand what happened when a memory leak originated a crash.

Fastest idempotent way to install nodejs in linux or MAC OSX

Simply run the below one-liner from my plain old bash (POB) recipe as shown below:

export NODE_VERSION=18.19.0; curl -sL https://raw.githubusercontent.com/nestoru/pob-recipes/master/common/nodejs.sh | sudo bash -s $NODE_VERSION $USER

Note that you might need to run 'npm rebuild' in your project(s) if you run into errors like:

Error: The module '/app/node_modules/node-expat/build/Release/node_expat.node'
was compiled against a different Node.js version using
NODE_MODULE_VERSION 64. This version of Node.js requires
NODE_MODULE_VERSION 83. Please try re-compiling or re-installing