Apache log rotation: when things go wrong

There was a time a mod-jk.log file got to 3.3GB because of a miss configuration (someone forgot to revert back to error level from trace level.

In linux is easy to rotate any logs and for apache in particular in this Ubuntu server we have:

$ cat /etc/logrotate.d/apache2
/var/log/apache2/*.log {
 weekly
 missingok
 rotate 52
 compress
 delaycompress
 notifempty
 create 640 root adm
 sharedscripts
 postrotate
  /etc/init.d/apache2 reload > /dev/null
 endscript
 prerotate
  if [ -d /etc/logrotate.d/httpd-prerotate ]; then \
   run-parts /etc/logrotate.d/httpd-prerotate; \
  fi; \
 endscript
}

This means every week the log will be gzipped and rotated, then recycled after number 52 is reached. However a week will be too much for a trace level. To correct this problem be sure mod-jk is configured to log as error then without disruption you can proceed as follows:

$ sudo vi /etc/logrotate.d/apache2
...
#replacing:
#JkLogFile /var/log/apache2/mod-jk.log
JkLogFile "|/usr/sbin/rotatelogs /var/log/apache2/mod-jk.log 86400"
...
$ sudo mv /var/log/apache2/mod-jk.log /var/log/apache2/mod-jk.log.todelete
$ sudo apachectl configtest
$ sudo apachectl graceful
$ sudo vi /etc/logrotate.d/apache2
...
#reverting:
JkLogFile /var/log/apache2/mod-jk.log
#JkLogFile "|/usr/sbin/rotatelogs /var/log/apache2/mod-jk.log 86400"
...
$ sudo apachectl configtest
$ sudo apachectl graceful

Take a look at how to avoid this issues through monit.

Quartz scheduler instance is still active but was recovered by another instance in the cluster

When scheduling tasks we should be sure clocks are synchronized across servers. I have seen so many issues because of clock synchronization that I am still surprised to see how small attention is put on server time synchronization. From security vulnerabilities exploits to serious business logic erros resulting in money loss the error always hits you hidden behind words like:

2012-09-12 19:13:09,193 WARN [org.springframework.scheduling.quartz.LocalDataSourceJobStore] - This scheduler instance (bhub-test11347491427777) is still active but was recovered by another instance in the cluster. This may cause inconsistent behavior.

And of course a simple ntpdate configuration will solve the issue. This ntpdate configuration can be deployed remotely in all your servers using Remoto-IT and a POB Recipe similar to the below:

#!/bin/bash -e
# ntp.sh

cd /etc/cron.daily
#assuming you have your configuration files on SVN, aren't you versioning your changes yet?
svn export http://svn.sample.com/environment/common/scripts/etc/cron.daily/ntpdate
chmod 755 ntpdate
ntpdate ntp.ubuntu.com pool.ntp.org

Here is how the ntpdate file looks like:

#!/bin/sh
#Please do use an internal NTP server for security reasons!
ntpdate ntp.ubuntu.com pool.ntp.org > /dev/null

Of course if the cron fails you should get an email so be sure MAIL_TO is working in cron.

Monitor your MySQL or MSSQL servers heartbeat

It is a central piece of most applications nowadays and yet you discover the DB is down because some other alert about a failing service or even worse from your client.

Here is a simple POB script to help you check the health of your MSSQL or MYSQL server.

If you want alerts you could just cron the script and on failure you will get an error in the email address configured in the MAILTO option however a more sophisticated and better approach (for one you will receive the alert just once and after corrected you will get feedback about the issue been solved) is to call the script from Monit for example as in:

#################################################################
#REMOTE MSSQL CHECK
################################################################
check program ssql-heartbeat with path "/bin/sql-heartbeat.sh"
  if status != 0 then alert

Naturally /bin/mssql-heartbeat.sh will need to have 700 permissions and owned by root as credentials will be kept inside. Here is an example of such wrapper:

#!/bin/bash -e
/bin/sql-heartbeat.sh MSSQL myserver.mydomain.com heartbeatUser heartbeatPassword

Feel free to modify the heartbeat script to customize the database for example in case you want to be sure a specific db is available.

Reinitializing monit daemon monit: No daemon process found

So here is the issue I faced today:

$ sudo monit reload
Reinitializing monit daemon
monit: No daemon process found

However "monit -Iv" would show monit able to read the configuration file and send alerts correctly.

As monit is installed as a service if such service is stopped the reload command would do nothing. So first the service should have been started:

sudo service monit start

The way monit restarts automatically is through a /etc/init.d/monit which sources the /etc/default/monit where in turn "startup=1" sets if monit should automatically start after system reboot.

As a reminder "/usr/sbin/service " is equivalent to call "/etc/init.d/ " so the below is perfectly fine as well:

sudo /etc/init.d/monit restart

There is one word of advise though. If you faced this issue be sure 'which monit' resolves to the same monit version in use when you call monit as a service. I have found dual installations in /usr/bin and /usr/local/bin perhaps related to ".configure" problems when installing monit from sources.

Security: Print context variables in Talend

The Talend tContextLoad component has a 'Print operations' option which should be used with caution. You could be revealing passwords in your log files otherwise.

In reality the "context" variable is accessible from Java as a regular java.util.Property object so you can get a greater control as to what to print or log but you have also "Advanced Settings" which you can use to state things like:

!(key_tContextLoad_1.contains("pass") || key_tContextLoad_1.contains("pwd"))

Which as you guessed is trying to exclude anything containing password related paramter keys.

Here you can find a project that illustrates the two alternatives. Remember that Security must be a top priority in your app.

Talend and SQL Transactions

It would be great if when setting a Talend Connection to a Database the "Auto Commit" setting in Advanced Settings of the tMySQLConnection would drive more in the rest of the components using such connection. Unfortunately at least for the tMySQLRow component that is not true.

I started putting together a proof of concept to demonstrate a bug in Talend Open Studio Version: 4.2.3 tMysqlRow component. Basically it does not support transactions.

So here is a project with two jobs. The first called "mysql_pojo_transaction" just uses JDBC while the second called "mysql_transaction" uses tMySQLRow to insert two identical records (just a name column) in table "names" in a mysql "test" database.

Using "mysql_transaction" project I was expecting to check and uncheck Autocommit checkbox from Advanced settings in the tMysqlConnection and see how a record will exist in "names" table only when autocommit is checked. However a record will exist there in any case:



Using "mysql_pojo_transaction" project and commenting out setAtocommit, commit and rollback lines will result in one record while uncommenting results in no records. This would be the expected behaviour in tMySQLRow I guess:



This project is useful as a test for a couple of reasons. First it shows how to prepare a test database from within talend without the need of external files (through the use of tFixedFlowInput which generates the statements to be run by a tMySQLRow component). Second it shows how to generate input data via tRowGenerator (Set schema with field "name", Use as Funtion "Talend Trim", use as Funtion Parameter origin the value "Foo". Clicking on Preview should show two rows with the word "Foo") without the need of extra files to insert records later on using again a tMysqlRow.

POB script to assert if user quota is exceeded

Here is a Plain Old Bash (POB) script to assert the user quotas are OK. Should quotas exceed certain threshold provided as a parameter the script will return an exit code 1 with the description in stderr. Here is how to call it to find out who is using more than 85% space:

/usr/sbin/find-exceeded-quota.sh 85

Motivation

When trying to put a file via SFTP or creating a remote directory via SFTP we were getting:

Couldn't write to remote file "filename": Failure Couldn't create directory: Failure

Increasing log level for SSH did not help to clarify the issue:

$sudo vi /etc/ssh/sshd_config
...
# Logging
SyslogFacility AUTH
#LogLevel INFO
LogLevel VERBOSE
...
$ sudo service ssh restart

However other user accounts did not show the issue. It ended up being related to the user quota in use where "used" was nearly the same as the "hard" setting:

sudo repquota -a 

This simple POB script then can be used to check for certain percentage as explained. After getting the alert all the sysadmin will do is either delete files or increase quota:

sudo setquota -u user2 200000 200000 1500 1500 -a /

It is a good idea to automatically delete of course old files if that is possible.