Friday, January 07, 2011

Use LogMonitor to email log errors

Inspecting application server logs to act proactively is a must do for developers. In their local environments they should inspect their logs before committing to the VCS.

The question is what is happening in other environments, especially in production. How the developer can be promptly notified about an issue.

I am not sure if I am just so "lucky" to find so many tech guys proposing monitoring the logs of applications from within the applications. It is simply like saying I am my own Doctor, period.

In my opinion log monitoring should be done from the outside.

Below are the instructions to get any log and send an email when a line meets certain criteria.
  1. Install perl script to send email. I haven't seen so far a most powerful unix sendmail command than this script.
  2. sudo vi /usr/sbin/smtp-cli
    sudo chmod +x /usr/sbin/smtp-cli
    sudo apt-get install libio-socket-ssl-perl  libdigest-hmac-perl  libterm-readkey-perl libmime-lite-perl libfile-type-perl libio-socket-inet6-perl
  3. Be sure logtail is installed in your system.
    apt-get install logtail
  4. Before creating the logMonitor script let us be sure we do not get too much information with the first email sent. Run just the logtail command, for example:
    apt-get install logtail
    logtail -f /opt/tomcat-6.0.18/logs/catalina.out -o /opt/tomcat-6.0.18/catalina.out.offset

  5. Create a logMonitor script
    # @fileName:
    # @description: Sends an email every time a new PATTERN is found in a growing log file.
    # @author: Nestor Urquiza
    # @date: Jan 7, 2011
    # Constants
    TEMP_FILE=`mktemp /tmp/logMonitor.XXXXXXXXXX`
    # Functions
    function usage {
      echo "Usage - $0 logFile  offsetFile includePattern excludePattern from to subject"
      echo "If you need no exclusions then simply put some string you know will not be matched"
      exit 1
    # Main program
    if [ $# -lt 7 ]
    command="/usr/sbin/logtail -f $logFile -o $offsetFile | egrep -v \"$excludePattern\"  | egrep \"$includePattern\" -A $LINES_AFTER > $TEMP_FILE"
    echo "Running: $command"
    eval $command
    echo "Result: `cat $TEMP_FILE`"
    #if [ -n "$body" ]
    if [ -s $TEMP_FILE ]; then
      command="/usr/sbin/smtp-cli --host=$SMART_HOST --from $from --to $to --subject \"$hostname: $subject\" --body-plain \"$TEMP_FILE\""
      echo "Running: $command"
      eval $command
    rm $TEMP_FILE

  6. Cron your script to run let us say every five minutes
    */5 * * * * /usr/sbin/logMonitor /opt/tomcat-6.0.18/logs/catalina.out /opt/tomcat-6.0.18/catalina.out.offset "SEVERE|ERROR|WARN" "thingsIwantToExclude" "catalina.out problems"

No comments: