Wednesday, November 26, 2014

Error level or priority for Linux syslog

If you grep your linux server logs from time to time you might be surprised at the lack of an error level. If you want to know for example all error logs currently in syslog, how would you go around it? Simple answer you cannot without changing the log format in /etc/syslog.conf.

Let us say you configured to see the priority (%syslogpriority%) as the first character in the log file:
$ vi /etc/rsyslog.conf
...
#
# Use traditional timestamp format.
# To enable high precision timestamps, comment out the following line.
#
$template custom,"%syslogpriority%,%syslogfacility%,%timegenerated%,%HOSTNAME%,%syslogtag%,%msg%\n"
$ActionFileDefaultTemplate custom
...
$ sudo service rsyslog restart
To filter information look at the description of priorities. From http://www.rsyslog.com/doc/queues.html:
        Numerical         Severity
          Code

           0       Emergency: system is unusable
           1       Alert: action must be taken immediately
           2       Critical: critical conditions
           3       Error: error conditions
           4       Warning: warning conditions
           5       Notice: normal but significant condition
           6       Informational: informational messages
           7       Debug: debug-level messages
A simple grep helps us now:
$ grep '^[0-3]' /var/log/syslog
...
3,3,Nov 26 11:56:15,myserver,monit[17496]:, 'myserver' mem usage of 96.3% matches resource limit [mem usage>80.0%]
...
A more readable format:
$ vi /etc/rsyslog.conf
...
#
# Use traditional timestamp format.
# To enable high precision timestamps, comment out the following line.
#
$template TraditionalFormatWithPRI,"%pri-text%: %timegenerated% %HOSTNAME% %syslogtag%%msg:::drop-last-lf%\n"
$ActionFileDefaultTemplate TraditionalFormatWithPRI
...
$ sudo service rsyslog restart
Would allow you to search as well:
$ grep -E '\.error|\.err|\.crit|\.alert|\.emerg|\.panic' /var/log/syslog
...
daemon.err<27>: Nov 26 13:10:18 myserver monit[17496]: 'myserver' mem usage of 95.9% matches resource limit [mem usage>80.0%]
From http://www.rsyslog.com/doc/rsyslog_conf_filter.html valid values are debug, info, notice, warning, warn (same as warning), err, error (same as err), crit, alert, emerg, panic (same as emerg).

Here is how to force an alert from the cron facility for testing purposes:
logger -p "cron.alert" "This is a test alert that should be identified by logMonitor" 
You will get:
cron.alert<73>: Aug 27 08:42:14 sftp krfsadmin: This is a test alert that should be identified by logMOnitor
Which you could inspect with logMonitor.

No comments:

Followers