Modlog – a BASH modular logger
by Hans Kokx on Nov.10, 2009, under Bash Scripts, Linux
Introduction
This summer, I had the good fortune to spend several months working at the University of Michigan. One of the tasks assigned to me was devising a method for data collection over time for certain types of data. In particular, we were looking to collect information from /proc/meminfo and out of our lm_sensors output.
The issue that we came across with logging these sets of data was that they were set up for one-time viewing. That is, they looked similar to the following:
Timestamp: 00:00:00 Header 1: data 1 Header 2: data 2 Header 3: data 3
Timestamp: 00:00:01 Header 1: data 4 Header 2: data 5 Header 3: data 6
Timestamp: 00:00:02 Header 1: data 7 Header 2: data 8 Header 3: data 9
and so on. Certainly, we had the ability to set up a cron job and dump the data we sought after into a text file, however that was a very messy solution over long periods of time. Ideally, we would like to pile up our data in a manner such as this:
Timestamp Header 1 Header 2 Header 3 00:00:00 Data 1 Data 2 Data 3 00:00:01 Data 4 Data 5 Data 6 00:00:02 Data 7 Data 8 Data 9
Over time, it is much easier to read the data, as well as build graphs from the dataset. You will be able to awk out a single column (or multiple columns) and compare certain data over a particular date range. Compared to the original method -- dumping to a text file in the original state -- this is much more usable.
Solution
Modlog, at it's heart, is a modular logging system (thus the name). It uses configuration files to generate output, then compiles the output in an easy to read and easy to store format. Furthermore, modlog will allow you to easily recall that data using regular expressions. Modlog is the brain child of Hans Kokx, with additional code snippets provided by Brian Ward and Ezekiel Hendrickson. There are a few components to modlog. First, there is the main script. This is the heart and soul of modlog, of which all other components are referenced. Second, there are the configuration files, and finally, a perl helper application. I will explain each of these in more detail.
Modlog
The main modlog script is written for BASH. It has been tested throughly, but that does not mean it will work in all circumstances without any hiccups. There are a few variables that can (and should) be set, depending on your configuration. They are as follows:
confdir is where your configuration files can be found.
confdir=/etc/sysconfig/modlog
kernlogger is what logs to your system logs. In this case, it is configured to use syslog-ng. You should check with your distrobution's documentation to find the correct command for logging to your system logs.
kernlogger="logger -t kernlog -p daemon.info $name[$$]: "
Most other variables should be fine with the defaults. Anything that is initialized blank should be left blank.
#!/bin/bash # Name : modlog # Description : A modular logging system # Version : 1.0.1 # Coded by : Hans Kokx, Brian Ward, Ezekiel Hendrickson # Date : Sept 24, 2009 ## Variables ## # Configuration file directory. confdir=/etc/sysconfig/modlog # Default column length used for writing files and displaying output. defaultcollength=25 # Kernlogger command kernlogger="logger -t kernlog -p daemon.info $name[$$]: " # Date format used in output file names. date=`date +%F%n` # Timestamp used as the first column in output files. time="date +%H:%M:%S" # Define a new line as a variable nl=" " optdate=$date optconfig= optfield= opt_c= opt_d= opt_f= opt_i= opt_l= opt_w= input= output= description= works= hrf=tee read= write= verbose= counter=1 running= # CONJUNCTION JUNCTION WHATS YOUR FUNCTION # Generic function to convert text to lowercase toLower() { echo $1 | tr "[:upper:]" "[:lower:]" } # Make sure the $collength variable is populated. colLength() { if [[ -z "${collength}" ]] then collength=$defaultcollength fi } main() { outfile=$output/$date.log if [ "$verbose" = "1" ] then echo -en "\033[1m*\033[0m Writing output to $outfile\n" fi if [[ $works = 0 ]] then mkdir -p $output if [[ "$input" != "" ]] then if [[ ! -s $outfile ]] # Print the timestamp header to our output, then write headers, and # finally start writing rows of data. then if [ "$verbose" = "1" ] then echo -en "\033[1m*\033[0m $outfile does not exist. Creating new file, and writing headers.\n" fi echo -n "Timestamp " | awk '{ printf "%9s", $1 " " }' >> $outfile $input | sed 's/:/ /g' | awk 'BEGIN{FS=" +"}NF>1{ printf "%'"$collength"'s", $1 " " }' >> $outfile echo "" >> $outfile fi $time | awk '{ printf "%9s", $1 " " }' >> $outfile $input | sed 's/:/ /g' | awk 'BEGIN{FS=" +"}NF>1{ printf "%'"$collength"'s", $2 " " }' >> $outfile echo "" >> $outfile fi else if [[ "$verbose" = "1" ]] then echo -en "\E[31m\033[1m*\E[31m\033[0m $input will not complete successfully on this machine. Skipping.\n" fi $kernlogger $input will not complete successfully on this machine. Skipping. fi works= } writeData() { if [[ "$optconfig" = "" ]] && [[ "$optfield" = "" ]] then # Read config files one by one, and run them as commands, dumping output # into a file. if [ "$verbose" = "1" ] then echo -en "\033[1m*\033[0m Reading configuration from $FILE\n" fi for FILE in $(find $confdir -name "*.conf" -type f); do works= if [ "$verbose" = "1" ] then echo -en "\033[1m*\033[0m Executing commands...\n" fi . $FILE main done elif [[ ! "$optconfig" = "" ]] && [[ "$optfield" = "" ]] && [[ "$optdate" = "$date" ]] then . $confdir/$optconfig.conf main else echo >&2 "Error!" fi } showData() { # We have -l input, so loop through arguments and return the matching # fields. if [ "$verbose" = "1" ] then echo -en "\033[1m*\033[0m Sourcing $confdir/$optconfig.conf\n" fi . $confdir/$optconfig.conf colLength # The following is all voodoo magic. # Loop over each line to see if it matches our regex, and determine whether # or not it should be printed. counter= if [ "$verbose" = "1" ] then echo -en "\033[1m*\033[0m Determining matching columns...\n" fi for re in ${ARGS[@]}; do forloop='for(i=1;i<=NF;i++)if(tolower($i)~/'$re'/)f[i]=1;' forloops="$forloops $forloop" counter=`expr $counter + 1` done # Start by setting our field separator as at least two whitespaces. # Then, we set each line to show, based on whether or not it matches our # regex. # Finally, we loop through our columns, match all lines that should be # visible, and then pipe it to our helper to help display them. if [ ! "$opt_w" = "1" ] then if [ "$verbose" = "1" ] then echo -en "\033[1m*\033[0m Automatic column width adjustments being calculated...\n" fi termcols=`stty -a| grep columns|awk '{print $7}'|sed 's/;//g'` collength=`expr $termcols / $counter - 65` # set a variable column fi # width based on the # current terminal width if [ "$verbose" = "1" ] then echo -en "\033[1m*\033[0m Printing matching fields\n" fi gawk 'BEGIN{FS=" +"} NR==1{ for(i=1;i<=NF;i++) f[i]=0;'"$forloops"' f[1]=0; } { printf"%9s",$1; for(i=1;i<=NF;i++) if(f[i]) printf"%'"$collength"'s",$i; print"" }' $output/$optdate.log | $hrf } info() { if [ -n "$optconfig" ] then . $confdir/$optconfig.conf echo -en "\033[1mKey:\033[0m Header ("'\E[31m'"\033[1mawk record"'\E[31m\033[0m'")\n" $input | sed 's/:/ /g' | awk 'BEGIN{FS=" +"}NF>1{ sub(/[:]/, ""); printf "%25s", $1 " (" "$"NR+1 ") \n"; }' opt_i=2 echo >&2 "REMINDER: The awk delimiter is two or more whitespaces. \(ex. -F \" +\"\)." echo >&2 "The awk record numbers indicated are pre-computed and should be used when trying to pull data directly from the log file." else echo >&2 "No configuration file selected. Use -l flag to view available configuration files." fi exit 1 } list() { echo Available configuration files: for FILE in $(find $confdir -name "*.conf" -type f); do . $FILE filename=`basename $FILE .conf` echo " $filename - $description" done exit 1 } # Subroutine to display usage information usage() { cat<<EOF Usage: $0 <options> This script can pull columns of data from a specified date on a specified configuration. OPTIONS: -c Use configuration for data as listed in -i. Case sensitive. -d Use date for data in the format YYYY-MM-DD. Defaults to today. -f Use field given with -i switch for information. Case insensitive. -h Human readable output. Displays headers every 20 lines. -i Lists possible field for a configuration. -l Lists available configurations. -R Read mode, for pulling data from logs. -v Verbose -W Write mode, for writing data to logs. -w Variable column width. EOF exit 1 } # If no arguments are given, display usage information [ $# -eq 0 ] && usage # Parse command line options while getopts "c:d:f:hilRvWw:" option do case $option in c) # Set configuration file to use for data here optconfig=$OPTARG opt_c=1 ;; d) # Set date to use for data here optdate=$OPTARG opt_d=1 ;; f) # Set field(s) to use for data here optfield=`toLower $OPTARG` typeset -a ARGS; n=0 ARGS[${n}]=${optfield} n=$(( $n+1 )) while [[ "${!OPTIND:0:1}" != "-" && "${!OPTIND}" != "" ]]; do # echo ${!OPTIND} ARGS[${n}]=${!OPTIND} OPTIND=$(( $OPTIND+1 )) n=$(( $n+1 )) done opt_f=1 ;; h) # Sets the output to human readable (headers every few lines). hrf="$confdir/helper.pl" ;; i) # List possible fields within a configuration opt_i=1 ;; l) # Lists available configuration files opt_l=1 ;; R) read=1 ;; v) verbose=1 ;; W) write=1 ;; w) # Set the column width collength=$OPTARG opt_w=1 ;; \?) # Displays a handy help message usage ;; esac done # Verbosity checks if [ "$opt_c" = 1 ] && [ "$verbose" = 1 ] then echo -en "\033[1m*\033[0m Using $optconfig as config\n" fi if [ "$opt_d" = 1 ] && [ "$verbose" = 1 ] then echo -en "\033[1m*\033[0m $optdate specified for review\n" fi if [ "$opt_f" = 1 ] && [ "$verbose" = 1 ] then echo -en "\033[1m*\033[0m Field(s) \033[1m$optfield\033[0m specified for review\n" fi if [ "$opt_h" = 1 ] && [ "$verbose" = 1 ] then echo -en "\033[1m*\033[0m Using $hrf for human-readable post-processing\n" fi # Process functions based on input if [ "$opt_i" = "1" ] then info fi if [ "$opt_l" = "1" ] then list fi if [ "$read" = "1" ] then if [ ! "$opt_c" = "1" ] then echo >&2 "No configuration file specified!" exit 1 else if [ "$opt_c" = "1" ] && [ -z "$opt_f" ] then echo >&2 "No fields specified!" exit 1 else showData fi fi elif [ "$write" = "1" ] then writeData elif [ -z "$write" ] || [ -z "$read" ] then echo "You must specify whether you would like to [R]ead or [W]rite data!" fi
Configuration
The configuration directory is located in modlog as a variable, near the top of the script. All configuration files are specified as .conf files within the directory. Modlog loops over each configuration file, performing the commands specified, and then goes on to the next. There are some limitations with modlog because of the setup of the configuration files. For example, the command that will be run must be encapsulated in quotes. For some more advanced commands, this has proven to be problematic. Your milage may vary. If, however, you run into this problem and decide to fix it, I would ask that the changes be merged into this document, and all credit for the fix will go to the author of that fix.
The following sample configuration will collect data from /proc/meminfo. Note the output directory; this is where the data will be logged to, and may be changed per configuration.
# Name : meminfo.conf # Description : Configuration file for modlog. This config file pulls meminfo. # Version : 0.1 # Coded by : Hans Kokx, Brian Ward # Date : May 21, 2009 # Prerequisite : none # Command that is run to gather data input="cat /proc/meminfo" # Where the outputted data is stored output=/var/log/archive/modlog/meminfo # Short description of this configuration file description="Memory usage information" # A test to make sure that our config file wont fail outright. # $works should be 0 if all is well, and 1 if the command will fail. if [ ! "$input" == "" ] then works=1 # fail, unless otherwise specified if [ -e "/proc/meminfo" ] then works=0 fi fi
Again, here is a second example configuration. This one is used to collect data from lm-sensors. NOTE: lm-sensors has a small bug in it, where beep is outputted as beep:enabled. Since there is no whitespace, modlog sees this as a header, and therefore does not properly log this data. If this is of concern to you, the source code is easily modified to fix the bug (in lm-sensors).
# Name : lmsensors.conf # Description : Configuration file for modlog. This config file pulls lm- # sensors data. # Version : 0.1 # Coded by : Hans Kokx, Brian Ward # Date : May 21, 2009 # Prerequisite : lm-sensors, `cd /dev && mknod i2c c 89 0` # Command that is run to gather data input="/usr/bin/sensors" # Where the outputted data is stored output=/var/log/archive/modlog/sensors # Short description of this configuration file description="lm-sensors data" # A test to make sure that our config file wont fail outright. # $works should be 0 if all is well, and 1 if the command will fail. if [ -z "$input" ] then works=1 # fail, unless otherwise specified if [ ! -s "$input" ] then if [ -s "/etc/sensors3.conf"-a -c "/dev/i2c" ] then works=0 fi fi fi
helper.pl
The final piece of the puzzle is a small perl script. When recalling data from modlog: if -h is invoked, every 20 or so lines (a variable that can be set in the main script), the data column headers will be printed. This bit of magic is accomplished through the following perl script, provided primarilly by Ezekiel Hendrickson:
#!/usr/bin/perl # Name : helper.pl # Description : Helper script for formatting the output of modlog for display. # Version : .1 # Coded by : Hans Kokx, Brian Ward, Ezekiel Hendrickson # Date : July 2, 2009 use strict; use warnings; # Declare variables my $i = 0; my $firstline; # Print a blank line, then the column header every 20 lines. foreach(<STDIN>) { $firstline = $_ if ($i == 0); print; print "\n",$firstline if ($i > 0 && ($i % 20) == 0); $i++; }
Conclusion
Modlog allows for easy, modular logging of a set of data that is intended to be viewed and then discarded. Furthermore, it extends the functionality by easily allowing you to pull data back out of the logs, using regular expressions, for easy comparison. Modlog solved a vexing issue for the University of Michigan, and I am proud to serve it up to all who need it. As with all things, if you find anything that should be changed, fixed, or amended, feel free to leave it in the comments.
