I often get questions around Log Insight agent configuration. While the options are clearly documented, it can be time-consuming and error prone to construct lengthy agent configuration files. In this post, I will introduce a CLI utility I wrote to make this process easier and less error prone.
The Log Insight agent configuration is made up of sections that monitor specific logging aspects on the client device. There are several important things to note about these configuration sections:
- Each configuration section must have a unique name.
- Today, the agent supports two types of event monitoring: winlog and filelog.
- Winlog requires exactly one channel to monitor
- Filelog requires a single directory to monitor (no globs)
- By default, winlog and filelog will collect everything in the configured channel/directory
- Winlog offers the option to limit what is collected
- Winlog and filelog offer the ability to pass tags
- Filelog supports three different character sets today: UTF-8, UTF-16LE, UTF-16BE
While the generate “rules” of the agent syntax are pretty easy to pick up, you may notice that multiple configuration sections may be needed to collect all of the events you care about on a client device. For example, with filelog, every directory you want to monitor events in needs its own configuration section. If you are running vSphere 6 and use the VCSA, you will notice that 28 different directories need to be monitored to collect all of the VMware specific log messages.
To make it easier to generate these configuration files, I have written a script that takes the inputs supported by both winlog and filelog today and spits out a configuration section. For example:
$ > ./liagentify.sh filelog apache /var/log/apache2 [filelog|apache] enabled=yes directory=/var/log/apache2
Now with a little CLI magic, you can easily generate configuration files for multiple directories.
Note: This magic assumes that options such as include, exclude, event_marker, charset and tags are identical
For example, let’s say I have directories /var/log/a and /var/log/b from which I wish to collect only .log files with an event_marker of ^\d. I can either write a for loop with the directories included around liagentify:
$ > for dir in a b; do ./liagentify.sh filelog $dir /var/log/$dir '' '' '^\d'; done [filelog|a] enabled=yes directory=/var/log/a event_marker=^\d [filelog|b] enabled=yes directory=/var/log/b event_marker=^\d
Or I can write the directories to a file and use a for loop:
$ > cat dirs a b $ > for dir in $(cat $dirs); do ./liagentify.sh filelog $dir /var/log/$dir '' '' '^\d'; done [filelog|a] enabled=yes directory=/var/log/a event_marker=^\d [filelog|b] enabled=yes directory=/var/log/b event_marker=^\d
As you can see, the script makes it very easy to generate agent configurations.
Important: The script to does not properly validate all inputs today. You should check the agent logs after applying the configuration to ensure you entered the correct values. For example, all winlog section and all filelog section names must be unique, however this script will allow the same name to be used more than once.
And with that, here is the script:
#!/usr/bin/env sh
#
# liagentify
# Steve Flanders (stevesflandersnet)
#
# Description - Constructs Log Insight agent configurations
# Limitations - parameters are not validated
#
# ***PLEASE DO NOT REMOVE THIS HEADER AND PLEASE CREDIT THE AUTHOR***
#
######################################
# DO NOT CHANGE ANYTHING BELOW HERE!!!
######################################
# Usage
SCRIPT=`basename $0`
USAGE_WINLOG="Usage: $SCRIPT winlog
Where:
* = name of the configuration section, cannot contain spaces and must be unique per configuration file
* = Windows event viewer channel
* = fields you wish to add to each event
* = include specific events
* = exclude specific events
* = any fields you do not wish to include
Examples:
* basename $0 winlog application application
* basename $0 winlog application application '{ms_product:windows}'\n\n"
USAGE_FILELOG="Usage: $SCRIPT filelog
Where:
* = name of the configuration section, cannot contain spaces and must be unique per configuration file
* = absolute path where the file(s) are located
* = a semicolon separated list of files to collect, supports globs
* = a semicolon separated list of files to ignore, supports globs
* = regex for what is unique per event
* = UTF-8, UTF-16LE, UTF-16BE
* = fields you wish to add to each event
* = any fields you do not wish to include
Examples:
* basename $0 filelog vCenter 'C:\\ProgramData\\VMware\\\\vCenter Server\\Logs'
* basename $0 filelog vCenter 'C:\\ProgramData\\VMware\\\\vCenter Server\\Logs' 'vpxd-*.log vpxd-alert*;vpxd-profiler*'
* basename $0 filelog vCenter 'C:\\ProgramData\\VMware\\\\vCenter Server\\Logs' 'vpxd-*.log vpxd-alert*;vpxd-profiler*' '' '{vmw_product:vcenter-server}'\n\n"
USAGE="\nUsage: $SCRIPT
Usage: $SCRIPT winlog
Usage: $SCRIPT filelog
Important:
* must be unique for each configuration section
* does NOT support globs (* or ?)
* Inputs are not validated\n\n$USAGE_WINLOG$USAGE_FILELOG"
# Set parameters
TYPE=$1
NAME=$2
if [ "$1" == "winlog" ]; then
if [ "$2" == "" -o "$3" == "" ]; then printf "$USAGE_WINLOG"; exit; fi
CHANNEL=$3
TAGS=$4
WHITELIST=$5
BLACKLIST=$6
EXCLUDEFIELDS=$7
elif [ "$1" == "filelog" ]; then
if [ "$2" == "" -o "$3" == "" ]; then printf "$USAGE_FILELOG"; exit; fi
DIRECTORY=$3
INCLUDE=$4
EXCLUDE=$5
EVENTMARKER=$6
CHARSET=$7
TAGS=$8
EXCLUDEFIELDS=$9
else
printf "$USAGE"; exit
fi
# Construct configuration
echo ""
echo "[$TYPE|$NAME]"
echo "enabled=yes"
if [ ! -z "$CHANNEL" ]; then echo "channel=$CHANNEL"; fi
if [ ! -z "$DIRECTORY" ]; then echo "directory=$DIRECTORY"; fi
if [ ! -z "$INCLUDE" ]; then echo "include=$INCLUDE"; fi
if [ ! -z "$EXCLUDE" ]; then echo "exclude=$EXCLUDE"; fi
if [ ! -z "$EVENTMARKER" ]; then echo "event_marker=$EVENTMARKER"; fi
if [ ! -z "$CHARSET" ]; then echo "tags=$CHARSET"; fi
if [ ! -z "$TAGS" ]; then echo "tags=$TAGS"; fi
if [ ! -z "$WHITELIST" ]; then echo "whitelist=$WHITELIST"; fi
if [ ! -z "$BLACKLIST" ]; then echo "blacklist=$BLACKLIST"; fi
if [ ! -z "$EXCLUDEFIELDS" ]; then echo "tags=$EXCLUDEFIELDS"; fi
echo ""
exit
© 2015, Steve Flanders. All rights reserved.
