Log Insight 3.3 brings some new features to the Log Insight agent. In this post I would like to cover the improvements. Read on to learn more!
New Parsers
A variety of additional parsers are now available including:
LTSV
LTSV or (Labeled Tab-Separated Values) is a logging format seen in some environments. The Log Insight agent now natively supports this format via a base_parser. This base_parser does not have any configuration options and can be called directly.
[filelog|ltsv] directory=/var/log/ltsv parser=ltsv
If you wish to debug the parser then you can call it in a dedicated parser section by using a name other than ltsv.
[filelog|ltsv] directory=/var/log/ltsv parser=ltsv-parser [parser|ltsv-parser] base_parser=ltsv debug=yes
Syslog
Syslog is a standard format in which logs are typically sent. There are a few RFCs to be aware for syslog including: 3164 and 5424. The Log Insight agent supports both via a base_parser. This base_parser does not have any configuration options and can be called directly.
[filelog|syslog] directory=/var/log/syslog parser=syslog
If you wish to debug the parser then you can call it in a dedicated parser section by using a name other than ltsv.
[filelog|syslog] directory=/var/log/syslog parser=syslog-parser [parser|syslog-parser] base_parser=syslog debug=yes
The STRUCTURED-DATA part of RFC5424 is not supported by the syslog parser today, but can be parsed with other existing parsers using the next parser option.
[filelog|syslog]
directory=/var/log/syslog
parser=syslog
next_parser=syslog5424-parser
[parser|syslog5424-parser]
base_parser=clf
format=%i %i %i %i %i [%{structured_data}i]%i
field_decoder={"structured_data":"kvp-all"}
exclude_fields=structured_data
[parser|kvp-all]
; Note parsing all fields with KVP may return undesired results
; Assuming RFC5424 this will work as expected
base_parser=kvp
delimiter=" []"
fields=*
Regex
Regular expressions provide a lot of flexibility, but often at the cost of resources and time. Sometimes it is impossible to avoid regular expressions. To deal with complex logs (i.e. logs with many different logging formats or non-standard log formats), the Log Insight agent now supports regex via a base_parser. Below is an example of how to use the regex parser to extract STRUCTURED-DATA from RFC5424 instead via the CLF parser (note this is NOT recommended for performance reasons — this is simply an example of how to use the regex parser).
[filelog|syslog]
directory=/var/log/syslog
parser=syslog
next_parser=syslog5424-parser
[parser|syslog5424-parser]
base_parser=regex
format=\S+ \S+ \S+ \S+ \S+ [(?<structured_data>.*?)]%i
field_decoder={"structured_data":"kvp-all"}
exclude_fields=structured_data
[parser|kvp-all]
; Note parsing all fields with KVP may return undesired results
; Assuming RFC5424 this will work as expected
base_parser=kvp
delimiter=" []"
fields=*
The regex parser has one mandatory parameter: format. The format supports naming fields through capture groups in the form (?) where everything in the will get the name . Again, please be advised this parser should only be used when the other parsers are not sufficient (i.e. rarely).
IPv6 Support
All Log Insight 3.3 agents support running on IPv6 clients. No configuration changes are needed for this support!
copytruncate Support
The Log Insight Linux agents support the copytruncate option from logrotate. Per logrotate’s man page, here is what the copytruncate option does:
Truncate the original log file to zero size in place after creating a copy, instead of moving the old log file and optionally creating a new one. It can be used when some program cannot be told to close its logfile and thus might continue writing (appending) to the previous log file forever. Note that there is a very small time slice between copying the file and truncating it, so some logging data might be lost. When this option is used, the create option will have no effect, as the old log file stays in place.
Here is an example of a logrotate configuration (typically found in /etc/logrotate.d) leveraging the copytruncate option:
/tmp/output.log {
size 1k
copytruncate
compress
compresscmd /bin/bzip2
compressext .bz2
rotate 4
maxage 100
}
If you have any logs rotated in this manner then the Log Insight Linux agent has you covered no additional configuration required! This one is a big deal — to my knowledge, the Log Insight agent is the ONLY agent on the market to support the copytruncate option. Other agents either do not support it and recommend changing the logrotate option or “support it” by ignoring the rotated file. The Log Insight agent supports it even across rotated files!!!
Windows 10 Support
The Log Insight 3.3 Windows agent supports Windows 10.
Summary
The Log Insight agent includes a lot of cool new features, not to mention bug fixes and performance improvements. If you are currently leveraging the agent it is recommended that you upgrade. Note the agent is forward and backwards compatible for two versions so you can use the 3.3 agent on LI 2.5 and the 2.5 agent on LI 3.3. Of course, it is always recommended that you run the latest version of the server and agent.
© 2016, Steve Flanders. All rights reserved.
