Now that you know what agent parsers are capable of you are probably wondering how to configure them. In this post, I will start with the basics including the common options for all parsers. Read on to learn more!
Using the parser option
In order to use parsers you need two things. The first is to call the parser option within each winlog or filelog section for which you wish to use a parser. This is as simple as:
parser=my_parser_name
For example:
[filelog|test] directory=/var/log/test parser=test_parser
Basic parser sections
The second thing you need is one or more parser sections. The bare requirements are:
[parser|my_parser_name] base_parser=type
I already covered the types in my last post. Here is a real example:
[parser|test_parser] base_parser=kvp
Note that every base_parser has different required options and I will be covering each parser in-depth in future posts.
Advanced parser sections
Besides the minimal options and those specific to each base_parser there are also some more advanced options to be aware of:
- field_decoder: This option makes nested parsers possible. For example, say you wanted to apply one base_parser to an event and then apply a separate — could be the same or different — against a subset of what is contained within an event. The field_decoder makes this possible.
- field_rename: Agent parsers can automatically name fields based on built-in logic of from information contained within an event such as the key in the KVP base_parser. Sometimes the field name is not what you want to appear as on Log Insight. This option allows you to change the name prior to it being sent. It is important to note that the following field names cannot be used: event_type, hostname, source, text, timestamp.
- next_parser: Not all events within the same log files can be treated equally. As such, it may become necessary to run more than one parser against a file to ensure parsing is done as desired. This is where the next_parser comes in. If an event matches the first parser then it is used then the next_parser option is used. If the next_parser has the same field names as the previous parser then the fields are overridden. This option is powerful, but should be used cautiously to prevent improper parsing.
- exclude_fields: An agent parsing operation is all or nothing. If configured properly, it parse all defined fields. Sometimes a parsed field is not needed and should not be sent to the Log Insight server. The exclude_fields option handles this scenario.
- debug: This option is set to “no” by default and should not be enabled unless agent parsing is not working as desired. If troubleshooting of a parser is needed, the debug option should be set to yes. Do not change the log_level option for parsing troubleshooting. All debug information is written to the agent log file.
The above advanced options will make more sense with examples. In my next series of posts I will be discussing each base_parsers in-depth and I will be including the above options so you can see how they work.
Summary
Agent parsers require a parser option within the winlog and filelog sections you wish to parse. In addition, parser configuration sections needs to be added to handle the parsing desired. While the parsing section options will depend on the base_parser being used, there are also some advanced options that allow for nested parsers and multiple parsers to be run against a single event as well as field renaming and exclusion. The most important parser section option may be the debug option as it makes it possible to easily troubleshoot parsing configuration issues via the agent log file. Have you tried any of the advanced parser options yet?
© 2015, Steve Flanders. All rights reserved.
