The second agent parser I want to take a look at is the KVP parser. Read to learn how it works!
How the Parser Works
The KVP parser is for events that follow a key = value delimiter format. The pieces are defined as:
- Key: Any characters except space or a double quoted set of characters
- Equal: The equal sign — no other characters allowed — can be prefixed or suffixed with a space
- Value: Any characters before a delimiter
- Delimiter: Defaults to space, tab, newline, comma, semicolon — can be changed
While the delimiter is an optional configuration parameter, the “fields” option is not. Like the CSV parser, you must specify the fields you want indexed. If you remember from my CSV post, the fields option was required because the key equivalent part of CSV events could not be easily determined. With the KVP clearly the key is known however some events contain a lot of KVP entries and a particular log file may contain a ton of KVP entries. While indexing fields is more performant for Log Insight, it does result in more space being consumed on the server.
The best practice is to only index fields that you care about (i.e. will query over). Because the KVP parser can return a lot of unique results you must explicitly select which fields you wish to index. The format of this option is the same as the CSV parser option, however the KVP option also allows the following:
fields=*
This configuration would allow for all KVP entires to be indexed. Note this is NOT a default option and is NOT recommended without first confirming that you have a limited number of unique KVP entries and that you care about the majority of them.
Basic Example
Given the above information, the most basic KVP parser configuration would be as follows:
[parser|simple_kvp] base_parser=kvp fields=*
Remember, this results in all KVP entries being indexed and may not work as desired. To clearly demonstrate why the above configuration should not be used by default, look at the example log file:
2015-01-26 23:08:45,048 | DEBUG | consoleproxy | SimpleProxyConnectionHandler | Initiated handling for channel 0xb8bdf4 [java.nio.channels.SocketChannel[connected local=/10.150.15.232:443 remote=/10.150.15.237:60713]] | 2015-01-26 23:08:47,631 | DEBUG | consoleproxy | AuthdUserMatch | Logging in via an Authd token: cst-ccVmij4kNtDni8StYA3fN+KtAznt7jMYiq3kXPV01N0ZWs0u2HN/Q/qasVqxhLPlOaB+OO0GZcnw90FniYvReS+d2m6Y6dFl/2N90h4rAm9sdG17b/D1swUzMOhHbXmCAm7QF6sJ7goby1SXCiGxOiux/CS/+Mmr5tdhSfHS4YWEELX3W+MumStJXGRlMWfkRbvmFiykmNzkKTb9rtl1hJG8+3Lii6zmJBJAuypAD6d7fqzlPq3d4TO2aq1hnSfYKcB3xJDzqaTGf9olirWGWXB1JkxyGwpz7H/mOjEXCPQ81LAFEt8hLudNwiDryImX-pszG506/LwlRDZrLgnVWM84PSHX+mRr4zTE5hw==--tp-FC:2F:57:44:96:DA:C5:03:5A:E8:EC:9B:DD:3A:B0:8F:C2:5C:E8:92-- |
This file contains valid KVP entries, but it also contains SSL certificate information. The SSL certificate information contains an equal sign and results in KVP entries that are not desired. If you have a lot of different SSL certificate information entries in your log files, this can result in a lot of indexed fields:
Advanced Example
Now let’s look at a more complex example:
Sample configuration scope=local abstract=false lazyInit=false autowireMode=0 dependencyCheck=0 Sample configuration scope=ad abstract=false lazyInit=false autowireMode=1 dependencyCheck=0
For the above log file the following filelog configuration can be defined:
[filelog|sample-conf] directory=/var/log/test event_marker=^Sample configuration parser=sample-conf-parser
The result is a multiline event containing KVP entries. The first event looks like:
Sample configuration scope=local abstract=false lazyInit=false autowireMode=0 dependencyCheck=0
Let’s assume I care about the scope, lazyInit and autowireMode keys only. My parser configuration would need to look like the following:
[parser|sample-conf-parser] base_parser=kvp delimiter="\n\t" fields=scope,lazyInit,autowireMode
As you can see, the default delimiters were not sufficient for the above example because KVP entries were separated by tab in addition to newline. As such, I had to define a new delimiter. Defining a delimiter overrides the entire default delimiter to I had to add newline back in. Finally, given I only cared about select fields, I statically defined them for indexing.
IMPORTANT: Given that newline, tab and return (\r) need to be escaped, this also means that forward slash (\) also needs to be escaped (\\).
Summary
As you can see, the KVP parser will be helpful for a variety of log files, but ensure only the fields you care about are extracted. Be sure to specify the fields option or else no fields will be sent. The KVP parser is only for events that follow a known key = value pattern. For other logs formats see my following posts on other agent parsing options. Do you have a need for the KVP parser?
© 2015, Steve Flanders. All rights reserved.
