I have spent the last three weeks talking about the new agent parsing feature in Log Insight 3.0. I wanted to wrap-up by discussing why you should use agent parsers.
- Static versus dynamic fields: As I have described before, static fields are way more efficient than dynamic fields when it comes to query performance.
- Pattern match versus regex: Regular expressions are hard, but defining a pattern is easy. Parsers do not require the knowledge of regex like extracted fields do.
- No need for keywords: In order to make extracted fields remotely efficient you need to leverage the keyword index, which means you need context. With parsers, you only need to know the format or the fields you care about.
- Subparsing: Since you do not need keywords, when defining parsers you can also easily subparse events regardless of the format or structure of the event.
- Parsers provide natively support for structured data and enhance support for unstructured data: While the message part of syslog events is unstructured by definition, it is common for syslog and non-syslog events to contain structured information. CSV events could not be efficiently parsed with extracted fields due to missing keywords and while KVPs could be extracted the process could not be automated.
- Parsers works with any protocol: By default the agent sends events over Log Insight’s ingestion API in which case parsed fields are sent as static fields. If the syslog protocol is used, parsed fields are included in the STRUCTURED-DATA section of the event based on syslog RFC 5424. This means parsing works whether you sent events from the agent to Log Insight or a third-party syslog destination.
- Client-side versus server-side: The more you do server-side the more resources you need. Log Insight is CPU and IOPS intensive so offloading new functionality makes it possible to leverage existing resources.
- Time: When analyzing events, time matters. By default server time is used, with the agent client time can be used, with parsers event time can be used.
- Unnamespaced fields: With parsers you can easily use the same field name across multiple different log formats. This makes it possible to parse something like a username across systems.
- Field renaming: Fields can also be renamed making it possible to leverage something like the KVP parser, but still get the benefit of unnamespaced fields.
- Unstructured data is hard: Maybe the pattern between events are all different, but you always want to extract key/value pairs. Maybe the same key/value pairs are used multiple times in the same event. Parsers make unstructured data easier to manage.
- Many content packs include agent groups with parsing by default: This means you did not need to determine the parsing configuration and you can enjoy all of the benefits of parsing just be configuring a content pack.
Summary
As you can see, agent parsers are extremely powerful — and in my opinion represent that best new feature in Log Insight 3.0! The best part is that content packs can now, and many do now, contain agent groups as well as agent parsers. I would encourage you to take a look at agent parsers when you have the chance.
© 2015, Steve Flanders. All rights reserved.
