Extracted fields provide a powerful way to construct queries in Log Insight. In Log Insight 2.5, two improvements have been made to extracted fields to make them easier to understand and more efficient to construct. I would like to discuss these changes.
Previously
Before Log Insight 2.5, constructing a field required four pieces of information (for more information about extracted fields see this post):
- Value – what you were trying to extract
- Pre-context – what appeared before the Value
- Post-context – what appeared after the Value
- Name – what the field was to be called
Since extracted fields are meant to be regular expressions that extract something within an event, it is critical to include as many keywords as possible in the pre/post context to optimize the query. In order to add required keywords, you often need to construct regular expressions in the pre/post context sections. For example, assume you want to extract the following value:
Upon selecting the Extract field option, LI attempts to make a pattern match for you. As you can see, the pattern match does not include all the events we care about:
To address this issue we need to provide more context like:
This requires manual changes and good knowledge of regular expressions as well as how to construct efficient queries.
New
One of the primary limitations of the previous extract field capabilities was that it did not allow for the construction of a regular expression like the Interactive Analytics page. What I mean is that you could not search for keywords anywhere within an event or filter by fields. In Log Insight 2.5, you now have these options when you select the Add additional context button:
Given the previous example, one can now add the appropriate keyword(s) without having to figure out the regular expression necessary to match the event. In addition, specified keywords can appear in the pre or post-context.
One other new option is the ability to add notes to the extracted field. Notes are a way of telling a user additional information that may be useful and are used in many other locations in Log Insight today including dashboard widgets and alerts. To add or edit notes for extracted fields, select the i button in the extracted field dialog box:
Usage
While the extracted field enhancements are meant to be used by everyone, they are especially beneficial to content pack authors as they provide a means to extract fields from events that do not contain keywords but could contain tags like those supported in the Log Insight agents. For example, here is an example Apache log from my server:
As you can see the event contains no real unique keywords. If I send apache events with the LI Linux agent configured to tag Apache events then I can extract fields from Apache events efficiently. Hmm… 🙂
© 2014 – 2021, Steve Flanders. All rights reserved.
