Log Insight Query Building: Fields

In the final installment of the Log Insight query building series, I would like to cover the use of fields. Fields in Log Insight are a way of providing structure to unstructured data. Fields represent a regular expression pattern match for a particular object within an event. I covered how to manage fields in this post and now I would like to cover the different types of fields Log Insight supports, how to construct fields, how to modify fields, and how to use fields to extend query building in Log Insight.
UPDATE: While this post is from version 1.x, it is applicable for 2.x and 3.x as well. The post has been updated to reflect changes over time. Note the screenshots are old but do represent the capabilities available.
li-logo

Types

Log Insight support multiple types of fields including:

  • Static fields
    • Static fields are fields that Log Insight knows about by default or is told about via Log Insight agent tags/parsers (version 2.0 or newer), Log Insight event forwarding tags (version 2.5 or newer), or Log Insight multi-VIP + tags (version 3.3 or newer)
    • Examples of static fields include syslog RFC metadata including source, hostname, and appname
    • Static fields are read-only — they exist during ingestion time and are stored in a query performant index
      log-insight-static-field
  • Content pack fields
    • Content pack field are fields that are defined within a content pack
    • An example of a content pack field is the vmw_device_id defined in the VMware – vSphere content pack
    • Content pack fields are read-only, but can be cloned by any user and can be removed by admin users by removing the content pack
    • Content pack fields have the name in parenthesis to the right of the field name
      log-insight-content-pack-field
  • User fields (aka custom fields)
    • User fields are defined by a user of Log Insight
    • There is no example of a user field by default in Log Insight
    • User fields can be defined under private or shared user spaces
    • User fields defined under private user space are editable only by the current user
    • User fields can only be defined and edited under shared user space by admin users though normal users have read-only access
    • Starting in version 2.5, additional context can be added to extracted fields to optimize query performance
  • Temporary fields (called orphaned fields in Log Insight 1.0)
    • Temporary fields are fields used in a save query, but fields that are not defined in a Log Insight instance
    • There is no example of a temporary field by default in Log Insight
    • Temporary fields are the result of saving a query with a field and later deleting the field from Log Insight while keeping the query
    • Temporary fields have the word Temporary in parenthesis to the right of the field name
      log-insight-temporary-field

Creating

Log Insight makes creating fields easy. Fields can be extracted from the Interactive Analytics page. The easiest way to extract a field is to highlight any part of an event that you want to extract. Upon doing so, an Extract Field button becomes available.

When the Extract Field button is selected, a dialog box opens under the Fields section of the Interactive Analytics page.

Note: In Log Insight 1.0, when extracting a field a test option was available. The test option has been removed in Log Insight 1.5.

An extracted field is made up of several components:

  • Value – a regular expression matching the object you selected in an event. Several optimized regular expression values are available or any custom value can be entered.
  • Context – what appears before and/or after the value. Ideally, this would contain one or more keywords.
  • Name – what you want to call the field. This is how the field will appear.
  • Permission – should the field defined by private or shared. Shared is only available for admin users.

By default, if a field is extracted by highlighting part of an event and selecting the Extract Field button, Log Insight automatically enters information for the value, context, and permission components. You have the ability to edit any of these components as necessary to match the object you are trying to extract. Once the field is given a name and saved the field is available for use.

log-insight-extract3

An alternative way to extract a field is with the Extract Field link to the right of the Fields section title. With this method, the value and context fields need to be manually entered as Log Insight has no information about the field that you want to extract.

Modifying

After a user field has been defined, you may at some point want to modify or even delete the field. Log Insight provides two ways of doing this:

  • If the field appears under the Fields section of the Interactive Analytics page then the pencil icon can be selected.
    log-insight-extracted-field
  • If the field does not appear under the Fields section then from the menu drop-down next to the Search button you can select Manage Extracted Fields.
    • You can delete a user field by hovering over the field and selecting the red X.
    • You can modify a user field by selecting the name and making changes in the dialog box presented under the Fields section.

It is important to note that in 1.5 versions of Log Insight, if you modify the definition of a field that is used by a saved query then you will be updating the definition of the field for all saved queries as well.
log-insight-update-field

Warning: In Log Insight 1.0, field modification worked differently. In Log Insight 1.0, modifying a field did NOT update any saved queries that leveraged the field.

Using

What value do Log Insight fields bring? Fields can be used in many different aspects of query building including:

  • Constraints (now called Filters) – constraints with fields allow for refined queries based on specific operators against defined fields (e.g. contains, >, etc).
  • Functions – mathematical functions (e.g. average) require a field to apply the function against.
  • Groupings – objects can be grouped by available fields.

The value of fields can be easily demonstrated with an example. Take, for example, ESXi SCSI latency events. Let’s say I search for: scsi deteriorated. The results might look something like the following:

What if I only want to see results when the latency being experienced is over a certain value? To address this, I can add a constraint looking for when the vmw_latency_in_micros, a field defined in the VMware – vSphere content pack, is > a certain value:

While the message query and aggregation query bring value, I cannot easily determine the average SCSI latency of the events even though the latency is listed in the event. To address this, I can create an aggregation query that shows the average latency using the vmw_latency_in_micros field:

This aggregation query makes it easy to see the average SCSI latency across all the datastores in my environment, but it does not give an indication if the latency is being experienced on one or more datastores even though the events contain datastore information. To address this, I can modify the aggregation query and group by the vmw_device_id field available in the VMware – vSphere content pack:

Now, I can easily see which datastores are experiencing latency. Finally, what if I want to be notified when the above query returns any results for a particular datastore? To do this, I can create an alert with a threshold based on the grouping specified in the aggregation query:
log-insight-query-alert

Summary

Fields provide a way to add structure to unstructured events. Some of the key takeaways for query building using fields are:

  • Fields consist of a regular expression value and context around the value
  • Multiple types of fields exist including static, content pack, user, and temporary
  • Fields can be used multiple ways in query building including constraints, functions, and groupings

© 2013 – 2021, Steve Flanders. All rights reserved.

9 comments on “Log Insight Query Building: Fields

Dan says:

Thanks for these articles. Not too many resources out there for Log Insight, and these are particularly helpful.

Glad you like them! Let me know if you are interested in anything else 🙂

Reza says:

Do you know if it is possible to modify the static fields? i found that fields are not correct all the time, and we need to change them.
For instance appname doesn’t resolve to postfix as it has postfix/xx format

Static fields cannot be changed today — if it is “wrong” then that likely means an RFC violation. Can you provide an example of what is wrong? Note, LI is alphanumeric, hyphen and/or underscore delimited so postfix/xx will treat postfix and xx separating in terms of queries.

SP says:

Hi Steve, trying to find out the destination host details for a vMotion task and the current query “migrate virtual machine” has just the source details of it. Is there a way to add a field to get this data?

In one query I have not found a way, but in two it is possible. I have uploaded a sample content pack here — if you extract, import and go to the Dashboard page then you can select a VM that appears in the “Migrated VMs by VM” widget and then change the filter from “vc_vm_name” to “text”, it will show you the hosts involved in the migration.

SP says:

Thanks Steve. The use case I am looking for is to fetch the details of VMs restarted by HA when there is a host failure. We can leverage powercli, but that is limited to the data retention and thought LI could be the best bet. Though we can fetch the VM details that got restarted, it only gives the details about the destination Host on which it got restarted but not the source.

Martin says:

Hi Steve. I am looking for solution to merge to type of events. One is coming from Palo Alto and second from Juniper. I trying to create new fields with the same data but from different systems. It is possible?

Hey Martin — if the goal is to create an extracted field that matches two different events then it is possible if you construct the proper query. The typical gotcha is performance — if the two different events do not share any common keywords outside of the value you are trying to extract then your only option is to use a regex, which is slow. Assuming they do share at least one keyword then you just need to construct an extracted field with pre/post context that matches both types of events you are trying to match. If you provide an example I can make a suggestion on how to achieve this.

Leave a Reply

Your email address will not be published. Required fields are marked *

Back To Top