Log Insight Query Building: Search Bar

If you missed the Deep Dive into vSphere Log Management with vCenter Log Insight session at VMworld 2013 then you missed out on some great information about Log Insight! If you are using or considering using Log Insight then I would highly recommend you view the recording as it will help you on your journey of managing unstructured data. In the session, I did a deep dive on query building in Log Insight and I would like to dig into some of the concepts covered in this and future posts. In short, once you download, deploy, configure, and start sending sources to Log Insight the next step is to query for important information in your events for information, troubleshooting, and root cause analysis.

Overview

To start, navigate to the Interactive Analytics page of Log Insight. From here you will notice the search bar that divides the visual representation of data on the top, from the events coming into the system on the bottom. You will notice that this post is called Query Building in Log Insight and not Log Insight Query Language. That is because Log Insight does not require you to learn a proprietary query language to use the product. Instead, Log Insight supports complete keyword queries or what I like to call natural-language queries, in addition to constraints and regular expressions.

In this post, I would like to cover actions that can be performed from the search bar. To do this, I will discuss the different types of queries possible from the search bar as well as how to string them together.

Keywords

One common complete keyword query to start with is: error. Simply enter error in the search bar and select Search. Upon entering the word error you will notice a couple things:

• As you start to type, results are auto-completed for you based on actual complete keywords in the events.
• For each complete keyword you will notice an approximation on the number of results that are returned to the right of the complete keyword.

After running the query, you will get results that match. From your results you will notice that every occurrence of the keyword ‘error’ is highlighted.

It is important to note that the results returned are based on the time range specified for the query. By default, the time range specified is the latest five minutes of data. The reason for this is because you want to limit the number of results to query over. In short, the more results, the resource intensive the operation. The best practice is always to use the most specific time range possible to optimize the query.

One other important thing to note is that keyword queries in Log Insight are case-insensitive. This means that the query error and the query ERROR will return the same results.

Phrases

Besides complete keyword queries phrase queries can also be used. A phrase query must be encapsulated in double quotations. For example, notice the different between the query: performance has deteriorated

and: “performance has deteriorated”

While the difference is subtle the difference is very important. Notice how only the phrase query highlights the spaces between the complete keywords. This difference will be elaborated on in the values section below.

Index

Now, I am sure you noticed I have used the phrase “complete keyword” multiple times. What does it means and why is it important? Let me define the terms:

• Keyword: means any alpha-numeric and/or hyphen and/or underscore value separated by space.
• Complete: an entire keyword (i.e. not-partial).

This concept is important because keywords are how Log Insight indexes data. As data is ingested in Log Insight, it is stored in a raw, but compressed format, and an index is created based on complete keywords. When constructing a query, complete keywords must be used. An example will be provided in the next section.

Globs

Try querying for the keyword: deteriorated. You may see results like the following:

Now try querying for the keyword: deteriorate. You will notice two things:

• ‘deteriorate’ does not show up as an auto-completed result

• the query does not return any results

The reason for this is because in my environment no events contain the complete keyword deteriorate. What if you want to match both deteriorate and deteriorated? You can do this through the use of globs. You can think of a glob as a simplified regular expression. Today, two different types of globs are allowed asterisk (*) and question mark (?). Asterisk means zero or more characters while question mark means exactly one character. So to match both complete keywords above you could use the query: deteriorate*:

It should be noted that multiple globs can be used at the same time and globs can be used in any position except to prefix a keyword. An example of an invalid query would be: *deteriorated:

Queries

Another thing I would like to point is how multiple keyword, phrase, and/or globs in a single query are interpreted. Keywords, phrases, and globs separated by space on the search bar are treated as logical AND queries. As an example, the query ‘scsi “performance has deteriorated”‘ literally means: SEARCH FOR scsi AND THE PHRASE “performance has deteriorated”. This means the complete keyword and the phrase must both be found in the event for the query to return results:

In the phrase section above, I showed the difference between the query performance has deteriorated and the query “performance has deteriorated”. In addition to the first query matching three separate keywords and the second query matching one phrase, the first query does not guarantee the order in which the keywords appear in the event either.

Special Characters

As mentioned above, phrases require double quotations. In the index section I stated that keywords support hyphen and/or underscore characters. In addition, globs support asterisk and question mark. All other special characters are ignored on the search bar.

This also means that searching for literal special characters in an event cannot be done from the search bar. To search for literal special characters in an event, a regular expression must be used.

Regular Expressions

Regular expression queries are possible using constraints. I will be covering constraints in a future post.

Summary

Some of the key takeaways for querying building on the search bar are:

© 2013 – 2021, Steve Flanders. All rights reserved.