I covered query building in Log Insight a long time ago. While the UI has changed significantly since I wrote those posts, the information contained within them remains true to the current version of Log Insight. In Log Insight 3.0, two new operators were added to certain places within the UI — in one instance replacing two existing operators. In this post, I would like to discuss these new operators. Read on to learn more!
New Operator
Both a “matches” and “does not match” operator have been added to the following pages:
- Administration > Event Forwarding (replaces “contains” and “does not contain”)
- Administration > Agents (new functionality added to page — for more information see this post)
Operator Definition
So what do the new operators mean exactly? Well, as the name implies, “matches” would match EXACTLY the value presented while “does not match” would NOT match EXACTLY the value presented. The key word in both cases is “exactly”. To understand the difference of these new operators be sure to read the globs section of this post first. Given that Log Insight requires keyword searching, does NOT support prefix globbing and the new operators described in this post require an exact match, you may be wondering how Log Insight supports the new operators. The answer is the “matches” and “does not match” operators *ONLY* do support prefix globbing. To make this clearer, let me use some examples:
- “test123” is equivalent to “matches test123” or “matches test*” or “matches *123” or “matches *test123*”
- “this is a test” is equivalent to “matches this is a test” or “matches this *” or “matches * is a *” — searching for a single keyword will NOT work (e.g. “matches test” will not return any results for this example)
Compare this against the existing contains operator:
- “test123” is equivalent to “contains test123” or “contains test*” — using a prefix glob is NOT supported (e.g. “contains *123” would result in a syntax error)
- “this is a test” is equivalent to “contains this is a test” or “contains this *” or “contains test”
Why A New Operator?
You may be wondering why a new operator was introduced instead of using the existing operators on these two pages. The reason is because these two Administration pages do not use keyword indexes for queries like the Dashboard and Interactive Analytics pages do. The net result would be some unexpected behavior with existing operators.
As an example, let’s say on the Administration > Agents page I have the following two IP addresses listed: 192.168.1.11 and 192.168.111. Let’s say I wished to create an agent filter that would match just the 192.168.1.11 agent. Given the behavior of the Dashboards and Interactive Analytics pages, you would expect that “contains 192.168.1.11” would match just the first agent and that “contains 192.168.1.11*” would match both agents, but because the data on the agents page is not keyword indexed, “contains 192.168.1.11” actually matches both 192.168.1.11 and 192.168.1.111 — yes this means that querying against indexed data is different than querying against non-indexed data. Clearly this could be problematic and result in a lot of confusion given the same operator would behave differently on different pages.
In the case of event forwarding, Log Insight 2.5 offered “contains” and “does not contain” operators, however they did NOT behave like the operators on the Dashboards and Interactive Analytics pages did. Event forwarding happens BEFORE the keyword index is generated (for more information about the ingestion pipeline see this post). Event forwarding filters ONLY support static fields — which you could consider to be pre-indexed — and text — which is NOT indexed. Because text is supported and indexes for text have not been created by the time event forwarding kicks in, a different query facility is needed just like for the Agents page described above. Long story short, while Log Insight 2.5 appeared to support “contains” and “does not contain” for event forwarding, it actually behaved like the new “matches” and “does not match” operators do now (this was a bug). This of course is confusing and also limiting as “contains” does not support prefix globbing. Both of these issues are fixed in Log Insight 3.0 by changing the event forwarding operators to the new “matches” ones.
Impact of New Operators
It is important to note that while filters to Agents is new in Log Insight 3.0, Event Forwarding existed previous to Log Insight 3.0 and supported filtering using the “contains” and “does not contain” filters. As a result, if you were leveraging those operators with event forwarding and you upgrade to Log Insight 3.0 those operators will be changed to the new “matches” and “does not match” operators. To ensure the experience is as seemless as possible, the value of such filters are now prefixed and suffixed with *.
For example, if you previously had an event forwarding filter of “contains foo” and you upgrade to Log Insight 3.0 then the filter will be changed to “matches *foo*”. While this should work in the majority of cases, this does mean that some queries may match slightly differently than before. To be clear, all queries that worked with “contains foo” will also work with “matches *foo*”, however “matches *foo*” may match more events than “contains foo” did. Here is an example:
- “this is a test” if you had a filter of “contains test” then upon upgrade you will have “matches *test*”. Both “contains test” and “matches *test*” will forward the event “this is a test”, however an event that contains the keyword “latest” would be sent using the matches operator where it would not with the contains operator
IMPORTANT: While event forwarding filters using “contains” or “does not contain” have been changed to “matches” and “does not match” with prefixed/suffixed globs the SAME event will still be forwarded. Some additional events MAY also be forwarded depending on the value used.
It is also important to note that selecting the “Run in Interactive Analytics” option under event forwarding does not result in the most accurate query alternative when using the new operators for Log Insight 3.0 (this is a bug). If you have an event forwarding filter of “matches *foo*” and select the option to run in Interactive Analytics then the Interactive Analytics query will show “contains foo*”. This may not match all the events given the prefix *. Another example would be “matches foo” which when run on IA results in “contains foo”. Since Interactive Analytics uses keyword queries, “contains foo” is likely to match significantly more events than “matches foo” — put another way, it is more likely that “foo” appears somewhere within a given field/text, but less likely to match exactly a given field/text. To address this, I suggest when selecting the run in Interactive Analytics option you should change the “contains” operator to “matches regex”, change “*” to “.*” and prefix with “.*” — for example “matches *foo*” should be “matches regex .*foo.*”. For the “does not match” operator, I suggest using the regex lookahead option so “does not match *foo*” should be “matches regex .*(?!foo).*” — for information on regex see this post.
Summary
The new operators in Log Insight 3.0 are featured for filters that do not support keyword queries. These new operators often need prefix and/or suffix globs to match the events expected. Note that event forwarding filters will be slightly different after upgrade, but all existing events forwarded will continue to be forwarded. Also note that testing event forwarding filters by running on Interactive Analytics will not work as expected in many cases in Log Insight 3.0, but can be adjusted to return similar results.
© 2015, Steve Flanders. All rights reserved.
