Another more subtle, but in my opinion amazing and very powerful, feature in Log Insight 2.0 beta is machine learning. So what does machine learning look like in Log Insight, what does machine learning provide, and why is it powerful? These are the questions I will address in this post.
What does machine learning look like?
On the Interactive Analytics page, you will notice a new Summary tab below the Search bar. Selecting this tab will present events as follows: 
What you see are similar events grouped together and events being shown from the most frequent to the least frequent by default (note the number to the left of the event). In addition, you will notice that events may contain one or more hyperlinks within the message. Each hyperlink represents a type that machine learning discovered within events that were grouped together. Such types include timestamp, string, int, and hex to name a few. Since events shown on the Summary tab have been modified to show types, an Expand option is available below each pattern that machine learning discovered, which will show actual events that match the given pattern. 
As you can see, then expanded, the actual events and all the associated fields are shown and can be used just like on the Events tab.
What else does machine learning provide?
Each type discovered within a pattern learned by machine learning represents a new type of field in Log Insight called smart fields. By selecting the type hyperlink within a summary event, you are presented with some options: 
The view chart option performs an aggregation query, just like the main chart on the Interactive Analytics page, for the current type against the time range specified on the Interactive Analytics page. In short, it runs a count of events over time query for the given type. You also have the option of promoting the chart to the main chart on the Interactive Analytics page. 
The name this field option for types gives you the ability to name a smart field so that it can be used in a query (either as a filter, a function, or a grouping). 
The default name for a smart field follows the format: smart field – <type> <number> [<event_type>]. Once a smart field has been named, it will appear under the Fields section just like other fields. A named smart field can be renamed or deleted, but the definition of the smart field cannot be modified: 
Finally, machine learning introduces a new static field called event_type. This static field is seen on the Events tabs just like other fields and makes it possible to look for or exclude certain event_types when querying. 
How is machine learning powerful?
So why would you want machine learning, how is it powerful, and how would you use it? Well, before you would run a query on the Interactive Analytics page and get back some number of events. The number of events returned depends on the query and the time range, but it is common for queries to return a large number of results. For example, let me look for all events in my small lab (less than 5 devices) in the last 24 hours: 
Over 600,000 events! Now, let me search for just error events in the same time period: 
Close to 24,000 events! As you can see, searching for errors reduced the number of events significantly, but still, there are too many events to review. Let me now switch to the new Summary tab: 
16 patterns are returned, a much more reasonable number to review! Looking at the results, three patterns make up the majority of the events returned. All three are not actually error events, but warning, verbose, and warning respectively. A quick Google search and you can see the events can either be ignored or are minor:
By removing these event types, what is left is 13 patterns to review. 
Now think about this from a troubleshooting or RCA perspective. You likely have hundreds if not thousands of devices sending events and millions if not billions of events coming in every day. Through the use of machine learning, it is much easier to see what types of events are coming in, it is much easier to detect anomalies, and it is much easier to troubleshoot and perform RCA.
Summary
Machine learning is in my opinion the best feature in the Log Insight 2.0 beta. It dynamically learns and adjusts patterns from events coming into the system, it is capable of learning types, and discovered types can be named in used in queries, and it makes troubleshooting and RCA so much easier. Of course, this is only the beginning of machine learning. I am sure you can think of all the possibilities that machine learning introduces!
© 2014 – 2021, Steve Flanders. All rights reserved.
