I have talked about Log Insight user alerts in the past, but I think it is important to re-enforce the user alert architecture as it may be a little different than you think. What I want to focus on in this post is how user alerts are triggered within Log Insight. Read on to learn more!
Parts
A user alert is a query that runs on a schedule. The parts of a user alert include:
- Query: What you want to be alerted on
- Name: What you want to name the alert
- Description: Any important information about the alert
- Threshold: When the alert should trigger
Besides the query, it is the threshold that dictates when a user alert is triggered. The threshold is also the source of some questions around user alerts.
Threshold
Log Insight offers three thresholds in the form of three radio buttons. Depending on the defined query either two or three radio buttons will be available. How frequently a user alert runs is stated by the gray text below the third threshold radio button. When the user alert runs the time range specified by the threshold is applied to the query and if the results match the defined threshold then the alert is triggered. To date, the most frequent a user alert can be run is once every minute.
Silence
The piece of the user alert architecture that is often missed is that once a user alert triggers, it is silenced for the next threshold period. This means if you have results for the user alert over the next threshold period, you WILL NOT receive an alert.
Let’s say I create a query and add it as an alarm using the “on any match” threshold option. I hit the save button at 12:36.01. At 12:41.36 Log Insight receives an event that matches the user alert. At 12:48.47 Log Insight receives another event that matches the user alert. At 12:57.13 Log Insight receives another event that matches the user alert. The question becomes, how many user alerts should I expect and at what time?
The answer is two user alerts and they should be received at 12:46.01 and 13:01.01 respectively. To see why, let’s play out the example:
- At 12:41.01 the user alert runs, but no matches are found
- At 12:46.01 the user alert runs and one match is found at 12:41.36 so a user alert is sent
- At 12:51.01 the user alert DOES NOT run because it triggered at 12:46.01 is now silenced through the next alert cycle, which is this example is 5 minutes
- At 12:56.01 the user alert runs, but no matches are found because the query looks over the last 5 minutes and no events matching were received between 12:51.01 and 12:56.01
- At 13:01.01 the user alert runs and one match is found at 12:57.13 so a user alert is sent
Summary
The Log Insight user alert architecture makes it possible to run queries on a schedule and also supports setting thresholds as appropriate. The two important things to note about user alerts are that the frequency at which user alerts are run is written as text below the third threshold radio button and that once a user alert triggers it is silenced during the next polling cycle.
© 2015, Steve Flanders. All rights reserved.
