Today, Log Insight system notifications consist of both informational messages and critical system alert messages. In this post, I will discuss how to only get notified for critical system alert messages when integrating Log Insight with an on-call product like PagerDuty.
PagerDuty
PagerDuty is a SaaS service used to alert an on-call of issues within an environment. There are many similar products on the market today, but the issue being discussed in this post will be applicable to all of them. Note that configuration of PagerDuty is beyond the scope of this post. The focus of this post will be primarily around configuring a service to get properly notified about real Log Insight issues.
Prerequisites
Before continuing be sure to read the following as this post is based primarily on these concepts:
Configuring Services
In PagerDuty, to get alerted about potential issues you need to configure a service. The first step is determine the integration type. Since Log Insight sends system notification and user alerts via email, it makes sense to use the email integration type in PagerDuty. There are two potential issues when sending email from Log Insight to PagerDuty:
- Some system notifications are informational and do not indicate a real issue
- Depending on user alert threshold configuration, repeat alerts (i.e. emails) can be sent for the same issue
To address these issues you should leverage the following PagerDuty service options:
- Email filters — this option makes it possible to ignore some events coming in so people do not get paged for informational events. For this to work properly, you need to configure email rules that can match parts of an incoming email. The following settings should be used for Log Insight:
- Email filters: Accept email only if it matches ONE OR MORE rules below
- The email subject: does not match the regex “(Log Insight Admin Alert: (Repository Retention Time|Oldest Data Will be Unsearchable Soon|Archive Failure)|Log Insight Successfully Upgraded)”
- AND the email body: is anything
- AND the from address: is anything
- Email management — this option makes to possible to have a single incident for identical emails while the incident remains open. When properly configured, this means as long as an incident is not resolved, any new emails that are identical to the email that triggered the currently active incident will get appended to the active incident instead of creating a new incident (default). The following setting should be used for Log Insight:
- Email management: Open a new incident for each new trigger email subject.
Summary
Integrating with a monitoring system like PagerDuty is easy thanks to Log Insight’s ability to send email notifications/alerts. While there are a few considerations that need to be taken into account when configuring a monitoring system like PagerDuty, such systems provide the tools necessary to ensure proper monitoring of an environment.
© 2015, Steve Flanders. All rights reserved.
