Log Insight Alerts: System Notifications

In addition to user alerts, Log Insight also offers system notifications through configured email addresses. I would like to discuss the different types of system notifications available today and how you can troubleshoot potential issues Log Insight notifies you about.

bell

Configuration

System notifications are configured under Administration > General:

li-general

Since system notifications are sent via email, it is important to properly configure and test the SMTP settings as well to ensure notifications will be received:

li-smtp

Types

Log Insight send two different types of system notifications:

  1. System alerts
    • Critical issues that need immediate attention
    • Warnings that may require attention
  2. Informational alerts
    • Normal activity to be aware of

All system notification are defined in the official documentation:

  • Critical alerts
    • Dropped events
    • Corrupt index buckets
    • Out of disk
    • Total disk space changed
    • License is expired
  • Warning alerts
    • Archive space will be full
    • Archive failure
    • Pending archives
    • License is about to expire
  • Information alerts
    • Oldest data will be unsearchable soon
    • Repository retention time

Location

As mentioned earlier, system notifications are primarily sent to the configured email address(es), but may also appear in the UI as a red exclamation mark for situations such as when email cannot be sent properly:

li-ui-alert

Resolving Issues

Sometimes the system notification does not provide enough information to determine if the issue is continuing and what needs to be done to resolve the problem. If this occurs, the logs on the Log Insight virtual appliance can be analyzed. I would like to walk through a couple common examples.

vCenter collection

When Log Insight is unable to collect vCenter events, tasks, and alarms, the following system notification is sent:

This alert is about your Log Insight installation on li.sflanders.net

vCenter collection failed triggered at 2014-09-19T06:54:40.878Z

vCenter task and event collection failed for the following host:

vcs01.matrix

This message was generated by your Log Insight installation, visit the Documentation Center for more information.

To look for the exact error that caused the collection failure and to see if collection is working currently, look in the /storage/var/loginsight/plugins/vsphere/li-vsphere.log file:

OK, looks like it is working now, so let’s see why it failed:

Looks like a temporary DNS issue.

Archiving failure

When Log Insight is unable to archive, the following system notification is sent:

This alert is about your Log Insight installation on li.sflanders.net

Archive Failure triggered at 2014-09-03T15:25:59.686Z

Action is required. Log Insight failed to connect to the archive storage: nfs://nfs.matrix/data/archive. Data might not be able to archive. This could indicate that the archiving destination is unavailable or having performance issues. If this problem is not resolved, the disk on your Log Insight installation will fill up and stop accepting data. Other unexpected issues may also occur. Please check the status of the archive destination.

Troubleshooting tips:

1. Is your NFS server reachable by Log Insight?

2. Is there sufficient space available?

3. Have permissions been properly configured to allow Log Insight to write and access the NFS server?

4. Is there sufficient end-to-end NFS throughput between the Log Insight appliance and the NFS server?

This message was generated by your Log Insight installation, visit the Documentation Center for more information.

To look for the exact error that caused the archive failure and to see if archiving is working currently, look in the /storage/var/loginsight/runtime.log file:

Well, that is not good! Archiving is failing because of a permissions issue, which means some configuration change was made. Let’s address the issue and then check the logs again:

Much better.

Summary

System notifications are critical to understand the health of your Log Insight environment. You should ensure that system notifications are properly configured and tested prior to putting a Log Insight environment into production. To understand what a system notifications means, see the Log Insight documentation. If the system notification does not provide enough information, look at the log files within the Log Insight virtual appliance.

© 2014 – 2015, Steve Flanders. All rights reserved.

2 thoughts on “Log Insight Alerts: System Notifications

Leave a Reply