Log Insight notifies users when it is unable to maintain the defined retention period by sending a system notification. In this post, I would like to walk through what the system notifications mean in a clustered environment. Read on to learn more!
System Notifications
Log Insight notifies users about the health of the instance via system notifications sent over email. System notifications are configured by setting the “Email system notifications to” option during initial configuration of the Log Insight instance or through the Administration > General section. Note that only email addresses specified under the “Email system notifications to” dialog box will receive notifications (e.g. setting the email address for the admin account will NOT insure system notifications are received). Given that system notifications are sent by email today, this means that the SMTP settings of Log Insight need to be set properly as well to ensure emails can be received.
Retention Period
Log Insight utilizes all of the storage space it is allocated for retention of events. By default, Log Insight attempts to maintain 1 month worth of logs. If Log Insight is unable to retain 1 month of logs then a system notification will be sent to notify the configured users. The retention period can be changed from the Administration > General section.
Standalone Retention System Notifications
If Log Insight is unable to retain the configured retention then a system notification is sent that looks similar to the following:
At the current ingestion rate, Log Insight will provide 29 days 15 hours of searchable log data on node node01. If you would like to retain searchable log data for a longer period of time at the current ingestion rate, you will need to add additional storage to the Log Insight virtual appliance on this node. The retention notification threshold can also be changed on the General Admin Configuration page. See the Online Help for instructions on how to increase storage of the appliance.
On this Log Insight node:
Current total storage space: 420.4 GB
Current used storage space: 404.1 GB
Current disk consumption rate: 13.6 GB/day
From the alert, you can see how much disk capacity the Log Insight node that generated the retention has, how much retention is currently used, what the current disk consumption rate is and how log Log Insight can provide retention given the numbers stated. Note that all numbers given are current so if the ingestion rates change so will the retention period. Also note that Log Insight sends one retention system notification per day every day for which the retention period cannot be met. To address retention system notifications, either add more space to Log Insight or change the retention period defined.
Clustered Retention System Notifications
In the case of a Log Insight cluster, each node checks to ensure it can meet the retention period defined. Every time an individual node cannot maintain the retention period it will individually send a retention system notification. Let’s assume I have a 3-node cluster and the retention period is set to the default which is 1 month. One day within the month I receive the following three system notifications:
At the current ingestion rate, Log Insight will provide 29 days 15 hours of searchable log data on node node02. If you would like to retain searchable log data for a longer period of time at the current ingestion rate, you will need to add additional storage to the Log Insight virtual appliance on this node. The retention notification threshold can also be changed on the General Admin Configuration page. See the Online Help for instructions on how to increase storage of the appliance.
On this Log Insight node:
Current total storage space: 420.4 GB
Current used storage space: 404.1 GB
Current disk consumption rate: 13.6 GB/day
At the current ingestion rate, Log Insight will provide 29 days 15 hours of searchable log data on node node03. If you would like to retain searchable log data for a longer period of time at the current ingestion rate, you will need to add additional storage to the Log Insight virtual appliance on this node. The retention notification threshold can also be changed on the General Admin Configuration page. See the Online Help for instructions on how to increase storage of the appliance.
On this Log Insight node:
Current total storage space: 420.4 GB
Current used storage space: 404.1 GB
Current disk consumption rate: 13.4 GB/day
At the current ingestion rate, Log Insight will provide 29 days 15 hours of searchable log data on node node01. If you would like to retain searchable log data for a longer period of time at the current ingestion rate, you will need to add additional storage to the Log Insight virtual appliance on this node. The retention notification threshold can also be changed on the General Admin Configuration page. See the Online Help for instructions on how to increase storage of the appliance.
On this Log Insight node:
Current total storage space: 420.4 GB
Current used storage space: 404.1 GB
Current disk consumption rate: 13.5 GB/day
Why did I receive 3 system notifications? As it turns out, based on the Log Insight architecture, if one node in a cluster cannot maintain the retention period then that means that all nodes cannot maintain the retention period. The reason for this has to do with the internal load balancing of a Log Insight cluster — not to be confused with the Log Insight integrated load balancer. While an external or integrated load balancer provides L4 load balancing, each Log Insight node performs L7 load balancing. The net result is that traffic should be fairly distributed across the cluster meaning the ingestion, and as such retention rate, should be the same across nodes within a cluster. An example will make this clearer.
Let’s say I have 3 clients send traffic to the ILB of my 3-node Log Insight cluster with client 1 sending 3 events per second (EPS), client 2 sending 6 EPS and client 3 sending 9 EPS. The ILB will send client 1 to one node in the cluster (e.g. node1), client 2 to a different node in the cluster (e.g. node2) and client 3 to a different node in the cluster (e.g. node3) — the integrated load balancer is performing L4 load balancing. The net result is that each LI node would receive a different number of EPS (3, 6 and 9 respectively). To address this each node rebalances the EPS across other nodes in the cluster — the net results is that each node will end up consuming 6 EPS ((3 + 6 + 9)) / 3 = 6) — this is L7 load balancing. The means the ingestion rate of every node in the cluster should be very close to the same. This also means if one node cannot maintain 30 days of retention then all nodes cannot maintain 30 days of retention.
Summary
Log Insight features system notifications which should be configured properly to ensure that important information about the status of Log Insight is received. A retention period can be set in Log Insight, which if not met will trigger a system notification. The retention period is measured and reported on a per node basis. Since Log Insight fairly distributes ingestion across a cluster, if one node cannot maintain the retention period then this means all nodes cannot maintain the retention period. This means in a clustered environment you will receive one retention notification per node within a cluster per day for which the retention period cannot be maintained.
© 2015, Steve Flanders. All rights reserved.
