Log Insight Agent: Unavailable Remote Destination

The Log Insight agent functions similar to a syslog agent though is much more powerful. One question I often get asked is what happens if the remote destination the agent is configured to send events to is down? In this post, I will cover this exact question.
li-agent

Configuration

First up, let me cover how to configure the agent to forward events to a remote destination. This is done through the client-side ligent.ini file. By default, the file contains the following configuration:

[server]
; Log Insight server hostname or ip address
; If omitted the default value is LOGINSIGHT
;hostname=LOGINSIGHT

If LOGINSIGHT resolves to a valid IP address than events will be sent to that destination, otherwise the agent will remain idle until that hostname becomes available or the agent is reconfigured. To change the configuration, edit the configuration and then save the file. As long as you are running an agent version 2.5 or newer you do not need to restart the agent for configuration changes to take effect.

Note: The [server] section can only be defined client-side.

Unavailable Remote Destination

If the agent is unable to send events to the remote destination then the events are stored in the agent disk buffer. If the remote destination is offline long enough that the disk buffer reaches capacity then the agent drops incoming new events until room exists in the buffer (i.e. when the remote destination comes back online). UPDATE: Stating with version 3.0, a bookmark is used instead of the buffer so events are only lost when logs rotate off the system or logs are rotated and compressed.

Note: The disk buffer can become full for a variety reasons beyond the remote destination being unavailable.

The disk buffer configuration is listed in the client-side agent configuration:

[storage]
;max_disk_buffer - max disk usage limit (data + logs) in MB:
; 100 - 2000 MB, default 200
;max_disk_buffer=200

The disk buffer size defaults to 200MB, but can be increased up to 2GB in order to buffer more events before they get dropped.

Summary

As you can see, the Log Insight agent has built-in capabilities to address the situation when the remote destination becomes unavailable. If you have the disk space, I would recommend configuring the disk buffer to the maximum allowed size in order to minimize the chance of events getting dropped.

© 2015, Steve Flanders. All rights reserved.

Leave a Reply

Your email address will not be published. Required fields are marked *

Back To Top