Log Insight 2.5 introduced an often requested forwarding feature. In this post, I will discuss how forwarding is implemented and how to configure forwarding.
Getting Started
Before attempting to configure event forwarding in Log Insight it is important to understand how it is implemented. In short, any Log Insight instance, whether standalone or clustered, can be configured to forward events. Even when forwarding events the Log Insight instance still ingests and stores events locally and archives if archiving has been configured. In addition, queries can be issued from Log Insight instances configured for event forwarding.
Important: There is no way to configure Log Insight to ONLY forward events (i.e. not store them locally)
Assuming you have an existing Log Insight instance, if you want to configure event forwarding to point to this existing instance (i.e. configure event forwarding in a different data center), this would require standing up a NEW AND COMPLETELY SEPARATE Log Insight instance. In short, event forwarders cannot be tacked onto a Log Insight instance if the purpose of the forwarder is to forward TO the existing Log Insight instance.
Important: Event forwarding starts working only after it has been configured. This means that previous events the Log Insight instance has ingested will NOT be forwarded even after event forwarding has been configured.
How to Configure
Configuration is done under Administration > Event Forwarding. From there, you can create a new destination.
Note: Event forwarding supports up to 10 destinations today.
Upon selecting the option to create a new destination you will be prompted to provide some information including:
- Name: What you would like to name the destination (some user-friendly name or alias)
- Host: The FQDN for the remote destination
- Protocol: How events should be sent to the remote destination
- Ingestion API (default) if the remote destination is another Log Insight instance
- Syslog (TCP) if the remote destination is something other than Log Insight
Note: Syslog forwarding over UDP is not supported today.
- Filters (optional): What events you would like to forward
- By default all events are sent
- Filters only support static fields such as syslog metadata fields or ingestion API tags
- Tags (optional): One or more fields to pass with the event
- Only supported if forwarding over the ingestion API
There are also several advanced options, which include:
- Port: In case you have a non-standard port requirement
- Cache: Disk-based cache in case the remote destination is unavailable (maximum allowed = 2000)
- Note: I would recommend always changing this to the maximum allowed (2000)
- Workers: Number of worker threads per node (in most cases should not be changed)
There are a few important things to note with event forwarding over syslog:
- When using the syslog protocol, the remote destination may need to be configured to properly handle multi-line messages
- When using the syslog protocol, Log Insight will add static fields into the STRUCTURED-DATA part of the forwarded event following RFC 5424 and there is no way to disable this today
Summary
Event forwarding can be configured on existing Log Insight instances running version 2.5 or new instances can be deployed and configured to forward events. Configuration is done through the Administration section of the Log Insight UI and offers a variety of options from ingestion API and syslog to filters and tags. In a future post, I will discuss several reasons why you might consider using forwarding.
© 2015 – 2021, Steve Flanders. All rights reserved.
