From time to time, I hear about the desire to configure two separate Log Insight instances to forward to one another without creating a loop. In this post, I will discuss the reasons for this desired state, the best practice to achieve this desired state and some alternatives to consider. Read on to learn more!
Understand the Ask
- Ask: Configure two separate Log Insight instances to forward to one another without creating a loop
- Use-case: Disaster recovery
Best practice: Event Forwarder
- Process: Follow the reference architectures. The references architectures state to always use a Log Insight event forwarder instance in front of your primary Log Insight instance. It also states to have the Log Insight event forwarder instance forward to two separate Log Insight instances to support DR.
- Pros
- More configuration flexibility
- No loops
- Cons
- Additional CapEx and OpEx to run Log Insight
- If Log Insight event forward instance has a cluster-wide issue (should be extremely rare) second Log Insight instance does not help
Alternative: Client-side (Not Recommended)
- Process: Configure all clients to send events to multiple destinations.
- Pros
- Most syslog agents support multiple destinations today
- No single point of failure
- Cons
- The Log Insight agent only supports a single destination
- Even in the case of syslog agents, you often get very little buffer on the clients should a remote destination be unavailable, which may increase the likelihood of dropped events
- Configuration complexity client-side
Alternative: Log Insight Loop (OK if configured properly)
- Process: Configure LI1 to event forward to LI2 with a specific tag (e.g. instance=li1), but with a filter to not forward a specific filter (e.g. instance=li2). Configure LI2 to event forward to LI1 with a specific tag (e.g. instance=li2), but with a filter to not forward a specific filter (e.g. instance=li1).
- Pros
- Requires less CapEx and OpEx to run Log Insight
- Cons
- Error prone configuration
- Requires a static field be added to all events
- If Log Insight has a cluster-wide issue (should be extremely rare) second Log Insight instance does not help
- Configured integrations (e.g. vSphere or vROps) may need to be manually moved over in the case of DR
General Guidance
- If you have more than two fault domains (e.g. data centers) or you want an enterprise-grade Log Insight deployment then I would recommend going with the best practice listed above
- If you have two or less fault domains and do not need an enterprise-grade Log Insight deployment then go with the Log Insight Loop alternative
© 2017, Steve Flanders. All rights reserved.
