Now that I have talked about the new forwarding feature in Log Insight 2.5, I would like to discuss why you should consider using it if you are not already.
- Forwarders support sending events over syslog (TCP only) and over Log Insight’s ingestion API: this means the forwarder will work with any remote syslog destination. A classic example of the value of supporting syslog is for an existing SIEM product used to analyze security events.
- Forwarders using the ingestion API send compressed events: In testing I have done, this can lead to up to 16 times less traffic being sent over the wire, with 5 times less traffic being the average. Even though log network bandwidth is low, this is huge! This is a compelling reason to deploy forwarders in every datacenter.
Note: Free syslog agents like Rsyslog and Syslog-NG do not offer compression.
- Forwarders using the ingestion API support sending encrypted events: because security matters.
- Forwarders minimize the attack vector of the primary Log Insight instance / remote syslog destination: because security matters. In short, if the only devices allowed to send events to the primary Log Insight / remote syslog destination are the forwarders then security becomes easier. This is a compelling reason to deploy forwarders in every datacenter.
- Forwarders using the ingestion API can add metadata to events – sometimes you may wish to add additional information to an event to make it easier to query for or correlate over. While many syslog agents provide a solution to this problem (see this post for more information), they do so by manipulating the actual event instead of providing supplemental or metadata information. The Log Insight forwarder offers a tags option and supports JSON key/value pairs for specifying tags. This makes it possible to add information such as which datacenter the event came from.
- Forwarders using the ingestion API can respond to throttle requests (i.e. back pressure) and can ensure no message loss during intermittent connection issues – back pressure is handled by sending a 503 HTTP status code while connection issues can be detected and prevented against through REST calls. The Log Insight forwarder provides a configurable disk-backed queue used to save events until the server has acknowledged ingestion. This disk-backed queue prevents drops of forwarded events during intermittent connection issues where syslog events already sent, but not acknowledged by the server, can be lost after the TCP ACK window expires. Free versions of syslog agents like Rsyslog and Syslog-NG do not provide back pressure functionality nor do they provide a disk-based queue (some do provide a configurable in-memory queue). In addition, these agents rely only on the retry abilities of the TCP protocol (e.g. TCP ACK window) if the TCP protocol is being used to send the traffic–none of this is available for UDP.
- Forwarders can serve as a backup: events that are forwarded are also stored on the forwarder. In addition, forwarders are complete Log Insight instances and offer the same UI as regular Log Insight instances.This means if the central Log Insight instance / remote syslog destination becomes temporarily or permanently unavailable the events still exist on the forwarder, users can be given direct access to forwarders, alerts can be configured on forwarders, etc. The forwarder local capacity can be configured to store events for as long as desired.
- Forwarders can be configured to integrate with different vSphere and/or vR Ops instances: forwarders are complete Log Insight instances and offer the same options as regular Log Insight instances. This means that vSphere integration can be configured on forwarders keeping traffic local to vSphere / forwarder. The same applies to vR Ops and makes it possible to support environments running more than on vR Ops instances.
- Forwarders make it easier to support disaster recovery: again, the forwarder is a Log Insight instance; it contains the events it forwarded and supports querying. In addition, forwarders can send events to up to 10 different destinations. This make it easy to forward events to two different remote destinations (e.g. Log Insight instances) to handle complete disaster recovery. This is important as many devices still only support sending events to a single destination.
- Forwarders support filtering of events: since forwarders support sending filters events it is possible to send security events to an existing SIEM and all other logs to a central Log Insight instance. In addition, filtering makes it possible to ensure “noise” is not ingested by the central Log Insight instance.
- Forwarders make it possible to support variable retention periods: again, since forwarded events can be filtered, it is possible to send security events to one Log Insight instance where events are kept for 90 days on the remote Log Insight instance and all other events to a separate Log Insight instance where events are kept for 30 days.
- Forwarders can be used as an alternative to archiving: forwarders store events locally, which is similar to having an archive. In addition, forwarders can send some/all events to a remote destination that acts as an archive. The benefits to this approach is that the “archive” is online and searchable immediately if needed. The downside to this approach is that it requires additional resources or more disk space than just archiving. In order to properly handle the archiving use-case this means the forwarder or the secondary instance has local capacity greater than the primary instance.
Summary
As you can see, the Log Insight forwarder feature, especially in conjunction with the Log Insight ingestion API, provides several benefits. In my opinion, if you are already using the Log Insight server, then you should also be leveraging the Log Insight forwarder in front of it.
© 2015, Steve Flanders. All rights reserved.
If you forward from A to B, can you forward from B to C? And if so, is it transitive (everything from A gets forwarded to C) or is it only anything that B received locally that is forwarded.
Lastly, how does licensing work in a LI forwarder setup?
Hey Chris — thanks for the comment and sorry for the delay! If I am understanding your first question correctly, the answer is yes. Could you clarify? As for licensing, my understanding is that if the forwarder is only used for forwarding then you do not need to license the forwarder separately.
Say you have two datacenters that back eachother up (both active active) and you have remote sites. I was thinking of having the remote LI forward to a LI cluster in each site (for DR).
Since I have hosts in the two datacenters as well, if I have them report to their local LI cluster, i would still need to get those local hosts information to the opposite cluster (either via LI or multiple syslog destinations). I think my question in this instance should have been, if i have forward LI info from A to B, can I forward (local stuff) to C and filter out the stuff from A.
It’s a bit overkill when I drew it on my whiteboard, but if someone really wants a central view of everything even in a DR, it’s really redundant.
Got it. The recommended approach is to deploy forwarders in the two datacenters as well. This way the primary LI clusters do not have to back each other up and you would get try DR. If you do want to back up two LI clusters to each other then this can be achieved as forwarders support tagging — so would could tag which instance it came from — and filtered forwarding — so you could use the tag to include/exclude what is forwarded. I hope this helps!
Will forwarding using the Syslog protocol change the source host to the Log Insight IP/Hostname?
Hey Rob — Thanks for the comment! The “source” field is not defined in the syslog RFC, it is a Log Insight added construct. If you forward from LI to another LI over syslog then yes the source will reflect the LI forwarder. If you forward from LI to another syslog destination then it depends on what features the remote syslog destination has. I hope this helps!