Log Insight has supported event forwarding since 3.0. Given the syslog RFC, it was common for Log Insight to add a prefix to log events it forwarded. An option exists in 4.6 to change this behavior. Read on to learn more!
The Problem
Let’s start with the problem statement. When you configure Log Insight to send events over the syslog protocol, Log Insight follows the syslog RFCs. Per the syslog RFC, if the syslog prefix is missing, then it MUST be added. Trying to determine whether the prefix exists or not is really expensive, so to adhere to the syslog RFCs, Log Insight adds a prefix to all events it forwards over the syslog protocol.
NOTE: This is not unique to Log Insight. For example, rsyslog does the exact same thing.
Let’s use an example to make this more clear. Assume you have the following event path: ESXi over syslog > LI forwarder over CFAPI > LI cluster over syslog > third-party destination. Assume the ESXi log event is the following:
2018-04-20T21:40:24.341Z esx03.sflanders.net Hostd: verbose hostd[BC81B70] [Originator@6876 sub=PropertyProvider] RecordOp ASSIGN: guest.disk, 232. Sent notification immediately.
At the third-party destination, the event will look like the following:
2018-04-20T21:40:46.343Z esx03.sflanders.net Hostd – – [Originator@6876 source_type=”syslog”] 2018-04-20T21:40:24.341Z esx03.sflanders.net Hostd: verbose hostd[BC81B70] [Originator@6876 sub=PropertyProvider] RecordOp ASSIGN: guest.disk, 232. Sent notification immediately.
As you can see, a prefix has been added as per the syslog RFC. While this does ensure the event sent is RFC compliant, it does result in the original event, which in this case was already RFC compliant, to be changed, which may cause issues for the third-party system receiving the event.
The Solution
To better support third-party destinations, Log Insight 4.6 offers the ability to forward events in raw format:
Raw format is the same as syslog, except it does not ensure syslog RFC compliance and just forwards the event the way it was received. This is confirmed, but the “i” dialog:
If this option were selected in the LI cluster example above, then the third-party destination would receive the same event that LI did.
Summary
This feature is another example of what you asked, Log Insight listened. With better support for third-party destinations, it is now easier than ever to integrate Log Insight into your environment. Will you be using this new feature?
© 2018 – 2021, Steve Flanders. All rights reserved.
This is the kind of features we use to allow us to KEEP using LogInsight. Our security team is nuts going with another product that they pay per gig to use. Even after showing them Log Insight features and licensing advantages… Fine, you can lead a horse to water but you can’t make them drink. But now I don’t have to tear down LI or duplicate our logging traffic.
Now that I can forward raw – I can do a simple raw forwarder and get on with my day.
Nice! Thanks for sharing.
Hi,
We use this for vCenter and ESXi forwarding and it works great.
When we use it for NSX Controller forwarding it still adds a prefix to the data for some reason (4.6.2)
Have you seen this?
The feature literally relays whatever it received. Do you have an example of a prefix? Do you have a LI instance forwarding to a LI instance?
NSX Controller and vCenter seem to do the same thing.
The following excerpts were received on a Kiwi syslog server from data received from vRLI. There is no difference in the data received between the syslog and the raw type.
SYSLOG format from NSX Controller – note the “1 ” in front of the 2019-04-18 portion of the data that was not on the source:
2019-04-18 14:36:04 User.Info 10.10.10.22 1 2019-04-18T13:37:53Z nsx-controller controller – api_request [niciraTag@39961 controller=
RAW format from NSX Controller – note it still has the “1 ” in front of the 2019-04-18 portion of the data that was not on the source:
2019-04-18 14:32:17 Local7.Debug 10.10.10.21 1 2019-04-18T13:34:07Z nsx-controller controller – api_request [niciraTag@39961 controller=
We’ve got a case open with VMware and they’re investigating.