The event forwarder feature of Log Insight is powerful for a variety of reasons (several covered in this post). When it comes to event forwarding over the syslog protocol (i.e. to a third-party destination), some people have commented on the prefix Log Insight adds to forwarder events. It turns out this is true for event forwarding as well as for the Log Insight agent. In this post, I would like to discuss the prefix and why it exists. Read on to learn more!
Event Forwarding
Not familiar with the event forwarding feature of Log Insight? Learn more about it in this post.
The Prefix
Let me start by showing some examples of log events before and after being forwarded over the syslog protocol (not ingestion API) via Log Insight — either event forwarder or agent.
Before
2016-09-08T11:49:55Z syslog.sflanders.net app: this event is in syslog RFC 3164 format
2016-09-08T11:49:55Z syslog.sflanders.net app - - - this event is in syslog RFC 5424 format
this event is not in syslog format
After
2016-09-08T11:45:44.384Z 192.168.1.35 app: - - [Originator@6876 source_type="syslog"] this event is in syslog RFC 3164 format
2016-09-08T11:45:44.384Z 192.168.1.35 app - - [Originator@6876 source_type="syslog"] this event is in syslog RFC 5424 format
2016-09-08T11:45:44.384Z 192.168.1.35 - - - [Originator@6876 source_type="syslog"] this event is not in syslog format
The Syslog RFC
Now, you may be wondering why Log Insight adds a prefix. Let me quote syslog RFC 3164:
When a relay receives a packet, it will check for a valid PRI. If
the first character is not a less-than sign, the relay MUST assume
that the packet does not contain a valid PRI. If the 3rd, 4th, or
5th character is not a right angle bracket character, the relay again
MUST assume that the PRI was not included in the original message.
If the relay does find a valid PRI part then it must check for a
valid TIMESTAMP in the HEADER part. From these rules, there will be
three general cases of received messages. Table 3 gives the general
characteristics of these cases and lists the subsequent section of
this document that describes the handling of that case.
Case Section
Valid PRI and TIMESTAMP 4.3.1
Valid PRI but no TIMESTAMP or invalid TIMESTAMP 4.3.2
No PRI or unidentifiable PRI 4.3.3
As you can see, the syslog RFC mandates a particular format when relaying over the syslog protocol (note that RFC 5424 does not change the behavior of relays defined in RFC 3164). Looking back at the example event that is not in syslog format, you can see it doesn’t adhere to the syslog RFC so it must be modified. Looking back at the syslog formatted events, you can see they do adhere to the syslog RFC. So why are the syslog formatted events modified as well?
Log Formats
Logs are inherently unstructured and their prefix format is only standard when sent across the syslog protocol. Many applications log directly to a filesystem and when doing so they do not adhere to the syslog format — because it is not mandated. The example event that is not in syslog format is a good example. If this event is sent over the syslog protocol then it would need to be modified since it does not adhere to the syslog RFC. If this event is sent over the ingestion API (cfapi) protocol then the message can be sent in its original format as this protocol does not have the same strict guidelines as the syslog RFC.
If you look at the events in syslog format, you can see they do adhere to the syslog RFC. So why are they modified as well?
Performance
The answer is performance. Event forwarding in Log Insight happens very early in the ingestion pipeline — before the events are even stored on the filesystem. If you forward an event in its original format and just add a prefix to adhere to the syslog RFC then the process is very efficient. The alternative solution would be to parse every event in the ingestion pipeline to:
- See if it adheres to the syslog RFC
- Ensure it adheres to the syslog RFC before it is sent
This would be extremely inefficient and would impact the ingestion and resource utilization of Log Insight. So what do other logging solutions do?
Third-Party Solutions
The answer is the exact same thing as Log Insight. Take for example rsyslog. Below is a before and after.
Before
$ tail -n 1 /tmp/originatorlog test event via file append
After
$ tail -n 1 /tmp/relaylog 2016-08-07T08:21:05.081711-08:00 localhost: test event via file append
Summary
Log Insight adheres to the syslog RFC. As a result, events forwarded over the syslog protocol are prefixed with a string that is syslog RFC compliant. This behavior as the same as other third-party solutions. If you use the cfapi then the syslog RFC does not apply and events are forwarded in their original format. All of this information is applicable to the event forwarder and Log Insight agent.
© 2016, Steve Flanders. All rights reserved.
Hope your doing well !!
We are trying to integrate log insight 4.0 with SIEM through event forwarding with syslog forwarder, UDP and events are forwarding to SIEM. But not in proper format . Example not able to find any of esxi server details (source) where those events are generated. Is there any special tags to be added. Please advise
Samething I was discussed with you TAM session in vmworld 2016. Hope you remember
Thanks in advance
Hey Pon — Thanks for the comment. If you are referring to specifically the source field then you should vote for this: http://loginsight.vmware.com/a/dtd/Feature-Request-Using-Log-Insight-as-a-Forwarder-and-retaining-source-IP/172908-24427. Everything else should be in the raw message. I hope this helps!