Log Insight: Syslog Event Forwarder Prefix

The event forwarder feature of Log Insight is powerful for a variety of reasons (several covered in this post). When it comes to event forwarding over the syslog protocol (i.e. to a third-party destination), some people have commented on the prefix Log Insight adds to forwarder events. It turns out this is true for event forwarding as well as for the Log Insight agent. In this post, I would like to discuss the prefix and why it exists. Read on to learn more!

li-forwarder

Event Forwarding

Not familiar with the event forwarding feature of Log Insight? Learn more about it in this post.

The Prefix

Let me start by showing some examples of log events before and after being forwarded over the syslog protocol (not ingestion API) via Log Insight — either event forwarder or agent.

Before

After

The Syslog RFC

Now, you may be wondering why Log Insight adds a prefix. Let me quote syslog RFC 3164:

As you can see, the syslog RFC mandates a particular format when relaying over the syslog protocol (note that RFC 5424 does not change the behavior of relays defined in RFC 3164). Looking back at the example event that is not in syslog format, you can see it doesn’t adhere to the syslog RFC so it must be modified. Looking back at the syslog formatted events, you can see they do adhere to the syslog RFC. So why are the syslog formatted events modified as well?

Log Formats

Logs are inherently unstructured and their prefix format is only standard when sent across the syslog protocol. Many applications log directly to a filesystem and when doing so they do not adhere to the syslog format — because it is not mandated. The example event that is not in syslog format is a good example. If this event is sent over the syslog protocol then it would need to be modified since it does not adhere to the syslog RFC. If this event is sent over the ingestion API (cfapi) protocol then the message can be sent in its original format as this protocol does not have the same strict guidelines as the syslog RFC.

If you look at the events in syslog format, you can see they do adhere to the syslog RFC. So why are they modified as well?

Performance

The answer is performance. Event forwarding in Log Insight happens very early in the ingestion pipeline — before the events are even stored on the filesystem. If you forward an event in its original format and just add a prefix to adhere to the syslog RFC then the process is very efficient. The alternative solution would be to parse every event in the ingestion pipeline to:

  1. See if it adheres to the syslog RFC
  2. Ensure it adheres to the syslog RFC before it is sent

This would be extremely inefficient and would impact the ingestion and resource utilization of Log Insight. So what do other logging solutions do?

Third-Party Solutions

The answer is the exact same thing as Log Insight. Take for example rsyslog. Below is a before and after.

Before

After

Summary

Log Insight adheres to the syslog RFC. As a result, events forwarded over the syslog protocol are prefixed with a string that is syslog RFC compliant. This behavior as the same as other third-party solutions. If you use the cfapi then the syslog RFC does not apply and events are forwarded in their original format. All of this information is applicable to the event forwarder and Log Insight agent.

© 2016, Steve Flanders. All rights reserved.

2 thoughts on “Log Insight: Syslog Event Forwarder Prefix

  1. Pon Elangovan says:

    Hope your doing well !!

    We are trying to integrate log insight 4.0 with SIEM through event forwarding with syslog forwarder, UDP and events are forwarding to SIEM. But not in proper format . Example not able to find any of esxi server details (source) where those events are generated. Is there any special tags to be added. Please advise

    Samething I was discussed with you TAM session in vmworld 2016. Hope you remember

    Thanks in advance

Leave a Reply