The event forwarder feature of Log Insight is powerful for a variety of reasons (several covered in this post). When it comes to event forwarding over the syslog protocol (i.e. to a third-party destination), some people have commented on the prefix Log Insight adds to forwarder events. It turns out this is true for event forwarding as well as for the Log Insight agent. In this post, I would like to discuss the prefix and why it exists. Read on to learn more!
Not familiar with the event forwarding feature of Log Insight? Learn more about it in this post.
Let me start by showing some examples of log events before and after being forwarded over the syslog protocol (not ingestion API) via Log Insight — either event forwarder or agent.
The Syslog RFC
Now, you may be wondering why Log Insight adds a prefix. Let me quote syslog RFC 3164:
When a relay receives a packet, it will check for a valid PRI. If the first character is not a less-than sign, the relay MUST assume that the packet does not contain a valid PRI. If the 3rd, 4th, or 5th character is not a right angle bracket character, the relay again MUST assume that the PRI was not included in the original message. If the relay does find a valid PRI part then it must check for a valid TIMESTAMP in the HEADER part. From these rules, there will be three general cases of received messages. Table 3 gives the general characteristics of these cases and lists the subsequent section of this document that describes the handling of that case. Case Section Valid PRI and TIMESTAMP 4.3.1 Valid PRI but no TIMESTAMP or invalid TIMESTAMP 4.3.2 No PRI or unidentifiable PRI 4.3.3
As you can see, the syslog RFC mandates a particular format when relaying over the syslog protocol (note that RFC 5424 does not change the behavior of relays defined in RFC 3164). Looking back at the example event that is not in syslog format, you can see it doesn’t adhere to the syslog RFC so it must be modified. Looking back at the syslog formatted events, you can see they do adhere to the syslog RFC. So why are the syslog formatted events modified as well?
Logs are inherently unstructured and their prefix format is only standard when sent across the syslog protocol. Many applications log directly to a filesystem and when doing so they do not adhere to the syslog format — because it is not mandated. The example event that is not in syslog format is a good example. If this event is sent over the syslog protocol then it would need to be modified since it does not adhere to the syslog RFC. If this event is sent over the ingestion API (cfapi) protocol then the message can be sent in its original format as this protocol does not have the same strict guidelines as the syslog RFC.
If you look at the events in syslog format, you can see they do adhere to the syslog RFC. So why are they modified as well?
The answer is performance. Event forwarding in Log Insight happens very early in the ingestion pipeline — before the events are even stored on the filesystem. If you forward an event in its original format and just add a prefix to adhere to the syslog RFC then the process is very efficient. The alternative solution would be to parse every event in the ingestion pipeline to:
- See if it adheres to the syslog RFC
- Ensure it adheres to the syslog RFC before it is sent
This would be extremely inefficient and would impact the ingestion and resource utilization of Log Insight. So what do other logging solutions do?
The answer is the exact same thing as Log Insight. Take for example rsyslog. Below is a before and after.
$ tail -n 1 /tmp/originatorlog test event via file append
$ tail -n 1 /tmp/relaylog 2016-08-07T08:21:05.081711-08:00 localhost: test event via file append
Log Insight adheres to the syslog RFC. As a result, events forwarded over the syslog protocol are prefixed with a string that is syslog RFC compliant. This behavior as the same as other third-party solutions. If you use the cfapi then the syslog RFC does not apply and events are forwarded in their original format. All of this information is applicable to the event forwarder and Log Insight agent.
© 2016, Steve Flanders. All rights reserved.