In my last post, I talked about the differences between how events are displayed over the syslog protocol, which has a strict format structure, and the ingestion API, which sends events as-is. In this post, I would like to talk about the differences between using the syslog protocol versus the ingestion API when it comes to the Log Insight agent and the Log Insight forwarder.
Agent
The Log Insight agent defaults to the cfapi (i.e. ingestion API) to send events. With the cfapi, events are sent in their original format to Log Insight. Events sent over the cfapi do not have to follow the guidelines of a syslog event and are not modified to comply with the syslog RFC. In addition to the event, the cfapi provides the ability to pass metadata with the event that gets added as static fields to the event in Log Insight. This metadata is similar to the metadata provided by the syslog protocol, but provides even more information including filepath and any configured tags.
It is important to note that when the agent is configured to use the syslog protocol static fields, such as filepath and configured tags, are treated in a different way. Since the syslog protocol does not understand the Log Insight notion of static fields, but adding more information to events can be important, another way to send metadata information is needed. To handle this, the Log Insight agent adds metadata to the STRUCTURED-DATA section of the syslog events per syslog RFC 5424.
As an example, let’s say I have the following log message:
sflanders.net:80 182.118.53.86 - - [16/Jan/2015:23:34:42 +0000] "GET / HTTP/1.1" 200 485 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2251.0 Safari/537.36"
The log message was written to /var/log/apache2/access.log. I would like to collect this log and add a tag where product=apache. To do this, my Log Insight agent configuration would look like the following:
[filelog|apache]
directory=/var/log/apache2
tags={"product":"apache"}
Note that in addition to the tag, the agent also supplies the filepath as a tag as well. If I configure the Log Insight agent to use the syslog protocol then the event will look like the following on the destination logging device:
sflanders.net:80 182.118.53.86 - - [liagent@6876 product="apache" filepath="/var/log/apache2/access.log"] [16/Jan/2015:23:34:42 +0000] "GET / HTTP/1.1" 200 485 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2251.0 Safari/537.36"
Notice the section in bold, which represents STRUCTURED-DATA the agent has added to apply metadata while conforming with the syslog RFC. Now, it is clearly easy to remove tags in the configuration to prevent that information from being passed, but how do you delete metadata that the agent sets automatically like filepath? Well, the agent provides an exclude_fields option. As such, you could change your agent configuration to:
[filelog|apache] directory=/var/log/apache2 exclude_fields=filepath
With the above configuration, events sent over the syslog protocol will not contain any STRUCTURED-DATA information:
sflanders.net:80 182.118.53.86 - - [16/Jan/2015:23:34:42 +0000] "GET / HTTP/1.1" 200 485 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2251.0 Safari/537.36"
Forwarder
The forwarder behaves similar to agents in that it supports both the ingestion API and syslog protocols. In addition, the forwarder leveraging the ingestion API allows for tags to be passed as metadata. The forwarder leveraging the syslog protocol does not support tags today, however when the syslog protocol is used with the forwarder, STRUCTURED-DATA will be forwarded in three specific cases:
- For any tags sent by an agent over the ingestion API
- For vR Ops inventory mapping static fields
- To indicate the source of syslog events — only if vR Ops inventory fields exist
For example, here is an example forwarded event that contains all cases 2 and 3 listed above:
sflanders.net:80 182.118.53.86 - - [filepath="/var/log/apache2/access.log" vmw_host="esx03.matrix" vmw_object_id="vm-44" vmw_vr_ops_id="aefc98d6-1128-432d-b244-c84f778938d3" vmw_vcenter="vcs01.matrix" product="apache" vmw_vcenter_id="FFB49D08-CBE1-417C-A1E0-5265DC1BDE12" vmw_datacenter="bos01"] [16/Jan/2015:23:34:42 +0000] "GET / HTTP/1.1" 200 485 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2251.0 Safari/537.36"
Note there is no way to prevent STRUCTURED-DATA from being forwarded today — like was demonstrated in the agent section above — if any of the three cases above are met.
Summary
As you can see, there are subtle differences between the cfapi and syslog protocols when it comes to the agent and/or forwarders in Log Insight. Careful consideration should be taken to leverage these features properly within your environment. Whenever possible, the Log Insight ingestion API should be used. If the syslog protocol needs to be used, the configuration should be validated for the most optimal experience.
© 2015, Steve Flanders. All rights reserved.
