I have compared the ingestion API to syslog and also explained why you should use the Log Insight agent, but I have never compared the ingestion API to the Log Insight agent. In this post, I will do exactly that. Read on to learn more!
What is the Ingestion API?
The ingestion API is a RESTful, JSON API that is proprietary to Log Insight. It allows for events to be ingested into Log Insight in the most efficient way possible and without requiring adherence to the syslog RFCs.
What is the Agent?
The Log Insight Agent is Windows and Linux-based and capable of collecting events from a filesystem and sending them to both ingestion API (called cfapi) or syslog destinations.
Ingestion API or Agent?
The answer is simple: the agent, but what I would like to answer is why. There are three primary reasons:
- Client-side improvements
- The agent is extremely resource efficient — it is unlikely you will be able to write something more efficient over the ingestion API
- Client-side disk-based queue — you would need to build this logic into the code calling the ingestion API
- Network improvements
- Client-side compression — the ingestion API does not provide a documented way to send compressed events over the network
- SSL validation — you would need to build this logic into the code calling the ingestion API
- Server-side capabilities
- LCM functionality (configuration, upgrade, etc)
- Statistics
In short, you could implement most of the functionality the agent natively has built-in, but why would you want to invest and support a custom application when the agent is free and fully supported by VMware?
I have heard a few reasons why a user believed they could not use the agent:
- Using different tags for different events in the same log file — it is true you can only set one set of tags per filelog section, however if you write the tag to the event then you can leverage parsers to “tag” (i.e. create a static field) the event being ingested.
- Run a script and log the output — just ensure the script writes to the filesystem and the agent monitors the written log file.
- The file to be monitored did not exist when the agent process starts — the agent will discovered newly created files even after the agent process has started
There was only one time when the agent hit a snag — when the directory the agent was monitoring did not exist before the agent process started. This snag was addressed in version 4.5.
When to use the Ingestion API
Now, do not get me wrong, the ingestion API is useful by itself and that is why it is an officially supported API. The question becomes when you should consider using the ingestion API. Some reasons that come to mind:
- You want to collect events from non-Windows and non-Linux systems. In this case, the agent cannot be used so the only option is the ingestion API.
- You do not want to write to the filesystem. Some systems are stateless and may not even have a filesystem to write to. The agent requires a writeable filesystem to work so in this case your only option is the ingestion API (or syslog).
- You want to prove you can perform the same or at least a subset of the functionality of the agent using the programming language of your choice.
In all of these cases, you suffer from the same limitations mentioned in the previous section. That is not to say you cannot implement feature parity, but again I am not sure why you would when the agent is constantly being improved.
© 2017, Steve Flanders. All rights reserved.
