As I am sure you know, Log Insight 2.0 features an ingestion API, which makes it possible to ingest information without use of the syslog protocol. The API uses a JSON string to send events to Log Insight and also supports the ability to pass fields during ingestion time. An example of a JSON message would be:
{"messages":[{"text":"Hello Log Insight"}]}
Depending on your operating system, you have a variety tools to send API events like the above. For example:
Depending on the method you choose and the format in which you pass the information you will get one of the following return codes:
- 200 OK
- 400 Bad Request
- 500 Internal Server Error
- 503 Service Unavailable
Unless you receive 200 OK something is wrong that needs to be corrected. If you get 503 Service Unavailable then the issue is either server-side or network related. The 400 and 500 error codes point to a client-side error. The question becomes, how do you fix client-side errors?
URL
Verify that the URL being used is in the right format:
http://_loginsight_:9000/api/v1/messages/ingest/_agentId_
Where:
- _loginsight_ is the FQDN/hostname/IP of your Log Insight instance
- _agentId_ is a unique identifier for the device sending events to Log Insight
Note:
The _agentId_ does not require the use of the Log Insight Windows agent. The <agentId> is meant to uniquely identify the device sending events to Log Insight like the hostname field does in the syslog protocol.
An example of a valid URL is provided in the documentation:
http://_loginsight_:9000/api/v1/messages/ingest/4C4C4544-0037-5910-805A-C4C04F585831
Note that any agentId can be used so the following is also valid:
http://_loginsight_:9000/api/v1/messages/ingest/1
Header
This part is critical and can be easily overlooked. You need to ensure that the header is set to the following:
Content-Type: application/json
In cURL, a header can be set using the -H flag. For example:
curl -H 'content-type:application/json'
Gotcha:
In the introduction I mentioned the use of the .NET REST post. It appears this client always sends it own header (x-www-form-urlencoded) even if you explicitly set a header (application/json). This results in ingestion API requests failing 400 Bad Request. If you believe you have formatted the event properly for the ingestion API and you are still receiving errors, use packet analysis (e.g. Wireshark on Windows or tcpdump on Linux) to see if the wrong header is being passed.
Event(s)
The event(s) must be in the following JSON format:
{"messages":
[{
"text": optional, message text as a string,
"timestamp": optional, timestamp encoded as number of milliseconds since Unix epoch,
"fields": optional array of
[{
"name": the name of the field,
"content": optional, the content of the field,
"start_position": optional, the start position in the "text",
"length": optional, the length of the string in the "text",
},...]
},...]
}
Note:
If “timestamp” is not present, the server uses arrival time. If “fields”[].”content” is not present, then”startPosition” and “length” must be present and must point to a valid position in the “text” field string.
Gotcha:
Be sure to escape nested quotations! For example, in Windows the cURL command would look like the following:
"c:\Program Files (x86)\Git\bin\curl.exe" -X POST -H "Content-Type: application/json" -d "{\"messages\":[{\"text\":\"Hello Log Insight\"}]}" http://loginsight:9000/api/v1/messages/ingest/F8B0CEE0-F018-4D53-92F6-7A162D0513D2
Summary
The Log Insight ingestion API is simple and powerful way to ingest any unstructured data. The Log Insight Windows agent defaults to using the ingestion API to provide capabilities beyond those provided in the syslog protocol. You can use the ingestion API to remove the dependency on using a syslog agent and the syslog protocol for sending events to Log Insight.
© 2014, Steve Flanders. All rights reserved.
