Log Insight: Importing Existing Logs

One question I get over and over again is can you / how do you import existing logs into Log Insight? The common use-cases are:

• Support bundle – someone has a support bundle and wants to analyze the logs
• RCA – an existing set of logs exist and analysis to determine the root cause of an issue is desired
• Analysis – a log analysis tool does / did not exist and analysis of previous logs is desired

So, how to you import existing logs into Log Insight?

Server-side

UPDATE: As of LI 3.3, an importer utility is available and is the recommended way to ingest existing logs.
In terms of the server-side, there is no officially supported way today. If you would like to see this feature supported in the product, please vote for the feature here. The good news is there are plenty of client-side ways of importing existing logs!

Client-side

UPDATE: As of LI 3.3, an importer utility is available and is the recommended way to ingest existing logs. This utility also addresses the timestamp issue that exists with the other methods below.

IMPORTANT: One thing to note about all methods below is how timestamps will be handled in existing log files. In all cases, the timestamp LI will use for queries will be the timestamp of when the existing log file was ingested into LI. This means if your log files are from last week, but you ingest them today then when running a query against the log the time range in LI should be set for today and not last week. For more information on how timestamps are used in LI, please see my time series blog posts here.

Syslog

UPDATE: As of LI 3.3, an importer utility is available and is the recommended way to ingest existing logs.
Syslog agents are extremely powerful and are capable of reading any standard log file. One simple way to ingest existing logs is have your syslog agent monitor a file and then cat the contents of your existing log file into the file that the syslog agent is monitoring. The syslog agent will in turn send all new events it sees to Log Insight.
For example, in Syslog-NG you can monitor a file with a configuration like (be sure to restart the syslog process afterwards):

source s_file { file(“/tmp/mylogs.txt” flags(no-parse)); };
log { source(src); source(s_file); destination(loginsight); };

Then you can add your existing logs to the file Syslog-NG is monitoring:

cat /tmp/myexistinglogs.log >>/tmp/mylogs.txt

Within a minute or so you should start to see the existing log messages in Log Insight.
For more information on how to configure syslog agents to monitor files, see this post.

liagent

UPDATE: As of LI 3.3, an importer utility is available and is the recommended way to ingest existing logs. The agent is meant for real-time log ingestion only. The below approach is not recommended or supported.
In Log Insight 2.0, a native Windows agent was introduced. This agent features the same ability as other syslog agents to monitor files. This means you can follow the same process as the one outlined above about syslog to send existing logs to Log Insight.
For example, you can monitor a file with a configuration like (be sure to restart the liagent service afterwards):

[filelog|mylogs]
directory=C:\Temp
include=mylogs.txt

Then you can add your existing logs to the file liagent is monitoring:

type C:\Temp\myexistinglogs.log > C:\Temp\mylogs.txt

Within a minute or so you should start to see the existing log messages in Log Insight.
For more information on how to configure syslog agents to monitor files, see this post.

Ingestion API

UPDATE: As of LI 3.3, an importer utility is available and is the recommended way to ingest existing logs.
In Log Insight 2.0, an ingestion API was introduced, which can be used to send any unstructured data to Log Insight. This means you can send your existing logs to Log Insight over the ingestion API.
For example, you can send events in an existing log file using the ingestion API with a script like:

#!/bin/bash
while read event; do json+="{\"text\":\"${event}\"},"; done </tmp/myexistinglogs.log
curl -H 'content-type:application/json' -d "{\"messages\":[${json%?}]}" http://loginsight:9000/api/v1/messages/ingest/1

Within a minute or so you should start to see the existing log messages in Log Insight.
For more information on how to configure syslog agents to monitor files, see this post.

© 2014, Steve Flanders. All rights reserved.