Skip to content

Log Insight: Importing Existing Logs

One question I get over and over again is can you / how do you import existing logs into Log Insight? The common use-cases are:

  • Support bundle – someone has a support bundle and wants to analyze the logs
  • RCA – an existing set of logs exist and analysis to determine the root cause of an issue is desired
  • Analysis – a log analysis tool does / did not exist and analysis of previous logs is desired

So, how to you import existing logs into Log Insight?
128

Server-side

UPDATE: As of LI 3.3, an importer utility is available and is the recommended way to ingest existing logs.
In terms of the server-side, there is no officially supported way today. If you would like to see this feature supported in the product, please vote for the feature here. The good news is there are plenty of client-side ways of importing existing logs!

Client-side

UPDATE: As of LI 3.3, an importer utility is available and is the recommended way to ingest existing logs. This utility also addresses the timestamp issue that exists with the other methods below.

IMPORTANT: One thing to note about all methods below is how timestamps will be handled in existing log files. In all cases, the timestamp LI will use for queries will be the timestamp of when the existing log file was ingested into LI. This means if your log files are from last week, but you ingest them today then when running a query against the log the time range in LI should be set for today and not last week. For more information on how timestamps are used in LI, please see my time series blog posts here.

Syslog

UPDATE: As of LI 3.3, an importer utility is available and is the recommended way to ingest existing logs.
Syslog agents are extremely powerful and are capable of reading any standard log file. One simple way to ingest existing logs is have your syslog agent monitor a file and then cat the contents of your existing log file into the file that the syslog agent is monitoring. The syslog agent will in turn send all new events it sees to Log Insight.
For example, in Syslog-NG you can monitor a file with a configuration like (be sure to restart the syslog process afterwards):

Then you can add your existing logs to the file Syslog-NG is monitoring:

Within a minute or so you should start to see the existing log messages in Log Insight.
For more information on how to configure syslog agents to monitor files, see this post.

liagent

UPDATE: As of LI 3.3, an importer utility is available and is the recommended way to ingest existing logs. The agent is meant for real-time log ingestion only. The below approach is not recommended or supported.
In Log Insight 2.0, a native Windows agent was introduced. This agent features the same ability as other syslog agents to monitor files. This means you can follow the same process as the one outlined above about syslog to send existing logs to Log Insight.
For example, you can monitor a file with a configuration like (be sure to restart the liagent service afterwards):

Then you can add your existing logs to the file liagent is monitoring:

Within a minute or so you should start to see the existing log messages in Log Insight.
For more information on how to configure syslog agents to monitor files, see this post.

Ingestion API

UPDATE: As of LI 3.3, an importer utility is available and is the recommended way to ingest existing logs.
In Log Insight 2.0, an ingestion API was introduced, which can be used to send any unstructured data to Log Insight. This means you can send your existing logs to Log Insight over the ingestion API.
For example, you can send events in an existing log file using the ingestion API with a script like:

Within a minute or so you should start to see the existing log messages in Log Insight.
For more information on how to configure syslog agents to monitor files, see this post.

© 2014, Steve Flanders. All rights reserved.

Published inVMware

4 Comments

  1. Hi SfLander.
    I am trying to ingest my existing logs to Loginsight using Ingestion API. But I get curl error as follows:
    curl: no URL specified!
    curl: try ‘curl –help’ or ‘curl –manual’ for more information
    Any help would be much appreciated

    • Hey Deepak — Thanks for the comment. Can you tell the complete curl command you are trying?

  2. Steve Steve

    Hey There
    I am trying to send my existing log to LI with following command:
    [filelog|tal-dhcp-int]
    enabled=yes
    directory=C:\Windows\System32\dhcp
    include=DhcpSrvLog-Thu.txt
    But it doesn’t work….
    Do you have any hints for me?
    Kind regards
    Steve

    • Hey Steve — Thanks for the comment. I think this piece of the release notes will help:

      The vRealize Log Insight Windows agent is a 32-bit application and all its requests for opening files from C:\Windows\System32 sub-directories are redirected by WOW64 to C:\Windows\SysWOW64. However, you can configure the vRealize Log Insight Windows agent to collect from C:\Windows\System32 by using the special alias C:\Windows\Sysnative. For example, to collect logs from their default location for the MS DHCP Server, add the following line to the corresponding section of the vRealize Log Insight Windows agent configuration file: =C:\Windows\Sysnative\dhcp.

Leave a Reply

Your email address will not be published. Required fields are marked *