Log Insight Importer: Examples

In my last post, I covered the new Log Insight Importer. In this post, I would like to show some examples on how to leverage the next Importer. Read on to learn more!

li-logo

Example 1: Single Directory

Let’s say I have the following directory structure:

Now let’s compare a Log Insight Agent versus Log Insight Importer configuration:

Here are the differences:

  • The Agent requires absolute directory paths, the Importer supports absolute and relative paths
  • The Agent does not support directory globbing, the Importer does

IMPORTANT: A single glob (*) for a directory paths means EXACTLY ONE subdirectory. This means if test1.log was at the test1 directory level it would NOT be collected. This also means if subdir2 was at the same level as subdir1 then any files in subdir2 would also be collected.

Example 2: Nested Directories

Let’s say I have the following directory structure:

Now let’s compare a Log Insight Agent versus Log Insight Importer configuration:

Here are the differences:

  • The Agent requires absolute directory paths, the Importer supports absolute and relative paths
  • The Agent requires a configuration section per directory, the Importer supports recursive directories in the same configuration section

IMPORTANT: A double glob (*) for a directory paths means ONE OR MORE subdirectories. This means if test2.log was at the test2 directory level it would NOT be collected.

If you only wanted to collect files in subdir1 then you would use a single glob (*) just like in Example 1:

If you only wanted to collect files in subdir1 and subdir3 then you either use a double glob (*) and either include the files you want or exclude the files you do not want, or you could use multiple configuration section:

Example 3: Compressed Nested Directories

Let’s say I have the following directory structure:

And if I uncompress I have:

And if I uncompress the nested directories I have:

Now let’s compare a Log Insight Agent versus Log Insight Importer configuration:

Here are the differences:

  • The Agent does not support compressed files/directories, the Importer does including nested compressed directories
  • The Agent requires absolute directory paths, the Importer supports absolute and relative paths
  • The Agent requires a configuration section per directory, the Importer supports recursive directories in the same configuration section

IMPORTANT: The Importer only supports tar, tar.gz, and zip compression formats today.

Example 4: Honor Timestamp

Let’s say I have the following directory structure:

Let’s look at the contents of test4.log:

Let’s assume we want to use the timestamp within the event. Now let’s compare a Log Insight Agent versus Log Insight Importer configuration:

Here are the differences:

  • The Agent requires absolute directory paths, the Importer supports absolute and relative paths
  • The Agent timestamp parser will only work if the event timestamp and the server time are within 10 minutes, the Importer timestamp parser supports arbitrary time differences when authenticated and using the honor_timestamp flag

IMPORTANT:

  • By default the Importer will NOT use the timestamp within the event
  • In order to use the timestamp within the event you MUST use the timestamp parser
  • For timestamps greater than 10 minutes from the LI server time you MUST pass the honor_timestamp, username, and password parameters to the Importer

Summary

As you can see, the Importer is very powerful and very flexible. Not only does it make it easy to support already generated logs, it also handles support bundle import and offers the ability to honor the timestamp written in the log files. If you are interested in the example code I used above, I have uploaded it all here.

© 2016, Steve Flanders. All rights reserved.

4 thoughts on “Log Insight Importer: Examples

  1. Anders says:

    Thanks for the info. Is there any way to import a complete vSphere Support Log Bundle in a simple way? I’m troubleshooting a multi-host issue, and would love to get all theESXi hosts’ logs into a LI instance for analysis.

    • Thanks for the comment! Absolutely, you can use the agent group that comes with the vSphere content pack. You just need to change the directory option from an absolute path to a relative path. I have a blog post lined up for this soon.

  2. Anders says:

    Great examples. This should be covered better in the official docs.

    One thing I don’t understand is how to get the host names into the logs. I’ve tried importing some customer logs into my lab environment, but they all end up with empty host names.

    The live syslog-fed logs look like:
    “date/timestamp hostname appname”
    but the imported logs look like:
    “2016-06-23T02:48:20.647Z: Jun 23 02:48:20 Fdm: 2016-06-23T02:48:20.641Z verbose fdm[FF97DB70]”

    If I click the hostname below each log line, it displays an internal host name from my lab environment.

Leave a Reply