Log Insight Importer: Examples

In my last post, I covered the new Log Insight Importer. In this post, I would like to show some examples on how to leverage the next Importer. Read on to learn more!

Example 1: Single Directory

Let’s say I have the following directory structure:

$ ls -R test1
test1:
subdir1/  test1.log
test1/subdir1:

Now let’s compare a Log Insight Agent versus Log Insight Importer configuration:

$ sdiff test1-agent.ini test1-importer.ini
[filelog|test1]							[filelog|test1]
directory=/tmp/importer/test1/subdir1 | directory=test1/*
include=*.log							include=*.log

Here are the differences:

• The Agent requires absolute directory paths, the Importer supports absolute and relative paths
• The Agent does not support directory globbing, the Importer does

IMPORTANT: A single glob (*) for a directory paths means EXACTLY ONE subdirectory. This means if test1.log was at the test1 directory level it would NOT be collected. This also means if subdir2 was at the same level as subdir1 then any files in subdir2 would also be collected.

Example 2: Nested Directories

Let’s say I have the following directory structure:

$ ls -R test2
test2:
subdir1/
test2/subdir1:
subdir1.log  subdir2/  subdir3/
test2/subdir1/subdir2:
subdir2.log
test2/subdir1/subdir3:
subdir3.log

Now let’s compare a Log Insight Agent versus Log Insight Importer configuration:

$ sdiff test2-agent.ini test2-importer.ini
[filelog|subdir1]					          | [filelog|test2]
directory=/tmp/importer/test2/subdir1         |	directory=test2/**
include=*.log						          <
							                  <
[filelog|subdir2]					          <
directory=/tmp/importer/test2/subdir1/subdir2 <
include=*.log						          <
							                  <
[filelog|subdir3]					          <
directory=/tmp/importer/test2/subdir1/subdir3 <
include=*.log							        include=*.log

Here are the differences:

• The Agent requires absolute directory paths, the Importer supports absolute and relative paths
• The Agent requires a configuration section per directory, the Importer supports recursive directories in the same configuration section

IMPORTANT: A double glob (*) for a directory paths means ONE OR MORE subdirectories. This means if test2.log was at the test2 directory level it would NOT be collected.
If you only wanted to collect files in subdir1 then you would use a single glob (*) just like in Example 1:

$ cat test2-subdir1-importer.ini
[filelog|test2-subdir1]
directory=test2/*
include=*.log

If you only wanted to collect files in subdir1 and subdir3 then you either use a double glob (*) and either include the files you want or exclude the files you do not want, or you could use multiple configuration section:

$ cat test2-nosubdir2-importer.ini
[filelog|test2-nosubdir2]
directory=test2/**
include=*.log
exclude=subdir2.log
; OR
[filelog|test2-subdir1]
directory=test2/*
include=*.log
[filelog|test2-subdir3]
directory=test2/*/subdir3/*
include=*.log

Example 3: Compressed Nested Directories

Let’s say I have the following directory structure:

$ ls -R test3
test3:
test3.tar.gz

And if I uncompress I have:

$ ls -R test3-nestedcompress/
test3-nestedcompress/:
tar.tar  test3.log  zip.zip

And if I uncompress the nested directories I have:

$ ls -R test3-uncompressed/
test3-uncompressed/:
tar/  test3.log  zip/
test3-uncompressed/tar:
tar.log
test3-uncompressed/zip:
zip.log

Now let’s compare a Log Insight Agent versus Log Insight Importer configuration:

$ sdiff test3-agent.ini test3-importer.ini
; Must manually decompress tar.gz,	      <
; manually uncompress both .tar and .zip, <
; only then can you use below		      <
							              <
[filelog|test3]							    [filelog|test3]
directory=/tmp/importer/test3		      |	directory=test3/**
include=*.log						      <
							              <
[filelog|tar]						      <
directory=/tmp/importer/test3/tar	      <
include=*.log						      <
							              <
[filelog|zip]						      <
directory=/tmp/importer/test3/zip	      <
include=*.log							    include=*.log

Here are the differences:

• The Agent does not support compressed files/directories, the Importer does including nested compressed directories
• The Agent requires absolute directory paths, the Importer supports absolute and relative paths
• The Agent requires a configuration section per directory, the Importer supports recursive directories in the same configuration section

IMPORTANT: The Importer only supports tar, tar.gz, and zip compression formats today.

Example 4: Honor Timestamp

Let’s say I have the following directory structure:

$ ls -R test4
test4:
subdir1/  test4.log
test1/subdir1:

Let’s look at the contents of test4.log:

$ cat test4/subdir1/test4.log
2016-04-05T04:46:59+0000 localhost test: hello from test4!

Let’s assume we want to use the timestamp within the event. Now let’s compare a Log Insight Agent versus Log Insight Importer configuration:

$ sdiff test4-agent.ini test4-importer.ini
[filelog|test4]				      [filelog|test4]
directory=/tmp/importer/test4/* | directory=test4/*
include=*.log				      include=*.log
parser=test4-parser			      parser=test4-parser
[parser|test4-parser]			  [parser|test4-parser]
base_parser=clf				      base_parser=clf
format=%t %i %{appname}i: %i		  format=%t %i %{appname}i: %i

Here are the differences:

• The Agent requires absolute directory paths, the Importer supports absolute and relative paths
• The Agent timestamp parser will only work if the event timestamp and the server time are within 10 minutes, the Importer timestamp parser supports arbitrary time differences when authenticated and using the honor_timestamp flag

IMPORTANT:

• By default the Importer will NOT use the timestamp within the event
• In order to use the timestamp within the event you MUST use the timestamp parser
• For timestamps greater than 10 minutes from the LI server time you MUST pass the honor_timestamp, username, and password parameters to the Importer

Summary

As you can see, the Importer is very powerful and very flexible. Not only does it make it easy to support already generated logs, it also handles support bundle import and offers the ability to honor the timestamp written in the log files. If you are interested in the example code I used above, I have uploaded it all here.

© 2016, Steve Flanders. All rights reserved.