The Log Insight Importer has always supported recursive directory support, however the Log Insight Agent has not. I am happy to announce that in Log Insight 4.5, the agent now supports this functionality as well. Read on to learn more!
Problems
- Dynamic directory naming — the name of the directory from which you collect logs may be different from system to system. Examples include:
- vRA — directory name is specified during installation
- Apache/Tomcat/NGINX — directory name is defined by virtual server / domain name
- Logrotate — to directory based on hostname
- Recursive log collection — ability to specific a parent directory and collect all logs within the tree. Examples include:
- Linux /var/log
Importer Solution
The Log Insight Importer has always supported this use-case as the directory structure of support bundles is dynamic and may change between systems even for the same application. To support this scenario, the importer manifest file support single glob (*) for one-level deep directory collection and double glob (**) for recursive directory collection. In fact, the only difference between an agent group and an importer manifest is the directory option.
The importer does need to support one additional directory globbing use-case though: relative versus absolute directory paths. To overcome this, importer directory options can start with a double glob (**) indicating that the path is relative.
More information about the importer can be found in my previous blog posts.
Agent Solution
In Log Insight 4.5, the agent now supports directory globbing options for just absolute paths, but not the same way as the Importer.
Note: The Log Insight Agent is meant for real-time log collection so relative path directory collection does not make sense.
While the agent now supports a single glob (*) to collecting all files one level deep in terms of directories, it does not support recursively navigating a tree (double glob). This means the agent can support any level of directory (e.g. /var/log/*/*/*/*) however each level for which files are to be collected must be specified in its own filelog section. To make this clearer, let me provide an example. Let’s say I have the following directory structure
- var
- log
- dir1
- dir11
- file111
- file11
- dir11
- dir2
- dir22
- file222
- file22
- dir22
- dir1
- log
If I wish to collect all the two digit files then I could use the following configuration:
[filelog|twoDigit] directory=/var/log/*
Note the above configuration will NOT collect the three digit files. If I wish to collect both the two digit and three digit files then I would need the following configuration:
[filelog|twoDigit] directory=/var/log/* [filelog|threeDigit] directory=/var/log/*/*
© 2017, Steve Flanders. All rights reserved.
when I try [filepath | rwoDigit] the agent complains about INI parser Error at line: 162. Unknown section ‘filepath’., ignoring line.
Hey Cary — “filepath” is not a valid section name. I believe you are looking for “filelog”. It might be easier to create your configuration from the /admin/agents page of the Log Insight server. The native configuration builder can assist with configuration errors. I hope this helps!
great article, i think cary is confused because your examples above say “filepath” not “filelog”
Hey Chris — thanks, totally missed that — fixed!
What about Windows? Are the \ bothering me?
I have this directory:
C:\Program Files\Microsoft SQL Server\130\Setup Bootstrap\Log\20180717_164639\Detail.txt
so the Folder 20180717_164639 is variable – how do I get it right?
C:\Program Files\Microsoft SQL Server\130\Setup Bootstrap\Log\*
C:\Program Files\Microsoft SQL Server\130\Setup Bootstrap\Log\*\*
C:\Program Files\Microsoft SQL Server\130\Setup Bootstrap\Log\\*
C:\Program Files\Microsoft SQL Server\130\Setup Bootstrap\Log\\*\\*
[filelog|MSSQL16InstallDetail]
directory=C:\Program Files\Microsoft SQL Server\130\Setup Bootstrap\Log\*
tags={“ms_product”:”mssql”}
charset=UTF-16LE
exclude=*.cab
include=Detail_Local.txt;Detail.txt
parser=auto
characterset of the file is ANSI – is that the problem?
Hey Markus — the configuration looks good so it could be the charset. Best bet would be to look at the logs, perhaps even turn up the logging level on the agent. It should clearly indicate what the issue is.
Hi, I oppened a case in VMWARE. they say that is not supported and your post contradict the official documentation of Log insight 4.7.1.
I oppened a case because a i needed to collect .log files from subfolders for example:
My .log file is inside:
C:\user\app1\file\archive1.log
C:\user\app1\file\archive2.log
But i have too many apps paths like:
C:\user\app2\file\archive1.log
C:\user\app3\file\archive1.log
When i configure my directory in this way
C:\user\**
The VMware Documentation says that i going to collect log files from “user” include subfolders. Then, i should see “archive.log” to all “appsX” but it doesn’t occur. The only way i can see the logs is setting agents by app folder like this:
C:\user\app1\file\**
That method is inappropriate when you have too many apps and also too many files inside each folder.
filelog example:
[filelog|com.microsoft.iis.IIS-Apps]
; IMPORTANT: Change the directory as per the environment
directory=c:\inetpub\wwwroot\**
include=*.log
In wwwroot are all apps like app1, app2,..
In Log Insight 4.5 were possible to do this?
Hey Fersan — the Log Insight agent never supported multiple recursions like C:\user\** it only supports single subdirectory. With that said, you can use a glob more than once. So if your example is C:\user\app2\file\archive1.log then you can use C:\user\*\* and the log file should be collected for app2 and app3. It is possible this logic was changed in Log Insight 4.7.1, but that would be a regression. I hope this helps!