Skip to content

Log Insight 4.5: Agent Recursive Directory Support

The Log Insight Importer has always supported recursive directory support, however the Log Insight Agent has not. I am happy to announce that in Log Insight 4.5, the agent now supports this functionality as well. Read on to learn more!

Problems

  1. Dynamic directory naming — the name of the directory from which you collect logs may be different from system to system. Examples include:
    • vRA — directory name is specified during installation
    • Apache/Tomcat/NGINX — directory name is defined by virtual server / domain name
    • Logrotate — to directory based on hostname
  2. Recursive log collection — ability to specific a parent directory and collect all logs within the tree. Examples include:
    • Linux /var/log

Importer Solution

The Log Insight Importer has always supported this use-case as the directory structure of support bundles is dynamic and may change between systems even for the same application. To support this scenario, the importer manifest file support single glob (*) for one-level deep directory collection and double glob (**) for recursive directory collection. In fact, the only difference between an agent group and an importer manifest is the directory option.
The importer does need to support one additional directory globbing use-case though: relative versus absolute directory paths. To overcome this, importer directory options can start with a double glob (**) indicating that the path is relative.
More information about the importer can be found in my previous blog posts.

Agent Solution

In Log Insight 4.5, the agent now supports directory globbing options for just absolute paths, but not the same way as the Importer.

Note: The Log Insight Agent is meant for real-time log collection so relative path directory collection does not make sense.

While the agent now supports a single glob (*) to collecting all files one level deep in terms of directories, it does not support recursively navigating a tree (double glob). This means the agent can support any level of directory (e.g. /var/log/*/*/*/*) however each level for which files are to be collected must be specified in its own filelog section. To make this clearer, let me provide an example. Let’s say I have the following directory structure

  • var
    • log
      • dir1
        • dir11
          • file111
        • file11
      • dir2
        • dir22
          • file222
        • file22

If I wish to collect all the two digit files then I could use the following configuration:

Note the above configuration will NOT collect the three digit files. If I wish to collect both the two digit and three digit files then I would need the following configuration:

© 2017, Steve Flanders. All rights reserved.

Published inVMware

6 Comments

  1. cary sweet cary sweet

    when I try [filepath | rwoDigit] the agent complains about INI parser Error at line: 162. Unknown section ‘filepath’., ignoring line.

    • Hey Cary — “filepath” is not a valid section name. I believe you are looking for “filelog”. It might be easier to create your configuration from the /admin/agents page of the Log Insight server. The native configuration builder can assist with configuration errors. I hope this helps!

      • Chris Chris

        great article, i think cary is confused because your examples above say “filepath” not “filelog”

  2. Markus Markus

    What about Windows? Are the \ bothering me?
    I have this directory:
    C:\Program Files\Microsoft SQL Server\130\Setup Bootstrap\Log\20180717_164639\Detail.txt
    so the Folder 20180717_164639 is variable – how do I get it right?
    C:\Program Files\Microsoft SQL Server\130\Setup Bootstrap\Log\*
    C:\Program Files\Microsoft SQL Server\130\Setup Bootstrap\Log\*\*
    C:\Program Files\Microsoft SQL Server\130\Setup Bootstrap\Log\\*
    C:\Program Files\Microsoft SQL Server\130\Setup Bootstrap\Log\\*\\*
    [filelog|MSSQL16InstallDetail]
    directory=C:\Program Files\Microsoft SQL Server\130\Setup Bootstrap\Log\*
    tags={“ms_product”:”mssql”}
    charset=UTF-16LE
    exclude=*.cab
    include=Detail_Local.txt;Detail.txt
    parser=auto
    characterset of the file is ANSI – is that the problem?

    • Hey Markus — the configuration looks good so it could be the charset. Best bet would be to look at the logs, perhaps even turn up the logging level on the agent. It should clearly indicate what the issue is.

Leave a Reply

Your email address will not be published. Required fields are marked *