Log Insight 4.5: Agent Recursive Directory Support

The Log Insight Importer has always supported recursive directory support, however the Log Insight Agent has not. I am happy to announce that in Log Insight 4.5, the agent now supports this functionality as well. Read on to learn more!

Problems

  1. Dynamic directory naming — the name of the directory from which you collect logs may be different from system to system. Examples include:
    • vRA — directory name is specified during installation
    • Apache/Tomcat/NGINX — directory name is defined by virtual server / domain name
    • Logrotate — to directory based on hostname
  2. Recursive log collection — ability to specific a parent directory and collect all logs within the tree. Examples include:
    • Linux /var/log

Importer Solution

The Log Insight Importer has always supported this use-case as the directory structure of support bundles is dynamic and may change between systems even for the same application. To support this scenario, the importer manifest file support single glob (*) for one-level deep directory collection and double glob (**) for recursive directory collection. In fact, the only difference between an agent group and an importer manifest is the directory option.

The importer does need to support one additional directory globbing use-case though: relative versus absolute directory paths. To overcome this, importer directory options can start with a double glob (**) indicating that the path is relative.

More information about the importer can be found in my previous blog posts.

Agent Solution

In Log Insight 4.5, the agent now supports directory globbing options for just absolute paths, but not the same way as the Importer.

Note: The Log Insight Agent is meant for real-time log collection so relative path directory collection does not make sense.

While the agent now supports a single glob (*) to collecting all files one level deep in terms of directories, it does not support recursively navigating a tree (double glob). This means the agent can support any level of directory (e.g. /var/log/*/*/*/*) however each level for which files are to be collected must be specified in its own filelog section. To make this clearer, let me provide an example. Let’s say I have the following directory structure

  • var
    • log
      • dir1
        • dir11
          • file111
        • file11
      • dir2
        • dir22
          • file222
        • file22

If I wish to collect all the two digit files then I could use the following configuration:

Note the above configuration will NOT collect the three digit files. If I wish to collect both the two digit and three digit files then I would need the following configuration:

© 2017, Steve Flanders. All rights reserved.

Leave a Reply