Log Insight Agent: Detecting Directories and Files

The Log Insight agent can be configured to monitor log files within specific directories. It is important to understand how and when the agent detects directories and files in order to ensure that events are monitored and collected properly. In this post, I will discuss the differences between how the agent handles directories versus files.

li-agent

Directories

If a directory to be monitored does not exist when the LI agent is started then the directory is permanently ignored. For example, let’s say your configuration file contains the following section:

Let’s also assume that when the agent is started, the directory specified in the configuration example above does not exist. If you look in the agent logs you will see an entry like:

If the directory is created after the agent has been started then the agent will not pick up the new directory or the log files it contains. In order to get the log files within a directory to be monitored by the agent in cases where the directory is created after the agent is started, the agent must be restarted.

Note: Every directory must have its own filelog section today. The directory option does not support globs for specifying one or more directories or recursing directory structures today.

Files

Files are automatically picked up even if they are not present when the agent starts as the agent subscribes to a directory. If you look in the agent logs you will see entires like:

The above event indicates that the agent is monitoring the directory specified in the messages section of the configuration file. The only deciding factor on what log files the agent collects within a directory is what the include parameter is set to. For example, if you have a configuration like:

and a log rotate program rotates the file bar.log to bar.log.1 then bar.log.1 will not be picked up by the agent as the include parameter specified does not include the rotated file. This may not seem like a problem, but the agent, like all other syslog agents, does not poll for changes real-time so some amount of logs will be rotated and not collected. To address this issue, ensure the include option is inclusive of rotated files like:

Note: Using include=bar.log.? may not be sufficient if a rotated file gets named something other than <file>.1

By default, the include option is set to collect all files within a directory (include=*), which means by default the agent properly monitors rotated files. The agent is aware of events it has previously collected so it will not re-collect events in rotated files. Also note that the agent does not support collecting compressed or binary files today. By default, the agent will ignore compressed files, but will not ignore binary files. If you are aware of binary files in a directory being monitored by the agent then you should ensure your include option does not include the binary files and/or you should specify an exclude option so binary files are not collected. For example, with Microsoft SQL events you need a configuration like:

Summary

The Log Insight agent:

  • Monitors existing directories when the agent starts
  • Requires a service restart to monitor new directories created after the agent was started
  • Requires a separate filelog option for every directory that is to be monitored
  • Monitors all files within an existing directory that match the include option specified (defaults to all)
  • May not monitor rotated files if the include parameter is not set properly (will by default)
  • Does not and should not monitor compress files
  • May, but should be configured not to, monitor binary files

If you ever want to know what directories Log Insight is monitoring or whether you have a configuration issue, check the latest log file for events like those described above.

© 2015, Steve Flanders. All rights reserved.

7 thoughts on “Log Insight Agent: Detecting Directories and Files

  1. Dan Y says:

    Hi Steve, so does Log Insight monitor when file contents change and report what that change was? Or is that where something like Configuration Manager comes into play instead?

    • The LI agent is meant to monitor for changes from the last event it has seen until the most recent event currently available. This means it is not meant for configuration management where text that has already been collected may later be changed. In short, for configuration changes use Configuration Manager and for logs use LI agent.

  2. Mohan Guttikonda says:

    Hi Steve, How should we handle if the rotated file is moved to different folder in the same directory? As per above note, agent does not look for recursive directories in a directory, please correct if my understand is wrong.

    In our environment, the log file is rotated and moved to a new folder(Archive) in the same logs folder. So, in this case how can we ensure that the logs are not lost during file rotation.
    Current Log file: E:\Logs\Test.log
    Archived Files: E:\Logs\Archive\0.Test.log, E:\Logs\Archive\1.Test.log ….etc.,

Leave a Reply