Log Insight Agent: Detecting Directories and Files

The Log Insight agent can be configured to monitor log files within specific directories. It is important to understand how and when the agent detects directories and files in order to ensure that events are monitored and collected properly. In this post, I will discuss the differences between how the agent handles directories versus files.
li-agent

Directories

If a directory to be monitored does not exist when the LI agent is started then the directory is permanently ignored. For example, let’s say your configuration file contains the following section:

[filelog|apache]
directory=/var/log/apache2

Let’s also assume that when the agent is started, the directory specified in the configuration example above does not exist. If you look in the agent logs you will see an entry like:

2014-12-21 09:05:18.801439 0x00002b86345bcd60  FLogCollector:217  | Invalid path specification was obtained. Channel <filelog|apache> will stay dormant until properly configured.

If the directory is created after the agent has been started then the agent will not pick up the new directory or the log files it contains. In order to get the log files within a directory to be monitored by the agent in cases where the directory is created after the agent is started, the agent must be restarted.

Note: Every directory must have its own filelog section today. The directory option does not support globs for specifying one or more directories or recursing directory structures today.

Files

Files are automatically picked up even if they are not present when the agent starts as the agent subscribes to a directory. If you look in the agent logs you will see entires like:

2014-12-21 09:05:19.471852 0x00002b86345bcd60  FLogCollector:246  | Subscribed to channel .

The above event indicates that the agent is monitoring the directory specified in the messages section of the configuration file. The only deciding factor on what log files the agent collects within a directory is what the include parameter is set to. For example, if you have a configuration like:

[filelog|test]
directory=/var/log/foo
include=bar.log

and a log rotate program rotates the file bar.log to bar.log.1 then bar.log.1 will not be picked up by the agent as the include parameter specified does not include the rotated file. This may not seem like a problem, but the agent, like all other syslog agents, does not poll for changes real-time so some amount of logs will be rotated and not collected. To address this issue, ensure the include option is inclusive of rotated files like:

[filelog|test]
directory=/var/log/foo
include=bar.log*

Note: Using include=bar.log.? may not be sufficient if a rotated file gets named something other than <file>.1

By default, the include option is set to collect all files within a directory (include=*), which means by default the agent properly monitors rotated files. The agent is aware of events it has previously collected so it will not re-collect events in rotated files. Also note that the agent does not support collecting compressed or binary files today. By default, the agent will ignore compressed files, but will not ignore binary files. If you are aware of binary files in a directory being monitored by the agent then you should ensure your include option does not include the binary files and/or you should specify an exclude option so binary files are not collected. For example, with Microsoft SQL events you need a configuration like:

[filelog|SQL2012]
directory=C:\Program Files\Microsoft SQL Server\MYSQL11.MSSQLSERVER\MSSQL\Log\
exclude=*.trc
charset=UTF-16LE
event_marker=^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}\.\d{2}

Summary

The Log Insight agent:

  • Monitors existing directories when the agent starts
  • Requires a service restart to monitor new directories created after the agent was started
  • Requires a separate filelog option for every directory that is to be monitored
  • Monitors all files within an existing directory that match the include option specified (defaults to all)
  • May not monitor rotated files if the include parameter is not set properly (will by default)
  • Does not and should not monitor compress files
  • May, but should be configured not to, monitor binary files

If you ever want to know what directories Log Insight is monitoring or whether you have a configuration issue, check the latest log file for events like those described above.

© 2015, Steve Flanders. All rights reserved.

7 comments on “Log Insight Agent: Detecting Directories and Files

Dan Y says:

Hi Steve, so does Log Insight monitor when file contents change and report what that change was? Or is that where something like Configuration Manager comes into play instead?

The LI agent is meant to monitor for changes from the last event it has seen until the most recent event currently available. This means it is not meant for configuration management where text that has already been collected may later be changed. In short, for configuration changes use Configuration Manager and for logs use LI agent.

Mohan Guttikonda says:

Hi Steve, How should we handle if the rotated file is moved to different folder in the same directory? As per above note, agent does not look for recursive directories in a directory, please correct if my understand is wrong.
In our environment, the log file is rotated and moved to a new folder(Archive) in the same logs folder. So, in this case how can we ensure that the logs are not lost during file rotation.
Current Log file: E:\Logs\Test.log
Archived Files: E:\Logs\Archive\0.Test.log, E:\Logs\Archive\1.Test.log ….etc.,

This would not work today — the solution would be to rotate the first file in the same directory and rotate the 2nd onward to the archive directory (logrotate supports this).

Mohan Guttikonda says:

Hi Steve,
Thanks for the reply.
We are using NLog where the numbering of archived files starts from zero(0.Test.log or Test.log.0) and does not support rotating the first file in the same directory and rotate the 2nd onward to the archive directory.
Also other applications are dependent on this archive folder structure, so it is not ideal for me to modify the archive structure.
If possible can you please clarify the following queries.
1. How does the agent look for log file changes? Is it based on file changed event?
2. Is there any plan to support this kind of log rotation structure in future releases?
Thanks,
Mohan G

Hey Mohan, 1) This is complicated — it is not based solely on file changed event because the agent is not a configuration management tool so it does not keep track of the whole file it keeps track of where it left off and events appended. 2) There is no plan at this time to my knowledge, but please open a feature request on https://loginsight.vmware.com!

Mohan Guttikonda says:

Leave a Reply

Your email address will not be published. Required fields are marked *

Back To Top