The more I use logrotate the less I like it. If you recall from my previous post on logrotate, I choose to leverage the copytruncate option. While this configuration seemed to work well when I tested it, I have now experienced some significant limitations that are not documented in the man page:
- After rotation, high volume logs files remained the same size and continued to grow
- Pre and post rotated high volume log files contained NUL characters
- Large sized log files lost messages during logrotate operation
So what caused the issues, what was the impact, and how can you rotate logs messages and not experience these issues?
The causes and impact of the issues were as follows:
- This issue was caused by open file handles. The impact of the issue was as log files continued to grow the amount of logs lost during log rotation increased
- This issue was a side effect of the first issue. In order to continue logging at the same spot in the log file, the log file was padded with NUL characters. While a log file with NUL characters could be tailed, other commands such as grep hang.
- Depending on your configuration, this issue is true whether you use the copytruncate option or not. The question becomes what are the requirements for the syslog service?
So how do you fix the issues? The answer is: macros. Syslog-NG has the notion of macros, which allow you to store log files in a directory structure / naming convention based on predefined variables. See below for examples.
- /logs/$HOST/auth.log = /logs/myVM.FQDN/auth.log
- /logs/$YEAR/$MONTH/$DAY/$HOST/auth.log = /logs/YYYY/MM/DD/myVM.FQDN/auth.log
The differences between the two examples are very important. In the first example, log files for a host always go to the same directory and file. This means logrotate must rotate a log file that is actively taking log messages (this configuration in combination with the copytruncate logrotate option was causing my issue). The second example, places log files for a host in a directory structure based on date. This means that when the date changes log files start writing in a new location. This type of configuration is known as dynamic logging. The benefit of this configuration is that logrotate only rotates logs from the previous day (because the new logs are not known to logrotate and as such they are not rotated) and these logs are no longer being written to. Based on the latter configuration, the copytruncate is no longer needed and all issues were resolved!
© 2012, Steve Flanders. All rights reserved.
