Logrotate Limitations

I having been working on a syslog architecture and one key component to the architect was leveraging log rotate for all log files. One section of my log rotate file looked like the following:

/path/to/logs/*
{
daily
dateext
missingok
copytruncate
compress
compresscmd /bin/bzip2
compressext .bz2
sharedscripts
lastaction
/usr/sbin/invoke-rc.d syslog-ng reload >/dev/null
endscript
}

The problem was, I noticed that after the cron for logrotate ran the system started to become slow. Looking at top I noticed several things: the load average continued going up, the logrotate process continued to run with the process consuming around 50% of memory, and the syslog process never restarted.
What was causing the problem?

From the logrotate man page:

Please use wildcards with caution. If you specify *, logrotate will rotate all files, including previously rotated ones. A way around this is to use the olddir directive or a more exact wildcard (such as *.log)
.
olddir directory
Logs are moved into directory for rotation. The  directory  must
be  on  the  same physical device as the log file being rotated,
and is assumed to be relative to the directory holding  the  log
file unless an absolute path name is specified. When this option
is used all old versions of the log end up in  directory.   This
option may be overriden by the noolddir option.

Initially, I looked into the olddir directive option. I quickly learned that the olddir directive had to exist (i.e. was not automatically created). This meant I either needed a centralized olddir directive or had to find a way to dynamically create the olddir directive within each host folder (I separated logs based on host).
Not liking the options the olddir directive left me with I looked into replacing the wildcard with something more specific. My only requirement was the ability to handle any log file. Since my syslog configuration only used standard handles, I changed the configuration to what is shown below which resolved the issue.

/path/to/logs/*.alert
/path/to/logs/*.crit
/path/to/logs/*.debug
/path/to/logs/*.emerg
/path/to/logs/*.err
/path/to/logs/*.info
/path/to/logs/*.notice
/path/to/logs/*.panic
/path/to/logs/*.warning
{
daily
dateext
missingok
copytruncate
compress
compresscmd /bin/bzip2
compressext .bz2
sharedscripts
lastaction
/usr/sbin/invoke-rc.d syslog-ng reload >/dev/null
endscript
}

In summary, the log rotate limitations I ran into included:

• Rotated logs must go to same drive as source
• Olddir directive is not created, it must exist
• Wildcards in the logrotate configuration file results in a process infinite loop

© 2011, Steve Flanders. All rights reserved.