A while back, I came across an entry from Michael White’s newsletter about changing VMware ESXi host logging levels:
Changing VMware ESXi host logging level
Someone was talking about doing this, and using this as a guide, but I would like to say you may not want to do this. There may be a reason for the log levels, and if you change them it may be harder to support you if you call VMware for help. And your syslog of choice should be able to handle the volume and I know – in fact better than most – that ESXi logs are noisy, but you can deal with that with good searching.
I completely agree with Michael and this post will explain why.
Developers configure logging on a device such that is outputs information necessary to troubleshoot a problem. In terms of logging verbosity, the most common levels are:
In my opinion, every device should be set to the information level. The reasons for this are straight-forward:
- If you get an error/warning message, it is often too late to prevent the issue
- If you get an error/warning message, you may not be able to get the root cause analysis without the information logs
- Some important logs are logged to the wrong facility (e.g. I have seen error message logged as info and info logs marked as error)
- Error/Warning messages may happen so infrequently you may not know that remote logging is working properly
So why do people choose the error or warning verbosity level? Some common reasons are:
- Too much information
- Too much storage space required to keep the messages
- Too expensive to query the information (e.g. products that charge per GB of logs)
- Too difficult to find the important logs (usually due to lack of an enterprise logging solution)
- Logs are used as a monitoring tool
- Only care about error and warning events (though also care about root cause analysis)
- Unless an error/warning is seen, logs are not/rarely analyzed
While I understand the reasons, I have lived in the operations world and know that being able to troubleshoot issues quickly and completely is critical. Storage is cheap and powerful analytics products such as Log Insight encourage, via licensing, more storage collection and quick log analysis. If you look specifically at ESXi, it defaults to information logging and such information is critical in support bundles that VMware support uses to troubleshoot problems.
So the next time you hear someone ask how to reduce the verbosity of logs ask them why they want to do it. Challenge them to defend their reasons and encourage them to look into other solutions if costs or other means are getting in the way. The amount of information in an environment is growing every day and reducing information by dropping it is not the solution.
UPDATE: VMware has several KB on log verbosity and I would like to call out http://kb.vmware.com/kb/1004795. It states that you should not increase the verbosity level — unless requested by VMware support — as it can cause issues (i.e. increase resource utilization, full disk, etc). Unfortunately, VMware to my knowledge does not have a KB stating you should not reduce the verbosity level — likely because this will not cause any problems in your environment per se. The issue with reducing verbosity is that VMware support may not be able to complete troubleshooting of an issue you reported. As such, they may require you to adjust the verbosity level and attempt to reproduce the issue. When you have a critical support case open with VMware, the last thing you want to do is make configuration changes, attempt to reproduce a problem, upload new support bundles to the case, and wait for an update. Again, leave the verbosity level alone.
© 2014, Steve Flanders. All rights reserved.