If you are upgrading or have upgraded to vSphere 6.0, you should be aware of a couple syslog gotchas. These will be especially important if you are running a central logging system like vRealize Log Insight. Read on to learn more.
vCenter Server / VCSA
If you are upgrading or have upgraded from vCenter Server / VCSA 5.x to 6.0 and had been sending logs to a remote syslog destination like Log Insight then you should know that the sending of logs will not work after the upgrade! Luckily, this appears to be a one-time, post-upgrade issue with an easy fix that requires only restarting the syslog service. VMware has released a KB for this issue with the steps necessary to fix it from the vSphere client as well as the CLI.
Credit: my colleague Alan Castonguay uncovered this issue and the content described below.
Once you upgrade to ESXi 6.0, you may notice that your forwarded hostd logs no longer follow the syslog RFC.
Important: This issue ONLY impacts hostd logs and not other logs like vpxd.
For example, when I sent them to my Log Insight instance they appeared like:
NoneZ esx03.matrix Hostd: 2015-03-17T01:14:32.870Z info hostd[59C40B70] [Originator@6876 sub=Vimsvc.TaskManager opID=3de02b57-61fe user=vpxuser] Task Created : haTask-ha-host-vim.host.NetworkSystem.commitTransaction-62674
Notice how the event starts with “NoneZ” instead of a timestamp. The good news is that the events do have a proper timestamp on ESXi (i.e. /var/log/hostd.log). This means the issue only impacts forwarded events. Since the forwarded events do not comply with the syslog RFC, this can cause issues on remote logging products such as Log Insight. For example, the appname (Hostd) will not be properly extracted because the events does not comply with the syslog RFC.
Now you might be wondering, how can you confirm this is an ESXi issue and not a remote logging product issue? Here is a little shell command you can run on ESXi to confirm:
[firstname.lastname@example.org:/var/log] tcpdump-uw -c 99999 -B 1500 -i vmk0 -s 1514 -vvv 'dst port 514 and ip host li01.matrix' | grep 'Hostd:'
tcpdump-uw: listening on vmk0, link-type EN10MB (Ethernet), capture size 1514 bytes
Msg: NoneZ esx03.matrix Hostd: [LikewiseGetDomainJoinInfo:355] QueryInformation(): ERROR_FILE_NOT_FOUND (2/0):\0x0a
This issue was just uncovered when doing some post vSphere 6 upgrade testing with Log Insight and I am not aware of a VMware KB at this time, but stay tuned!
© 2015, Steve Flanders. All rights reserved.