As you know, Log Insight allows for user alerts to be configured for queries through the UI. These alerts can trigger an email and/or vC Ops notification event. In Log Insight 2.0, it is possible to configure additional ways to trigger a user alert through the CLI. I would like to discuss these additional options available.
User alerts in the UI
For information about alerts in the UI, see this post.
User alerts log file
In Log Insight 2.0, an alert.log file was introduced and can be found at:
/storage/var/loginsight/alert.log
Within the file each line represents a user alert. A line consists of a JSON string that looks like:
{“Data”:{“Name”:”High SCSI Latency”,”Data”:{“Count”:1206317,”Fields”:{“deviceId”:”naa.6001405b9e05a9cd8c34d38e8daad8d5″}},”HitCount”:1,”Url”:”http://192.168.1.25/explorer?existingChartQuery=SELECT+MAX%28item0.ibadox3wonygqzlsmvwgc5dfnzrxsslojvuwg4tpom000000%29+FROM+%28%28%28%28%28%28timestamp%3E%3D1408371127191+%26+timestamp%3C%3D1408371427190%29+%26+text%3A%22scsi%22%29+%26+text%3A%22performance%22%29+%26+text%3A%22deteriorated%22%29+%26+%28ibadox3wonygqzlsmvwgc5dfnzrxsslojvuwg4tpom000000%3E1000000+%26+text%3D%7E%22to+%28%3F%3Cibadox3wonygqzlsmvwgc5dfnzrxsslojvuwg4tpom000000%3E-%3F%5Cd%2B%29+microseconds%22%29%29+%26+text%3D%7E%22%28%3F%3Cibadox3wonygqzlsmvsgk5tjmnsusza0%3E%28%28naa%7Ceui%7Csym%7Csns%29%5C.%5B0-9a-z%5D%2B%7Cmpx%5C.vmhba%5B0-9%5D%2B%3AC%5B0-9%5D%2B%3AT%5B0-9%5D%2B%3AL%5B0-9%5D%2B%29%29%22%29+AS+item0+WHERE+APPEAR%28item0%29%3E0+GROUP+BY+item0.timestamp%2F86400000.000000%2C+item0.ibadox3wonygqzlsmvsgk5tjmnsusza0+HAVING+COUNT%28item0%29%3E0+ORDER+BY+item0.ibadox3wonygqzlsmvsgk5tjmnsusza0+DESC%2C+item0.timestamp+DESC+LIMIT+1%2C+2000″,”EditUrl”:”http://192.168.1.25/explorer?existingChartQuery=SELECT+MAX%28item0.ibadox3wonygqzlsmvwgc5dfnzrxsslojvuwg4tpom000000%29+FROM+%28%28%28%28%28%28timestamp%3E%3D1408371127191+%26+timestamp%3C%3D1408371427190%29+%26+text%3A%22scsi%22%29+%26+text%3A%22performance%22%29+%26+text%3A%22deteriorated%22%29+%26+%28ibadox3wonygqzlsmvwgc5dfnzrxsslojvuwg4tpom000000%3E1000000+%26+text%3D%7E%22to+%28%3F%3Cibadox3wonygqzlsmvwgc5dfnzrxsslojvuwg4tpom000000%3E-%3F%5Cd%2B%29+microseconds%22%29%29+%26+text%3D%7E%22%28%3F%3Cibadox3wonygqzlsmvsgk5tjmnsusza0%3E%28%28naa%7Ceui%7Csym%7Csns%29%5C.%5B0-9a-z%5D%2B%7Cmpx%5C.vmhba%5B0-9%5D%2B%3AC%5B0-9%5D%2B%3AT%5B0-9%5D%2B%3AL%5B0-9%5D%2B%29%29%22%29+AS+item0+WHERE+APPEAR%28item0%29%3E0+GROUP+BY+item0.timestamp%2F86400000.000000%2C+item0.ibadox3wonygqzlsmvsgk5tjmnsusza0+HAVING+COUNT%28item0%29%3E0+ORDER+BY+item0.ibadox3wonygqzlsmvsgk5tjmnsusza0+DESC%2C+item0.timestamp+DESC+LIMIT+1%2C+2000&editAlertId=14″},”Timestamp”:”2014-08-18T14:17:08.081Z”}
The format of the JSON string is:
- Data
- Name (string)
- Data
- Count (number)
- Fields (key:value)
- HitCount (number)
- URL (string)
- EditURL (string)
- Timestamp (string)
SNMP
Given the alert.log file, if you compile snmptrapd and copy the executable to the LI VA then you could send SNMP traps in response to user alerts.
As a side note, you can also send snmptraps through vCenter Operations Manager integration by configuring an alert be sent to vCenter Operations Manager and configuring vCenter Operations Manager to send an SNMP trap as desired.
Scripts
The alert.log file allows for a custom script to be run from the Log Insight virtual appliance. This custom script could be used to notify another system or could execute some set of commands in response to an alert (e.g. to attempt to fix the problem discovered). For example,
#!/usr/bin/env bash
for a in $(tail -f /storage/var/loginsight/alert.log); do
if [ $(echo $a | awk '{ split($0,a,"hostname"); split(a[2],b,"\""); print b[3]}') == "ESXi syslog stopped working" ]; then
ssh root@$(echo $b | awk '{ split($0,a,"hostname"); split(a[2],b,"\""); print b[3]}') "esxcli system syslog reload"
fi
done
In the above example, Log Insight will restart the syslog process on ESXi hosts if they stop forwarding syslog events (a known issue in certain versions of ESXi).
Summary
As you can see there are other ways to trigger and respond to alerts in Log Insight. If you are interested in seeing other options natively in the UI, be sure to vote from them on https://loginsight.vmware.com:
If you are looking for other options on how to leverage the alerts.log file, be sure to check out Matt Cowger’s post here.
© 2014, Steve Flanders. All rights reserved.
What would be awesome if you could automate a powercli command (sort of like the power actions fling but alert driven).
Well, technically you could – have a PS script monitor the alerts.log file on the LI VA 🙂 Also looks like there are a couple ways to get PS in Linux though I have not tried them out yet.
I think the biggest thing is the linux Windows part. To monitor the alerts.log would need some sort of CIFS share on LI or go the other way and have LI push the alerts.log to a location that the Windows server running PS can get to it (using ftp or samba mounts or something of that nature).
A couple of options today would be to forward the log via syslog to another destination or via PS (and a SSH key) connect via the CLI.
I was thinking of the latter (ssh key on the windows PS system), but would have to figure out how to keep track of the location in the alert.log file.