As you probably know, there is a vRealize Operations Manager (vR Ops) content pack for Log Insight. In fact, one exists for vR Ops 5.x and a separate one exists for vR Ops 6.x. In this post, I would like to talk about the 6.x version of the content pack and also point out an important piece of information regarding configuration starting with vR Ops 6.0.1.
Why two Content Packs?
If you have used both vR Ops 5.x and vR Ops 6.x then it would be very clear the versions are very different. As a result, the log messages are very different. The easiest way to address this is to have two different content packs instead of trying to combine into one. If you are running both versions you can install both content packs into Log Insight.
What are the Requirements?
For the 6.x version of the content pack, the Log Insight agent must be installed and configured a specific way on each vR Ops node. If you checkout the Configuration Guide under the Resources tab of the Solution Exchange listing for the content pack, you will see the exact agent configuration required. It is important to note that the configuration requires agent tags with information unique to each vR Ops node/cluster. It is also important to note that the Log Insight agent must send events over the cfapi (default).
Changes in vR Ops 6.0.1 and newer!
As of vR Ops 6.0.1, the Log Insight agent comes installed on the vR Ops virtual appliance by default. This is nice because it eliminates the question of whether or not the agent is supported within the appliance — the answer is yes! It also eliminates the step of finding, copying and installing the agent. A real nice integration between vR Ops and the Log Insight agent is when syslog export is configured through the vR Ops UI, the configuration is set within the Log Insight agent. This means the syslog export option uses the Log Insight agent to forward events over the syslog protocol to the remote destination of your choice.
Now, this does bring up a subtle, but very important question: if you want to use the vR Ops 6.x content pack for Log Insight, can/should you configure syslog export through the vR Ops UI? The answer is: NO. As stated previously, the 6.x content pack requires a very specific agent configuration to work properly and also requires the cfapi. If syslog export is configured then the specific agent configuration required is not set and the syslog protocol is used instead of the cfapi. It is also very important to note that configurations can and will be overridden! For example, if you configure the agent to work with the content pack client-side (eg. liagent.ini) and later someone configures syslog export from the UI, the client-side agent configuration will be overridden with the UI settings breaking the content pack integration. Of course this works both ways in that a client-side configuration after syslog export configuration would result in a broken syslog export configuration.
So, what is the best practice? Well, to prevent filelog configuration from being overridden and to provide centralized agent configuration, it is recommended to configure the agent (for content pack integration) via server-side configuration (e.g. /admin/agents page) instead of client-side (e.g. liagent.ini) — note that non-vROps systems will ignore the agent configuration as it is not applicable. This will result in the syslog export configuration getting merged with content pack configuration so both can work properly. However, some settings such as the [server] section can only be set in the liagent.ini file (i.e. client-side). This means an important caveat to be aware of is that if syslog export is configured after the content pack configuration is applied — specifically the cfapi protocol setting required — then the agent protocol will be changed to syslog today. In case you are wondering, when configuring syslog export from the UI setting the port to 9000 or 9543 does not set the protocol to cfapi in the liagent.ini — I have a bug open for this and hope to have it fixed soon.
Summary
Two Log Insight content packs exist for vR Ops one for 5.x and one for 6.x. The 6.x version of the content pack requires the Log Insight agent with a specific configuration and the use of the cfapi protocol. Starting with vR Ops 6.0.1, the Log Insight agent is installed by default. In addition, syslog export configuration from the vR Ops UI results in configuration of the Log Insight agent. When configuring the Log Insight agent for the vR Ops content pack it is recommended to use Log Insight server-side configuration (e.g. /admin/agents). In addition, use of the syslog export feature from the UI should be avoided as it will override agent configuration required for the vR Ops content pack to work properly — even if the port is set to 9000 or 9543.
© 2015 – 2021, Steve Flanders. All rights reserved.
Steve – Is there a way to force on the vrops nodes agent configuration to only send “error” events/logs to log insight? Say using whitelist parameter (but not sure what field to use to define the expression e.g. level == “ERROR”).
We’ve been getting too much logs (within a certain period of time) from the vrops nodes because of the content pack agent group.
Hey Lyndon — I would advise trying to determine what is generating the most logs. The easiest way to do this is to add a filter on IA for the hostname(s) of your vROps nodes and then group by the filepath field. You can also switch to the event types tabs to determine if it is file issue or a particular set of events causing the issue. Is it possible you are running an older version of the content pack and/or agent group? There was an issue where garbage collection logs were be collected, but this has since been resolved.
Hi Steve,
From what I observed, it’s not a particular set of events coming from the vROP nodes. Different types of logs come in including TRACE and INFO coming from different file paths or file logs.
We are running vROP 6.4 (is this the version you’re referring to?) but would be upgrading to 6.6.1 as we are following VMware Validated Design 4.1.
The last resort I can think of is to disable these specific filelogs from the agent group configuration. Still wondering though if there is still a way to whitelist from the agent configuration part (can’t find any further resource on this).
If it is logging level then you are talking about verbosity. I assume you are using the default logging settings, which means all the logs are important for daily operation. You could reduce the verbosity level, but that will reduce your visibility as well as your ability to perform troubleshooting and root cause analysis (see https://sflanders.net/2014/04/08/changing-vmware-esxi-host-logging-level/). If you really want to do this then you could update the agent configuration, but this would require adding parsers. Instead, I would recommend changing the verbosity level in vROps: https://docs.vmware.com/en/vRealize-Operations-Manager/6.6/com.vmware.vcom.core.doc/GUID-D731C0C2-C0A1-4D69-A685-DCCDC5F11408.html. I hope this helps.