A commonly requested feature in Log Insight is the ability to blacklist incoming events. In this post, I will suggest a workaround to get this functionality. Read on to learn more!
Understanding the Requirements
For those desiring the ability to blacklist events it is important to understand the business requirements. The most fundamental question to understand is why is blacklisting being requested? The most common reasons I have heard to date are:
- The event is noise: Perhaps based on experience or from an official source such as a KB an event may not provide any valuable information. The idea is to remove the noise at ingestion time.
- The event is incorrect: Perhaps an event states it is an error when actually it is just informational. This reason is similar to noise.
- The event is repetitive: Perhaps the event happens frequently, but provides little to no value. To save disk space it may be desirable to drop the event.
- The event contains private information: Perhaps some events contain information that should not be made available to others.
- The event is not needed at all destinations: Perhaps certain events are analyzed in certain places within the environment.
Options Today
Log Insight does not provide native blacklisting capabilities today. With that said, there are a variety of workarounds depending on the use-case:
- The event is noise: If you determine an event is noise then you have a couple of options:
- If you are using the Log Insight agent you can leverage the blacklist option.
- If you are using Log Insight fowarders you can leverage the filtering ability to prevent events from being sent upstream.
- The event is incorrect: If you determine an event is incorrect then you have a couple of options:
- If you are using the Log Insight agent you can leverage the blacklist option.
- If you are using Log Insight fowarders you can leverage the filtering ability to prevent events from being sent upstream.
- The event is repetitive: While there is no way to extract a message once and then drop it, you do have a couple of options to drop all such events:
- If you are using the Log Insight agent you can leverage the blacklist option.
- If you are using Log Insight fowarders you can leverage the filtering ability to prevent events from being sent upstream.
- The event contains private information: In this case you typically want select people to see the events in which case Log Insight RBAC should be used. If you really want to exclude from everyone, the following options exist:
- If you are using the Log Insight agent you can leverage the blacklist option.
- If you are using Log Insight fowarders you can leverage the filtering ability to prevent events from being sent upstream.
- The event is not needed at all destinations: In this case typically you are leveraging Log Insight’s event forwarding capabilities in which case you can leverage the filtering ability to prevent events from being sent upstream.
Want More?
If you are interested in native blacklist capabilities server-side then vote for this option.
© 2015, Steve Flanders. All rights reserved.
Another great sharing Steve, thank!
Just 1 question. Could you elaborate on “If you are using the Log Insight agent you can leverage the blacklist option.” What’s the blacklist option in the 3.0 agent? I’m not able to figure out how to configure blacklist in the 3.0 agent.
Thanks from Singapore Steve!
e1
Hey Iwan — Thanks for the comment! “blacklist” is an option available in each section of the agent configuration. It only works on fields, but with agent parsers in 3.0 this means it pretty much works on everything. Interestingly, I do not see the option under filelog, but it was added in 3.0 so looks like there is a documentation bug. For more information see this link.
I know this is an old topic, and I know the answer will be “learn how to create filters”, but do you know of any forums that teach how to make these Log Insight Forwarder filters? There are a few fluff events in hosts that are in the thousands of events that I’d like to block from going to our retention cluster. One is : Hostd: –> error = (vmodl.MethodFault) null.
I want to start thinning out our logs so I don’t have to increase our LI storage space.
Hey Craig — Honestly, this needs to be fixed on the ESXi side. I would submit a SR though doubt it will be fixed. On the LI side, this is non-trivial because you really need to filter by regex and that is not efficient. The most efficient solution would be to exclude forwarding all events where appname=hostd, but then you will lose hostd messages you really want to keep. In general, ESXi generates about 250MB of logs a day. Unless you are dealing with more than 100 hosts, it is not worth the effort to filter out these events. Instead, you should just plan for this disk space to be consumed. One other potential workaround would be to exclude appname=hostd from being sent from the forwarder and querying from the forwarder directly if you ever need hostd events. I hope this helps!