Log Insight: Mass Export Events

As you may know, the Log Insight UI only allows a maximum of 20K events to be exported at any given time. I often hear the request to export more than 20K events. In this post, I will cover other options that are available today. Read on to learn more!

Today

1. Don’t: In many cases it is usually not necessary to export events
   • Support bundle: Good use-case for mass export
   • Reporting: Potentially good use-case for mass export (screenshot of dashboards page is preferred — yes this can be automated)
   • Perform some query: In most cases Log Insight natively supports this (exceptions: transactions and joins)
2. Webhooks: Log Insight 3.3 introduced webhooks which can be used for alerts. The webhook alert does not limit the number of results returned and could be used to support mass export. For more information on how to accomplish this see my post on webhook shims here.
3. Dedicated Log Insight: If the Log Insight instance only contains events you wish to export then you can use the CLI loginsight-dump-repo.sh script. Note this script dumps Log Insight repository buckets so additional filtering (e.g. by time) may be needed post-export.
4. Shared Log Insight: You can use the same loginsight-dump-repo.sh script as mentioned above however you will need to filter out the events you care about post-export
5. Query API: This is not the use-case of the API. Note the API does limit to 20K events just like the UI.
6. Export Utility: A command-line exporter of log events in VMware vRealize Log Insight. Exceed the 20k UI limit. Write local files. You can download the utility here.

Future

If you would prefer to see native UI support for mass export then you should vote for the feature request here.

© 2016, Steve Flanders. All rights reserved.