Log Insight Use Case: Authentication Logs

I have received a few comments that while I post a lot of technical information about Log Insight, I do not post a lot of examples and use cases. To begin to address this, I would like to demonstrate how to handle authentication logs, more specific Linux SSH logs, in Log Insight as I recently had someone ask me about this particular use case.

assetuploadfile83232718

Requirements

Let’s say you are forwarding Linux logs to Log Insight and you want to analyze authentication over SSH. You are specifically interested in learning about:

  • Who attempted to authenticate
  • If they were successful or not
  • Where they authenticated from

In addition, you want to be notified about:

  • Multiple (>10) failed login attempts for the same user (may be different sources)
  • Multiple (>10) failed login attempts for different users, but the same source

So how do you do this in Log Insight?

Step 1: Query for your events

Start by going to the Interactive Analytics page and constructing a query to return the events you care about. This could be a particular hostname or in the case of authentication events, one or more keywords. Let’s start by querying for failed SSH events through the use of a keyword query:

li-sshd-failed

Step 2: Extract useful fields

From these events we can see some useful bits of information that help address our requirements defined above. Let’s start by extracting the username:

li-sshd-failed-user0

Upon doing this, Log Insight attempts to create a pattern:

li-sshd-failed-user1

While the pattern is valid, it does not include all the events we care about and can be made specific to SSH events:

li-sshd-failed-user2

We give the extracted field a name and save it.┬áNext, let’s extract something a little harder – the status of the operation:

li-sshd-failed-status0

Again, Log Insight attempts to match a pattern for us:

li-sshd-failed-status1

Again the pattern does not match all the events we care about and is too specific so it should be changed to:

li-sshd-failed-status2

As you can see for the above example, changing the pre and post context is key to ensure that all desired events are matched as expected. Note, this is not always the case and depends on the events for which fields are being extracted.

The exercise of extracting fields should be repeated for all relevant fields.

Step 3: Create message queries

Depending on how you defined your extracted fields, creating message queries could be as simple as creating a filter that checks if an extracted field exists (notice how the keyword query is no longer needed):

li-sshd-failed-exists

Step 4: Create aggregation queries

With the right events being shown, you now need to visually represent the events in a meaningful way. Going back to the requirements, there was a need to know who attempted to authenticate. This is not a specific requirement and can be represented in a variety of ways including count of events over time grouped by username (a standard bar chart works well here):

li-sshd-failed-bar

It is also possible to combine two requirements into a single visualization with something like count of events over time grouped by username and source (a bubble chart best represents this data):

li-sshd-failed-bubble

Step 5: Save final queries

Now that you have the events you want presented in the format you want, you need to save the queries for future reference. You can save queries in a variety of ways depending on the requirements. For example, you can save the visual representation by selecting the Add to Dashboard option:

li-sshd-failed-add-to-dashboard

You can save the most recent authentication attempts through the use of a field table:

li-sshd-failed-field-table

You can create an alert for queries you want to be notified about. The threshold for the alert can be set as defined in the requirements:

li-sshd-failed-10-alert

Step 6: Share

Now that you have the queries you care about in the format and presentation you need:

li-sshd-failed-dashboards

You can export the saved information as a content pack that you can install on your Log Insight instances, including the one you created the content pack on, or share it more broadly on the Log Insight community or VMware Solution Exchange.

li-sshd-failed-export

Summary

As you can see, Log Insight makes it quick and easy to query for the data you care about. Log Insight is not meant for just VMware logs, but any unstructured data from any device. Audit/Security logs are a great example use case for Log Insight. You can see it leveraged in the vSphere content pack today and in this blog post I demonstrated how to handle Linux SSH logs as well. While the purpose of this post was to show a particular Log Insight use case, in the process I actually demonstrated how to build a content as well. For more information on how to build content packs in Log Insight, see this link.

I hope you found this Log Insight use case helpful and I would love to hear about the use cases you have come up with as well! In addition, if there are any other particular use cases you are interested in, please let me know!

© 2014, Steve Flanders. All rights reserved.

Leave a Reply