Skip to content

Log Insight: Syslog Agents and Multiline Messages

There appears to be some confusion on Log Insight’s support of multiline messages that I would like to clear up. The title gives it away, but I would like to discuss the details!
multiline-example

What is a multiline message?

In syslog, multiline messages are single events that span multiple lines. A log file may span multiple lines for many reasons including:

  • Ease of reading – Sometimes log messages are formatted in a way to promote readability and this may lead to the inclusion of multiline messages. Examples of this include, Java stack traces, Windows event logs, and some formatted messages including JSON, HTML and XML.
  • Perceived syslog agent limit – The syslog RFC  does not set an upper limit on message size, but it does set a minimum maximum of 480 octets that MUST be supported and a minimum maximum of 2048 octets that SHOULD be supported. Anything over 2048 octets MAY be supported. If the agent does have an upper limit then it SHOULD truncate the payload, but MAY discard the message entirely. It should also be noted that the UDP syslog RFC states that IPv6 syslog messages MUST support a minimum maximum of 1180 octets.
  • Actual limit of protocol – For UDP, the syslog RFC states that a messages can be up to 65535 octets minus the UDP header length. This limitation is due to UDP only supporting a maximum size of 65535 octets.

What is the problem with multiline messages?

The problem with multiline messages is that since they are not defined in the syslog RFC they are not well handled. What typically happens is that each newline signifies the end of a message. The result is that a multiline message appears as several separate messages. Since the messages are not joined, performing queries over the message is difficult.
li-20-multiline-bad

Wait, Log Insight does not support multiline messages?

No, Log Insight does support multiline. For example:
li-20-multiline-good

So why does Log Insight sometimes break up multiline messages?

Log Insight does not break up multiline messages. Log Insight ingests messages as they arrive and treats each line that starts with a <PRI> (defined in the syslog RFC) as a new event. For example, if the markup of the event looks like the following, it will be displayed as a single event in Log Insight:

If the markup of the event looks like the following, it will be displayed as multiple (4) events in Log Insight:

As you can also see, a multiline message may contain items that should only appear at the beginning of the event (e.g. timestamp, hostname and app name). While this is technically supported, it is not recommended and can lead to confusion. As such, the above multiline message should look like:

In short, Log Insight does not attempt to connect messages that are multiline if each line starts with a <PRI>. The issue is with the syslog agent (i.e. the client of the syslog aggregator) that sent the message to Log Insight.

Properly configuring syslog agents to send multiline

To address the issue, ensure your syslog agent is configured to send multiline messages properly. Some examples are listed below.

Rsyslog

Rsyslog supports converting multiline messages into single messages using the imfile module. It does this by embedding LF characters. The problem is that the syslog RFC does not state that embedded LF characters should be used and most other syslog agent/servers do not support the use of LF characters. As such, unless you are running rsyslog agents and rsyslog servers, multiline while likely not function for you properly.
To address this, rsyslog 7.5.3 supports the ability of escaping LF characters into a universally accepted four character sequence. In addition to running rsyslog 7.5.3 or newer, you also need to be using the new rsyslog configuration format as the legacy format does not support this new functionality. If you are running rsyslog 7.5.3 or newer with the new rsyslog configuration then you should be all set as the LF character escaping feature is enabled by default.
For more information, see this link.

Syslog-NG

Syslog-ng supports a no-multi-line flag for source definitions to support multiline messages. Unfortunately, this flag does not work with the TCP driver (protocol).
In addition to the no-multi-line flag, syslog-ng supports of variety of other options to ensure multiline message are handled the way you want including multi-lint-garbage, multi-line-mode, and multi-line-prefix. These options make it possible to properly distinguish a multiline message as every product may log multiline messages differently.
For more information, see this link.

liagent (LI Windows Agent)

The Log Insight Windows agent, available in the Log Insight 2.0 beta, supports multiline characters natively! Winlog events are notorious for being multilined, but when using the liagent, multiline messages come in as a single event. Filelog events support multiline grouping through the use of the event_marker configuration option. This option tells the liagent what distinguishes a new event. The option supports complete regular expressions and can be as simple as:

or as complicated as:

A complete example of using the event_marker configuration option is:

Other

What if you are not running one of the above clients or do not have the ability to configure the agent (one classic example of this is ESXi – see the hostd image earlier in this post)? The workaround in this case is to use a syslog aggregator in front of Log Insight and configure the aggregator to properly pass the multiline messages. If you would like to see functionality built natively into Log Insight, please vote for the feature here.

What about the release notes?

If you check the Log Insight release notes, you will notice that is also states that Log Insight does not support multiline messages. This was added because Log Insight does not attempt to join multiline messages that are not sent properly. I have notified the documentation team and the Log Insight 2.0 release notes will properly state how Log Insight supports multiline messages.

Summary

Log Insight fully supports multiline messages. Log Insight treats each line that starts with a <PRI> as a new event so ensure your syslog agent is configured to properly send multiline messages. If you are running Windows, use the new liagent for proper multiline handling.

© 2014, Steve Flanders. All rights reserved.

Published inVMware

6 Comments

  1. Matt F Matt F

    Have you found a good multi-line event indicator for SQL Server error logs?

    • Thanks for the comment! The SQL content pack does not current list one so I would suspect multi-line is not common. Based on your comment though, looks like you are seeing them. Can you email a sample of logs so I can take a look and make a recommendation?

  2. Thanks for all of the great information about log insight. Primarily based on your great articles, we have chosen log insight as our syslog agent on windows. One issue we are having is that when sending the windows event logs to ELK, we are seeing the windows event log broken into multi-line messages. Any thoughts on how to get around this? I assume we could use event markers, but a little unsure where to start. Any advice is appreciated.

    • Hey Dusty — Thanks for the comment and I am glad you are enjoying my blog and Log Insight! The issue you are experiencing forwarding events to ELK is the same issue covered in this blog post about Log Insight. If you replace Log Insight with ELK the solution will be the same 🙂 are multiline events being shown properly within Log Insight? If so you do not need to worry about the event_marker. If not then I am happy to assist. The problem you are experiencing is that Log Insight is forwarding events over syslog and the syslog RFC does not dictate how to handle multiline. By default, ELK is doing the same in as Log Insight and is treating each newline within a sent event as a new event. If multiline is configured properly on the Log Insight side then you will need to adjust the multiline settings on the ELK side to address the issue. I hope this helps!

  3. Mark Wolfe Mark Wolfe

    Is there a way to address multiline messages coming from ESXi’s own vmsyslogd?

    It looks like your “bad” examples are coming from ESXi hosts, but the suggested fixes are for other agents.

    • Hey Mark — that would be under the “Other” section of the post. Technically, this is an issue on the client-side so you could file a bug against the ESXi team. Alternatively, you can be a non-LI relay between ESXi and LI to manually parse the events. Final option would be for LI to natively support, but I do not see that happening.

Leave a Reply

Your email address will not be published. Required fields are marked *