Log Insight: Syslog Agents and Multiline Messages

There appears to be some confusion on Log Insight’s support of multiline messages that I would like to clear up. The title gives it away, but I would like to discuss the details!

What is a multiline message?

In syslog, multiline messages are single events that span multiple lines. A log file may span multiple lines for many reasons including:

• Ease of reading – Sometimes log messages are formatted in a way to promote readability and this may lead to the inclusion of multiline messages. Examples of this include, Java stack traces, Windows event logs, and some formatted messages including JSON, HTML, and XML.
• Perceived syslog agent limit – The syslog RFC  does not set an upper limit on message size, but it does set a minimum maximum of 480 octets that MUST be supported and a minimum maximum of 2048 octets that SHOULD be supported. Anything over 2048 octets MAY be supported. If the agent does have an upper limit then it SHOULD truncate the payload, but MAY discard the message entirely. It should also be noted that the UDP syslog RFC states that IPv6 syslog messages MUST support a minimum maximum of 1180 octets.
• Actual limit of protocol – For UDP, the syslog RFC states that a messages can be up to 65535 octets minus the UDP header length. This limitation is due to UDP only supporting a maximum size of 65535 octets.

What is the problem with multiline messages?

The problem with multiline messages is that since they are not defined in the syslog RFC they are not well handled. What typically happens is that each newline signifies the end of a message. The result is that a multiline message appears as several separate messages. Since the messages are not joined, performing queries over the message is difficult.

Wait, Log Insight does not support multiline messages?

No, Log Insight does support multiline. For example:

So why does Log Insight sometimes break up multiline messages?

Log Insight does not break up multiline messages. Log Insight ingests messages as they arrive and treats each line that starts with a <PRI> (defined in the syslog RFC) as a new event. For example, if the markup of the event looks like the following, it will be displayed as a single event in Log Insight:

<165> 2014-05-16T21:33:39.531Z smith01.matrix Hostd: [FFC38B70 info 'Hostsvc' opID=hostd-facb] VsanSystemVmkProvider : GetRuntimeInfo: Complete, runtime info: (vim.vsan.host.VsanRuntimeInfo) {
2014-05-16T21:33:39.531Z smith01.matrix Hostd: --> dynamicType = <unset>,
2014-05-16T21:33:39.531Z smith01.matrix Hostd: --> accessGenNo = 0,
2014-05-16T21:33:39.531Z smith01.matrix Hostd: --> }

If the markup of the event looks like the following, it will be displayed as multiple (4) events in Log Insight:

<165> 2014-05-16T21:33:39.531Z smith01.matrix Hostd: [FFC38B70 info 'Hostsvc' opID=hostd-facb] VsanSystemVmkProvider : GetRuntimeInfo: Complete, runtime info: (vim.vsan.host.VsanRuntimeInfo) {
<165> 2014-05-16T21:33:39.531Z smith01.matrix Hostd: --> dynamicType = <unset>,
<165> 2014-05-16T21:33:39.531Z smith01.matrix Hostd: --> accessGenNo = 0,
<165> 2014-05-16T21:33:39.531Z smith01.matrix Hostd: --> }

As you can also see, a multiline message may contain items that should only appear at the beginning of the event (e.g. timestamp, hostname, and app name). While this is technically supported, it is not recommended and can lead to confusion. As such, the above multiline message should look like:

<165> 2014-05-16T21:33:39.531Z smith01.matrix Hostd: [FFC38B70 info 'Hostsvc' opID=hostd-facb] VsanSystemVmkProvider : GetRuntimeInfo: Complete, runtime info: (vim.vsan.host.VsanRuntimeInfo) {
     --> dynamicType = <unset>,
     --> accessGenNo = 0,
     --> }

In short, Log Insight does not attempt to connect messages that are multiline if each line starts with a <PRI>. The issue is with the syslog agent (i.e. the client of the syslog aggregator) that sent the message to Log Insight.

Properly configuring syslog agents to send multiline

To address the issue, ensure your syslog agent is configured to send multiline messages properly. Some examples are listed below.

Rsyslog

Rsyslog supports converting multiline messages into single messages using the imfile module. It does this by embedding LF characters. The problem is that the syslog RFC does not state that embedded LF characters should be used and most other syslog agent/servers do not support the use of LF characters. As such, unless you are running rsyslog agents and rsyslog servers, multiline while likely not function for you properly.
To address this, rsyslog 7.5.3 supports the ability to escape LF characters into a universally accepted four-character sequence. In addition to running rsyslog 7.5.3 or newer, you also need to be using the new rsyslog configuration format as the legacy format does not support this new functionality. If you are running rsyslog 7.5.3 or newer with the new rsyslog configuration then you should be all set as the LF character escaping feature is enabled by default.
For more information, see this link.

Syslog-NG

Syslog-ng supports a no-multi-line flag for source definitions to support multiline messages. Unfortunately, this flag does not work with the TCP driver (protocol).
In addition to the no-multi-line flag, syslog-ng supports of variety of other options to ensure multiline messages are handled the way you want including multi-lint-garbage, multi-line-mode, and multi-line-prefix. These options make it possible to properly distinguish a multiline message as every product may log multiline messages differently.
For more information, see this link.

liagent (LI Windows Agent)

The Log Insight Windows agent, available in the Log Insight 2.0 beta, supports multiline characters natively! Winlog events are notorious for being multilined, but when using the liagent, multiline messages come in as a single event. Filelog events support multiline grouping through the use of the event_marker configuration option. This option tells the liagent what distinguishes a new event. The option supports complete regular expressions and can be as simple as:

event_marker=\n

or as complicated as:

event_marker=^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3}

A complete example of using the event_marker configuration option is:

[filelog|vCenterMain]
directory=C:\ProgramData\VMware\VMware VirtualCenter\Logs
include=vpxd-*.log
exclude=vpxd-alert*;vpxd-profiler*
event_marker=^\d{4}-\d{2}-\d{2}[A-Z]\d{2}:\d{2}:\d{2}\.\d{3}

Other

What if you are not running one of the above clients or do not have the ability to configure the agent (one classic example of this is ESXi – see the hostd image earlier in this post)? The workaround, in this case, is to use a syslog aggregator in front of Log Insight and configure the aggregator to properly pass the multiline messages. If you would like to see functionality built natively into Log Insight, please vote for the feature here.

What about the release notes?

If you check the Log Insight release notes, you will notice that it also states that Log Insight does not support multiline messages. This was added because Log Insight does not attempt to join multiline messages that are not sent properly. I have notified the documentation team and the Log Insight 2.0 release notes will properly state how Log Insight supports multiline messages.

Summary

Log Insight fully supports multiline messages. Log Insight treats each line that starts with a <PRI> as a new event so ensure your syslog agent is configured to properly send multiline messages. If you are running Windows, use the new liagent for proper multiline handling.

© 2014 – 2021, Steve Flanders. All rights reserved.