Syslog Agents on Linux

In order to send events from a Linux device to a remote syslog server like Log Insight, you need a syslog agent. Most Linux operating systems ship with a syslog agent and if one is not available, one can be easily installed. The two most common syslog agents used on Linux systems today are rsyslog and syslog-ng. I would like to cover how to configure these syslog agents to send events to a remote syslog server.

Introduction

Syslog agents are capable of handling a lot more then just syslog messages. The flexibility of syslog agents makes it the ideal solution for forwarding any type of event. Some examples of what syslog agents are capable of doing are listed below.

Forward Events

Forwarding events is critical for aggregation, correlation, and querying of events throughout an environment. Forwarding events is as simple as specifying what file(s) to forward, to what destination(s), and in what format.

Monitor Directories

In addition to syslog messages, directories containing files can be monitored and treated like a syslog message.

Monitor Files

In addition to syslog messages, any file can be monitored and treated like a syslog message.

Tag Logs

Syslog messages can be modified and support the addition of tags making querying easier. For a great example on how and why to tag logs see: http://www.virtuallyghetto.com/2013/05/how-to-add-tag-log-prefix-to-syslog.html.

IMPORTANT: Per RFC 3164 and 5424, tags are to be limited to 32 characters.

JSON to Syslog

JSON formatting to be converted to single line syslog format for forwarding and querying capabilities. To do so, a syslog parser/template needs to be created.

NetFlow to Syslog

A NetFlow collector can be installed on a system and either the collector can be configured to save NetFlow events as syslog events or the collector can save events to a file or directory and the file or directory can be monitored by a syslog agent for event forwarding. For more information on NetFlow see this post: http://sflanders.net/2013/11/04/sending-netflow-syslog.

SNMP to Syslog

snmptrapd can be installed on a system and syslog agents can be configured to convert an SNMP trap to a syslog message that in turn can be forwarded to a remote destination. This solution can be used instead of or in addition to a SNMP monitoring tool.

Log Insight

Syslog agents are important for Log Insight because Log Insight supports ingestion of events over the syslog protocol today and more specifically udp/514, tcp/514, and tcp(ssl)/1514. Log Insight choose to start with syslog as it is the standard for remote message logging and because syslog agents are flexible enough to handle more than just logs messages. I have often heard people ask how to monitor configuration files, JSON files, and SNMP traps in Log Insight. What a lot of people may not realize is that syslog agents are capable of handling all of this and since syslog agents support it so does Log Insight. To be clear, Log Insight is capable of handling more than just syslog events, but was optimized for syslog events as part of the 1.0 GA release.

Rsyslog

Forward Logs

UDP

TCP

Example

For more information see: http://www.rsyslog.com/ sending-messages-to-a-remote-syslog-server/

Monitor Directories

I am not aware of a way to do this in Rsyslog. See Monitor Files below.

Monitor Files

Legacy:

For more information see: http://www.rsyslog.com/doc/imfile.html

Tag logs

To overcome tag limitations see: http://www.rsyslog.com/sende-messages-with-tags-larger-than-32-characters/

JSON to Syslog

IMPORTANT: Certain version of rsyslog have a page when handling JSON. At the very least rsyslog 6.6 has a known issue so be sure to research beforehand.

CEE Format

For more information see: http://www.rsyslog.com/doc/mmjsonparse.html

Custom

Create a template for the JSON format like this:

Then forward the events using the template:

SNMP to Syslog

Syslog-NG

Syslog-NG comes in an open-source edition (OSE), which is free and premium edition (PE), which is charged per agent. In most cases, OSE should be sufficient. A comparison of the differences is available here: http://www.balabit.com/network-security/syslog-ng/comparing/detailed.

Forward logs

Monitor Directories

Syslog-NG PE natively supports monitoring directories. Syslog-NG OSE does not, but there is a workaround. First configure Syslog-NG:

Next, create a script to do all the heavy lifting:

Finally, monitor the directories you care about:

More information can be found here: http://asylum.madhouse-project.org/blog/2013/09/10/howto-monitor-a-directory-of-logfiles/

Monitor Files

For each file to monitor add a line like:

Then modify the forward logs line in above like:

For more information see: http://www.syslog.org/logged/reading-logs-from-a-file-in-syslog-ng/

Tag logs

Using tags

Using templates

JSON to Syslog

For more information see: http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.4-guides/en/syslog-ng-ose-v3.4-guide-admin/html/json-parser.html

SNMP to Syslog

IMPORTANT: Must be running syslog-ng v3 or newer and have snmptrapd configured.

For more information see: http://bazsi.blogs.balabit.com/2008/11/syslog-ng-3-0-and-snmp-traps/

Summary

When choosing a syslog agent it really comes down to requirements. Both Rsyslog and Syslog-NG offer syslog agents that are capable of addressing a majority of remote syslog needs and best of all they both offer free solutions! I hope this entry highlights that syslog agents are capable of handling way more than just syslog. If you are looking for additional types of files to monitor let me know and I will update this post.

© 2013 – 2014, Steve Flanders. All rights reserved.

Leave a Reply