Skip to content

Syslog Agents on Linux

In order to send events from a Linux device to a remote syslog server like Log Insight, you need a syslog agent. Most Linux operating systems ship with a syslog agent and if one is not available, one can be easily installed. The two most common syslog agents used on Linux systems today are rsyslog and syslog-ng. I would like to cover how to configure these syslog agents to send events to a remote syslog server.

Introduction

Syslog agents are capable of handling a lot more then just syslog messages. The flexibility of syslog agents makes it the ideal solution for forwarding any type of event. Some examples of what syslog agents are capable of doing are listed below.

Forward Events

Forwarding events is critical for aggregation, correlation, and querying of events throughout an environment. Forwarding events is as simple as specifying what file(s) to forward, to what destination(s), and in what format.

Monitor Directories

In addition to syslog messages, directories containing files can be monitored and treated like a syslog message.

Monitor Files

In addition to syslog messages, any file can be monitored and treated like a syslog message.

Tag Logs

Syslog messages can be modified and support the addition of tags making querying easier. For a great example on how and why to tag logs see: http://www.virtuallyghetto.com/2013/05/how-to-add-tag-log-prefix-to-syslog.html.
IMPORTANT: Per RFC 3164 and 5424, tags are to be limited to 32 characters.

JSON to Syslog

JSON formatting to be converted to single line syslog format for forwarding and querying capabilities. To do so, a syslog parser/template needs to be created.

NetFlow to Syslog

A NetFlow collector can be installed on a system and either the collector can be configured to save NetFlow events as syslog events or the collector can save events to a file or directory and the file or directory can be monitored by a syslog agent for event forwarding. For more information on NetFlow see this post: https://sflanders.net/2013/11/04/sending-netflow-syslog.

SNMP to Syslog

snmptrapd can be installed on a system and syslog agents can be configured to convert an SNMP trap to a syslog message that in turn can be forwarded to a remote destination. This solution can be used instead of or in addition to a SNMP monitoring tool.

Log Insight

Syslog agents are important for Log Insight because Log Insight supports ingestion of events over the syslog protocol today and more specifically udp/514, tcp/514, and tcp(ssl)/1514. Log Insight choose to start with syslog as it is the standard for remote message logging and because syslog agents are flexible enough to handle more than just logs messages. I have often heard people ask how to monitor configuration files, JSON files, and SNMP traps in Log Insight. What a lot of people may not realize is that syslog agents are capable of handling all of this and since syslog agents support it so does Log Insight. To be clear, Log Insight is capable of handling more than just syslog events, but was optimized for syslog events as part of the 1.0 GA release.

Rsyslog

Forward Logs

UDP

<what>;<to>;<forward> @server.example.com:514

TCP

<what>;<to>;<forward> @@server.example.com:514

Example

*.* @@server.example.com:514

For more information see: http://www.rsyslog.com/ sending-messages-to-a-remote-syslog-server/

Monitor Directories

I am not aware of a way to do this in Rsyslog. See Monitor Files below.

Monitor Files

module(load="imfile" PollingInterval="10") # needs to be done just once and polling interval can be adjusted as needed
input(type="imfile" File="/path/to/file1" # for each file to be monitored, change /path/to/file1 as appropriate
Tag="tag1" # optional
StateFile="/var/spool/rsyslog/statefile1" # must be unique per file being monitored
Severity="error"
Facility="local7")

Legacy:

$ModLoad imfile # needs to be done just once
# File 1
$InputFileName /path/to/file1
$InputFileTag tag1:
$InputFileStateFile stat-file1
$InputFileSeverity error
$InputFileFacility local7
$InputRunFileMonitor
# File 2
$InputFileName /path/to/file2
$InputFileTag tag2:
$InputFileStateFile stat-file2
$InputRunFileMonitor
# ... and so on ...
#
# check for new lines every 10 seconds
$InputFilePollInterval 10

For more information see: http://www.rsyslog.com/doc/imfile.html

Tag logs

template(name="FileFormat" type="string"
string= "%TIMESTAMP% %HOSTNAME% %syslogtag%%msg%\n"
)

To overcome tag limitations see: http://www.rsyslog.com/sende-messages-with-tags-larger-than-32-characters/

JSON to Syslog

IMPORTANT: Certain version of rsyslog have a page when handling JSON. At the very least rsyslog 6.6 has a known issue so be sure to research beforehand.

CEE Format

module(load="mmjsonparse")
action(type="mmjsonparse")

For more information see: http://www.rsyslog.com/doc/mmjsonparse.html

Custom

Create a template for the JSON format like this:

$template ls_json,"{%timestamp:::date-rfc3339,jsonf:@timestamp%,%source:::jsonf:@source_host%,\"@source\":\"syslog://%fromhost-ip:::json%\",\"@message\":\"%timestamp% %app-name%:%msg:::json%\",\"@fields\":{%syslogfacility-text:::jsonf:facility%,%syslogseverity-text:::jsonf:severity%,%app-name:::jsonf:program%,%procid:::jsonf:processid%}}"

Then forward the events using the template:

*.* @@server.example.com:514;ls_json

SNMP to Syslog

$template mkeventd,"<%PRI%>%TIMESTAMP% %HOSTNAME% %syslogtag% %msg%\n"
$template mkeventdsnmp,"<%PRI%>%TIMESTAMP% %msg:F,58:1$% %syslogtag%%msg%\n"
:programname,isequal,"snmptrapd" ^/omd/sites/mysite/bin/mkevent;mkeventdsnmp
:programname,!isequal,"snmptrapd" ^/omd/sites/mysite/bin/mkevent;mkeventdSources

Syslog-NG

Syslog-NG comes in an open-source edition (OSE), which is free and premium edition (PE), which is charged per agent. In most cases, OSE should be sufficient. A comparison of the differences is available here: http://www.balabit.com/network-security/syslog-ng/comparing/detailed.

Forward logs

#
# Enable this and adopt IP to send log messages to a log server.
#
destination logserver { udp("10.10.10.10" port(514)); }; # change everything within { } as appropriate
log { source(src); destination(logserver); }; # adjust source as appropriate

Monitor Directories

Syslog-NG PE natively supports monitoring directories. Syslog-NG OSE does not, but there is a workaround. First configure Syslog-NG:

@version: 3.5
destination d_all_logs {
  file("/var/log/all.log");
};
include "/etc/syslog-ng/conf.d/notify-*.conf";

Next, create a script to do all the heavy lifting:

#! /bin/sh
set -e
CONFDIR="/etc/syslog-ng/conf.d"
SNGCTL=/usr/sbin/syslog-ng-ctl
id="$(uuidgen | tr '-' '_')"
conffn="${CONFDIR}/notify-${id}.conf"
fn="$1"
cat >${conffn} <<EOF
source s_${id} {
 file("${fn}");
};
log {
 source(s_${id});
 destination(d_all_logs);
};
EOF
${SNGCTL} reload

Finally, monitor the directories you care about:

/var/log/remote.d/*.log  IN_CREATE,IN_MOVED_TO /usr/local/sbin/syslog-ng-wildcard-notify [email protected]/$#

More information can be found here: http://asylum.madhouse-project.org/blog/2013/09/10/howto-monitor-a-directory-of-logfiles/

Monitor Files

For each file to monitor add a line like:

source s_file { file(“/path/to/app.log” flags(no-parse)); };

Then modify the forward logs line in above like:

log { source(src); source(s_file); destination(logserver); };

For more information see: http://www.syslog.org/logged/reading-logs-from-a-file-in-syslog-ng/

Tag logs

Using tags

source s_file { file(“/path/to/app.log” flags(no-parse) log_prefix(“APP: “); };
source s_file { file(“/path/to/app.log” flags(no-parse) program_override(“APP: “); };

Using templates

destination my_file { # one per file
file("/path/to/app.log" template("$ISODATE $FULLHOST $TAG $MESSAGE")); # adjust /path/to/app.log as appropriate; $TAG optional
};

JSON to Syslog

source s_tcp_json { tcp(port(10514) flags(no-parse)); }; # adjust protocol and port as appropriate
parser p_json { json-parser(); }; # additional flags available if needed
destination d_file { file("/var/log/remote.log"); };
log { source(s_tcp_json); parser(p_json); destination(logserver); };

For more information see: http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.4-guides/en/syslog-ng-ose-v3.4-guide-admin/html/json-parser.html

SNMP to Syslog

IMPORTANT: Must be running syslog-ng v3 or newer and have snmptrapd configured.

filter f_snmptrapd { program(“snmptrapd”); };
rewrite r_snmptrapd { subst(“^([^ ]+) (.*)$ “, “${2}”); set(“${1}” value(“HOST”)); };

For more information see: http://bazsi.blogs.balabit.com/2008/11/syslog-ng-3-0-and-snmp-traps/

Summary

When choosing a syslog agent it really comes down to requirements. Both Rsyslog and Syslog-NG offer syslog agents that are capable of addressing a majority of remote syslog needs and best of all they both offer free solutions! I hope this entry highlights that syslog agents are capable of handling way more than just syslog. If you are looking for additional types of files to monitor let me know and I will update this post.

© 2013, Steve Flanders. All rights reserved.

Published inSystem Administration

Be First to Comment

Leave a Reply

Your email address will not be published. Required fields are marked *