In order to send events from a Linux device to a remote syslog server like Log Insight, you need a syslog agent. Most Linux operating systems ship with a syslog agent and if one is not available, one can be easily installed. The two most common syslog agents used on Linux systems today are rsyslog and syslog-ng. I would like to cover how to configure these syslog agents to send events to a remote syslog server.
Introduction
Syslog agents are capable of handling a lot more then just syslog messages. The flexibility of syslog agents makes it the ideal solution for forwarding any type of event. Some examples of what syslog agents are capable of doing are listed below.
Forward Events
Forwarding events is critical for aggregation, correlation, and querying of events throughout an environment. Forwarding events is as simple as specifying what file(s) to forward, to what destination(s), and in what format.
Monitor Directories
In addition to syslog messages, directories containing files can be monitored and treated like a syslog message.
Monitor Files
In addition to syslog messages, any file can be monitored and treated like a syslog message.
Tag Logs
Syslog messages can be modified and support the addition of tags making querying easier. For a great example on how and why to tag logs see: http://www.virtuallyghetto.com/2013/05/how-to-add-tag-log-prefix-to-syslog.html.
IMPORTANT: Per RFC 3164 and 5424, tags are to be limited to 32 characters.
JSON to Syslog
JSON formatting to be converted to single line syslog format for forwarding and querying capabilities. To do so, a syslog parser/template needs to be created.
NetFlow to Syslog
A NetFlow collector can be installed on a system and either the collector can be configured to save NetFlow events as syslog events or the collector can save events to a file or directory and the file or directory can be monitored by a syslog agent for event forwarding. For more information on NetFlow see this post: https://sflanders.net/2013/11/04/sending-netflow-syslog.
SNMP to Syslog
snmptrapd can be installed on a system and syslog agents can be configured to convert an SNMP trap to a syslog message that in turn can be forwarded to a remote destination. This solution can be used instead of or in addition to a SNMP monitoring tool.
Log Insight
Syslog agents are important for Log Insight because Log Insight supports ingestion of events over the syslog protocol today and more specifically udp/514, tcp/514, and tcp(ssl)/1514. Log Insight choose to start with syslog as it is the standard for remote message logging and because syslog agents are flexible enough to handle more than just logs messages. I have often heard people ask how to monitor configuration files, JSON files, and SNMP traps in Log Insight. What a lot of people may not realize is that syslog agents are capable of handling all of this and since syslog agents support it so does Log Insight. To be clear, Log Insight is capable of handling more than just syslog events, but was optimized for syslog events as part of the 1.0 GA release.
Rsyslog
Forward Logs
UDP
<what>;<to>;<forward> @server.example.com:514
TCP
<what>;<to>;<forward> @@server.example.com:514
Example
*.* @@server.example.com:514
For more information see: http://www.rsyslog.com/ sending-messages-to-a-remote-syslog-server/
Monitor Directories
I am not aware of a way to do this in Rsyslog. See Monitor Files below.
Monitor Files
module(load="imfile" PollingInterval="10") # needs to be done just once and polling interval can be adjusted as needed input(type="imfile" File="/path/to/file1" # for each file to be monitored, change /path/to/file1 as appropriate Tag="tag1" # optional StateFile="/var/spool/rsyslog/statefile1" # must be unique per file being monitored Severity="error" Facility="local7")
Legacy:
$ModLoad imfile # needs to be done just once # File 1 $InputFileName /path/to/file1 $InputFileTag tag1: $InputFileStateFile stat-file1 $InputFileSeverity error $InputFileFacility local7 $InputRunFileMonitor # File 2 $InputFileName /path/to/file2 $InputFileTag tag2: $InputFileStateFile stat-file2 $InputRunFileMonitor # ... and so on ... # # check for new lines every 10 seconds $InputFilePollInterval 10
For more information see: http://www.rsyslog.com/doc/imfile.html
Tag logs
template(name="FileFormat" type="string" string= "%TIMESTAMP% %HOSTNAME% %syslogtag%%msg%\n" )
To overcome tag limitations see: http://www.rsyslog.com/sende-messages-with-tags-larger-than-32-characters/
JSON to Syslog
IMPORTANT: Certain version of rsyslog have a page when handling JSON. At the very least rsyslog 6.6 has a known issue so be sure to research beforehand.
CEE Format
module(load="mmjsonparse") action(type="mmjsonparse")
For more information see: http://www.rsyslog.com/doc/mmjsonparse.html
Custom
Create a template for the JSON format like this:
$template ls_json,"{%timestamp:::date-rfc3339,jsonf:@timestamp%,%source:::jsonf:@source_host%,\"@source\":\"syslog://%fromhost-ip:::json%\",\"@message\":\"%timestamp% %app-name%:%msg:::json%\",\"@fields\":{%syslogfacility-text:::jsonf:facility%,%syslogseverity-text:::jsonf:severity%,%app-name:::jsonf:program%,%procid:::jsonf:processid%}}"
Then forward the events using the template:
*.* @@server.example.com:514;ls_json
SNMP to Syslog
$template mkeventd,"<%PRI%>%TIMESTAMP% %HOSTNAME% %syslogtag% %msg%\n" $template mkeventdsnmp,"<%PRI%>%TIMESTAMP% %msg:F,58:1$% %syslogtag%%msg%\n" :programname,isequal,"snmptrapd" ^/omd/sites/mysite/bin/mkevent;mkeventdsnmp :programname,!isequal,"snmptrapd" ^/omd/sites/mysite/bin/mkevent;mkeventdSources
Syslog-NG
Syslog-NG comes in an open-source edition (OSE), which is free and premium edition (PE), which is charged per agent. In most cases, OSE should be sufficient. A comparison of the differences is available here: http://www.balabit.com/network-security/syslog-ng/comparing/detailed.
Forward logs
#
# Enable this and adopt IP to send log messages to a log server.
#
destination logserver { udp("10.10.10.10" port(514)); }; # change everything within { } as appropriate
log { source(src); destination(logserver); }; # adjust source as appropriate
Monitor Directories
Syslog-NG PE natively supports monitoring directories. Syslog-NG OSE does not, but there is a workaround. First configure Syslog-NG:
@version: 3.5
destination d_all_logs {
file("/var/log/all.log");
};
include "/etc/syslog-ng/conf.d/notify-*.conf";
Next, create a script to do all the heavy lifting:
#! /bin/sh
set -e
CONFDIR="/etc/syslog-ng/conf.d"
SNGCTL=/usr/sbin/syslog-ng-ctl
id="$(uuidgen | tr '-' '_')"
conffn="${CONFDIR}/notify-${id}.conf"
fn="$1"
cat >${conffn} <<EOF
source s_${id} {
file("${fn}");
};
log {
source(s_${id});
destination(d_all_logs);
};
EOF
${SNGCTL} reload
Finally, monitor the directories you care about:
/var/log/remote.d/*.log IN_CREATE,IN_MOVED_TO /usr/local/sbin/syslog-ng-wildcard-notify $@/$#
More information can be found here: http://asylum.madhouse-project.org/blog/2013/09/10/howto-monitor-a-directory-of-logfiles/
Monitor Files
For each file to monitor add a line like:
source s_file { file(“/path/to/app.log” flags(no-parse)); };
Then modify the forward logs line in above like:
log { source(src); source(s_file); destination(logserver); };
For more information see: http://www.syslog.org/logged/reading-logs-from-a-file-in-syslog-ng/
Tag logs
Using tags
source s_file { file(“/path/to/app.log” flags(no-parse) log_prefix(“APP: “); };
source s_file { file(“/path/to/app.log” flags(no-parse) program_override(“APP: “); };
Using templates
destination my_file { # one per file
file("/path/to/app.log" template("$ISODATE $FULLHOST $TAG $MESSAGE")); # adjust /path/to/app.log as appropriate; $TAG optional
};
JSON to Syslog
source s_tcp_json { tcp(port(10514) flags(no-parse)); }; # adjust protocol and port as appropriate
parser p_json { json-parser(); }; # additional flags available if needed
destination d_file { file("/var/log/remote.log"); };
log { source(s_tcp_json); parser(p_json); destination(logserver); };
For more information see: http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.4-guides/en/syslog-ng-ose-v3.4-guide-admin/html/json-parser.html
SNMP to Syslog
IMPORTANT: Must be running syslog-ng v3 or newer and have snmptrapd configured.
filter f_snmptrapd { program(“snmptrapd”); };
rewrite r_snmptrapd { subst(“^([^ ]+) (.*)$ “, “${2}”); set(“${1}” value(“HOST”)); };
For more information see: http://bazsi.blogs.balabit.com/2008/11/syslog-ng-3-0-and-snmp-traps/
Summary
When choosing a syslog agent it really comes down to requirements. Both Rsyslog and Syslog-NG offer syslog agents that are capable of addressing a majority of remote syslog needs and best of all they both offer free solutions! I hope this entry highlights that syslog agents are capable of handling way more than just syslog. If you are looking for additional types of files to monitor let me know and I will update this post.
© 2013, Steve Flanders. All rights reserved.
