Someone recently asked me if Microsoft’s Log Parser application could be used as a syslog agent. To be honest, I had not heard of the application so I looked it up and tried it out. This post is a result of what I learned.
How do you collect NetFlow events over the syslog protocol so you can analysis them with a tool like Log Insight? That is the question I would like to answer in this post!
In order to send events from a Windows device to a remote syslog server like Log Insight, you need a syslog agent. Windows does not natively support syslog. The good news is that several syslog agents for Windows exist. I would like to cover my considerations and recommendations for a syslog agent on Windows.
UPDATE: As of Log Insight 2.0, Log Insight offers a free Windows agent that supports the syslog protocol and Log Insight’s ingestion API. For more information see these posts.
In order to send events from a Linux device to a remote syslog server like Log Insight, you need a syslog agent. Most Linux operating systems ship with a syslog agent and if one is not available, one can be easily installed. The two most common syslog agents used on Linux systems today are rsyslog and syslog-ng. I would like to cover how to configure these syslog agents to send events to a remote syslog server.
Logging is important for any device, but often considered critical for enterprise applications. One issue with configuring logging on enterprise applications is determining how to configure logging on enterprise applications. I would like to provide a reference guide to ease this pain point.
When architecting any syslog solution several things need to be taken into consideration. Outside of requirements, I would like to discuss some of the technical considerations that need to be taken into account.
The more I use logrotate the less I like it. If you recall from my previous post on logrotate, I choose to leverage the copytruncate option. While this configuration seemed to work well when I tested it, I have now experienced some significant limitations that are not documented in the man page:
So what caused the issues, what was the impact, and how can you rotate logs messages and not experience these issues?
I having been working on a syslog architecture and one key component to the architect was leveraging log rotate for all log files. One section of my log rotate file looked like the following:
/path/to/logs/*
{
daily
dateext
missingok
copytruncate
compress
compresscmd /bin/bzip2
compressext .bz2
sharedscripts
lastaction
/usr/sbin/invoke-rc.d syslog-ng reload >/dev/null
endscript
}
The problem was, I noticed that after the cron for logrotate ran the system started to become slow. Looking at top I noticed several things: the load average continued going up, the logrotate process continued to run with the process consuming around 50% of memory, and the syslog process never restarted.
What was causing the problem?