When architecting any syslog solution several things need to be taken into consideration. Outside of requirements, I would like to discuss some of the technical considerations that need to be taken into account.

IP vs FQDN

When you configure a syslog agent to forward events to a remote syslog target you have the option of specifying the destination using either an IP address or a FQDN. It is important to understand the differences between these two options. When using a FQDN you become reliant on DNS. If DNS is unavailable then you may not be able to resolve the FQDN, which means that events cannot be forwarded. Most syslog agents support some amount of DNS caching, which makes it possible for syslog agents to sustain some amount of DNS downtime. The advantage to using DNS is that it gives you the flexibility to change either the syslog server or the IP address of the syslog server without needing to update the syslog agent configuration on devices connected to that syslog server instance.
Many products are dependent on DNS and flexibility in an environment is always something nice to have. My recommendation would be to use a FQDN unless you are in a fully automated environment, in which case either option would work.

UDP vs TCP

The connection between a client to a server, a client to a syslog aggregator, and a syslog aggregator to a syslog server is critical to ensuring that events arrive at the remote destination. Several factors can prevent events from arriving as expected including network latency, server load, and transport protocol used. With UDP, events may be dropped and there is no easy way to determine when this occurs because of how the UDP protocol was designed. TCP can guarantee that events make it from the source to the destination reducing the failure domain to just the server. To elaborate, TCP can be used to overcome network issues, however it does not guarantee that the server keeps the event even if the server acknowledges it. In either case, TCP is preferable to UDP when transporting events. This is especially true when the connection between the source and the destination is over a WAN link.
In general, my recommendation would be:

• If devices do not support TCP use UDP
• Use TCP whenever possible, especially between syslog aggregators and syslog servers or any traffic going over a WAN
• Understand that TCP does not guarantee events are not dropped

Single remote syslog target support

Some devices support only logging to a single remote syslog target. An example of such a product is VMware ESXi 4.x. Depending on business requirements this can be an issue as events will be dropped if the single syslog server is unavailable. To overcome this, load balancing can be leveraged. In short, you can configure a HA load balancer with a VIP between two or more syslog servers. This way if one syslog server is unavailable the other syslog server can receive the events.

© 2013, Steve Flanders. All rights reserved.