Logs are a powerful way to investigate system issues as well as perform system auditing, but what protocol should you send events over? In this post, I would like to discuss the topic and make recommendations. Read on to learn more!
UDP
It makes sense to start with UDP because that is where logging started (see: RFC 3164). Why was UDP chosen as the protocol of choice back in the day? I assume for two primary reasons:
- Volume of data
- OK to lose some of the data
Here are some examples of other systems that typically use UDP:
- Live media streaming
- VPN
- Broadcasting/multicasting
All of these systems send a lot of data, and losing a few packets is typically OK as long as most packets are received.
Now, some of you are probably screaming, “WHAT DO YOU MEAN IT IS OK TO LOSE SOME LOGS!!!” While it is true that losing a couple of logs (e.g., some that report metrics) is OK (because you will have most of the metric logs and can still compute things like averages), losing a single audit log may prove to be problematic. It turns out, there are a variety of reasons why UDP may not be the ideal choice for log messages:
- Lost/corrupt messages — see https://tools.ietf.org/html/rfc5426#section-4
- Event delivery order — see https://tools.ietf.org/html/rfc5426#section-4
- Security Incident & Event Management (SIEM) — see https://tools.ietf.org/html/rfc5426#section-5
- Congestion control — see https://tools.ietf.org/html/rfc5424#section-8.6
- Truncated messages — see section 6.1 https://www.ietf.org/rfc/rfc3164.txt and https://tools.ietf.org/html/rfc5426#section-3.2
It is worth noting that RFC5424 obsoletes RFC3164 — YOU SHOULD NO LONGER FOLLOW RFC 3164 except for legacy reasons (i.e., backwards compatibility).
TCP
RFC5424 removed the requirement of using only UDP for log sending but still mandates UDP be supported (for at least backwards compatibility). One option available starting with RFC 5424 is TCP. Why might you want to use TCP to send events? To guarantee message delivery, right? WRONG! TCP DOES NOT GUARANTEE MESSAGE DELIVERY. Some benefits of TCP include:
- Message receipt acknowledgment
- Reliability
- Flow control
VERY IMPORTANT: Message receipt acknowledgment (i.e. TCP handshake) != guaranteed message delivery
It is worth noting:
- TCP for log events existed well before RFC5424 was created, however not all vendors implemented it, given RFC3164 had been created 11 years earlier
- TCP does not have a dedicated port assignment (514/TCP is actually reserved for something else though it is often used for TCP logging as well as X514)
- RFC5424 does not recommend TCP — more on this below
In addition, there are two different primary format options seen with TCP logs events (https://tools.ietf.org/html/rfc6587#section-1):
- Non-transparent-framing
- Octet-framing
In general, octet-framing is the recommended approach, but most systems still leverage non-transparent-framing today.
Finally, like UDP, TCP has some limitations to be aware of, including:
- Guaranteed message delivery
- Security Incident & Event Management (SIEM)
TLS
RFC5424 recommends TLS for log delivery (as defined in RFC 5425). TLS offers the same benefits as TCP plus the following security threat remediations (see: https://tools.ietf.org/html/rfc5425#section-2)
- Masquerade
- Modification
- Disclosure
It has the same limitations as TCP (minus some of the SIEM limitations) but introduces more complexity to offer enhanced security. Many systems do not properly/fully implement RFC5425 today (e.g., many do not implement certificate-based authentication).
HTTPS
Finally, we enter the land of application-level protocols. While several exist, HTTPS is the most commonly used for logging today. HTTPS offers all the benefits of TLS but also guarantees message delivery. It also is not bound to a single message per event, so batching, compression, and other techniques may be offered (potentially at the cost of slightly delayed delivery).
RELP
Another less widely known protocol is RELP. RELP leverages TCP but guarantees message delivery. It was originally created for Rsyslog, but several other logging tools offer support for RELP today. RELP does not have an RFC, but you can find the specification here: https://www.rsyslog.com/doc/relp.html.
Recommendation
Now that we have covered the most common protocols used for logging today, which should you use? The answer, like most things, is: it depends.
- For log communication on the same system (e.g., docker side-car), UDP is recommended
- For communication between systems on a LAN, HTTPS is preferred, TLS is recommended, TCP secondary, UDP last resort
- For communication between systems on a WAN, HTTPS is preferred, TLS is recommended, TCP/UDP NOT RECOMMENDED
- For SIEM use-cases, you typically want guaranteed message delivery, so for LAN/WAN, something like HTTPS or RELP is likely required
In all cases, it is recommended to use RFC5424 (not RFC3164) or newer and in the case of TCP octet-framing (RFC6587) whenever possible.
Still have questions? Leave a comment!
© 2018 – 2021, Steve Flanders. All rights reserved.
