Log Insight Agents: Client-side versus Server-side Configuration

I have received some questions around agent configuration that I wanted to clear up. Read on to learn more!

Background

The Log Insight agent has been around since Log Insight 2.0 and features the ability to collect logs from clients. In Log Insight 2.0 the client was Microsoft Windows and in Log Insight 2.5 Linux support was added. While syslog agents exist today, mostly they target the Linux market and many suffer from some significant limitations in part because they must abide by the syslog RFC. The Log Insight agent supports and abides by the syslog RFC, but it also provides many benefits over other agents — for more information see this post.

Configuration

Like all other syslog agents, the Log Insight agent requires configuration to work properly. Since the Log Insight agent was first introduced it supported two ways to perform agent configuration:

1. Client-side: Directly from the client on which the agent is running
2. Server-side: Centrally from the Log Insight server

I would like to dive into each of these options to provide a better understanding of each. Before I do, it is important to understand the configuration sections used by the agent. These sections can be broken into two categories:

• Agent Settings: [server], [storage], [logging]
• Collection Settings: [winlog], [filelog], [parsers]

Client-side

Agent configuration can be done entirely client-side or only partially client-side. At a minimum, all Agent Settings defined above must be done client-side. Of those options, the hostname option under the [server] section needs to be properly set to forward logs to the right destination. By default, the hostname option is set to “LOGINSIGHT” meaning if the client can resolve “LOGINSIGHT” then the agent will send events to that destination. The hostname option can be set in a variety of ways:

• Automatically during installation by keeping default: As described above, the hostname defaults to “LOGINSIGHT”.
• Automatically during installation by installing package downloaded from a Log Insight server: Starting with Log Insight 3.0, if you download the agent from the server, the agent package name contains the IP of the Log Insight server which is used during installation instead of the default option.
• Automatically during installation by passing a parameter during installation: This process is outlined in the Log Insight documentation.
• Manually by editing the liagent.ini file: Note as of Log Insight 2.5 you do not need to restart the agent for the change to take effect.

As of Log Insight 2.5, any change made to the liagent.ini file takes effect immediately without a restart. Note that client-side and server-side configurations get merged so if you want to know the current configuration of the agent you need to check the liagent-effective.ini.
IMPORTANT: Never edit the liagent-effective.ini file. Changes made to the file are not applied to the agent and will be overridden.

Server-side

Partial agent configuration — Collection Settings — can be done from the Agents page under the Administration section of the Log Insight UI (/admin/agents). Until Log Insight 3.0, the Agent page featured a single configuration box for all reporting agents. This meant you had the option to configure all reporting agents server-side or none. While this may not sound ideal, it really was not as bad as it sounded. The agent is smart and will ignore agent configurations that are not applicable to it. The problem with a single configuration was when you wished to collect different logs for different agents where both agents contained the same logs (e.g. development versus production systems). To address the limitations of a single configuration option, Log Insight 3.0 features agent groups. Agent groups make it possible to determine which agents receive which configuration. They also make it possible to group configuration to specific events such as those found in content packs.
Once server-side configuration has been applied, it is important to note that the agents pull the configuration from the server — the server does not push configuration to the agents. Also, server-side agent configuration only works for agent configured to use the cfapi protocol (default) — if you have switched to the syslog protocol then server-side configuration is not possible and the agents will not even appear on the Agents page.

Can You Use Both?

The next logical question that typically gets asked is can you use client-side and server-side configuration at the same time? The answer is of course yes as you must perform Agent Settings client-side and you have option to perform Collection Settings server-side. But can you do Collection Settings on both? The answer is yes!
Given that you can configure Collection Settings client-side and server-side the next question is how is merging performed? The answers to this question are:

• If a Collection Setting is unique then it is kept: For example if a [filelog|client] section is created client-side and a [filelog|server] section is created server-side then the agent configuration will contain both filelog sections. Note this can lead to duplicate collection if the directory and include setting in each configuration section are the same.
• If a Collection Setting is not unique then the server-side wins during conflict: If a conflict does exist then the server will always win. For example:

  [filelog|test]
  directory=/var/log/test
  include=client.log

  [filelog|test]
  directory=/var/log/test
  include=server.log

  Results in:

  [filelog|test]
  directory=/var/log/test
  include=server.log

• You cannot have the same section name defined multiple times server-side: To prevent server-side conflicts as an agent can belong to multiple agent groups, a section name created server-side can only be defined once. If you attempt to define it twice you will receive an error:

  1: section with 'filelog|test' name is already defined in 'test' group

• Content Pack agent groups are never merged with client-side configuration: Even if a content pack agent group contains the same section name as a client-side configuration section name no merging takes place and they behave as though both are unique. The reason for this is because content pack agent groups are namespaced so the section names are actually unique.

Recommendation

You can use client-side and server-side configuration so what is the recommendation? Well, there are a few considerations:

• Do you want configuration management? If yes, server-side configuration can provide that for the Collection Settings. While Agent Settings can not be configured server-side today, they are often only set during installation time.
• Do you have configuration management today? Given that you may want to change Agent Settings having a configuration management tool may be nice. If you already have a configuration management tool like Puppet, Chef, Ansible, Salt, etc then you can use it to server similar capabilities to the server-side configuration.
• Are you leveraging content pack that require agent configuration? If so, it is really easy to configure the agent configuration server-side as agent groups can be included in content packs.

In general, the recommendation is to use server-side configuration, but this is in no way a requirement.
IMPORTANT: Be careful not to collect the same files multiple times by using different section names — if this happens you will have duplicated events in Log Insight.

Summary

For Log Insight, flexibility and ease of use are critical. As such, Log Insight agents provide both client-side and server-side configuration which can be used together. Merging is handled automatically and it is easy to see the actual configuration of the agent. In addition, agent groups provide flexible configuration management options and agent groups are part of content packs to make data ingestion easy. In general, the recommendation is to use server-side configuration for all Collection Settings, but this is in no way a requirement.
Do you use server-side configuration today?

© 2015, Steve Flanders. All rights reserved.