A question that comes up from time-to-time is how do you backup and restore Log Insight? I would like to cover backup and recovery over the next three posts.
Deployment Types
Log Insight is made up of a variety of components and two primary configurations including:
- Server
- Standalone
- Cluster (Master/Worker)
- Forwarder
- Standalone
- Cluster (Master/Worker)
- Agent
Before discussing how to properly backup each component and configuration, let me discuss some important backup considerations.
Backup Considerations
When you think about what you might backup, the following likely come to mind:
- Configuration – in the case of:
- Configuration changes – like deleting a user and losing all user content
- Configuration loss – in the case of permanent data loss
- Data – more specifically:
- Events – ingested messages
- User data – anything saved by a user
- Procedures – how to:
- Backup – does it matter what order things are backed up? does everything need to be backed up in sync?
- Restore – does it matter what order things are restored? does everything need to be restored in sync?
Understanding the Architecture
Before I outline the steps to backup and restore Log Insight, it is important to take a step back and understand how Log Insight is architected.
IMPORTANT: The architecture of Log Insight changes with every release. The information in this post applies to Log Insight 2.5. If you are running an older version of the Log Insight then the architecture and procedure to backup and restore will be very different.
Server vs. Forwarder
A Log Insight server and a Log Insight forwarder are the same thing. Both are deployed from the same Log Insight OVA. In terms of function, a Log Insight server does not forward events while a Log Insight forwarder performs all of the functions of a Log Insight server and can be used as a Log Insight server, but also forwards events. This means the backup and restore procedures are identical between server and forwarder. Going forward, when I say server I mean server and/or forwarder.
IMPORTANT: Servers and forwarders are completely separate Log Insight instances even if the forwarder is forwarding to a server. This means that servers and forwarders need to be backed up and restored separately.
A server is only available via an OVA package today. The virtual appliance is made of three virtual disks. The important directories for the server are:
- /storage/core – where all ingested events are stored as well as some configuration information and the Cassandra database
- /storage/var – where all LI logs are stored
The most important virtual disks are the second virtual disk and the fourth or greater virtual disk which is where the /storage/core and /storage/var partitions come from. While the OVA only comes with three virtual disks by default, additional virtual disks can be added to the OVA to increase the size of the /storage/core partition, which would allow for longer retention of events.
The majority of space consumed by the OVA is by the second and fourth or later virtual disks if additional virtual disks have been added to increase the capacity of a node. The data on the first and third virtual disks can be recovered just by deploying a new OVA. This means everything available in /usr/lib/loginsight can be replaced simply by deploying a new OVA.
IMPORTANT: The SSL certificate used by Tomcat is stored in cp /usr/lib/loginsight/application/3rd_party/apache-tomcat-*/conf/keystore. You should already have a copy of the certificate outside of Log Insight, but if you do not then you will want to ensure you backup this file as well. Also note that the SSL certificate is uploaded per node — uploading through the Administration section only uploads to the master node today.
Configuration
Log Insight configuration information is automatically stored across all nodes in a cluster. This means backing up the configuration information so you could replace a permanently lost Log Insight node is unnecessary. The location of the configuration is:
- /storage/core/loginsight/config/loginsight-config.xml#
User information including save queries are stored in Cassandra. The server replicates data in Cassandra across multiple — not all — nodes in the cluster.This means backing up the user information so you could replace a permanently lost Log Insight node is unnecessary. Of course, if you were to permanently lose multiple nodes or you wanted to be able to recovery configuration that was changed by a user then you would need a backup of the Cassandra database.
Log Insight does not backup the data in Cassandra today. If the data in Cassandra were to be lost for some reason then Event Types and Event Trends on Interactive Analytics may not work properly until Log Insight is able to repair the data — it will do this automatically — and any users along with saved queries by users would be lost. Again permanent loss of a node is not a concern, the concern is if multiple nodes are permanently lost and/or an administrator removes a user accidentally.
As mentioned in the Server vs. Forwarder section above, Cassandra data is stored on the /storage/core partition.
Agent
A Log Insight agent is installed per device and the only configuration information it requires is how to contact the remote destination (e.g. Log Insight server). Additional configuration, including what files should be monitored and sent to the remote destination, can be specified either client-side or on the Log Insight server. If additional configuration is handled partially or completed server-side then backing up the server/forwarder will handle backing up the agent additional configuration.
In addition to the client-side configuration, the agent also stores logs on the client device and has a database to queue events when necessary. From a backup perspective, logs can be sent via the agent to a remote destination or could be discarded in the case of data loss. If desired, agent logs could also be backed up. The agent database is meant to store events that have been read by the agent, but not accepted by the remote destination yet. Losing the database means that any events in the database that were to be sent to the remote destination will be lost. In the case of data loss, loss of the agent database may be acceptable. If not, then the agent database can be backed up as well.
The important directories for the agent are:
- Linux
- /var/lib/loginsight-agent – client-side agent configuration and database
- /var/log/loginsight-agent – agent logs
- Windows
- %PROGRAMDATA%\VMware\Log Insight Agent – client-side agent configuration, database and logs
IMPORTANT: The %PROGRAMDATA% directory is often hidden by default. Be sure to show hidden files and folders.
Summary
As you can see, Log Insight is made of various components and supports a variety of deployment architectures. Now that you know what the components are, next I would like to discuss how to back them up.
UPDATE: Added information about the SSL certificate.
© 2015, Steve Flanders. All rights reserved.
