If you started with Log Insight 1.x or with an environment that a single Log Insight node could handle then you likely are running a standalone node. Upon upgrading to Log Insight 2.x, when ingestion increases above the amount a single node can handle or when business requirements change (e.g. requiring ingestion HA), moving from a standalone Log Insight instance to a cluster may become necessary. While building a cluster, even with an already leveraged standalone node, is very straightforward, there are several considerations to take into account. In this post, I will walk through these considerations as well as the process to perform the transformation.
Before you being
Before starting, you should to take a look at how your environment is configured. Questions you should answer include:
- How many standalone Log Insight instances are you running today?
- Are all instances running in the same datacenter?
- If running multiple standalone instances, does this provide any advantage (e.g. knowing which environment logs are coming from)?
- Are all devices connecting to Log Insight via a FQDN?
- How easy it is for you to change the FQDN on the devices?
While I will talk about reference architectures in a future post, here are some important notes:
- For most environments, running multiple, separate Log Insight instances should not be necessary
- A Log Insight cluster must be in the same data center and same layer 2 network
- If you have multiple datacenters then you should consider using Log Insight forwarders in each datacenter — depending on business requirements the forwarder in each datacenter may need to be a cluster
- You cannot join already configured standalone nodes together, but you can join new nodes to an already configured standalone node
- If devices are not connecting to Log Insight via FQDN then this should be changed
- Since standalone nodes do not provide redundancy, any downtime to the standalone node will result in an outage
Before modifying any existing instances, you need to prepare the additional changes required: At least three more IPs addresses with FQDNs for the nodes (1 for existing standalone and 2 for workers given that a 3-node cluster is the minimum supported today.
Process
You are now ready to perform the transformation:
- Deploy two new Log Insight OVAs with static IP addresses
- Go to the UI of each new OVA and select Join Existing Deployment then leave them for now
- Shutdown the standalone node
- Edit the settings of the standalone node to change the IP and FQDN
- Power on the standalone node
WARNING: If the standalone node has been running for several months than during start it may go through a forced fsck. The size of the virtual disks attached to the node dictates the time it takes to complete the fsck.
- Confirm the standalone node’s UI is accessible and you can log in
- Go the each new worker UI and submit the request to join the standalone node
- Go back to the standalone node’s UI and navigate to Administration > Cluster
- Accept each join request
NOTE: While the UI allows you to connect multiple workers at the same time, I recommend adding them one at a time and waiting two minutes between each join accept.
- Enable ILB and enter the old standalone node IP as the VIP
That’s it! Easy, right?
© 2015, Steve Flanders. All rights reserved.
