Log Insight: SSL Certificate Management

I just concluded a three part series on how to backup and restore Log Insight. I just realized that I missed how to backup and restore the SSL certificate on the Log Insight virtual appliance. I will address this oversight in this post and then update the previous posts.

As you know, Log Insight is primarily used through its HTML5 interface. By default, Log Insight ships with a unique SSL certificate per appliance. The Administration section of the Log Insight UI allows a user to upload a PEM certificate to use in place of the self-signed certificate. In this post, I would like to discuss operations you may desire to perform in regards to SSL in Log Insight as well as share a script on how to properly manage all the available options.

li-logo

Why Certificates Matter in Log Insight

I think (and hope) we all know why SSL certificates are important. In the case of Log Insight, the SSL certificate option in the Administration section is used for two purposes:

  1. For connections to the GUI
  2. By anything — like the Log Insight agents — using the (ingestion) API (TCP/9543)

The second reason listed above is especially significant if you are using the integrated load balancer as agents could be sending traffic to any node in the cluster, which means that all nodes in the cluster need to have the same SSL certificate.

IMPORTANT: Remember, by default every virtual appliance has a unique SSL certificate. Even if you upload a custom SSL certificate through the Administration section, the workers will NOT get updated. To change the SSL certificates on workers you must directly access the workers UI and upload the certificate through the UI to each worker.

Certificate Requirements

Log Insight SSL certificates MUST adhere to the following guidelines:

  1. The PEM file contains both a valid private key and a valid certificate chain.
  2. The private key is generated by the RSA or the DSA algorithm.
  3. The private key is not encrypted by a pass phrase.
  4. If the certificate is signed by a chain of other certificates, all other certificates must be included in the PEM file that you plan to import.
  5. All the certificates and the private key that are included in the certificate file are must be PEM-encoded. DER, PFX, PKCS12, PKCS7, or other formats for certificates and private keys are not supported.

A couple notes:

  • PEM-encoded means human readable
  • The certificate file order matters, it should be: cat domain.crt domain.key domain.ca-bundle >>domain.pem

Desirable Certificate Operations

I suspect most people operating Log Insight or responsible for the security of Log Insight will care about the following SSL certificate operations:

  1. Backup: of the certificate and where it is used on the virtual appliance
  2. Check: when the certificate expires/expired and if certificates are the same on all nodes in a cluster
  3. Replace: replace the original self-signed certificate with a new certificate because of expiration or other security concern
  4. Restore: put the original self-signed certificate back in case of accidental upload or other security concern
  5. Upload: upload a signed certificate to replace the self-signed certificate

Backup can technically be handled by virtual appliance backups, checks can be performed by client browsers and uploading can be done from the Administration section of the Log Insight UI (on each node). Explicit replace and restore operations are not provided by Log Insight today though the tools necessary to perform these operations are available on the virtual appliance. The end result is that you may need to use multiple tools as well as know some CLI commands to perform all of the above operations.

Automated Certificate Management

To handle all of the desirable certificate operations, I have put together a CLI script that can be run. The script is meant to be run from a Log Insight virtual appliance and can be used as follows:

I would like to cover each of these options in more details.

DISCLAIMER: This script is not officially supported by VMware or me. Use at your own risk.

Backup

Given that SSL certificates are local to a virtual appliance, even in the case of a Log Insight cluster, it is important to ensure that the certificate is backed up. Now of course you should have a copy of the certificate outside of Log Insight anyway, but if you do not this option is for you. The backup copies all keystores and truststores from the Log Insight virtual appliance. In addition, the command will extract the SSL certificate from the Tomcat keystore and store the certificate in the backup. When the command has finished, a tarball available at /tmp/li-ssl-certs.tar.gz will be produced. Remember to store the tarball outside of the /tmp directory and ideally off of the virtual appliance.

Check

The check commands dumps information about the Tomcat keystore as well as the SSL certificate within the keystore. This information is helpful in determining which certificate is currently in use as well as when it expires. The check command also has a short flag which can be used to return only the SHA1 for the SSL certificate in the Tomcat keystore. This flag is helpful when used in conjunction with the li_rexec command as it will make it easy to see if nodes in a cluster have the same SSL certificate or not — which is important when you are using the integrated load balancer and the ingestion API over SSL as described above.

Replace

WARNING: This command requires a restart of the Log Insight service to take effect and would need to be run on all nodes if using the integrated load balancer and the ingestion API over SSL.

The replace command is primarily available to replace the default self-signed certificate with a new self-signed certificate. Two primary reasons come to mind on why you might want to do this:

  1. Because the original self-signed certificate has expired — the original certificate is good for 10 years so it is unlikely this is your reason today.
  2. You wish to get a vRealize Log Insight self-signed certificate instead of a vCenter Log Insight self-signed certificate.

Now, you may also wish to change the self-signed certificate in some way. If this is the case you will need to modify this script to meet your needs.

Restore

WARNING: This command requires a restart of the Log Insight service to take effect and would need to be run on all nodes if using the integrated load balancer and the ingestion API over SSL.

The restore command allows you to go back to the default self-signed certificate that existed when you deployed the virtual appliance. This command may be helpful if you are testing signed certificates.

Upload

WARNING: This command requires a restart of the Log Insight service to take effect and would need to be run on all nodes if using the integrated load balancer and the ingestion API over SSL.

The upload command will take a PEM file, that you would have otherwise uploaded through the Administration section of the Log Insight UI, and attempt to install it. The benefits of using this option instead of the UI are:

  1. Easier to upload to all nodes in a cluster — you could also automate against the UI using cURL commands instead of using this script
  2. Clearer error messages — when an upload attempt fails on the UI you will typically see “Failed to upload certificate”. While the reason for the failure is always one of the certificate requirements above, the exact reason is unknown. Through the CLI, the error message tells you specifically what failed.

Script

And with that, here is the script:

© 2015 – 2016, Steve Flanders. All rights reserved.

4 thoughts on “Log Insight: SSL Certificate Management

  1. Vidya Nayak says:

    Hi – Thanks for the script. Pretty helpful. When I run this script on LI ver 3.6, the cert upload just fine, with no visible errors. The –check option also displays the correct thumbprint. But after a minute or so, I see the original (default) keystore being replaced. This does not happen when cert is uploaded via the UI.
    Any ideas ?

    Thanks,

Leave a Reply