I just concluded a three part series on how to backup and restore Log Insight. I just realized that I missed how to backup and restore the SSL certificate on the Log Insight virtual appliance. I will address this oversight in this post and then update the previous posts.
As you know, Log Insight is primarily used through its HTML5 interface. By default, Log Insight ships with a unique SSL certificate per appliance. The Administration section of the Log Insight UI allows a user to upload a PEM certificate to use in place of the self-signed certificate. In this post, I would like to discuss operations you may desire to perform in regards to SSL in Log Insight as well as share a script on how to properly manage all the available options.
Why Certificates Matter in Log Insight
I think (and hope) we all know why SSL certificates are important. In the case of Log Insight, the SSL certificate option in the Administration section is used for two purposes:
- For connections to the GUI
- By anything — like the Log Insight agents — using the (ingestion) API (TCP/9543)
The second reason listed above is especially significant if you are using the integrated load balancer as agents could be sending traffic to any node in the cluster, which means that all nodes in the cluster need to have the same SSL certificate.
IMPORTANT: Remember, by default every virtual appliance has a unique SSL certificate. Even if you upload a custom SSL certificate through the Administration section, the workers will NOT get updated. To change the SSL certificates on workers you must directly access the workers UI and upload the certificate through the UI to each worker.
Certificate Requirements
Log Insight SSL certificates MUST adhere to the following guidelines:
- The PEM file contains both a valid private key and a valid certificate chain.
- The private key is generated by the RSA or the DSA algorithm.
- The private key is not encrypted by a pass phrase.
- If the certificate is signed by a chain of other certificates, all other certificates must be included in the PEM file that you plan to import.
- All the certificates and the private key that are included in the certificate file are must be PEM-encoded. DER, PFX, PKCS12, PKCS7, or other formats for certificates and private keys are not supported.
A couple notes:
- PEM-encoded means human readable
- The certificate file order matters, it should be: cat domain.crt domain.key domain.ca-bundle >>domain.pem
Desirable Certificate Operations
I suspect most people operating Log Insight or responsible for the security of Log Insight will care about the following SSL certificate operations:
- Backup: of the certificate and where it is used on the virtual appliance
- Check: when the certificate expires/expired and if certificates are the same on all nodes in a cluster
- Replace: replace the original self-signed certificate with a new certificate because of expiration or other security concern
- Restore: put the original self-signed certificate back in case of accidental upload or other security concern
- Upload: upload a signed certificate to replace the self-signed certificate
Backup can technically be handled by virtual appliance backups, checks can be performed by client browsers and uploading can be done from the Administration section of the Log Insight UI (on each node). Explicit replace and restore operations are not provided by Log Insight today though the tools necessary to perform these operations are available on the virtual appliance. The end result is that you may need to use multiple tools as well as know some CLI commands to perform all of the above operations.
Automated Certificate Management
To handle all of the desirable certificate operations, I have put together a CLI script that can be run. The script is meant to be run from a Log Insight virtual appliance and can be used as follows:
USAGE: ./li-ssl-cert.sh [--backup|--check [--short]|--replace|--restore|--upload <pem>]
WHERE:
--backup Backs up all keystore and SSL certificate information including
default and custom certificates.
--check [--short] Prints the current keystore and SSL certificate information. Use
the --short flag to just return the certificate SHA1 (most useful
when ensuring all nodes in a cluster have the same certificate).
--replace Creates and installs a new, self-signed SSL certificate. THIS
COMMAND RESTARTS LOG INSIGHT.
--restore Puts the default, self-signed SSL certificate that originally came
with the VA back. THIS COMMAND RESTARTS LOG INSIGHT.
--upload <pem> Attempts to install the specified PEM certificate. THIS COMMAND
RESTARTS LOG INSIGHT.
I would like to cover each of these options in more details.
DISCLAIMER: This script is not officially supported by VMware or me. Use at your own risk.
Backup
Given that SSL certificates are local to a virtual appliance, even in the case of a Log Insight cluster, it is important to ensure that the certificate is backed up. Now of course you should have a copy of the certificate outside of Log Insight anyway, but if you do not this option is for you. The backup copies all keystores and truststores from the Log Insight virtual appliance. In addition, the command will extract the SSL certificate from the Tomcat keystore and store the certificate in the backup. When the command has finished, a tarball available at /tmp/li-ssl-certs.tar.gz will be produced. Remember to store the tarball outside of the /tmp directory and ideally off of the virtual appliance.
# ./li-ssl-cert.sh --backup SSL certificates where successfully backed up to /tmp/li-ssl-certs.tar.gz
Check
The check commands dumps information about the Tomcat keystore as well as the SSL certificate within the keystore. This information is helpful in determining which certificate is currently in use as well as when it expires. The check command also has a short flag which can be used to return only the SHA1 for the SSL certificate in the Tomcat keystore. This flag is helpful when used in conjunction with the li_rexec command as it will make it easy to see if nodes in a cluster have the same SSL certificate or not — which is important when you are using the integrated load balancer and the ingestion API over SSL as described above.
# ./li-ssl-cert.sh --check === Keystore contents ========================================================================================= Keystore type: JKS Keystore provider: SUN Your keystore contains 1 entry loginsight, Apr 4, 2015, PrivateKeyEntry, Certificate fingerprint (SHA1): 22:A6:53:44:E4:22:55:AD:66:45:41:23:18:1A:9E:C0:43:EC:DA:F7 === Certificate contents ====================================================================================== Owner: CN=VMware vRealize Log Insight, OU=vRealize Log Insight, O="VMware, Inc.", L=Palo Alto, ST=California, C=US Issuer: CN=VMware vRealize Log Insight, OU=vRealize Log Insight, O="VMware, Inc.", L=Palo Alto, ST=California, C=US Serial number: 29a0410c Valid from: Sat Apr 04 03:27:51 UTC 2015 until: Tue Apr 01 03:27:51 UTC 2025 Certificate fingerprints: MD5: CA:30:8D:89:C3:1B:18:48:15:B2:23:70:25:A3:2E:A7 SHA1: 22:A6:53:44:E4:22:55:AD:66:45:41:23:18:1A:9E:C0:43:EC:DA:F7 SHA256: 6F:59:DB:86:1E:54:4D:27:DA:8D:9E:D9:63:64:79:38:1F:6D:72:70:DC:9E:D3:71:9A:A9:BA:2F:37:1D:4B:43 Signature algorithm name: SHA256withRSA Version: 3 Extensions: #1: ObjectId: 2.5.29.14 Criticality=false SubjectKeyIdentifier [ KeyIdentifier [ 0000: 04 4B 0F 4C 07 B5 C6 C9 6E 31 AC 60 7D DA D4 6A .K.L....n1.`...j 0010: 3A E1 98 60 :..` ] ]
Replace
WARNING: This command requires a restart of the Log Insight service to take effect and would need to be run on all nodes if using the integrated load balancer and the ingestion API over SSL.
The replace command is primarily available to replace the default self-signed certificate with a new self-signed certificate. Two primary reasons come to mind on why you might want to do this:
- Because the original self-signed certificate has expired — the original certificate is good for 10 years so it is unlikely this is your reason today.
- You wish to get a vRealize Log Insight self-signed certificate instead of a vCenter Log Insight self-signed certificate.
Now, you may also wish to change the self-signed certificate in some way. If this is the case you will need to modify this script to meet your needs.
# ./li-ssl-cert.sh --replace
USAGE: ./li-ssl-cert.sh --replace --force
WARNING: This command will restart the Log Insight service.
This command should be run on every node in a cluster.
# ./li-ssl-cert.sh --replace --force
Backing up existing certificates...done
Generating self-signed SSL certificate keystore...done
Exporting self-signed SSL certificate...done
Generating certificate authority...done
Moving certificate stores into place...done
Cleaning up exported SSL certificate...done
Stopping Log Insight...
Log Insight was stopped successfully.
Starting Log Insight...
Log Insight started successfully.
Web application running on port 80.
Restore
WARNING: This command requires a restart of the Log Insight service to take effect and would need to be run on all nodes if using the integrated load balancer and the ingestion API over SSL.
The restore command allows you to go back to the default self-signed certificate that existed when you deployed the virtual appliance. This command may be helpful if you are testing signed certificates.
# ./li-ssl-cert.sh --restore Default certificate already in place...exiting
# ./li-ssl-cert.sh --restore
USAGE: ./li-ssl-cert.sh --restore --force
WARNING: This command will restart the Log Insight service.
This command should be run on every node in a cluster.
# ./li-ssl-cert.sh --restore --force
Backing up existing certificate...done
Restoring default certificate...done
Stopping Log Insight...
Log Insight was stopped successfully.
Starting Log Insight...
Log Insight started successfully.
Web application running on port 80.
Upload
WARNING: This command requires a restart of the Log Insight service to take effect and would need to be run on all nodes if using the integrated load balancer and the ingestion API over SSL.
The upload command will take a PEM file, that you would have otherwise uploaded through the Administration section of the Log Insight UI, and attempt to install it. The benefits of using this option instead of the UI are:
- Easier to upload to all nodes in a cluster — you could also automate against the UI using cURL commands instead of using this script
- Clearer error messages — when an upload attempt fails on the UI you will typically see “Failed to upload certificate”. While the reason for the failure is always one of the certificate requirements above, the exact reason is unknown. Through the CLI, the error message tells you specifically what failed.
# ./li-ssl-cert.sh --upload test1.pem
USAGE: ./li-ssl-cert.sh --upload <pem> --force
WARNING: This command will restart the Log Insight service.
This command should be run on every node in a cluster.
# ./li-ssl-cert.sh --upload test1.pem --force
Checking for PEM file...done
Checking for previously uploaded certificate...not found
Attempting to install new certificate...Unable to find the private key or private key in wrong format
Invalid PEM file
mv: cannot stat `/usr/lib/loginsight/application/3rd_party/apache-tomcat-6.0.36/conf/keystore.bak': No such file or directory
rm: cannot remove `/usr/lib/loginsight/application/3rd_party/apache-tomcat-6.0.36/conf/keystore.pkcs12': No such file or directory
Verify that your custom SSL certificate meets the following requirements.
1. The certificate file contains both a valid private key and a valid certificate chain.
2. The private key is generated by the RSA or the DSA algorithm.
3. The private key is not encrypted by a pass phrase.
4. If the certificate is signed by a chain of other certificates, all other certificates must be included in the certificate file that you plan to import.
5. All the certificates and the private key that are included in the certificate file are must be PEM-encoded. DER-encoded certificates and private keys are not supported.
6. All the certificates and the private key that are included in the certificate file must be in the PEM format. Certificates in the PFX, PKCS12, PKCS7, or other formats are not supported.
# ./li-ssl-cert.sh --upload ccbill-cacert.pem --force
Checking for PEM file...done
Checking for previously uploaded certificate...found
Backing up previously uploaded certificate...done
Attempting to install new certificate...done
Stopping Log Insight...
Log Insight was stopped successfully.
Starting Log Insight...
Log Insight started successfully.
Web application running on port 80.
Script
And with that, here is the script:
#!/usr/bin/env bash
DATE=$(date +%s)
DEFAULT_KEYSTORE_STOREPASS=vmware
KEYSTORE_STOREPASS=`grep keystorePass /usr/lib/loginsight/application/etc/3rd_config/server.xml 2>/dev/null | gawk -F'"' '{print $4}' || echo ${DEFAULT_KEYSTORE_STOREPASS}`
TOMCAT=$(ls /usr/lib/loginsight/application/3rd_party 2>/dev/null | grep tomcat)
[ ! -d "/usr/lib/loginsight/application/3rd_party/${TOMCAT}" ] && \
echo >/dev/stderr "ERROR: Unable to locate Tomcat directory, you must run this on the Log Insight virtual appliance...exiting" && \
exit 255
backup() {
BACKUPDIR="/tmp/li-ssl-certs"
BACKUPTAR="/tmp/li-ssl-certs.tar.gz"
for X in $BACKUPDIR $BACKUPTAR; do
if [ -d "$X" -o -f "$X" ]; then
echo "ERROR: $X already exists. Please remove and try again."
exit 1
fi
done
mkdir $BACKUPDIR
cp /usr/lib/loginsight/application/3rd_party/$TOMCAT/conf/custom* $BACKUPDIR
cp /usr/lib/loginsight/application/3rd_party/$TOMCAT/conf/keystore* $BACKUPDIR 2>&1
cp /usr/lib/loginsight/application/etc/truststore $BACKUPDIR 2>&1
check
tar cfz /tmp/li-ssl-certs.tar.gz $BACKUPDIR
rm -rf $BACKUPDIR
echo "SSL certificates where successfully backed up to /tmp/li-ssl-certs.tar.gz"
exit 0
}
check() {
if [ "$1" == "--short" ]; then
if [ ! -z "$2" ]; then
echo $KEYSTORE_STOREPASS | /usr/java/default/bin/keytool -list -keystore $2 | grep "Certificate fingerprint" | awk '{split($0,a," "); print a[4]}'
else
echo $KEYSTORE_STOREPASS | /usr/java/default/bin/keytool -list -keystore /usr/lib/loginsight/application/3rd_party/$TOMCAT/conf/keystore | grep "Certificate fingerprint" | awk '{split($0,a," "); print a[4]}'
fi
exit 0
fi
if [ -z "$BACKUPDIR" ]; then
echo "=== Keystore contents ========================================================================================="
echo ""
echo $KEYSTORE_STOREPASS | /usr/java/default/bin/keytool -list -keystore /usr/lib/loginsight/application/3rd_party/$TOMCAT/conf/keystore
fi
ALIAS=$(echo $KEYSTORE_STOREPASS | /usr/java/default/bin/keytool -list -keystore /usr/lib/loginsight/application/3rd_party/$TOMCAT/conf/keystore | grep PrivateKeyEntry | awk '{split($0,a,","); print a[1]}')
if [ -z "$BACKUPDIR" ]; then echo ""; fi
echo $KEYSTORE_STOREPASS | /usr/java/default/bin/keytool -alias $ALIAS -export -file /tmp/$ALIAS.crt -keystore /usr/lib/loginsight/application/3rd_party/$TOMCAT/conf/keystore
if [ -z "$BACKUPDIR" ]; then
echo "=== Certificate contents ======================================================================================"
echo ""
/usr/java/default/bin/keytool -printcert -v -file /tmp/$ALIAS.crt
rm -rf /tmp/$ALIAS.crt
exit 0
else
cp /tmp/$ALIAS.crt $BACKUPDIR
rm -rf /tmp/$ALIAS.crt
return
fi
}
replace() {
if [ "$1" != "--force" ]; then
echo "USAGE: $0 --replace --force"
echo "WARNING: This command will restart the Log Insight service."
echo " This command should be run on every node in a cluster."
exit 2
fi
echo -n "Backing up existing certificates..."
cp /usr/lib/loginsight/application/etc/3rd_config/keystore /usr/lib/loginsight/application/etc/3rd_config/keystore.$DATE 2>&1
KEYSTORE=$(ls /usr/lib/loginsight/application/3rd_party/$TOMCAT/conf/keystore)
cp $KEYSTORE $KEYSTORE.$DATE 2>&1
cp /usr/lib/loginsight/application/etc/truststore /usr/lib/loginsight/application/etc/truststore.$DATE 2>&1
echo "done"
echo -n "Generating self-signed SSL certificate keystore..."
/usr/java/default/bin/keytool -genkey -alias loginsight -keyalg RSA -validity 3650 -keysize 4096 -keystore /tmp/keystore -keypass $KEYSTORE_STOREPASS -storepass $KEYSTORE_STOREPASS -dname "CN=VMware vRealize Log Insight, OU=vRealize Log Insight, O=VMware\, Inc., L=Palo Alto, S=California, C=US" 2>&1
echo "done"
echo -n "Exporting self-signed SSL certificate..."
/usr/java/default/bin/keytool -export -alias loginsight -file /tmp/loginsight.cer -keystore /tmp/keystore -storepass $KEYSTORE_STOREPASS >/dev/null 2>&1
echo "done"
echo -n "Generating certificate authority..."
/usr/java/default/bin/keytool -import -noprompt -alias loginsight -v -trustcacerts -file /tmp/loginsight.cer -keystore /tmp/truststore -keypass $KEYSTORE_STOREPASS -storepass $KEYSTORE_STOREPASS >/dev/null 2>&1
echo "done"
echo -n "Moving certificate stores into place..."
cp /tmp/keystore /usr/lib/loginsight/application/etc/3rd_config/keystore
mv /tmp/keystore /usr/lib/loginsight/application/3rd_party/$TOMCAT/conf/keystore
mv /tmp/truststore /usr/lib/loginsight/application/etc/truststore
echo "done"
echo -n "Cleaning up exported SSL certificate..."
rm -rf /tmp/loginsight.cer
echo "done"
service loginsight restart
exit 0
}
restore() {
SHA1=$(check --short)
SHA2=$(check --short /usr/lib/loginsight/application/etc/3rd_config/keystore)
if [ "$SHA1" == "$SHA2" ]; then echo "Default certificate already in place...exiting"; exit 2; fi
if [ "$1" != "--force" ]; then
echo "USAGE: $0 --restore --force"
echo "WARNING: This command will restart the Log Insight service."
echo " This command should be run on every node in a cluster."
exit 2
fi
echo -n "Backing up existing certificate..."
cp /usr/lib/loginsight/application/etc/3rd_config/keystore /usr/lib/loginsight/application/etc/3rd_config/keystore.$DATE 2>&1
echo "done"
echo -n "Restoring default certificate..."
cp /usr/lib/loginsight/application/etc/3rd_config/keystore /usr/lib/loginsight/application/3rd_party/$TOMCAT/conf/keystore
echo "done"
service loginsight restart
exit 0
}
upload() {
if [ "$2" != "--force" ]; then
echo "USAGE: $0 --upload <pem> --force"
echo "WARNING: This command will restart the Log Insight service."
echo " This command should be run on every node in a cluster."
exit 2
fi
echo -n "Checking for PEM file..."
PEM=$1
if [ ! -f "$PEM" ]; then
echo "file specified not found"
exit 1
fi
echo "done"
echo -n "Checking for previously uploaded certificate..."
CUSTOM=$(ls /usr/lib/loginsight/application/3rd_party/$TOMCAT/conf/custom.pem)
if [ -f "$CUSTOM" ]; then
echo "found"
echo -n "Backing up previously uploaded certificate..."
cp /usr/lib/loginsight/application/3rd_party/$TOMCAT/conf/custom.pem /usr/lib/loginsight/application/3rd_party/$TOMCAT/conf/custom.pem.$DATE
cp /usr/lib/loginsight/application/3rd_party/$TOMCAT/conf/custom-key.pem /usr/lib/loginsight/application/3rd_party/$TOMCAT/conf/custom-key.pem.$DATE
echo "done"
else
echo "not found"
fi
echo -n "Attempting to install new certificate..."
cp $PEM /usr/lib/loginsight/application/3rd_party/$TOMCAT/conf/custom.pem
/usr/lib/loginsight/application/sbin/custom-ssl-cerf 2>&1
if [ "$?" != "1" ]; then
echo "done"
service loginsight restart
exit 0
else
echo ""
echo "Verify that your custom SSL certificate meets the following requirements."
echo ""
echo "1. The certificate file contains both a valid private key and a valid certificate chain."
echo "2. The private key is generated by the RSA or the DSA algorithm."
echo "3. The private key is not encrypted by a pass phrase."
echo "4. If the certificate is signed by a chain of other certificates, all other certificates must be included in the certificate file that you plan to import."
echo "5. All the certificates and the private key that are included in the certificate file are must be PEM-encoded. DER-encoded certificates and private keys are not supported."
echo "6. All the certificates and the private key that are included in the certificate file must be in the PEM format. Certificates in the PFX, PKCS12, PKCS7, or other formats are not supported."
exit 1
fi
}
func=$(echo $1 | awk '{split($0,a,"-"); print a[3]}')
$func $2 $3 2>/dev/null
echo "USAGE: $0 [--backup|--check [--short]|--replace|--restore|--upload <pem>]"
echo "WHERE:"
echo " --backup Backs up all keystore and SSL certificate information including"
echo " default and custom certificates."
echo " --check [--short] Prints the current keystore and SSL certificate information. Use"
echo " the --short flag to just return the certificate SHA1 (most useful"
echo " when ensuring all nodes in a cluster have the same certificate)."
echo " --replace Creates and installs a new, self-signed SSL certificate. THIS"
echo " COMMAND RESTARTS LOG INSIGHT."
echo " --restore Puts the default, self-signed SSL certificate that originally came"
echo " with the VA back. THIS COMMAND RESTARTS LOG INSIGHT."
echo " --upload <pem> Attempts to install the specified PEM certificate. THIS COMMAND"
echo " RESTARTS LOG INSIGHT."
exit 2
© 2015, Steve Flanders. All rights reserved.
Will be using this, thanks!
Awesome — let me know if you need any additional functions!
Hi – Thanks for the script. Pretty helpful. When I run this script on LI ver 3.6, the cert upload just fine, with no visible errors. The –check option also displays the correct thumbprint. But after a minute or so, I see the original (default) keystore being replaced. This does not happen when cert is uploaded via the UI.
Any ideas ?
Thanks,
Hey Vidya — thanks for the comment. The script should work. The replacement a minute later would indicate a sync issue with the upload. Let me investigate and get back to you.
Can you please share the script
See the end of the post — the script is embedded