I was recently asked to assist in configuring a wildcard SSL certificate on a pair of vCloud Director (vCD) cells. While the certificate had been installed on the cells, some browsers were displaying SSL errors such as the following:
| While other browsers appeared to work: | Until you drilled down a little further: |
  |
 |
In addition, while uploading/downloading VMs SSL errors like the following were displayed:
So what was going on and how can it be fixed?
Troubleshooting
To start the troubleshooting process, I attempted to recurse the certificate chain using the openssl command line:
{codecitation class=”brush:bash”}$ openssl s_client -verify 5 -connect vcd.example.com:443
verify depth is 5
CONNECTED(00000003)
depth=0 /C=US/ST=Nevada/L=Las Vegas/O=Organization/CN=*.example.com
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 /C=US/ST=Nevada/L=Las Vegas/O=Organization/CN=*.example.com
verify error:num=27:certificate not trusted
verify return:1
depth=0 /C=US/ST=Nevada/L=Las Vegas/O=Organization/CN=*.example.com
verify error:num=21:unable to verify the first certificate
verify return:1
—
Certificate chain
0 s:/C=US/ST=Nevada/L=Las Vegas/O=Organization/CN=*.example.com
i:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert High Assurance CA-3{/codecitation}
What I noticed was that while the wildcard SSL certificate was seen no intermediate SSL certificates were. To explain what I mean, let me give a brief background on SSL: Every browser has a set list of known authoritative SSL certificates sometimes referred to as Root CA Certificates. These top-level certificates are the heart of SSL’s security and as such are kept under extremely limited access. As such, Intermediate root CA certificates are created and used to sign a customer’s Certificate Signing Request (CSR). Since Intermediate root CA certificates are not known by browsers they must be included on the server-side certificate.
To confirm the intermediate certificate was indeed the issue, I consulted the issuer of the certificate (http://www.digicert.com/help/):
? DNS resolves ‘vcd.example.com’ to 1.2.3.4 ? SSL certificate Common Name = *.example.com
Subject Alternative Names = *.example.com, example.com
Issuer = DigiCert High Assurance CA-3
Serial Number = XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
SHA1 Thumbprint = XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
Key Length = 2048 bit
Signature algorithm = SHA1 + RSA (good)
Secure Renegotiation: Supported
? This certificate does not use a vulnerable Debian key (this is good)
? SSL Certificate expiration
The certificate expires April 3, 2013 (349 days from today)
? Certificate Name matches vcd.example.com
Subject *.example.com
Valid from 29/Mar/2012 to 03/Apr/2013
Issuer DigiCert High Assurance CA-3
X The server is not sending the required intermediate certificate.
This server needs to be configured to include DigiCert’s intermediate certificates during SSL handshakes. You may not notice a problem when using Internet Explorer because it can follow the http link to the intermediate certificate embedded in the certificate’s ‘Authority Information Access’ extension, but Firefox, Safari, and other browsers will likely complain until the intermediate certificates are installed and configured on the server. For instructions on how to achieve this, please check the installation guide for your platform in the SSL certificate installation section of our site. If you have any problems correcting this issue, please contact our helpful support team and we would be happy to assist.
Fix Attempt #1
With the issue confirmed, I want back to the SSL configuration on the vCD cells. Unfortunately, the person who configured the cells was on vacation so I checked the VMware site for relevant documentation and came across KB#1026309. Step 3 stated to perform the following steps:
When you receive the signed certificates, import them into the keystore.
To import the Certification Authority’s root certificate into the keystore file, run the command:
keytool -storetype JCEKS -storepass passwd -keystore certificates.ks -import -alias root -file root.cer
To import the Certification Authority’s intermediate certificates into the keystore file, run the command:
keytool -storetype JCEKS -storepass passwd -keystore certificates.ks -import -alias intermediate -file intermediate.cer
To import the host-specific certificate for the HTTP service, run the command:
keytool -storetype JCEKS -storepass passwd -keystore certificates.ks -import -alias http -file http.cer
To import the host-specific certificate for the console proxy service, run the command:
keytool -storetype JCEKS -storepass passwd -keystore certificates.ks -import -alias consoleproxy -file consoleproxy.cer
NOTE: One important point of clarification is that the root.cer referred to above is the Intermediate root CA certificate and not the Root CA certificate. In my particular case the intermediate already included the Intermediate root CA and Intermediate certificate in the same file. You can validate this by opening the file supplied by the certificate issuer and ensuring you have at least two certificates in the file. Each certificate will start with a ‘—– BEGIN CERTIFICATE —–‘ header. As such, I broke the file into two separate files so I could follow the steps exactly.
What the KB article did not state was that once the new keystore had been created you need to reconfigure vCD to use it. A step 5 should be added that states something like the following:
In order for the keystore to be used each vCD cell needs to be reconfigured
Stop the vCD service by running the command:
service vmware-vcd stop
Reconfigure vCD by running the command:
/opt/vmware/vcloud-director/bin/configure
When prompted for the certificate enter the appropriate path. For example:
/opt/vmware/vcloud-director/jre/bin/certificates.ks
Ensure there are not errors and when prompted to start the cell, press “y” and hit enter.
Upon reconfiguring vCD, I received the following error: Cryptographic error. While it would still let me start the cell the error was enough to tell me something was wrong so I went back to the drawing board.
Fix Attempt #2
Next, I came across the following link: http://virtualyzation.com/?p=449.
Step 1 states to cat the root and intermediate certificates into the same file. Two important points of clarification in this step:
- The root.crt is the Intermediate root CA certificate and not the Root CA certificate. In my particular case the intermediate already included the Intermediate root CA and Intermediate certificate in the same file. You can validate this by opening the file supplied by the certificate issuer and ensuring you have at least two certificates in the file. Each certificate will start with a ‘—– BEGIN CERTIFICATE —–‘ header.
- The certificate.chain.txt is not some random empty file. It should actually be the signed certificate you received back from the certificate issuer (star_example_com in my case). This becomes important in step 2.
If your put the root and intermediate certificates into a file named certificate.chain.txt or some other file that does not contained your signed SSL certificate then when you run the command in step 2 you will receive the following error: No certificate matches private key. Step 2 should also note that you need to enter the pass phrase for the key.
Besides these clarifications, upon following the steps outlined in the above link the SSL issue was resolved:
Even the openssl command line looked a lot better:
{codecitation class=”brush:bash”}$ openssl s_client -verify 5 -connect vcd.example.com:443
verify depth is 5
CONNECTED(00000003)
depth=2 /C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert High Assurance EV Root CA
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=2 /C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert High Assurance EV Root CA
verify error:num=27:certificate not trusted
verify return:1
depth=1 /C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert High Assurance CA-3
verify return:1
depth=0 /C=US/ST=Nevada/L=Las Vegas/O=Organization/CN=*.example.com verify return:1
—
Certificate chain
0 s:/C=US/ST=Nevada/L=Las Vegas/O=Organization/CN=*.example.com
i:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert High Assurance CA-3
1 s:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert High Assurance CA-3
i:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert High Assurance EV Root CA
2 s:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert High Assurance EV Root CA
i:/C=US/O=Entrust.net/OU=www.entrust.net/CPS incorp. by ref. (limits liab.)/OU=(c) 1999 Entrust.net Limited/CN=Entrust.net Secure Server Certification Authority{/codecitation}
Additional Information
In case you are curious why the command line still states that the certificate is not trusted the issue is that the Root CA was not specified. Unlike the browser which has the Root CA builtin, the command line needs to be told were to locate the Root CA. For example, if you downloaded the Root CA to /etc/ssl/certs you could verify by running:
{codecitation class=”brush:bash”}$ openssl s_client -verify 5 -connect vcd.example.com:443 -CApath /etc/ssl/certsverify
depth is 5
CONNECTED(00000003)
depth=3 /C=US/O=Entrust.net/OU=www.entrust.net/CPS incorp. by ref. (limits liab.)/OU=(c) 1999 Entrust.net Limited/CN=Entrust.net Secure Server Certification Authority
verify return:1
depth=2 /C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert High Assurance EV Root CAverify return:1depth=1 /C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert High Assurance CA-3
verify return:1
depth=0 /C=US/ST=Nevada/L=Las Vegas/O=Organization/CN=*.example.com
verify return:1
—
Certificate chain
0 s:/C=US/ST=Nevada/L=Las Vegas/O=Organization/CN=*.example.com
i:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert High Assurance CA-3
1 s:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert High Assurance CA-3
i:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert High Assurance EV Root CA
2 s:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert High Assurance EV Root CA
i:/C=US/O=Entrust.net/OU=www.entrust.net/CPS incorp. by ref. (limits liab.)/OU=(c) 1999 Entrust.net Limited/CN=Entrust.net Secure Server Certification Authority{/codecitation}
NOTE: One final important note is that you should avoid SSL acceleration/offloading as it will not work with console proxy connections (VMRC) per http://www.vmware.com/files/pdf/vcat/Architecting-VMware-vCloud.pdf.
© 2012, Steve Flanders. All rights reserved.
