Replacing SSL certificates for vRealize Orchestrator Appliance

It’s a common practice to replace self-signed certificates, that are used in several VMware products, with CA signed certificates. I did this in my lab for my vCenter Server Appliance and my VMware Update Manager. While I was working with vRealize Orchestrator I noticed, that it is also using self-signed certificates (what else?). For completeness, I decided to replace the self-signed certificates with CA signed.

My lab environment

1. VMware vSphere 5.5 environment running a vCenter Server appliance (already using CA signed certificates)
2. vRealize Orchestrator Appliance 5.5.2 (not version 5.5.2.1,  because I had problems with this release)
3. Microsoft Windows CA running on a Windows 2012 R2 Standard server

You don’t need a Microsoft Windows CA. You can use any other CA. There is no need to use a special vendor. I use a windows-based CA in my lab, so the screenshots reflect this fact. The way how certificates are replaced differs between vRealize Orchestrator Appliance and the windows-based standalone or vCenter Server embedded version. If you use the in the vCenter Server embedded or Standalone Orchestrator check Derek Seamans VMware vSphere 5.5 SSL Toolkit. I used the Orchestrator appliance.

I will only highlight the necessary steps to replace the certificates. I assume that you have a running Orchestrator appliance.

Create the package signing certificate

This certificate is used to sign packages. This certificate is NOT used with HTTPS.

1. Log into the Orchestrator Configuration website using the username “vmware” and click “Server Certificate” on the left navigation page. On the right side appears the server package signing certificate.

2. Create a new certificate. Otherwise, if you directly export the CSR, the CSR would include the organization, common name, OU etc. from the self-signed certificate. Choose the fourth option “Create a certificate database and self-signed server certificate”.

3. Enter at least the common name (FQDN of your Orchestrator appliance) and click “Create” (on the right at the end of the page).

4. Now the CSR can be exported. The CSR is saved into a file called “vCO_SigningRequest.csr”.

5. Take the CSR and submit a certificate request at your CA. In my case I took the content of the file and copied it into the corresponding text box of my CA. Make sure that you only use the content between “—–BEGIN NEW CERTIFICATE REQUEST—–” and “—–END NEW CERTIFICATE REQUEST—–”. I used a customized certificate template (check Derek Seamans blog for more information about VMware and SSL certificates!).

6. Download the Base 64 encoded certificate and give it a meaningful name (certnew.cer is NOT meaningful…).

Import the CA certificate

1. Now we have to import the CA certificate. Otherwise we would get an error message when we try to import the CA signed certificate. If you use a Microsoft CA, you can get the CA certificate from the “Active Directory Certificate Services” website. Simply click “Download a CA certificate, certificate chain, or CRL” from the “Select a task:” list. Then save the Base 64 encoded certificate file by choosing “Download CA certificate”. Give the file a meaningful name.

2. Start the Orchestrator Client and login with an account, that has administrator privileges. In my case this is my domain-admin account (Administrator@lab.local) which is member of the Orchestrator administrator group.

3. Select “Tools” > “Certificate manager…” from the right top of the Orchestrator client.

4. Click “Import certificate…”, choose the certificate file you saves some seconds ago and import it.

That’s it. Now we can move forward and replace the package signing certificate.

Replace the package signing certificate

1. Switch back to the Orchestrator Configuration website and choose the third option: “Import a certificate signing request signed by a certificate authority”.

2. Choose the saved certificate for your Orchestrator appliance and click “Import” (on the right at the end of the page).

That’s it! The package signing certificate is now replaced by a CA signed one.

As I already wrote: This certificate is not used to secure HTTPS. To get rid of the certificate warning when using the Orchestrator Client or the Orchestrator Configuration website, we need some additional steps.

Replace the client certificate

This certificate is used to for HTTPS. After replacing this certificate, the certificate warning for the Orchestrator configuration page (port 8283), the application page (port 8281) and the appliance management page (port 5480) should disappear.

These steps can’t be done using the Orchestrator Configuration, or the appliance management website. Let’s start a SSH session to the Orchestrator appliance.

1. Use SSH, connect to the Orchestrator appliance and login with root credentials. Change to the directory /etc/vco/app-server/security and take a backup of the Java Keystore (JKS).

vco:~ # cd /etc/vco/app-server/security
vco:/etc/vco/app-server/security # cp -a jssecacerts jssecacerts.old

2. Stop the Orchestrator service

vco:/etc/vco/app-server/security # service vco-server stop
Stopping tcServer
Instance is running as PID=3454, shutting down...
Instance is running PID=3454, sleeping for up to 60 seconds waiting for shutdown
Instance shut down gracefully

3. The utility “keytool” is used to manage the Java Keystore. The certificate we want to replace has the alias “dunes”. The password for the Java Keystore is “dunesdunes”. This password is valid for every Orchestrator installation! Before we can create a new keypair and export the CSR, the old key needs to be removed from the Java Keystore.

vco:/etc/vco/app-server/security # keytool -delete -alias dunes -keystore jssecacerts -storepass dunesdunes

4. Now a new keypair must be created.

vco:/etc/vco/app-server/security # keytool -keystore jssecacerts -storepass dunesdunes -genkey -alias dunes -keyalg RSA -sigalg SHA512withRSA
What is your first and last name?
[Unknown]: vco.lab.local
What is the name of your organizational unit?
[Unknown]: Lab
What is the name of your organization?
[Unknown]: vcloudnine.de
What is the name of your City or Locality?
[Unknown]:
What is the name of your State or Province?
[Unknown]:
What is the two-letter country code for this unit?
[Unknown]: DE
Is CN=vco.lab.local, OU=Lab, O=vcloudnine.de, L=Unknown, ST=Unknown, C=DE correct?
[no]: yes

Enter key password for <dunes>
(RETURN if same as keystore password):

Make sure that you hit RETURN keytool asks for the password! Just accept, that the same password is used as for the Java Keystore. btw: “dunes” is a hint to the company who originally developed the Orchestrator. This compay was bought by VMware some years ago.

5. Export the CSR to a file.

vco:/etc/vco/app-server/security # keytool -keystore jssecacerts -storepass dunesdunes -certreq -alias dunes -file vco-dunes.csr

You can copy the file to your CA by using SCP. Otherwise use a simple cat and copy the content between “—–BEGIN NEW CERTIFICATE REQUEST—–” and “—–END NEW CERTIFICATE REQUEST—–” directly into the corresponding text box of the CA.

vco:/etc/vco/app-server/security # cat vco-dunes.csr
-----BEGIN NEW CERTIFICATE REQUEST-----
MIICpDCCAmICAQAwbzELMAkGA1UEBhMCREUxEDAOBgNVBAgTB1Vua25vd24xEDAOBgNVBAcTB1Vu
a25vd24xFjAUBgNVBAoTDXZjbG91ZG5pbmUuZGUxDDAKBgNVBAsTA0xhYjEWMBQGA1UEAxMNdmNv
---snip---
WU/mcQcQgYC0SRZxI+hMKBYTt88JMozIpuE8FnqLVHyNKOCjrh4rs6Z1kW6jfwv6ITVi8ftiegEk
O8yk8b6oUZCJqIPf4VrlnwaSi2ZegHtVJWQBTDv+z0kqA4GFAAKBgQCU1o/X0gdOU7AECclXb7FM
WJlxNSNs2mJfvvsXjh+hinY+SNA3k1QnZ2oLdlW/BM81KlQgO3i4tS1R08WC9UJ0VeEDPhNWkD0V
LRPPCfjT1jTdo+UFsWIJz/XAX0ATiVyDVnktToRTXuFaKUYTz7eU80H0Wp67pr1L0oV5mr5Q5aAw
MC4GCSqGSIb3DQEJDjEhMB8wHQYDVR0OBBYEFMWW1WR7ropcYC5jzBxicMi/LL3lMAsGByqGSM44
BAMFAAMvADAsAhRJYkO/JB7xnjYGlC9b5TYJJIQDXAIUCpCX4p/MV85glncQpc2deDlUfyY=
-----END NEW CERTIFICATE REQUEST-----

6. Use the CSR to issue a new certificate.

1. Download the Base 64 encloded certificate.

8. Copy the certificate (using SCP) to the Orchestrator appliance, e.g. to /root or /etc/vco/app-server/security. Depending on the path, you have to change the “-file” parameter! I’ve copied the certificate to /etc/vco/app-server/security.

vco:/etc/vco/app-server/security # keytool -keystore jssecacerts -storepass dunesdunes -importcert -alias dunes -file vco-client-cert.cer
Certificate reply was installed in keystore

Please note that you also have to import the CA certificate into the Java Keystore! In my case, the CA certificate was already imported during the initial certificate import from my vCenter Server Appliance, where I also use CA signed certificates. You can import the CA certificate using the “SSL Tab” on the Orchestrator Configuration website.

9. Start the Orchestrator service.

service vco-server start

10. Navigate to the Orchestrator website and check the success of the certificate import.

I still got a certificate warning when starting the Orchestrator client. But I am sure that this behavior is due to Java, because Java doesn’t know the CA.

Replace the appliance management website certificate

The appliance management (port 5480) is also secured with HTTPS. By default the certificate and private key are stored in a PEM file (the file is not protected by a passphrase), which is located at /opt/vmware/etc/lighttpd/server.pem. The PEM file includes the certificate AND the private key. It’s a bit tricky to export a PEM file with the private key from the Java Keystore.

1. First of all: Backup the old PEM file. I assume that you are still logged in on the Orchestrator appliance and still located at /etc/vco/app-server/security.

vco:/etc/vco/app-server/security # cp -a /opt/vmware/etc/lighttpd/server.pem /opt/vmware/etc/lighttpd/server.pem.old

2. Export the dunes key from the Java Keystore to a PKCS#12 store.

vco:/etc/vco/app-server/security # keytool -importkeystore -srckeystore jssecacerts -destkeystore dunes.p12 -deststoretype PKCS12 -srcalias dunes

3. Export a PEM file from the PCKS12 keystore. Make sure that you add the “-nodes” parameter.

vco:/etc/vco/app-server/security # openssl pkcs12 -in dunes.p12 -passin pass:dunesdunes -out server.pem -nodes
MAC verified OK

4. Copy the PEM file to /opt/vmware/etc/lighttpd/server.pem.

vco:/etc/vco/app-server/security # cp server.pem /opt/vmware/etc/lighttpd/server.pem

5. Restart the lighttpd.

vco:/etc/vco/app-server/security # service vami-lighttp restart
Shutting down vami-lighttpd:done.
Starting vami-lighttpd:2014-11-30 14:57:09: (/build/mts/release/bora-1191928/vadk/src/vami/apps/lighttpd/1.4.29/src/network.c.239) warning: please use server.use-ipv6 only for hostnames, not without server.bind / empty address; your config will break if the kernel default for IPV6_V6ONLY changes
done.

You can safly ignore the warning. Check the state of the daemon using this command:

vco:/etc/vco/app-server/security # service vami-lighttp status
Checking vami-lighttpd status: 5838 ? 00:00:00 vami-lighttpd

Lighttpd is running.

6. Check the status of the appliance management website.

Congratulations! The certificate is working.

Final words

As always, working with certificates is challenging. My first attempts have cost me an entire Sunday, especially because the documentation didn’t cover all aspects. I hope this blog post helps you to get through the certificate jungle. Feel free to provide feedback!