HTTPS does not initiate

[VincentHermes]

ISSUE TYPE

• Other / HTTPS

COMPONENT NAME

• Webserver

CLOUDSTACK VERSION

• 4.14

CONFIGURATION

• server.properties:
  https.enable=true
https.port=8443
https.keystore=/etc/cloudstack/management/thecurrentkeystore.pkcs12
https.keystore.password=currentkeystorepassword

• Firewall disabled

OS / ENVIRONMENT

• CentOS7
• MGMT Server on VMWare
• Cloudstack 4.14
• Java 11
• OpenSSL 1.0.2k-fips

SUMMARY

We are not able to access the 4.14 Webserver over HTTPS after upgrading from a functioning 4.13. The Webserver seems to not send anything back. If we curl the https it just loads infinitely:
# curl -v https://localhost:8443/client
* About to connect() to localhost port 8443 (#0)
* Trying ::1...
* Connected to localhost (::1) port 8443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
....(waits)

Previous investigations regarding networking etc can be seen here:
https://lists.apache.org/thread.html/r50fa6f94dae308a598eb2eeb738f4325e29814d8da83de8558bccfb2%40%3Cusers.cloudstack.apache.org%3E

Using a self-signed certificate actually works, only when using an adequate certificate (wildcard in our case) it stops functioning

Commands used:

Combine Files
cat key.key servercert.crt intermediate.crt root.crt > combined.crt

Create Keystore
openssl pkcs12 -in combined.crt -export -out combined.pkcs12

Import Keystore
keytool -importkeystore -srckeystore combined.pkcs12 -srcstoretype PKCS12 -destkeystore /etc/cloudstack/management/combined.pkcs12 -deststoretype pkcs12

Then change https.keystore= and https.keystore.password= accordingly and restart cloudstack-management

Logs found:

2020-06-29 12:01:02,052 INFO [o.e.j.s.h.ContextHandler] (main:null) (logid:) Started o.e.j.w.WebAppContext@311bf055{/client,file:///usr/share/cloudstack-management/webapp/,AVAILABLE}{/usr/share/cloudstack-management/webapp}
2020-06-29 12:01:02,053 INFO [o.e.j.s.h.ContextHandler] (main:null) (logid:) Started o.e.j.s.h.MovedContextHandler@451001e5{/,null,AVAILABLE}
2020-06-29 12:01:02,076 INFO [o.e.j.s.AbstractConnector] (main:null) (logid:) Started ServerConnector@6f46426d{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}
2020-06-29 12:01:02,090 INFO [o.e.j.u.s.SslContextFactory] (main:null) (logid:) x509=X509@25c6abfa(1,h=[our acual domain name],w=[our domain name again]) for SslContextFactory@4991c0f7[provider=null,keyStore=file:///etc/cloudstack/management/combined.pkcs12,trustStore=null]

Things tried:

• Tested Firefox, Chrome, Edge, IE
• Cache cleared / Private Mode
• Multiple Client Systems
• Numerous combinations of key-cert-intermediate-root when generating the pkcs12
• Keystore only with key and server certificate without any CA's
• Different Certificate Vendors (2 different Wildcard Certificates, Sectigo and Digicert/RapidSSL)
• Generated Certificate via internal Domains Certificate Authority
• Checked all certificate combinations via certutil in Windows Powershell
• Changed keystore password to minimal ones without special characters (e.g. 123456)
• Changed the https.port to any other Port in server.properties
• Switched back to java-1.8.0 - of course management server failed to start
• Uploaded certificate chain via 8080 Web GUI - Found working combination of cert-intermediate-root to work with by opening console proxy in separate window and checking certificate validity
• Definitely working combination (for firefox) gets the same outcome when being used as pkcs12
• Updating to a newer Openssl Version fails because its the newest for CentOS7
• Just in case it has an impact - Outcommented jdk.tls.disabledAlgorithms in java.security.ciphers

Further Investigations:

• If we change the keystore password to nonsense it makes no difference
• If we change the keystore name or path to nonsense it almost instantly says that the Website is unreachable and does not load infinitely and curl says Connection Refused
• Fresh Install shows the same behaviour

STEPS TO REPRODUCE

• Install Cloudstack 4.14 on CentOS7 (No need to configure Infrastructure)
• Create Keystore and configure HTTPS
• Try to get https://mymgmtserver.mydomain.de:8443/client to respond

EXPECTED RESULTS

• A response from 8443 as it has been before the upgrade or at least an SSL error regarding a wrong certificate

ACTUAL RESULTS

• Webserver not responding / loading forever no matter what certificate is used
• Actually it isn't really "not responding" as it is in fact trying to communicate but nothing happens

[VincentHermes]

After additional investigation with our very capable colleague we seem to have found the error, but are not able to solve it. There is an exception on jetty regarding keystores:

Exception in thread "main" java.lang.IllegalStateException: KeyStores with multiple certificates are not supported on the
base class org.eclipse.jetty.util.ssl.SslContextFactory. (Use org.eclipse.jetty.util.ssl.SslContextFactory$Server or org.eclipse.jetty.util.ssl.SslContextFactory$Client instead)
at org.eclipse.jetty.util.ssl.SslContextFactory.newSniX509ExtendedKeyManager(SslContextFactory.java:1275)
....
KeyStores with multiple certificates are not supported on the base class org.eclipse.jetty.util.ssl.SslContextFactory

Starting with jetty-9.4.23.v20191118 the usage of SslContextFactory is deprecated and SslContextFactory.Server must be used instead. The embedded Version in CS 4.14 is jetty-9.4.26.v20200117.

I think this is the root of our HTTPS Issue. We are not able to solve this but we are using an Apache Proxy now, which however does not allow to upload from local but that's another story.

[yadvr]

@VincentHermes thanks for sharing, does this work for you if you use a single certificate?

[VincentHermes]

@rhtyd We tried these certificate combinations:

• Key - Server Cert - Intermediate Cert - RootCA
• Key - Server Cert - Intermediate Cert
• Key - Server Cert - RootCA
• Key - Server Cert

Only the server cert doesn't work of course because generating the keystore requires a fitting key. None of these worked however.

The creation of the Self Signed happened like that:
openssl genrsa -out selfsigned.key 2048
openssl req -new -key selfsigned.key -out selfsigned.csr
openssl x509 -req -days 3650 -in selfsigned.csr -signkey selfsigned.key -out selfsigned.crt
cat selfsigned.key selfsigned.crt > selfsignedcombined.crt
openssl pkcs12 -in selfsignedcombined.crt -export -out selfsignedcombined.pkcs12
keytool -importkeystore -srckeystore selfsignedcombined.pkcs12 -srcstoretype PKCS12 -destkeystore /etc/cloudstack/management/selfsignedcombined.pkcs12 -deststoretype pkcs12

And after editing server.properties it launched HTTPS with that self signed cert.

[andrijapanicsb]

Can you try a clean 4.14 install and ssl setup by following this https://www.shapeblue.com/securing-cloudstack-4-11-with-https-tls/ - just the later part (skip the securing system VMs)

[dhslove]

@andrijapanicsb
I tested on my cloudstack 4.14 environments.
As @VincentHermes reported, if the SSL certificate was a single certificate, then SSL is set. However, in the case of multiple certificates, SSL configuration fails.

[yadvr]

I'll take it up, I think on master I also found that SSL did not work for one of our use-cases (wildcard cert).

[rvalle]

I am also seeing this issue on a clean 4.14

furthermore I am also testing my keystore on a vanilla distribution of jetty 9.4 and it does work.

SSL on system VMs does work OK too.

It is something specific to the Jetty SSL connector in the manager.

[yadvr]

Yes @rvalle I've reproduced this with a bunch of keystore combinations with the latest jetty 9.x and JDK11, it does not happen with older releases that used JDK8 such as 4.13. I've tested the following combinations:

• with a single self-signed cert
• with a cert and chain of intermediate and root-ca cert
• keystore created in two formats (the jks and PKCS12, I'm exploring this now)

[rvalle]

@rhtyd I downloaded a vanilla jetty 9 and jdk 11 and it works
With certificate, key, intermediate certificate and root ca.

[rvalle]

Here is the working configuration, in case you find it useful:

• jetty-9.4.31.v20200723
• openjdk version "11.0.8" 2020-07-14

the keystore created like this:

  - name: Generate PKCS#12 file
    openssl_pkcs12:
      action: export
      path: jetty.pkcs12
      friendly_name: jetty
      privatekey_path: Test_Server_Key.pem
      certificate_path: VDC_Test_Server.crt
      other_certificates: 
        - VDC_Test_Intermediate_CA.crt
        - VDC_Test_CA.crt
      state: present
      passphrase: Secret

  - name: Generate Java Key Store
    shell: keytool -importkeystore -srckeystore jetty.pkcs12 -srcstoretype PKCS12 -destkeystore keystore

Which is basically the same thing that jetty documentation says but in ansible commands

I got stuck in the vanilla jetty configuration becuase I had to set the keystore password at 3 different places and I have to enable 2 different modules:

• java -jar start.jar --add-to-start=ssl,https to enable the modules
• jetty.sslContext.keyStorePassword
• jetty.sslContext.keyManagerPassword
• jetty.sslContext.trustStorePassword

but after that it works and the cert chain reported to the browser is correct.

[rvalle]

@rhtyd while you fix this I am trying to find out if there is a workaround for 4.14
I am going to start by downloading vanilla jetty-9.4.26.v20200117 and testing the same configuration I have
Then check if this is due to any setting that can be added....

[rvalle]

yes, it does work with jetty-9.4.26.v20200117, the same packaged with the manager.
So, it must be an instantiation issue, hopefully a configuration issue that can be fixed with a system property...