Jetty.project: Support certificates hot reload

[Horcrux7]

Because certificates from Let's Encrypt has a lifetime of 90 days only a automatic renew is required without a server restart. This should be handle from Jetty server self.

A generic implementation of the ACME protocol should be the best solution. There are Java client libraries for the ACME protocol.

The minimum is a dynamic replacement of the current used certificate.

[sbordet]

See also: https://dev.eclipse.org/mhonarc/lists/jetty-users/msg07260.html

[Horcrux7]

@sbordet Do you means this is the recommended solution also for the future?

[sbordet]

@Horcrux7, no I just meant to gather together references to the same problem.
We will analyze all data, proposed solutions, etc. and come up with the right solution.

[fhossain]

I have a similar situation with Jetty using short lived certificates. In my case I am not using ACME directly as it is an internal private CA for service to service mutual authentication. I have control over the client portion and can deal with short lived certificates but the Jetty server portion needs a solution. I have dynamic keystores that update the key/certificate and can make a callback call when they are ready to use. What I am missing is a mechanism for the callback to notify Jetty listeners to reinitialize the SSLContext.

[sbordet]

@Horcrux7 for Jetty implementing the ACME protocol I think it's overkill. We are not in that business, and Let's Encrypt has already provided tools that people can use alongside with Jetty to implement the certificate renewal/revocation.

The idea is that you will install Jetty and a _certificate management agent_ (as defined by Let's Encrypt); the latter will take care of the ACME part and generate the new keystore.

We will improve Jetty's SslContextFactory with a way to be reconfigured and reloaded so that the the certificate management agent, once received the new certificate and generated the new keystore, can reconfigure Jetty's SslContextFactory with the new keystore path, password and other configuration options, and then reload the new configuration.

This will allow new connections to use the new key material, while older connections will continue to use the old key material until closed.
The ServerConnector will _not_ be stopped, and will atomically switch to the new key material.

I think this will cover @fhossain use case, as well as the original poster on the mailing list (thread referenced above), and with a bit of glue code, also @Horcrux7 use case.

[sbordet]

Introduced SslContextFactory.reload(Consumer<SslContextFactory>) to allow reconfiguration and reload of the SslContextFactory.

There is a gotcha, though.

The JDK by default supports TLS session resumption.
A Java client connects to a TLS server with the pair (host, port). When the first connection to this pair completes successfully and the connection is then closed, the TLS session ID is cached by the JDK.
Subsequent connections make use of the cached TLS session ID and will be able to perform the "fast" TLS handshake via session resumption.

A new connection established by the client will use session resumption; the client will also check that the certificate received by the server is equal to the certificate that it cached previously.
This check may fail when the certificate on the server has changed as part of a reload.

Turns out that the JDK chokes if this check fails, and throws a SSLHandshakeException.

On the other hand, browsers are perfectly capable of handling this case by falling back to the full TLS handshake instead of the fast one that uses session resumption.

@gregw has already verified that the JDK offers no API to disable the caching of TLS sessions (see #519).
The closest solution is to use [SSLSocket|SSLEngine].getSession().invalidate() but that may not apply to all cases.

In summary:

• It is possible to change the server keystore without restarting or suspending the ServerConnector via the new SslContextFactory.reload(...) method.
• Browser clients should work fine (we have tested Firefox).
• JDK clients must be prepared to get an SSLHandshakeException if that happens, and somehow try to invalidate the TLS session to avoid session resumption.

I would appreciate if people can test this new feature and report their feedback.

Thanks !

[Horcrux7]

@sbordet Sounds you should open a bug report/feature request for Java by Oracle.

[fhossain]

@sbordet thanks for addressing it. I have tested it out and it works like a charm. I am using this inside SpringFramework and have wired it to use both Keystore from disk as well as use SslStoreProvider() which allows me to have in memory version of Keystore. They both work great. I tested with standard HttpsURLConnection but could not reproduce the problem you mentioned. I ran a loop with System.setProperty("http.keepAlive", "false"); on the client with -Dssl.debug=true -Djavax.net.debug=ssl:handshake. I can see repeated calls with logs like %% Try resuming [Session-1, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384] from port 61259. In each round the port number seems to change but not the Session number. However, after a SslContextFactory.reload(...) I can see the client session number going up by one. The exception seems to be handled transparently from the client side without any exception propagated to the client. Let me know if you think I made any mistakes in my testing. Otherwise, everything LGTM 👍

[sbordet]

@fhossain, glad it works for you.

It is strange that the TLS session number goes up by one: it should be a random-like number every time it is generated by the server.

Can you share your code that uses HttpsURLConnection ?

Other testers ?

[fhossain]

Here is the gut of my test.

public void goRun(String url, int count, SSLSocketFactory sslSocketFactory) throws Exception {
    System.setProperty("http.keepAlive", "false");
    long start = System.currentTimeMillis();
    for (int i = 0; i < count; i++) {
        URL myurl = new URL(url);
        HttpsURLConnection con = (HttpsURLConnection) myurl.openConnection();
        //con.setRequestProperty("Connection", "close");
        con.setSSLSocketFactory(sslSocketFactory);

        InputStream ins = con.getInputStream();
        InputStreamReader isr = new InputStreamReader(ins);
        BufferedReader in = new BufferedReader(isr);
        String inputLine;
        while ((inputLine = in.readLine()) != null) {
            if (count == 1) {
                System.out.print(inputLine);
            }
        }
        in.close();
        Thread.sleep(5000);
    }
    System.out.println("Total time: " + (System.currentTimeMillis() - start) + " ms");
}

[sbordet]

What happens if you uncomment

con.setRequestProperty("Connection", "close");

?

[fhossain]

It behaves the same as using System.setProperty("http.keepAlive", "false");. In both cases the session re-establish is successful until the certificate is updated. Upon which the session count goes up by one. I think the actual session number might be random but the debug printout sanitized it for readability.
However, there is one change needed in the code. The Thread.sleep() has to be < 5 seconds. Otherwise, the keep-alive timer kicks in and closes connection and you can't see the different between System.setProperty("http.keepAlive", "false")/con.setRequestProperty("Connection", "close"); and commenting out both and just using the default. The system default is keep-alive on. In that case the connection is kept open and certificates change is not detected as expected.

[johngmyers]

Could the server invalidate all TLS session IDs whenever the certificate changes?

[joakime]

@johngmyers there's no API to either purge the TLS session ID cache, set it to 0 entries, or to access the list of sessions and invalidate each.

If you know of a way, let us know.

[amandeep33]

@sbordet wanted to know if we can have the same functionality in 9.2.x release also as version 9.3.x requires JDK 8.

[sbordet]

No, we won't backport this to 9.2.x because Jetty 9.2.x is End-Of-Life.

[ramtaneja]

@sbordet , I couldn't get this work for me,
IN my case I just need same keystore to be reloaded after I replace it manually
so I just used a dummy consumer
sslContextFactory.reload(scf -> {});

I could see log in at INFO level of SslContextFactory for change in X509 for my SslContextFactory, but browser wasn't able to detect change in certs, is there anything else needs to be done

[sbordet]

I could see log in at INFO level of SslContextFactory for change in X509 for my SslContextFactory, but browser wasn't able to detect change in certs, is there anything else needs to be done

So the server did the right thing, but the browser did not.
Perhaps the browser is using existing connections that won't pick up the new configuration (only new connections will).
Have you tried restarting the browser?
Since the server did the right thing, it's not a Jetty issue.

[ramtaneja]

well, actually its working perfectly fine...!
That was my bad, I was running two servers on different ports, reloading on one of them and was browsing the other one...