Node: Reload certificate files of https.createServer() without restarting node server

[nolimitdev]

Hi, letsencrypt certificate files expires each 3 months. Is there any way to refresh certificate files without restarting node server? Because using stale/expired certificate causes error ERR_INSECURE_RESPONSE in browser.

var fs = require('fs');
var https = require('https');
var ws = require('ws').Server;
var config = require('config.js');
var certificate = {
    key: fs.readFileSync(config.sslKeyPath),
    cert: fs.readFileSync(config.sslCrtPath),
}
var httpsServer = https.createServer(certificate).listen(config.port),
var wssServer = new ws({ server : httpsServer });

// I would like to reload certificate monthly...

// solution A): just update certificate.cer since variable certificate is passed to createServer() as reference because it is Object (not primitive value)
setInterval(function() { certificate.cert = fs.readFileSync(config.sslCrtPath); console.log("reload cerfificate A"); }, 1000 * 60 * 60 * 24 * 30);
// ... no success

// solution B): update directly httpsServer.cert (yes, this property exists when you console.log(httpsServer))
setInterval(function() { httpsServer.cert = fs.readFileSync(config.sslCrtPath); console.log("reload cerfificate B"); }, 1000 * 60 * 60 * 24 * 30);
// ... property is updated but no success

No solution works and node always use stale certificate for new incoming https requests and websocket connections too . It would be great to have a new method in returned Object from https.createServer() to reload certificate files e.g.:
httpsServer.reloadCertificate({key: fs.readFileSync(config.sslKeyPath), cert: fs.readFileSync(config.sslCrtPath)})
... now, new incoming https requests or websocket connections should be handled with new certificate files

[benjamingr]

That's interesting.

In practice, people solve this by running Node.js servers in a cluster of several instances, and then do a one-by-one update where severs individually go down and up with the new certificate so there is no runtime.

WebSocket requests require "draining", so you stop routing connections to the server and then restart it once all the existing requests are done.

I can see the use case here.

[mscdex]

FWIW you can already do this with SNICallback():

const https = require('https');
const tls = require('tls');
const fs = require('fs');
var ctx = tls.createSecureContext({
  key: fs.readFileSync(config.sslKeyPath),
  cert: fs.readFileSync(config.sslCrtPath)
});
https.createServer({
  SNICallback: (servername, cb) => {
    // here you can even change up the `SecureContext`
    // based on `servername` if you want
    cb(null, ctx);
  }
});

With that, all you have to do is re-assign ctx and then it will get used for any future requests.

[nolimitdev]

@mscdex thank you for great working fallback (already implemented) :)
So clients without SNI support will not connect now, yes? I know, it must be very old device which do not support SNI. Also option SNICallback seems to mean a little overhead or?

[mscdex]

@nolimitdev You should be able to still supply key and cert in the createServer() options in addition to SNICallback(). That pair should be used for non-SNI clients IIRC (this should be rare in this day and age though).

As far as overhead goes, it's just a couple of extra function calls really, I wouldn't be so worried about it.

[silverwind]

I wouldn't rule out non-SNI completely. There are still clients that don't support it or handle it inproperly, like the OpenSSL version included on the latest macOS ref.

I think adding a method to update server options at runtime would be nice to have. Similar to what is possible for ticket keys using setTicketKeys.

[bnoordhuis]

Duplicate of #4464?

[silverwind]

Yes, seems like a dupe.

[oxygen]

SNICallback is only invoked once when the https server is created.
Even running .close() and then .listen() won't determine SNICallback to be called again for the same domain.

As a result, SNICallback cannot be used to reload certificates.

I couldn't find any API supported way of reloading certificates.

[bnoordhuis]

SNICallback is only invoked once when the https server is created.

I'm not sure where you get that from. The SNICallback is inherited by incoming connections from the tls.Server instance on which they arrive:

https://github.com/nodejs/node/blob/138eb32be1a912f1b8a551f657880f19c12f864c/lib/_tls_wrap.js#L1054-L1076

https.Server inherits that behavior from tls.Server.

[oxygen]

I tried using it.
SNICallback is called only once per domain on the https server, that is, on the first connection with a particular domain.

Subsequent connections on the same domain do not go to SNICallback and so are using some kind of cache.

NodeJS 11.7

[bnoordhuis]

It sounds like you're getting tripped up by TLS session resumption, either server-side (session IDs) or client-side (ticket keys.)

It cuts short the TLS handshake but it means SNICallback won't be invoked because there's no need, that part of the handshake is skipped.

You can disable it (at the cost of reduced handshake performance) but you don't have to. Sessions resume from the key exchange step of the handshake, they don't look (and don't have to look) at the certificate. In other words, it's immaterial that SNICallback isn't called.

https://hpbn.co/transport-layer-security-tls/#tls-session-resumption is the best overview I could find in 30 seconds of googling, hope that helps. :-)

[oxygen]

Thanks for taking the time to explain it to me. I’ll try with multiple clients and see how it goes.

[masx200]

https://github.com/nodejs/node/pull/34444

[luisfonsivevo]

I was finally able to get this working without a full restart using tls.Server.setSecureContext. WebSocket connections using the old secure context remain open, while new WebSocket connections will use the fresh certificate.

Time will tell if it works specifically with certbot, but I'm able to trigger a reload by overwriting the certificates with self-signed garbage (obviously back up the old ones first if you want to test). cat fullchainbad.pem > fullchain1.pem && cat privkeybad.pem > privkey1.pem

var wsServer = new ws.Server(...

var keyMod = false;
var certMod = false;
var reloading = false;

wsServer.reloadCerts = async function()
{
    reloading = true;

    //keep trying until success
    while(true)
    {
        try
        {
            //sometimes throws an error if certs haven't finised writing to disk
            httpServer.setSecureContext(
            {
                key: fs.readFileSync(settings.PRIVKEY),
                cert: fs.readFileSync(settings.FULLCHAIN)
            });

            keyMod = certMod = reloading = false;

            console.log("Certificates reloaded");
            break;
        }
        catch(swallow) {} 

        await new Promise((resolve) => setTimeout(resolve, 100));
    }
};

wsServer.reloadCerts(); //initial load

fs.watch(settings.PRIVKEY, { persistent: false }, function()
{
    keyMod = true;
    if(certMod && !reloading)
        wsServer.reloadCerts();
});

fs.watch(settings.FULLCHAIN, { persistent: false }, function()
{
    certMod = true;
    if(keyMod && !reloading)
        wsServer.reloadCerts();
});

[nolimitdev]

@luisfonsivevo I think your implementation is unclear and complicated. Try to check this https://github.com/nodejs/node/issues/4464#issuecomment-357975317 I do not use setSecureContext() because it was added in node v11 and my post is from january 2018 but you can use algorithm. I just used setTimeout()+clearTimeout() without needing sth. like your "reloading". You uselessly combined setTimeout() and "reloading". I also wrote there that it is useless to watch both private key and cert because when private key is changed also certificate must be changed, so vars "keyMod" and "certMod" are also useless. Maybe you should also use chokidar instead of native watch which is platform unreliable (I fixed possible problems by clearTimeout()). I use my algorithm for several years without problems.

[luisfonsivevo]

I think that hoping both certificates are finished writing to disk after 5 seconds would be unreliable. It's pretty unlikely to ever take longer, but still. I could definitely lose an fs.watch though.