Kibana: [FEATURE]: Kibana to watch certificate changes and reload them on-the-fly

[danlsgiga]

Describe the feature:
Kibana should reload certificates on-the-fly if the certificate files are renewed / replaced. Right now it requires a full restart to reload certificates.

Describe a specific use case for the feature:
We have a very dynamic environment where certificates are short lived and managed by Vault. Every time a certificate is renewed and/or replaced by Vault it requires a full Kibana restart. I've tried doing a SIGHUP and SIGUSR1 in the kibana process to force a certificate reload but that didn't help.

Note: Elasticsearch is able to reload certificates on-the-fly already so I just assumed Kibana would be able to do the same.

[elasticmachine]

Pinging @elastic/kibana-operations (Team:Operations)

[tylersmalley]

SIGHUP should reload the certificate - if it's not, that is a bug.

[danlsgiga]

I'm running Kibana 7.5.1 and a SIGHUP does not reload the certs. Here is the error after the SIGHUP.

{"type":"error","@timestamp":"2020-01-08T21:41:49Z","tags":["connection","client","error"],"pid":10585,"level":"error","error":{"message":"140388375709568:error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate:../deps/openssl/openssl/ssl/record/rec_layer_s3.c:1407:SSL alert number 42\n","name":"Error","stack":"Error: 140388375709568:error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate:../deps/openssl/openssl/ssl/record/rec_layer_s3.c:1407:SSL alert number 42\n"},"message":"140388375709568:error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate:../deps/openssl/openssl/ssl/record/rec_layer_s3.c:1407:SSL alert number 42\n"}

[J7mbo]

It shoudn't need a SIGx though. Ideally Kibana would do the same as elasticsearch and reload automatically when the certs change.

[danlsgiga]

Agreed, and that was my expectation but it's not the case. Kibana is not detecting certificate changes and consequently not reloading them. Currently I need a full restart to get them updated.

[J7mbo]

Hacky workaround until it's implemented, (if we're lucky enough!): use inotifywait to watch for your certificates changing and then kill / restart kibana or use the SIGHUP if it works.

[danlsgiga]

That's what I have in place atm, so once a month (we have short lived certs) we do a full restart to reload the cert. SIGHUP or SIGUSR1 doesn't work!

[Funbit]

I can't understand why this ticket hasn't been resolved yet... Everybody who uses LetsEncrypt to update the certificates faces with this reloading issue. Taking into account that Elasticsearch instance is already capable of on-the-fly reloading (and all other major open source servers I know), it should be a piece of cake to fix.
Please, make the fix happen 🥺 🙏

[cataclysdom]

We're trying to move to automating Let's Encrypt certificate renewals since most browsers have announced their intent to limit the lifetime of certificates. However, rebooting Kibana once a month is not an option.

Read more here: Mozilla Joins Apple, Google in Reducing TLS Certificate Lifespans.