Cloud-on-k8s: Expose the Kibana keystore

[nkvoll]

It would be great if users could configure secure settings for Kibana through its keystore feature (similar to how we do it via Secret for the ES keystore): https://www.elastic.co/guide/en/kibana/current/secure-settings.html

[sebgl]

Things I've learned while looking into this:

• Kibana loads the keystore when starting.
• There is not hot-reload keystore feature like Elasticsearch has.
• Key/values are strings, there is no "file" type as in Elasticsearch keystore.
• kibana-keystore CLI and underlying file format is not the same as elasticsearch-keystore.
• Any setting from kibana.yml can be set in the keystore, using a reference in kibana.yml: elasticsearch.password: ${elasticsearch.password}

I think there is no good reason to reuse our Elasticsearch keystore client implementation for Kibana.
Also it's probably fine to recreate a Kibana instance automatically on Keystore update.

What I have in mind for the implementation.

• Keystore key-values can be provided by the user in a secret, referenced from the SecureSettings field of Kibana CRD, similar to Elasticsearch CRD. This secret is watched by the Kibana controller.
• ~Values are reconciled into an internal secret, in which we could also inject our own secrets (Elasticsearch password is a good example).~ We probably don't need to inject our secrets there since we already rely on env vars mounted from a secret (same security protection: cleartext in the secret resource & volume mount). Let's mount the user-provided secret into the pod directly?
• This secret is mounted into Kibana pod.
• We create the keystore on pod startup (./bin/kinana-keystore create), and add all secrets present in the secure settings mounted volume (cat my.secret | ./bin/kibana-keystore add my.secret --stdin)
• This can be done either through an init container where we mount Kibana data volume as an EmptyDir, or through overriding Kibana container command. I tend to think the init container approach (while slightly more complicated due to volume mount propagation, and a bit longer in time) is cleaner?
• We store ~a hash~ the version of the secure settings secret into the Kibana Deployment pod template labels: if anything changes in this secret, this will lead to the deployment being recreated, hence keystores to be recreated for all Kibana instances.

[thbkrkr]

Note: a secret for the Kibana config ${name}-kb-config is already in place and its checksum is already used as a pod label.

I'm in favor of the init container that looks cleaner.

[sebgl]

Note: a secret for the Kibana config ${name}-kb-config is already in place and its checksum is already used as a pod label.

Do you think we should reuse it?
It currently contains a single kibana.yml entry. We could reconcile all secure settings entries in there and ignore the kibana.yml one in the init container.
It seems a bit cleaner to have a dedicated secret imo (even more since we can mount the user-provided secure settings secret directly without reconciling into our own secret first)?

[pebrc]

I am in favour of two separate secrets (config + secure settings) + using an init container