Cloud-on-k8s: TestUpdateKibanaSecureSettings is flaky

[sebgl]

https://devops-ci.elastic.co/job/cloud-on-k8s-e2e-tests-snapshot-versions/106/testReport/

=== RUN   TestUpdateKibanaSecureSettings/secure_settings_should_eventually_be_set_in_all_nodes_keystore
Retries (5m0s timeout): ................................................................................
    utils.go:84: 
            Error Trace:    utils.go:84
            Error:          Received unexpected error:
                            invalid keystore entries. Expected: [logging.verbose]. Actual: []
            Test:           TestUpdateKibanaSecureSettings/secure_settings_should_eventually_be_set_in_all_nodes_keystore

[sebgl]

Happened again on stack snapshots tests: https://devops-ci.elastic.co/job/cloud-on-k8s-e2e-tests-snapshot-versions/107/testReport/

[barkbay]

I can reproduce the issue locally with the latest version of 7.9.0-SNAPSHOT :

=== RUN   TestUpdateKibanaSecureSettings/secure_settings_should_eventually_be_set_in_all_nodes_keystore
Retries (5m0s timeout): ...........................................................................
{"log.level":"error","@timestamp":"2020-07-28T11:03:25.392+0200","message":"stopping early","service.version":"0.0.0-00000000","service.type":"eck","ecs.version":"1.4.0","error":"test failure","error.stack_trace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/Users/michael/go/pkg/mod/github.com/go-logr/[email protected]/zapr.go:128\ngithub.com/elastic/cloud-on-k8s/test/e2e/test.StepList.RunSequential\n\t/Users/michael/go/src/github.com/elastic/cloud-on-k8s/test/e2e/test/step.go:43\ngithub.com/elastic/cloud-on-k8s/test/e2e/kb.TestUpdateKibanaSecureSettings\n\t/Users/michael/go/src/github.com/elastic/cloud-on-k8s/test/e2e/kb/keystore_test.go:114\ntesting.tRunner\n\t/usr/local/Cellar/go/1.13.7/libexec/src/testing/testing.go:909"}
    --- FAIL: TestUpdateKibanaSecureSettings/secure_settings_should_eventually_be_set_in_all_nodes_keystore (300.00s)
        utils.go:84: 
                Error Trace:    utils.go:84
                Error:          Received unexpected error:
                                invalid keystore entries. Expected: [logging.verbose]. Actual: []
                Test:           TestUpdateKibanaSecureSettings/secure_settings_should_eventually_be_set_in_all_nodes_keystore

I have the following event emitted on the cluster:

default       0s          Warning   OOMKilling                node/gke-michael-dev-2-default-pool-c3327033-hjlg                                Memory cgroup out of memory: Kill process 31303 (node) score 1965 or sacrifice child
Killed process 31303 (node) total-vm:2188540kB, anon-rss:1006452kB, file-rss:28584kB, shmem-rss:0kB

Not sure if it is related to #3344, still investigating...

[barkbay]

Actually the OOMKilling event is not the culprit, it seems that the directory where the keystore is created has changed from /usr/share/kibana/data to /usr/share/kibana/config

7.8.0 :

bash-4.2$ kibana --version
7.8.0

bash-4.2$ ls -l data/
total 12
-rw-r--r-- 1 kibana kibana    0 Jul 28 09:39 elastic-internal-init-keystore.ok
drwxrwsr-x 4 kibana kibana 4096 Jul 28 09:39 headless_shell-linux
-rw-r--r-- 1 kibana kibana  190 Jul 28 09:39 kibana.keystore
-rw-rw-r-- 1 kibana kibana   36 Jul 28 09:39 uuid

7.9.0-SNAPSHOT :

bash-4.2$ kibana --version
7.9.0-SNAPSHOT

bash-4.2$ ls -l data/
total 4
-rw-r--r-- 1 kibana kibana  0 Jul 28 09:38 elastic-internal-init-keystore.ok
-rw-rw-r-- 1 kibana kibana 36 Jul 28 09:39 uuid

bash-4.2$ kibana-keystore create
FATAL CLI ERROR Error: EROFS: read-only file system, open '/usr/share/kibana/config/kibana.keystore'
    at Object.openSync (fs.js:443:3)
    at writeFileSync (fs.js:1194:35)
    at Keystore.save (/usr/share/kibana/src/legacy/server/keystore/keystore.js:65:27)
    at create (/usr/share/kibana/src/cli_keystore/create.js:45:12)
    at Command.<anonymous> (/usr/share/kibana/src/cli/command.js:113:20)
    at Command.listener (/usr/share/kibana/node_modules/commander/index.js:291:8)
    at Command.emit (events.js:198:13)
    at Command.parseArgs (/usr/share/kibana/node_modules/commander/index.js:672:12)
    at Command.parse (/usr/share/kibana/node_modules/commander/index.js:459:21)
    at Object.<anonymous> (/usr/share/kibana/src/cli_keystore/cli_keystore.js:69:9)

[barkbay]

I think the related change is https://github.com/elastic/kibana/commit/94ef03dbd3ab57426fc04bbf0d6c11a8e12e11ac

[barkbay]

Right now ECK exposes the configuration directory /usr/share/kibana/config through a Secret.
If we want to let the keystore being created in this directory I think that we have to do something along those lines:

1. In a prepare-fs-container container:

   1. mount the config Secret in /mnt/elastic-internal/kibana-config instead of /usr/share/kibana/config

   2. mount an empty dir elastic-internal-kibana-config-local in /mnt/elastic-internal/kibana-config-local

   3. create a copy of /usr/share/kibana/config/* to /mnt/elastic-internal/kibana-config-local/

   4. copy the content of theconfig Secret /mnt/elastic-internal/kibana-config-local, mostly:

      • /mnt/elastic-internal/kibana-config/elasticsearch.yml to /mnt/elastic-internal/kibana-config-local/kibana.yml
      • /mnt/elastic-internal/kibana-config/telemetry.yml to /mnt/elastic-internal/kibana-config-local/telemetry.yml

2. The keystore init container mount elastic-internal-kibana-config-local in /usr/share/kibana/config, this way the file created by the keystore is persisted in the empty directory.

3. kibana container is started with elastic-internal-kibana-config-local mounted in /usr/share/kibana/config

_(this is more or less what we are doing for Elasticsearch without the symbolic links)_