Kibana: Error running Kibana in Docker w/ User Namespace Isolation

[drum7]

Kibana version: 7.5.0

Elasticsearch version: 7.5.0

Server OS version: Ubuntu 18.04 (Docker 19.03.5)

Browser version:

Browser OS version:

Original install method (e.g. download page, yum, from source, etc.): Docker Image from elastic repo.

Description of the problem including expected versus actual behavior:
Unable to run Kibana image as discussed here. When i spin up kib01 at step 7, kib01 fails to extract plugins due to a permissions error. Permissions of /usr/share/kibana are 0:0 instead of kibana user. I have user namespace isolation enabled - unclear if that's the cause.

Steps to reproduce:

1. Enable user namespace isolation (if relevant)
2. Follow steps as outlined on elastic wiki.

Errors in browser console (if relevant):

Provide logs and/or server output (if relevant):

{"type":"error","@timestamp":"2019-12-03T16:56:47Z","tags":["error","write-uuid"],"pid":6,"level":"error","error":{"message":"EACCES: permission denied, open '/usr/share/kibana/data/uuid'","name":"Error","stack":"Error: EACCES: permission denied, open '/usr/share/kibana/data/uuid'","code":"EACCES"},"message":"EACCES: permission denied, open '/usr/share/kibana/data/uuid'"}
{"type":"error","@timestamp":"2019-12-03T16:56:47Z","tags":["warning","process"],"pid":6,"level":"error","error":{"message":"Error: EACCES: permission denied, open '/usr/share/kibana/data/uuid'","name":"UnhandledPromiseRejectionWarning","stack":"UnhandledPromiseRejectionWarning: Error: EACCES: permission denied, open '/usr/share/kibana/data/uuid'\n    at emitWarning (internal/process/promises.js:81:15)\n    at emitPromiseRejectionWarnings (internal/process/promises.js:120:9)\n    at process._tickCallback (internal/process/next_tick.js:69:34)"},"message":"Error: EACCES: permission denied, open '/usr/share/kibana/data/uuid'"}
{"type":"error","@timestamp":"2019-12-03T16:56:47Z","tags":["warning","process"],"pid":6,"level":"error","error":{"message":"Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)","name":"UnhandledPromiseRejectionWarning","stack":"Error: EACCES: permission denied, open '/usr/share/kibana/data/uuid'"},"message":"Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)"}
{"type":"log","@timestamp":"2019-12-03T16:56:47Z","tags":["reporting","browser-driver","error"],"pid":6,"message":"Error code EACCES: Insufficient permissions for extracting the browser archive. Make sure the Kibana data directory (path.data) is owned by the same user that is running Kibana."}
{"type":"log","@timestamp":"2019-12-03T16:56:47Z","tags":["fatal","root"],"pid":6,"message":"{ ExtractError: Failed to extract the browser archive\n    at err (/usr/share/kibana/x-pack/legacy/plugins/reporting/server/browsers/extract/unzip.js:14:23)\n    at /usr/share/kibana/node_modules/extract-zip/index.js:64:20\n    at /usr/share/kibana/node_modules/mkdirp/index.js:47:53\n    at FSReqWrap.oncomplete (fs.js:154:21)\n  name: 'ExtractError',\n  cause:\n   { Error: EACCES: permission denied, mkdir '/usr/share/kibana/data/headless_shell-linux'\n     errno: -13,\n     code: 'EACCES',\n     syscall: 'mkdir',\n     path: '/usr/share/kibana/data/headless_shell-linux' } }"}
{"type":"log","@timestamp":"2019-12-03T16:56:47Z","tags":["info","plugins-system"],"pid":6,"message":"Stopping all plugins."}
{"type":"log","@timestamp":"2019-12-03T16:56:47Z","tags":["info","plugins","data"],"pid":6,"message":"Stopping plugin"}
{"type":"log","@timestamp":"2019-12-03T16:56:47Z","tags":["info","plugins","translations"],"pid":6,"message":"Stopping plugin"}
{"type":"log","@timestamp":"2019-12-03T16:56:47Z","tags":["info","plugins","spaces"],"pid":6,"message":"Stopping plugin"}
{"type":"log","@timestamp":"2019-12-03T16:56:47Z","tags":["info","plugins","features"],"pid":6,"message":"Stopping plugin"}
{"type":"log","@timestamp":"2019-12-03T16:56:47Z","tags":["info","plugins","timelion"],"pid":6,"message":"Stopping plugin"}
{"type":"log","@timestamp":"2019-12-03T16:56:47Z","tags":["info","plugins","code"],"pid":6,"message":"Stopping plugin"}
{"type":"log","@timestamp":"2019-12-03T16:56:47Z","tags":["info","plugins","licensing"],"pid":6,"message":"Stopping plugin"}
{"type":"log","@timestamp":"2019-12-03T16:56:47Z","tags":["info","plugins","security"],"pid":6,"message":"Stopping plugin"}

 FATAL  ExtractError: Failed to extract the browser archive

Describe the feature:

[elasticmachine]

Pinging @elastic/kibana-operations (Team:Operations)

[spalger]

@jarpy it looks like you wrote most of the permission-management stuff in our Dockerfile, do you happen to know what users need to do so that they can use "user namespace isolation" with our images?

[jarpy]

Looking at the Dockerfile, there's plenty of places where we assume that the UID is, deterministically, 1000. As a first guess, if I wanted to run with arbitrary, nondeterministic UIDs, I would probably create a derivative image that relied on the appropriate files and directories being readable and (where appropriate) writable by the primary group of the Kibana process.

[drum7]

changing "user" to 1000 seemed to solve the problem. it also seems to be more in-line w/ the RH documentation w/r/t openshift.

[jarpy]

Oh, that's good news.

Yes, we try to follow the good-practice of running as a non-root user by default. That can trigger problems in some environments, though, because so many public images are run as UID 0 that sometimes that assumption is baked in to the environment.

[drum7]

Yep you guys did great. Rereading the comment I made...I wasn't trying to be snarky, sorry if it came off that way. I'll rerun another build / test tonight to confirm.

Assuming it works, did you guys want to update or have me do a PR?

[jarpy]

Not at all! I detected no snark, whatsoever. ❤️

[jarpy]

Assuming it works, did you guys want to update or have me do a PR?

Sorry, I don't think I'm following. What did you end up changing? I assumed that you had changed your orchestrator to use UID 1000, rather than making changes to the image itself.

[jarpy]

Oh, you set this to numeric 1000, rather than the string kibana? Interesting!

I'm not the maintainer these days, so I'll defer to @spalger about what to do next.

[drum7]

Exactly - I got the idea when I went and read the OKD docs ref'd here.

Lastly, the final USER declaration in the Dockerfile should specify the user ID (numeric value) and not the user name.

[grahamrhay]

I followed the instructions here, and hit the same issue. Adding --user 1000 or --user kibana to the docker run incantation had no effect.

Am I missing something obvious?

[drum7]

I followed the instructions here, and hit the same issue. Adding --user 1000 or --user kibana to the docker run incantation had no effect.

Am I missing something obvious?

I believe I edited the Dockerfile and built it locally. You can see this referenced in their template or at the prebuilt file.

Instead of USER kibana I used USER 1000