Nomad: Docker Driver: Cann't override seccomp profile via security_opt

[merryjane]

Nomad version

Nomad v0.9.1 (4b2bdbd9ab68a27b10c2ee781cceaaf62e114399)

Operating system and Environment details

Ubuntu 16.04.6 LTS

Issue

Cann't override seccomp profile via security_opt.

Reproduction steps

Added next configuration to job (https://www.nomadproject.io/docs/drivers/docker.html#security_opt):

security_opt = [
      "seccomp=/etc/docker/seccomp_overridden.json",
]

Got error:

[ERROR] client.alloc_runner.task_runner: running driver failed: alloc_id=510103f3-27ba-97fa-fd48-7d26cc532edb task=ibiryulin-test-task error="Failed to start container 7caa1d690ba738563f0328832b0ecdf8276e05112128a5d43ffdd78602ed4488: API error (500): linux seccomp: Decoding seccomp profile failed: invalid character '/' looking for beginning of value"

With next configuration job works fine:

 security_opt = [
      "seccomp=unconfined",
 ]

but I need to override seccomp profile, not turn it off.

I tried to run docker container with security-opt without nomad on the same server and it run without errors:
docker run --security-opt="seccomp=/etc/docker/seccomp_overridden.json" --name test -it registry.xxx/ubuntu:xenial bash

Nomad Client logs (if appropriate)

Jun 10 17:38:02 docker-09-sas.test.xxx nomad[16095]:     2019-06-10T17:38:02.927+0300 [DEBUG] client.alloc_migrator: waiting for remote previous alloc to terminate: alloc_id=510103f3-27ba-97fa-fd48-7d26cc532edb previous_alloc=cc709860-4bdd-0028-e395-fbe42f6a8f2b
Jun 10 17:38:02 docker-09-sas.test.xxx nomad[16095]:     2019-06-10T17:38:02.952+0300 [DEBUG] client.alloc_migrator: waiting for remote previous alloc to terminate: alloc_id=510103f3-27ba-97fa-fd48-7d26cc532edb previous_alloc=cc709860-4bdd-0028-e395-fbe42f6a8f2b
Jun 10 17:38:03 docker-09-sas.test.xxx nomad[16095]:     2019-06-10T17:38:03.047+0300 [DEBUG] client.alloc_runner.task_runner.task_hook.logmon: starting plugin: alloc_id=510103f3-27ba-97fa-fd48-7d26cc532edb task=mytest-test-task path=/usr/bin/nomad args="[/usr/bin/nomad logmon]"
Jun 10 17:38:03 docker-09-sas.test.xxx nomad[16095]:     2019-06-10T17:38:03.048+0300 [DEBUG] client.alloc_runner.task_runner.task_hook.logmon: plugin started: alloc_id=510103f3-27ba-97fa-fd48-7d26cc532edb task=mytest-test-task path=/usr/bin/nomad pid=25899
Jun 10 17:38:03 docker-09-sas.test.xxx nomad[16095]:     2019-06-10T17:38:03.048+0300 [DEBUG] client.alloc_runner.task_runner.task_hook.logmon: waiting for RPC address: alloc_id=510103f3-27ba-97fa-fd48-7d26cc532edb task=mytest-test-task path=/usr/bin/nomad
Jun 10 17:38:03 docker-09-sas.test.xxx nomad[16095]:     2019-06-10T17:38:03.107+0300 [DEBUG] client.alloc_runner.task_runner.task_hook.logmon.nomad: plugin address: alloc_id=510103f3-27ba-97fa-fd48-7d26cc532edb task=mytest-test-task @module=logmon address=/tmp/plugin786308656 network=unix timestamp=2019-06-10T17:38:03.106+0300
Jun 10 17:38:03 docker-09-sas.test.xxx nomad[16095]:     2019-06-10T17:38:03.107+0300 [DEBUG] client.alloc_runner.task_runner.task_hook.logmon: using plugin: alloc_id=510103f3-27ba-97fa-fd48-7d26cc532edb task=mytest-test-task version=2
Jun 10 17:38:03 docker-09-sas.test.xxx nomad[16095]:     2019-06-10T17:38:03.110+0300 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=510103f3-27ba-97fa-fd48-7d26cc532edb task=mytest-test-task @module=logmon path=/var/lib/nomad/alloc/510103f3-27ba-97fa-fd48-7d26cc532edb/alloc/logs/.mytest-test-task.stdout.fifo timestamp=2019-06-10T17:38:03.110+0300
Jun 10 17:38:03 docker-09-sas.test.xxx nomad[16095]:     2019-06-10T17:38:03.111+0300 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=510103f3-27ba-97fa-fd48-7d26cc532edb task=mytest-test-task @module=logmon path=/var/lib/nomad/alloc/510103f3-27ba-97fa-fd48-7d26cc532edb/alloc/logs/.mytest-test-task.stderr.fifo timestamp=2019-06-10T17:38:03.111+0300
Jun 10 17:38:03 docker-09-sas.test.xxx nomad[16095]:     2019-06-10T17:38:03.298+0300 [DEBUG] client.driver_mgr.docker: binding directories: driver=docker task_name=mytest-test-task binds="[]string{"/var/lib/nomad/alloc/510103f3-27ba-97fa-fd48-7d26cc532edb/alloc:/alloc", "/var/lib/nomad/alloc/510103f3-27ba-97fa-fd48-7d26cc532edb/mytest-test-task/local:/local", "/var/lib/nomad/alloc/510103f3-27ba-97fa-fd48-7d26cc532edb/mytest-test-task/secrets:/secrets"}"
Jun 10 17:38:03 docker-09-sas.test.xxx nomad[16095]:     2019-06-10T17:38:03.298+0300 [DEBUG] client.driver_mgr.docker: setting container name: driver=docker task_name=mytest-test-task container_name=mytest-test-task-510103f3-27ba-97fa-fd48-7d26cc532edb
Jun 10 17:38:15 docker-09-sas.test.xxx nomad[16095]:     2019-06-10T17:38:15.207+0300 [ERROR] client.alloc_runner.task_runner: running driver failed: alloc_id=510103f3-27ba-97fa-fd48-7d26cc532edb task=mytest-test-task error="Failed to start container 1dace0dfb3cd4797be90c71824a54dbf2cb621d55f7e3ce8cca002f5199e02b1: API error (500): linux seccomp: Decoding seccomp profile failed: invalid character '/' looking for beginning of value"
Jun 10 17:38:15 docker-09-sas.test.xxx nomad[16095]:     2019-06-10T17:38:15.209+0300 [INFO ] client.alloc_runner.task_runner: restarting task: alloc_id=510103f3-27ba-97fa-fd48-7d26cc532edb task=mytest-test-task reason="Restart within policy" delay=15.223301763s
Jun 10 17:38:30 docker-09-sas.test.xxx nomad[16095]:     2019-06-10T17:38:30.565+0300 [DEBUG] client.driver_mgr.docker: binding directories: driver=docker task_name=mytest-test-task binds="[]string{"/var/lib/nomad/alloc/510103f3-27ba-97fa-fd48-7d26cc532edb/alloc:/alloc", "/var/lib/nomad/alloc/510103f3-27ba-97fa-fd48-7d26cc532edb/mytest-test-task/local:/local", "/var/lib/nomad/alloc/510103f3-27ba-97fa-fd48-7d26cc532edb/mytest-test-task/secrets:/secrets"}"
Jun 10 17:38:30 docker-09-sas.test.xxx nomad[16095]:     2019-06-10T17:38:30.565+0300 [DEBUG] client.driver_mgr.docker: setting container name: driver=docker task_name=mytest-test-task container_name=mytest-test-task-510103f3-27ba-97fa-fd48-7d26cc532edb
Jun 10 17:38:42 docker-09-sas.test.xxx nomad[16095]:     2019-06-10T17:38:42.282+0300 [ERROR] client.alloc_runner.task_runner: running driver failed: alloc_id=510103f3-27ba-97fa-fd48-7d26cc532edb task=mytest-test-task error="Failed to start container 21efac48c0caacc8798ac75a119d13e30f79b748b22e62d688968b8fe6680471: API error (500): linux seccomp: Decoding seccomp profile failed: invalid character '/' looking for beginning of value"
Jun 10 17:38:42 docker-09-sas.test.xxx nomad[16095]:     2019-06-10T17:38:42.285+0300 [INFO ] client.alloc_runner.task_runner: restarting task: alloc_id=510103f3-27ba-97fa-fd48-7d26cc532edb task=mytest-test-task reason="Restart within policy" delay=16.997866978s
Jun 10 17:38:59 docker-09-sas.test.xxx nomad[16095]:     2019-06-10T17:38:59.369+0300 [DEBUG] client.driver_mgr.docker: binding directories: driver=docker task_name=mytest-test-task binds="[]string{"/var/lib/nomad/alloc/510103f3-27ba-97fa-fd48-7d26cc532edb/alloc:/alloc", "/var/lib/nomad/alloc/510103f3-27ba-97fa-fd48-7d26cc532edb/mytest-test-task/local:/local", "/var/lib/nomad/alloc/510103f3-27ba-97fa-fd48-7d26cc532edb/mytest-test-task/secrets:/secrets"}"
Jun 10 17:38:59 docker-09-sas.test.xxx nomad[16095]:     2019-06-10T17:38:59.369+0300 [DEBUG] client.driver_mgr.docker: setting container name: driver=docker task_name=mytest-test-task container_name=mytest-test-task-510103f3-27ba-97fa-fd48-7d26cc532edb
Jun 10 17:39:11 docker-09-sas.test.xxx nomad[16095]:     2019-06-10T17:39:11.138+0300 [ERROR] client.alloc_runner.task_runner: running driver failed: alloc_id=510103f3-27ba-97fa-fd48-7d26cc532edb task=mytest-test-task error="Failed to start container 7caa1d690ba738563f0328832b0ecdf8276e05112128a5d43ffdd78602ed4488: API error (500): linux seccomp: Decoding seccomp profile failed: invalid character '/' looking for beginning of value"
Jun 10 17:39:11 docker-09-sas.test.xxx nomad[16095]:     2019-06-10T17:39:11.152+0300 [INFO ] client.alloc_runner.task_runner: not restarting task: alloc_id=510103f3-27ba-97fa-fd48-7d26cc532edb task=mytest-test-task reason="Exceeded allowed attempts 2 in interval 30m0s and mode is "fail""
Jun 10 17:39:11 docker-09-sas.test.xxx nomad[16095]:     2019-06-10T17:39:11.167+0300 [INFO ] client.gc: marking allocation for GC: alloc_id=510103f3-27ba-97fa-fd48-7d26cc532edb
Jun 10 17:39:11 docker-09-sas.test.xxx nomad[16095]:     2019-06-10T17:39:11.169+0300 [INFO ] client.gc: garbage collecting allocation: alloc_id=510103f3-27ba-97fa-fd48-7d26cc532edb reason="number of allocations (56) is over the limit (50)"
Jun 10 17:39:11 docker-09-sas.test.xxx nomad[16095]:     2019-06-10T17:39:11.186+0300 [INFO ] client.gc: marking allocation for GC: alloc_id=510103f3-27ba-97fa-fd48-7d26cc532edb
Jun 10 17:39:15 docker-09-sas.test.xxx nomad[16095]:     2019-06-10T17:39:15.167+0300 [WARN ] client.alloc_runner.task_runner.task_hook.logmon.nomad: timed out waiting for read-side of process output pipe to close: alloc_id=510103f3-27ba-97fa-fd48-7d26cc532edb task=mytest-test-task @module=logmon timestamp=2019-06-10T17:39:15.167+0300
Jun 10 17:39:15 docker-09-sas.test.xxx nomad[16095]:     2019-06-10T17:39:15.167+0300 [WARN ] client.alloc_runner.task_runner.task_hook.logmon.nomad: timed out waiting for read-side of process output pipe to close: alloc_id=510103f3-27ba-97fa-fd48-7d26cc532edb task=mytest-test-task @module=logmon timestamp=2019-06-10T17:39:15.167+0300
Jun 10 17:39:15 docker-09-sas.test.xxx nomad[16095]:     2019-06-10T17:39:15.170+0300 [DEBUG] client.alloc_runner.task_runner.task_hook.logmon: plugin process exited: alloc_id=510103f3-27ba-97fa-fd48-7d26cc532edb task=mytest-test-task path=/usr/bin/nomad pid=25899
Jun 10 17:39:15 docker-09-sas.test.xxx nomad[16095]:     2019-06-10T17:39:15.170+0300 [DEBUG] client.alloc_runner.task_runner.task_hook.logmon: plugin exited: alloc_id=510103f3-27ba-97fa-fd48-7d26cc532edb task=mytest-test-task
Jun 10 17:39:15 docker-09-sas.test.xxx nomad[16095]:     2019-06-10T17:39:15.170+0300 [DEBUG] client.alloc_runner.task_runner: task run loop exiting: alloc_id=510103f3-27ba-97fa-fd48-7d26cc532edb task=mytest-test-task
Jun 10 17:39:15 docker-09-sas.test.xxx nomad[16095]:     2019-06-10T17:39:15.245+0300 [DEBUG] client.gc: alloc garbage collected: alloc_id=510103f3-27ba-97fa-fd48-7d26cc532edb
Jun 10 17:39:15 docker-09-sas.test.xxx nomad[16095]:     2019-06-10T17:39:15.248+0300 [INFO ] client.gc: garbage collecting allocation: alloc_id=510103f3-27ba-97fa-fd48-7d26cc532edb reason="number of allocations (55) is over the limit (50)"
Jun 10 17:39:15 docker-09-sas.test.xxx nomad[16095]:     2019-06-10T17:39:15.248+0300 [DEBUG] client.gc: alloc garbage collected: alloc_id=510103f3-27ba-97fa-fd48-7d26cc532edb

[bogdanov1609]

+1
Any ETA or update here?

[bogdanov1609]

Somebody...

[powellchristoph]

I am running into the exact same issue. Any thoughts?

[powellchristoph]

So we did some digging and it appears that Nomad's docker client is expecting the json string to be passed when setting the seccomp while Nomad is just passing the file path along. At no point is it opening the actual json file and sending the contents for unmarshaling.

If you put the json in-line in the job file, it works fine.

         security_opt = [
           "seccomp={\"defaultAction\":\"SCMP_ACT_ERRNO\",\"syscalls\":[{\"name\":\"accept\",\"action\":\"SCMP_ACT_ALLOW\",\"args\":null},{\"name\":\"acc....         
         ]

[tgross]

Hi @merryjane and @powellchristoph! Sorry about the delay in getting back to y'all on this.

So we did some digging and it appears that Nomad's docker client is expecting the json string to be passed when setting the seccomp while Nomad is just passing the file path along.

Yes, this is generally the case for everything in the config stanza for Nomad task drivers. The docs for the task config stanza could probably be a bit more clear here:

Specifies the driver configuration, which is passed directly to the driver to start the task.

Another way to solve this should be to add the file:// in front of the path, so you may want something more like:

security_opt = [
      "seccomp=file:///etc/docker/seccomp_overridden.json",
]

As @powellchristoph provided what should be a working answer a while back now, I'm going to close this issue. But feel free to re-open if there are more questions!

[merryjane]

Thank you very much for your advice.
Solution with json in-line works perfectly. However it looks a little bit messy if we are passing many keys, and demands accuracy in escaping quotes.
Unfortunately solution with "seccomp=file:///etc/docker/seccomp_overridden.json" doesn't work. We got error [ERROR] client.driver_mgr.docker: failed to start container: driver=docker container_id=3ba4814bd5c74e340b5e066089b997ea9b8d469e0676122956a98bb3f429dae1 attempt=1 error="API error (500): linux seccomp: Decoding seccomp profile failed: invalid character 'i' in literal false (expecting 'a')".

[tgross]

Yeah, that's definitely an unfortunate usability "papercut". Let me take a look and see how feasible it would be to correct, otherwise we'll at least make a note in the documentation for it.

[tgross]

I looked into this and it looks like the Docker CLI may do some parsing of the --security-opt flag before passing it into the Docker HTTP API, which is why the behavior is different on Nomad and Docker. Our documentation is slightly wrong as a result because we don't pass anything into Docker's --security-opt flag (because we don't call the Docker CLI at all!).

I've opened https://github.com/hashicorp/nomad/issues/6720 to follow up on this.