Nomad: Bad UX when rendering template to shared alloc directory with Docker driver

[ctlajoie]

Based on conversation on gitter.

Relevant parts of my job file:

job "sf-demo" {
  group "frontend" {
    task "web" {
      template {
        destination = "${NOMAD_ALLOC_DIR}/conf.d/db.php"
        change_mode = "noop"
        data = "some stuff"
      }
    } #task web

    task "redis" {
       ...
    }
  }
}

The directory tree looks like:

├── alloc
│   ├── data
│   ├── logs
│   │   ├── redis.stderr.0
│   │   └── redis.stdout.0
│   └── tmp
├── redis
│   ├── executor.out
│   ├── local
│   ├── secrets
│   └── tmp
└── web
    ├── alloc
    │   └── conf.d
    │       └── db.php
    ├── local
    ├── secrets
    └── tmp

Nomad version 0.6.0

[dadgar]

Hey,

When rendering the template to the shared allocation directory with the docker driver, you need to change the path to be ../alloc/<path you want>. This unfortunately has to do with the fact that the environment variable is set to the path that exists in the container but the rendering is happening rooted at the task directory. This is causing the allocation to be rendered to an alloc directory under the web task instead of the shared alloc directory.

[k3n]

Huge +1 here. It's really annoying that NOMAD_ALLOC_DIR is /alloc, but the _actual_ location is/${NOMAD_TASK_NAME}/${NOMAD_ALLOC_DIR}, and that there's no env var for the root /alloc.

[maximka777]

Hey @dadgar , is this behaviour documented somewhere?

[danlsgiga]

I faced this same issue now. I already use the shared allocation dir to have a filebeat sidecar to various other tasks but when I tried to share an artifact among tasks (one being exec and the other one being docker) the job failed hard because the service could not start for not being able to find the files.

[ccakes]

☝️

Same - even after this thread and knowing what to look for I couldn't find this documented _anywhere_...

        destination = "../alloc/app.conf"

Using that in your template stanza causes /alloc/app.conf to exist inside the Docker container for all tasks within a group. This isn't really very intuitive and given that Nomad (mostly) isolates the host filesystem with Docker, I didn't even consider trying ../ because I assumed it'd be rejected on eval.

Voting to get this fixed to be consisted with the other task drivers or at least documented under the Docker driver

[tgross]

Cross-referencing https://github.com/hashicorp/nomad/issues/8919