Nomad: template not working with absolute directories

[hongkongkiwi]

Nomad version

Nomad v0.12.4 (8efaee4ba5e9727ab323aaba2ac91c2d7b572d84)

(also same with v0.12.5)

Operating system and Environment details

Linux builder0 4.15.0-112-generic 113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Issue

Strangely, the template cannot be rendered when using an absolute path for this specific job. The agent is using default settings, but for testing I also tried this config to client:

"options": {
  "env.blacklist": "CONSUL_TOKEN,CONSUL_HTTP_TOKEN,VAULT_TOKEN,AWS_ACCESS_KEY_ID,AWS_SECRET_ACCESS_KEY,AWS_SESSION_TOKEN,GOOGLE_APPLICATION_CREDENTIALS",
  "template.allow_host_source": true
}

The template simply does not appear when using any absolute path. It only seems to work with this docker when using local/..... Unfortunately in this case gitlab-runner cannot use a custom config location.

Reproduction steps

Run the below job spec, template will not appear, no matter what path is set.

Job file (if appropriate)

job "gitlab-runner" {
  region = "dev"
  datacenters = ["sfo2"]
  type = "service"
  update {
    max_parallel = 1
    min_healthy_time = "10s"
    healthy_deadline = "3m"
    auto_revert = false
    canary = 0
  }
  group "gitlab-runner" {
    count = 1
    restart {
      attempts = 10
      interval = "5m"
      delay = "25s"
      mode = "delay"
    }
    network {
      # mbits = 10
      dns = {
        servers = ["127.0.0.1"]
      }
    }
    service {
      name = "gitlab-runner"
      tags = [
        "nomad"
      ]
      address_mode = "host"
      task = "gitlab-runner"
    }
    volume "data_mount" {
      type      = "csi"
      source    = "builder0-gitlab-runner"
      read_only = false
    }
    task "gitlab-runner" {
      volume_mount {
        volume      = "data_mount"
        destination = "/data"
    read_only   = false
      }
      driver = "docker"
      config {
        network_mode = "host"
        image = "gitlab/gitlab-runner:latest"
    security_opt = ["label=disable"]
        mounts = [{
          type = "bind"
          target = "/var/run/docker.sock"
          source = "/var/run/docker.sock"
          readonly = false
    }]
      }
      template {
        destination   = "/etc/gitlab-runner/config.toml"
        change_mode   = "noop"
        perms = "777"
        data = <<EOH
concurrent = 10
check_interval = 0
log_level = "warning"
listen_address = "0.0.0.0:8092"

[session_server]
  session_timeout = 1800
  listen_address = "0.0.0.0:8093"
  advertise_address = "${attr.unique.network.ip-address}:8093"

[[runners]]
  name = "${meta.service_name}${meta.server_number} DigitalOcean Builder"
  url = "https://www.gitlab.com"
  token = "<my_gitlab_token>"
  limit = 0
  executor = "docker"
  builds_dir = "/data/builds"
  cache_dir = "/data/cache"
  output_limit = 40960
  [runners.custom_build_dir]
    enabled = true
  [runners.docker]
    tls_verify = false
    image = "library/bash"
    privileged = false
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/data:/data", "/var/run/docker.sock:/var/run/docker.sock"]
    shm_size = 0
  [runners.cache]
    [runners.cache.s3]
EOH
      }
      resources {
        cpu    = 100
      }
    }
  }
}

[tgross]

Hi @hongkongkiwi, the template destination is relative to the "task directory". See https://github.com/hashicorp/nomad/issues/8919 for some ambiguity we need to clear up around what "task directory" means, but tl;dr it's the directory above ./local and ./secrets.

Emitting a template into a location other than local/ or secrets/ isn't going to work with the Docker task driver. The typical workaround for this is to either have a symlink to the expected location baked into the container, or to copy the file into the appropriate location at container startup.

If you were using an exec or raw_exec driver, you could set the destination to etc/gitlab-runner/config.toml, because that path would be relative to the directory where Nomad binds in /bin, /etc, and so forth from the host. But in Docker, the template stanza executes before the Docker container is built, so the overlay filesystem that the Docker task driver creates is in a location owned by Docker.

To demonstrate this, consider the following jobspec:

job "example" {
  datacenters = ["dc1"]

  group "groupname" {

    task "taskname" {
      driver = "docker"

      config {
        image = "busybox:1"
        command = "/bin/sh"
        args = ["-c", "sleep 3600"]
      }

      template {
        destination = "etc/runner.toml"
        data = "some data"
      }

      resources {
        cpu    = 500
        memory = 256
      }
    }
  }
}

If we run this and look at the file system on the host, we see:

# tree /var/nomad/alloc/74b4d7a7-7b9f-52fd-c1a1-dca8ab773bc1/
/var/nomad/alloc/74b4d7a7-7b9f-52fd-c1a1-dca8ab773bc1/
├── alloc
│   ├── data
│   ├── logs
│   │   ├── task.stderr.0
│   │   └── task.stdout.0
│   └── tmp
└── taskname
    ├── etc
    │   └── runner.toml
    ├── local
    ├── secrets
    └── tmp

9 directories, 3 files

But if we look at the /etc directory inside the container, we see:

$ nomad alloc exec 74b /bin/sh
/ # ls etc
group        hosts        mtab         passwd       shadow
hostname     localtime    network      resolv.conf

This is because only the following directories are mounts into the container:

$ docker inspect d29 | jq '.[0].Mounts'
[
  {
    "Type": "bind",
    "Source": "/var/nomad/alloc/74b4d7a7-7b9f-52fd-c1a1-dca8ab773bc1/alloc",
    "Destination": "/alloc",
    "Mode": "",
    "RW": true,
    "Propagation": "rprivate"
  },
  {
    "Type": "bind",
    "Source": "/var/nomad/alloc/74b4d7a7-7b9f-52fd-c1a1-dca8ab773bc1/taskname/local",
    "Destination": "/local",
    "Mode": "",
    "RW": true,
    "Propagation": "rprivate"
  },
  {
    "Type": "bind",
    "Source": "/var/nomad/alloc/74b4d7a7-7b9f-52fd-c1a1-dca8ab773bc1/taskname/secrets",
    "Destination": "/secrets",
    "Mode": "",
    "RW": true,
    "Propagation": "rprivate"
  }
]

[angrycub]

@hongkongkiwi - I also encourage people to avoid bind mounting templated files into docker containers, because of how the consul-template library does "atomic" writes by creating a new file and replacing the original. Since Docker mounts single files using their inodes, this can cause your container to not see an update to the templated file.

Also, inside the template stanza, you need to use the env function to get the values for interpolated variables.

I took a quick pass at updating your job file I made a few changes, but this one one seems like it might get you unblocked. I've attached it to this comment inside a <details> block

Key changes:

• Used the local directory for templating into.
• Bind mounted the entire local directory into /etc/gitlab-runner.
• Changed the ${ ... } references inside the template to {{ env ... }} template actions.

👀 Click here to see job

job "gitlab-runner" {
  datacenters = ["dc1"]
  type = "service"
  update {
    max_parallel = 1
    min_healthy_time = "10s"
    healthy_deadline = "3m"
    auto_revert = false
    canary = 0
  }
  group "gitlab-runner" {
    count = 1
    restart {
      attempts = 10
      interval = "5m"
      delay = "25s"
      mode = "delay"
    }
    network {
      # mbits = 10
      dns = {
        servers = ["127.0.0.1"]
      }
    }
    service {
      name = "gitlab-runner"
      tags = [
        "nomad"
      ]
      address_mode = "host"
      task = "gitlab-runner"
    }
    task "gitlab-runner" {
      driver = "docker"
      config {
        network_mode = "host"
        image = "gitlab/gitlab-runner:latest"
  security_opt = ["label=disable"]
        mounts = [
        {
          type = "bind"
          target = "/var/run/docker.sock"
          source = "/var/run/docker.sock"
          readonly = false
  },
          {
          type = "bind"
          source = "local"
          target = "/etc/gitlab-runner"
          readonly = false
  }
  ]
      }
      template {
        destination   = "local/config.toml"
        change_mode   = "noop"
        perms = "777"
        data = <<EOH
concurrent = 10
check_interval = 0
log_level = "warning"
listen_address = "0.0.0.0:8092"

[session_server]
  session_timeout = 1800
  listen_address = "0.0.0.0:8093"
  advertise_address = "{{ env "attr.unique.network.ip-address" }}:8093"

[[runners]]
  name = "{{env "meta.service_name"}}{{ env "meta.server_number"}} DigitalOcean Builder"
  url = "https://www.gitlab.com"
  token = "<my_gitlab_token>"
  limit = 0
  executor = "docker"
  builds_dir = "/data/builds"
  cache_dir = "/data/cache"
  output_limit = 40960
  [runners.custom_build_dir]
    enabled = true
  [runners.docker]
    tls_verify = false
    image = "library/bash"
    privileged = false
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/data:/data", "/var/run/docker.sock:/var/run/docker.sock"]
    shm_size = 0
  [runners.cache]
    [runners.cache.s3]
EOH
      }
      resources {
        cpu    = 100
      }
    }
  }
}

[tgross]

I think we've covered this one sufficiently and I see a 🎉 from @hongkongkiwi in reply to @angrycub, so I'm going to close this out. I'm definitely going to take a crack at improving the documentation around this as noted in #8919