Nomad: Docker volumes (bind mounts) pointing to a concrete file aren't updated when the file changes

[rkettelerij]

Nomad version

Nomad v0.8.3 (c85483da3471f4bd3a7c3de112e95f551071769f)

Operating system and Environment details

Ubuntu 16.04
Docker version 18.03.0-ce, build 0520e24

Issue

If you create a file using the template stanza and map this to a location in a Docker container then the contents of the file in Docker are never updated, even though Nomad (through the template stanza) has rendered a new file. So basically you're dealing with stale files in Docker.

Reproduction steps

• Create a job file with a docker container, see example below.
• Put a template stanza in the job that reads something from an external source (like Consul or Vault). Set the change_mode to signal and choose SIGHUP as the signal.
• Map this file to a location in the Docker container, e.g: volumes = [ "local/foo.txt:/var/foo.txt" ].
• Deploy the job to Nomad
• Enter the Docker container e.g. docker exec -it <name> bash and observe the contents of /var/foo.txt
• Now trigger a re-render of the template stanza (consul-template). For example by killing a service in Consul.
• Observe that the contents on the host has changed (in /var/lib/nomad/alloc/.../.../foo.txt)
• Observe that the contents of /var/foo.txt hasn't changed! << this is the bug
• Observe that the contents of /local/foo.txt has changed, the default mappings (alloc/, local/, secrets/) are always correctly updated.

One way around this bug to use restart as the template's change_mode since this kills the docker container and therefore always creates a new volume mount. But this causes downtime. Since I'm using the template to render a Nginx config I'd rather not kill the Docker container but prefer to sent a SIGHUP to trigger a reload in Nginx.

Note that this is a Nomad issue not a Docker issue. When I manually create a Docker container and map a concrete file to a location in the container (e.g. docker run -d --name devtest --mount type=bind,source=/home/myname/foo/foo.conf,target=/etc/foo/foo.conf nginx:mainline) and update the file (e.g. echo 'test' >> /home/myname/foo/foo.conf) the changes are always reflected in the container.

Job file (if appropriate)

job "nomad-issue" {
  datacenters = ["dc1"]
  type = "service"

  constraint {
    distinct_hosts = true
  }

  group "nginx" {
    count = "1"

    task "nginx" {
      driver = "docker"

      config {
        image = "nginx:mainline"
        port_map {
          nginx = 443
        }
        volumes = [
          "local/foo.txt:/var/foo.txt",
        ]
      }

      template {
        data = <<EOF
time: {{ timestamp }}
{{range service "myservice"}}server {{.Address}}:{{.Port}};{{end}}
EOF
        destination   = "local/foo.txt"
        change_mode   = "signal"
        change_signal = "SIGHUP"
      }

      service {
        name = "nomad-issue"
        port = "nginx"
      }

      resources {
        cpu = 500
        memory = 256
        network {
          mbits = 500
          port "nginx" {
            static = 443
          }
        }
      }
    }

  }
}

[preetapan]

Looks similar to #4770

[rkettelerij]

I've stumbled upon a user with the exact same issue: https://groups.google.com/forum/#!topic/nomad-tool/s9cw-GeFF8M

[pmcatominey]

I had this issue, mounting the directory rather than the file directly resolved it.

https://github.com/moby/moby/issues/15793

[pmcatominey]

Seems that consul template copies the file into place rather than editing in place.

[vkiranananda]

The same problem.

      template {
        data        = "{{ range ls \"fedsp/apache2/sites-enabled\" }} {{ .Value }} \n\n {{ end }}"
        destination   = "sites.conf"
        change_mode   = "signal"
        change_signal = "SIGUSR1"
      }

file don't update (

[mud5150]

I am also experiencing this issue. Current workaround is to build my own container layer that symlinks from /local to the location I need in /etc. @preetapan Any idea when someone will look at this?

[polarathene]

@mud5150 it's not really solvable. It's described in the issue that was linked to in an earlier comment.

Basically when you do a bind mount, the directory or file has this thing called an inode which points to the file/directory. If you make a change to this file or directory, some software may replace the file causing a new inode to appear. For a single file, the filepath that was mounted, points to the original file inode, but on the host system, your filepath may now point to a new inode, but the docker container is only aware of the inode of the original file.

Some text editors have different settings to avoid the changing of an inode, thus it's not an issue with Docker or Nomad, you must take care of it on your end.

Mounting a directory instead should work, as the inode is now being watched on the directory, which the software like text editors don't tamper with, thus notifications of updates to it's contents works properly.

[rkettelerij]

@polarathene I'm not sure you're right in saying this isn't a Nomad issue. See my original comment:

Note that this is a Nomad issue not a Docker issue. When I manually create a Docker container and map a concrete file to a location in the container (e.g. docker run -d --name devtest --mount type=bind,source=/home/myname/foo/foo.conf,target=/etc/foo/foo.conf nginx:mainline) and update the file (e.g. echo 'test' >> /home/myname/foo/foo.conf) the changes are always reflected in the container.

Also as others have pointed out it could be an issue with consul-template. Either way I think this issue should remain open for investigation. Especially since it's something quite a few folks have run into.

[polarathene]

@rkettelerij I'm not actually a Nomad or Consul user (I am aware of them and might check them out sometime in future) I just noticed this issue on the thread/issue that the comment I mentioned linked to.

Your comparing the echo update to another method updating the file, it's not the same. Verify the files inode for both types of modification and I bet you'll have a difference in behavior. If the inode remains the same, it'll update and work as expected by anything watching it.

Alternatively, mount a directory with the file instead, and you should also note that any updates to the file should now behave as you expect them to? If so that'll confirm that what I chimed in to share is correct.

[stale[bot]]

Hey there

Since this issue hasn't had any activity in a while - we're going to automatically close it in 30 days. If you're still seeing this issue with the latest version of Nomad, please respond here and we'll keep this open and take another look at this.

Thanks!

[rkettelerij]

I believe this is still the case. Re-open.

[stale[bot]]

Hey there

Since this issue hasn't had any activity in a while - we're going to automatically close it in 30 days. If you're still seeing this issue with the latest version of Nomad, please respond here and we'll keep this open and take another look at this.

Thanks!

[stale[bot]]

This issue will be auto-closed because there hasn't been any activity for a few months. Feel free to open a new one if you still experience this problem :+1:

[urusha]

Still affected

[angrycub]

This is not a bug. It is an unfortunate interaction between two foundational design choices in two different products. Both of them are performing as designed.

• Consul-template does atomic writes using a write-then-move scheme. This prevents your workload from seeing a half-rendered template at any point. However, this causes the new version of the file to be at a new FS inode.

• Docker mounts single files into the container by using their inodes. So if the file is not updated in place, docker will not see the changes. However, as mentioned before, if consul-template writes the template into the existing file, it would be possible to see work in progress while the template is being written.

The combination of these two factors is what prevents consul-template and docker single file mounts from working like people expect. The workaround from Nomad and Docker's perspective is to use directory mounts in these cases.