Nomad: Docker volume not being mounted on container start

[sarneeh]

Nomad version

0.12.7

Operating system and Environment details

Mac OSX Catalina

Issue

When I try to mount a host volume into a docker task that is dependent on the volume to start - it will fail. For some reason the volume is not being mounted in the moment when the app starts - I get an no such file or directory error.

Reproduction steps

1. Run a Nomad agent with a caddy host volume mounted to a desired path
2. Run the job below

Job file (if appropriate)

job caddy {
  datacenters = ["dc1"]

  group caddy {
    volume "caddy" {
      type      = "host"
      read_only = false
      source    = "caddy"
    }

    task caddy {
      driver = "docker"

      config {
        image   = "caddy"
        command = "ls /opt/caddy"
      }

      volume_mount {
        volume      = "caddy"
        destination = "/opt/caddy"
      }
    }
  }
}

Nomad Client logs (if appropriate)

Failed to start container cf98ce4bb9ecde29891d3935b3013ad3163ee798a8521df03735b90251708ef0: API error (400): OCI runtime create failed: container_linux.go:349: starting container process caused "exec: \"ls /opt\": stat ls /opt: no such file or directory": unknown

[tgross]

Hi @sarneeh!

The error message you're receiving is bubbling up from the OS and has to do with the way the command is being parsed by Docker. It's a little subtle, but it's trying to stat ls /opt (as in, a path with a space and a slash in it!), not /opt. That's because the jobspec is passing the arguments to ls as the command instead of the args. I've tested this jobspec and see a successful mount:

job caddy {
  datacenters = ["dc1"]

  group caddy {
    volume "caddy" {
      type      = "host"
      read_only = false
      source    = "caddy"
    }

    task caddy {
      driver = "docker"

      config {
        image   = "caddy"
        command = "ls"
        args = ["/opt/caddy"]
      }

      volume_mount {
        volume      = "caddy"
        destination = "/opt/caddy"
      }
    }
  }
}

[sarneeh]

Oh, wow! I totally missed that property - I thought that the command works exactly as the Dockerfile's RUN. It works for me now, thank you very much for pointing this out and explaining the case 😄