Nomad: named docker volume doesn't work anymore with 0.9.0

[MorphBonehunter]

Nomad version

Nomad v0.8.7 (21a2d93eecf018ad2209a5eab6aae6c359267933+CHANGES)
Nomad v0.9.0 (18dd59056ee1d7b2df51256fe900a98460d3d6b9)

Operating system and Environment details

ArchLinux

Issue

After upgrading to nomad 0.9.0 named docker volumes are not working anymore.

Reproduction steps

• nomad config:

data_dir  = "/var/lib/nomad"
log_level = "debug"
server {
    enabled = true
    bootstrap_expect = 1
}
client {
    enabled = true
}

• run the job with nomad 0.8.7 and inspect resulting docker container:

---snip---
        "HostConfig": {
            "Binds": [
                "/var/lib/nomad/alloc/d6d8e0cf-61fb-8c46-d278-e56ab11337f6/alloc:/alloc",
                "/var/lib/nomad/alloc/d6d8e0cf-61fb-8c46-d278-e56ab11337f6/test/local:/local",
                "/var/lib/nomad/alloc/d6d8e0cf-61fb-8c46-d278-e56ab11337f6/test/secrets:/secrets",
                "test:/testing"
            ],
---snip---
        "Mounts": [
            {
                "Type": "bind",
                "Source": "/var/lib/nomad/alloc/d6d8e0cf-61fb-8c46-d278-e56ab11337f6/alloc",
                "Destination": "/alloc",
                "Mode": "",
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/var/lib/nomad/alloc/d6d8e0cf-61fb-8c46-d278-e56ab11337f6/test/local",
                "Destination": "/local",
                "Mode": "",
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/var/lib/nomad/alloc/d6d8e0cf-61fb-8c46-d278-e56ab11337f6/test/secrets",
                "Destination": "/secrets",
                "Mode": "",
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Type": "volume",
                "Name": "test",
                "Source": "/var/lib/docker/volumes/test/_data",
                "Destination": "/testing",
                "Driver": "local",
                "Mode": "z",
                "RW": true,
                "Propagation": ""
            }
        ],
---snip---

• deploy the same job on nomad 0.9.0 and inspect resulting docker container:

---snip---
        "HostConfig": {
            "Binds": [
                "/var/lib/nomad/alloc/4d497db5-12fc-9895-0fe0-790089845842/alloc:/alloc",
                "/var/lib/nomad/alloc/4d497db5-12fc-9895-0fe0-790089845842/test/local:/local",
                "/var/lib/nomad/alloc/4d497db5-12fc-9895-0fe0-790089845842/test/secrets:/secrets",
                "/var/lib/nomad/alloc/4d497db5-12fc-9895-0fe0-790089845842/test/test:/testing"
            ],
---snip---
        "Mounts": [
            {
                "Type": "bind",
                "Source": "/var/lib/nomad/alloc/4d497db5-12fc-9895-0fe0-790089845842/test/test",
                "Destination": "/testing",
                "Mode": "",
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/var/lib/nomad/alloc/4d497db5-12fc-9895-0fe0-790089845842/alloc",
                "Destination": "/alloc",
                "Mode": "",
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/var/lib/nomad/alloc/4d497db5-12fc-9895-0fe0-790089845842/test/local",
                "Destination": "/local",
                "Mode": "",
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/var/lib/nomad/alloc/4d497db5-12fc-9895-0fe0-790089845842/test/secrets",
                "Destination": "/secrets",
                "Mode": "",
                "RW": true,
                "Propagation": "rprivate"
            }
        ],

Job file (if appropriate)

cat example.job 
job "test" {
    datacenters = ["dc1"]
    type = "service"
    group "test" {
        task "test" {
            driver = "docker" 
            config {
                image = "alpine:3.9"
                args = [ "sleep", "1000" ]
                volumes = [ "test:/testing" ]
                volume_driver = "local"
            }
        }
    }
}

In nomad 0.9.0 the same job result not in an volume mount but an bind mount.

[endocrimes]

Hey @MorphBonehunter,

Sorry about this, it looks to be a regression in how we handle generating container binds in 0.9 due to some consistency changes around how we handled an empty driver vs "local" drivers.

While we figure out the correct fix, it should be possible for you to work around this by using the mounts stanza like so:

    task "test" {
      driver = "docker"

      config {
        image = "redis:3.2"

        mounts = [
          {
            type   = "volume"
            target = "/testing"
            source = "test"
          },
        ]
      }

[MorphBonehunter]

Thanks for your suggestion @dantoml, this is working for me.

[schmichael]

As this was an unintentional backward compatibility break we intend to restore the 0.8.7 behavior in Nomad 0.9.1.

We still recommend using mounts when possible as it has much easier to reason about behavior. The pre-0.9.0 Nomad volumes behavior is unintentionally subtle and tricky, but we intend to maintain it.

[MorphBonehunter]

hey @schmichael should ne no problem to use the mounts over the volumes option.
Maybe this could be mentioned in the docs on both places that the mount option is the better one so that in an future version the volume option could be deprecated :)
Thanks for quick response!