Nomad: [0.9.3] [question] Debug Docker plugin with auth config file

[kaspergrubbe]

Nomad version

Nomad v0.9.3 (c5e8b66c3789e4e7f9a83b4e188e9a937eea43ce)

Operating system and Environment details

Amazon Linux 2 on AWS.

Issue

I am having trouble understanding the format of the Docker auth configuration file, and how to debug the issues I am seeing.

First I did docker login on one of our servers, and that wrote a nice file to /home/ec2-user/.docker/config.json:

{
  "auths": {
    "https://index.docker.io/v1/": {
      "auth": "base64redacted=="
    }
  },
  "HttpHeaders": {
    "User-Agent": "Docker-Client/18.06.1-ce (linux)"
  }
}

When I base64decode the data inside auth, it comes out as dockerhubusername:password.

This is the file I need to feed into the Nomad Docker driver, right? So I did just that, I took the file from /home/ec2-user/.docker/config.json and moved it to /etc/docker-auth.json and this is a snippet from my Nomad configuration:

plugin "docker" {
  auth {
    config = "/etc/docker-auth.json"
  }

  gc {
    image       = true
    image_delay = "120m"
    container   = true
  }
}

However when I try to deploy my application I just see authorization errors:

Jul 11 18:31:52 ip-10-1-1-239.eu-central-1.compute.internal nomad[4581]: 2019-07-11T18:31:52.440Z [ERROR] client.alloc_runner.task_runner: running driver failed: alloc_id=ccc98b4e-3921-1fbf-fedf-948579fb6b97 task=irssi error="Failed to pull `kaspergrubbe/sshd-irssi:0.0.3`: API error (404): pull access denied for kaspergrubbe/sshd-irssi, repository does not exist or may require 'docker login'"

Nowhere in the logs can I see anything about the /etc/docker-auth.json file, or if the format is wrong, or if the file is being read:

[ec2-user@ip-10-1-1-239 ~]$ journalctl -u nomad.service | grep docker-auth.json
[ec2-user@ip-10-1-1-239 ~]$
[ec2-user@ip-10-1-1-239 ~]$ journalctl -u nomad.service | grep auth
[ec2-user@ip-10-1-1-239 ~]$

Am I missing something really obvious?

[kaspergrubbe]

Here is my job-file:

job "irssi" {
  datacenters = ["eu-central-1"]
  type = "service"

  update {
    max_parallel = 1
    min_healthy_time = "10s"
    healthy_deadline = "3m"
    progress_deadline = "10m"
    auto_revert = false
    canary = 0
  }

  migrate {
    max_parallel = 1
    health_check = "checks"
    min_healthy_time = "10s"
    healthy_deadline = "5m"
  }

  group "group" {
    count = 1

    restart {
      attempts = 2
      interval = "30m"
      delay = "15s"
      mode = "fail"
    }

    ephemeral_disk {
      size = 300
    }

    task "irssi" {
      driver = "docker"

      config {
        image = "kaspergrubbe/sshd-irssi:0.0.3"

        port_map {
          sshd = 2222
        }
      }

      resources {
        cpu    = 200
        memory = 100
        network {
          mbits = 10
          port "sshd" {}
        }
      }

      service {
        name = "irssi"
        port = "sshd"
        check {
          name     = "alive"
          type     = "tcp"
          interval = "60s"
          timeout  = "2s"
        }
      }
    }
  }
}

[kaspergrubbe]

Not sure if https://github.com/hashicorp/nomad/issues/2957 is somehow relevant.

[kaspergrubbe]

Non working config (with the plugin stanza):

log_level = "DEBUG"

region     = "europe"
datacenter = "eu-central-1"

data_dir = "/opt/nomad"
leave_on_terminate = true

client {
  enabled = true
}

consul {
  address = "127.0.0.1:8500"
  server_service_name = "nomad"
  client_service_name = "nomad-client"
  auto_advertise = true
  server_auto_join = true
  client_auto_join = true
}

plugin "docker" {
  auth {
    config = "/etc/docker-auth.json"
  }
}

Working config (with the deprecated syntax without the stanza):

log_level = "DEBUG"

region     = "europe"
datacenter = "eu-central-1"

data_dir = "/opt/nomad"
leave_on_terminate = true

client {
  enabled = true

+  options {
+    "docker.auth.config" = "/etc/docker-auth.json"
+  }
}

consul {
  address = "127.0.0.1:8500"
  server_service_name = "nomad"
  client_service_name = "nomad-client"
  auto_advertise = true
  server_auto_join = true
  client_auto_join = true
}

Maybe it is an issue with the plugin stanza configuration somehow?

[kaspergrubbe]

By reading this I realised my mistake:

https://github.com/hashicorp/nomad/blob/88c03d088850a07a505fff91ac3909cbfe71b656/drivers/docker/config.go#L137-L165

This fails silently:

plugin "docker" {
  auth {
    config = "/etc/docker-auth.json"
  }
}

This works:

plugin "docker" {
+  config {
    auth {
      config = "/etc/docker-auth.json"
    }
+  }
}

Shouldn't the docker plugin have thrown an error instead of silently ignoring a configuration mistake?

[preetapan]

@kaspergrubbe We have server side driver config validation slated for a future release - since drivers can be external the validation for config that's opaque to Nomad at job submission time is tricky to get right. Servers would need to be able to instantiate the driver to validate its config.

Closing this issue since you fixed the syntax and its not a bug.

[kaspergrubbe]

@preetapan Could we perhaps mark it as a documentation bug? Where can I find the exact format of the plugin configuration for the Docker driver (other than the source code)?