Nomad: [Feature] vault secrets in artifact stanza

[ryanmickler]

I've been trying to find a way to securely pull artifacts

artifact {
  source      = "..."
    destination = "..."
    options { 
      aws_access_key_id = "XYXYX"
      aws_access_key_secret = "XYXYXYX"
  }
}

however, this requires me to hard code in my access keys. What i'd love to be able to do is something like:

vault {
  policies = ["s3_artifact_bucket"]
}

artifact {
  source      = "..."
    destination = "..."
    options { 
      aws_access_key_id = "<pulled from vault>"
      aws_access_key_secret = "<pulled from vault>"
  }
}

Of course, this would never word for a few reasons, mainly because id' have to do two 'reads' of the credentials, which would generate new keys.

Perhaps i could propose something of the form:

vault {
  policies = ["s3_artifact_bucket"]
}

artifact {
  source      = "..."
    destination = "..."
    options {
       env = <<BLOCK
{{ with secret "aws/creds/s3_artifacts" }}
aws_access_key_id = "{{.Data.access_key}}"
aws_access_key_secret = "{{.Data.secret_key}}"{{end}}
BLOCK
    }
}

Although, there's is probably an issue here with regards this go-getter pull is happening on the host rather than in the container, so we'll need the host to have the keys injects.

Is there anyway to achieve this in nomad without having to pull the artifact from inside the container itself?

[latchmihay]

You can probably add a template stanza like this:

template {
  data = <<BLOCK
{{ with secret "aws/creds/s3_artifacts" }}
aws_access_key_id="{{.Data.access_key}}"
aws_access_key_secret="{{.Data.secret_key}}"{{end}}
BLOCK
  destination = "secrets/aws.env"
  env         = true
}

This in term will create 2 env interpolation variables that you can use like this

artifact {
  source      = "..."
    destination = "..."
    options { 
      aws_access_key_id = "${aws_access_key_id}"
      aws_access_key_secret = "${aws_access_key_secret}"
  }
}

I am using this method to set splunk token for Logging. Here is my working example in json format just for reference.

"logging": [
                {
                  "config": [
                    {
                      ....
                      "splunk-token": "${SPLUNK_TOKEN}",
                      ...
                      ...
                    }
                  ],
                  "type": "splunk"
                }
              ],

            "Templates": [
              {
                "ChangeMode": "restart",
                "ChangeSignal": "",
                "DestPath": "secrets/SPLUNK_TOKEN.env",
                "EmbeddedTmpl": "SPLUNK_TOKEN={{with secret \"/secret/jobs/splunk\"}}{{.Data.token}}{{end}}\n",
                "Envvars": true,
                "LeftDelim": "{{",
                "Perms": "0644",
                "RightDelim": "}}",
                "SourcePath": "",
                "Splay": 5000000000,
                "VaultGrace": 15000000000
              }
            ],

[nicolai86]

@latchmihay I tried this to retrieve S3 credentials from vault but it did not work for me combined with the artifact element...

[dmartin-isp]

++
It's painful to have an artifact section that supports S3 and not have the ability to pull the aws credentials from Vault.

[jhitt25]

Beyond AWS credentials, you can't secure git ssh keys or https credentials either. There appears to be no variable interpolation in the artifact stanza (at least not that i can get working). Trying to put variables in the environment then pulling in an env var does not work in either source or options.

[ybovard]

could also be nice to have a way to use it for docker hub downloads:

...
  task "xxx" {
            driver = "docker"
            config {
                image = "my/private:image"
                ssl = true
                auth {
                    server_address = "dockerhub.io"
                    username = "{{with secret \"/secret/docker/hub\"}}{{.Data.user}}{{end}}"
                    password = "{{with secret \"/secret/docker/hub\"}}{{.Data.password}}{{end}}"
                }
                port_map = {
                    http = 8080
                }
            }

[radcool]

We'd also love this to be possible as we pull artifacts from GitLab using its API and right now we have to put the API key in the clear in the jobspec. We'd much rather store in Vault...

[sam701]

@dadgar It seems this request does not get much attention. What is the recommended approach for handling secrets in artifact stanza for now? Should this stanza be better avoided? Small artifacts can be easily inlined in templates. But what about large ones? Is it preferred to pack everything into a docker image? I'd appreciate your thoughts on this topic.

[ryanmickler]

What I've ended up doing is using a local proxy http server that mirrors the contents of the s3 bucket on a in-cluster http address. I use vault to inject s3 credentials into the proxy server, and then i can pull artifacts without credentials locally just from the http server. Its not wildy secure, as any in-cluster user can pull the artifacts, but it works.

[pauldon2]

Recently also faced this problem. I cannot pass data with authorization parameters to the artifact stanza. Is there really no progress on this issue in two years? :(

[JohnyEvil]

@dadgar Any update regarding this issue? Is there any plan to implement this feature?

[schmichael]

Just wanted to share some brainstorming we've been doing internally. No timeline yet, but this is a topic of active discussion. Here are 3 ideas we're kicking around:

1. Retry failed artifacts

This is the hackiest approach, but also the easiest to implement: when an artifact fails we could continue with the task setup ("prestart hooks" in internal jargon) and retry failed artifacts after the template stanza has been processed and populates the environment with necessary secrets.

The big pro here is that artifacts could use secrets from the environment and it would Just Work. Nothing new to learn.

The con is probably obvious: ew. "Expected failures" is never a user experience we want at HashiCorp. Not to mention the template stanza is already a source of significant debugging pain since it cannot be statically validated and instead you have to run your job and monitor its progress to see if the template executed as desired. Now as you're monitoring template execution you first see an artifact failure which even if expected really doesn't make for a pleasant operational experience.

We may be able to special case artifact failures and make this approach a lot more palatable. We could also add a new flag to artifacts like artifact.post_template = true to force the intended ordering

2. New credentials stanza

Does anybody else find using template for all secrets a bit painful? It's an unnecessary layer of abstraction when you just want to use secrets elsewhere in your jobspec (eg artifact or task.config).

We could add a new credential (or secret ... naming is hard) stanza to allow using Vault secrets _without_ having to create a template.

By default these secrets would not be exposed to the task via environment variables. I think this would be a nice security benefit to keep from having to expose S3 and Docker Hub credentials to services when they're only needed internally by the Nomad client. They'd only ever be in memory, and ephemerally during task prestart at that.

Expand for an example that covers the artifact and task.config use cases above:

task "xxx" {
    vault {
        policies = ["s3_artifact_bucket"]

    credentials "aws/creds/s3_artifacts" {
            # A bit unfortunate to mix hardcoded fields like `name` with dynamic fields like `access_key`.
            # Probably want to iterate on this design.
            name = "s3"

            # Do we need full templating here or just implicitly reference
            # fields on .Data?
            access_key = "access_key" 
            secret_key = "secret_key"
        }
    credentials "secret/docker/hub" {
        name = "docker"
            user = "user" 
            password = "password"
        }
    }
    artifact {
      source      = "..."
        destination = "..."
        options { 
          aws_access_key_id = "${credentials.s3.access_key}"
          aws_access_key_secret = "${credentials.s3.secret_key}"
      }
    }
    driver = "docker"
    config {
        image = "my/private:image"
        ssl = true
        auth {
            server_address = "dockerhub.io"
            username = "${credentials.docker.user}"
            password = "${credentials.docker.password}"
        }
        port_map = {
            http = 8080
        }
    }
}

We may want to bubble credential up a level in case we ever migrate secrets to a plugin/provider model instead of just Vault.

3. HCL2 vault_secret function

Similar to above but we could use a vault_secret function instead of a new stanza:

dockerRegCredential = vault_secret("foo/bar")
config {
  username = ${dockerRegCredential.user_name}
  password = ${dockerRegCredential.password}
}

The pro is an optimally concise syntax. Secrets can be fetched right where they're used.

The con is that it depends on HCL2 and might be slightly harder to debug than a purely declarative approach like the new stanza. This approach has the most unknowns since Nomad is still evaluating how best to fully integrate HCL2 (it's used internally a bit already).

Feedback welcome!

[FernandoMiguel]

For future clarity, I believe you wanted to use docker creds in the second template

[schmichael]

For future clarity, I believe you wanted to use docker creds in the second template

@FernandoMiguel Good catch! Thanks, updated.

[ryanmickler]

Thanks so much @schmichael for addressing this. I really like solution 2, I think that's quite close to what i was hoping for in the OP.

[ryanmickler]

One minor point, I'd prefer the following syntax

    credentials "docker" {
        path = "secret/docker/hub"
        }

and use full templating, as you suggest

[spuder]

Here is a workaround until this is implemented

• Store git user password in vault
• Nomad fetches git password from vault and inject it as environment variable
• Template creates script to clone git repo
• Life cycle hook 'prestart' ensures the git repo is started before the other services

    task "git-clone" {
      template {
        data = <<EOH
DEPLOY_PASSWORD="{{with secret "secret/data/git_password"}}{{.Data.data.git_password}}{{end}}"
        EOH
        destination = "secret/deploypass.sh"
        env = true
      }
      template {
        data = <<EOH
#!/bin/bash
git --version
CODE_REPO=/alloc
if [ -z ${DEPLOY_PASSWORD+x} ]
then 
  echo "DEPLOY_PASSWORD is not set, script most likely not running under nomad"
else
  if [ ! -d ${CODE_REPO}/.git ]
  then
    git clone --depth 5 --branch ${BRANCH} "https://foobar:${DEPLOY_PASSWORD}@gitlab.example.com/foo/foobar.git"
  else
    cd $CODE_REPO
    git checkout ${BRANCH}
    git pull
  fi
fi
        EOH
        destination = "alloc/git-clone/clone.sh"
        perms = "755"
      }
      driver = "exec"
      config {
        command = "/alloc/git-clone/clone.sh"
      }
      env {
        "PATH" = "/bin:/sbin:/usr/bin:/usr/local/bin"
      }
      lifecycle {
        hook = "prestart"
        sidecar = false

      }
    }