Nomad: Cannot download private git repo with artifact stanza

[amelnikov-mylivn]

Nomad version

Nomad v0.5.6

Operating system and Environment details

Client running in Docker (17.06 CE):
Linux b82934a8bd68 4.8.0-1-amd64 #1 SMP Debian 4.8.7-1 (2016-11-13) x86_64 x86_64 x86_64 GNU/Linux
Server running agent and nodes:
Linux my-hostname 4.8.0-1-amd64 #1 SMP Debian 4.8.7-1 (2016-11-13) x86_64 GNU/Linux

Issue

Nomad or maybe go-getter (cannot check) fails when specifying private git repo in artifact stanza

Reproduction steps

• create job with artifact stanza using private git repo following Nomad documentation
• run job
• see results

Expected result

• job is allocated and running

Actual result

• got error message:

Recent Events:
Time                    Type                      Description
07/10/17 19:15:39 CEST  Not Restarting            Error was unrecoverable
07/10/17 19:15:39 CEST  Failed Artifact Download  failed to download artifact "[email protected]:group/repo-name.git": failed to parse source URL "[email protected]:group/repo-name.git": parse [email protected]:group/repo-name.git: first path segment in URL cannot contain colon
07/10/17 19:15:39 CEST  Downloading Artifacts     Client is downloading artifacts
07/10/17 19:15:39 CEST  Task Setup                Building Task Directory
07/10/17 19:15:39 CEST  Received                  Task received by client

Job file

...
      artifact {
        source      = "[email protected]:group/repo-name.git"
        destination = "repo"
        options {
          sshkey = "${CONFIG_READER_SSH_KEY}"
          ref = "${CI_COMMIT_SHA}"
        }
      }
...

[amelnikov-mylivn]

While digging into unit tests of go-getter I found that the right (probably) way to get private git repo is

...
      artifact {
        source      = "git::ssh://[email protected]/group/repo-name"
        destination = "repo"
        options {
          sshkey = "${CONFIG_READER_SSH_KEY}"
          ref = "${CI_COMMIT_SHA}"
        }
      }
...

BUT
It seems interpolation is not supported in sshkey and ref ...

Recent Events:
Time                    Type                      Description
07/10/17 21:56:19 CEST  Restarting                Task restarting in 29.086400192s
07/10/17 21:56:19 CEST  Failed Artifact Download  failed to download artifact "git::ssh://[email protected]/group/repo-name": error downloading 'ssh://[email protected]/group/repo-name?ref=%24%7BCI_COMMIT_SHA%7D&sshkey=%24%7BCONFIG_READER_SSH_KEY%7D': illegal base64 data at input byte 0
07/10/17 21:56:19 CEST  Downloading Artifacts     Client is downloading artifacts
07/10/17 21:55:49 CEST  Restarting                Task restarting in 29.432326454s
07/10/17 21:55:49 CEST  Failed Artifact Download  failed to download artifact "git::ssh://[email protected]/group/repo-name": error downloading 'ssh://[email protected]/group/repo-name?ref=%24%7BCI_COMMIT_SHA%7D&sshkey=%24%7BCONFIG_READER_SSH_KEY%7D': illegal base64 data at input byte 0
07/10/17 21:55:49 CEST  Downloading Artifacts     Client is downloading artifacts
07/10/17 21:55:49 CEST  Task Setup                Building Task Directory
07/10/17 21:55:49 CEST  Received                  Task received by client

[lfarnell]

is the sshkey value base64 encoded?

An SSH private key to use during clones. The provided key must be a base64-encoded string. For example, to generate a suitable sshkey from a private key file on disk, you would run base64 -w0 .

[amelnikov-mylivn]

Yes. It is.
However you can see in logs I attached, that value whatever it is (base64 or not) was not inserted instead of anchor ${CONFIG_READER_SSH_KEY}.
Another option is that they log values without interpolation. But I 100% sure I provide base64 string.

[amelnikov-mylivn]

I am really confused now. I looked into 0.5.6 release code and found that URLs and options should be interpolated:

client/getter/getter.go:54

func getGetterUrl(taskEnv *env.TaskEnvironment, artifact *structs.TaskArtifact) (string, error) {
    taskEnv.Build()
    source := taskEnv.ReplaceEnv(artifact.GetterSource)

    // Handle an invalid URL when given a go-getter url such as
    // [email protected]:hashicorp/nomad.git
    gitSSH := false
    if strings.HasPrefix(source, gitSSHPrefix) {
        gitSSH = true
        source = source[len(gitSSHPrefix):]
    }

    u, err := url.Parse(source)
    if err != nil {
        return "", fmt.Errorf("failed to parse source URL %q: %v", artifact.GetterSource, err)
    }

    // Build the url
    q := u.Query()
    for k, v := range artifact.GetterOptions {
        q.Add(k, taskEnv.ReplaceEnv(v))
    }
    u.RawQuery = q.Encode()

    // Add the prefix back
    url := u.String()
    if gitSSH {
        url = fmt.Sprintf("%s%s", gitSSHPrefix, url)
    }

    return url, nil
}

Either I am doing something wrong or my environment variables are not in taskEnv at the moment of this function execution.

[srivignessh]

I am able to pull using this job template

      artifact {
        source      = "git::ssh://[email protected]/group/repo-name"
        destination = "repo"
        options {
          sshkey = "<Base 64 string>"
        }
      }

However, I can't use vault template stanza to pull the repo.
ITs prompts the following error in log

Enter passphrase for key '/tmp/go-getter336367246': 2017/07/17 23:14:48.124869 [DEBUG] http: Request /v1/jobs?prefix=docs (1.425495ms)

[dadgar]

@srivignessh Can you show me what you mean "use vault template stanza". Not sure I know what you mean.

[srivignessh]

This fails.

   artifact {
     source="git::ssh://[email protected]/group/repo-name"
     destination="/local/repo/"
     options {
       sshkey="${NOMAD_SECRETS_DIR}/pwd"
     }
   }
   template {
     data = "{{ with secret \"secret/key\" }}{{ .Data.value }}{{ end }}"
     destination = "${NOMAD_SECRETS_DIR}/pwd"
   }

Vault 'secret/key' directory is stored the value="base64 string", Vault policy is appropriately configured.

It fails with this error
Enter passphrase for key '/tmp/go-getter336367246': 2017/07/17 23:14:48.124869 [DEBUG] http: Request /v1/jobs?prefix=docs (1.425495ms)

[dadgar]

@srivignessh Ah, yes that is not supported. The content of a file is not read. There is an issue tracking using Vault values in the job file which is what you really need. For the time being the sshkey has to be in plaintext in the file.

[srivignessh]

Please update the issue when it clears.

[tad-lispy]

I'm also unable to get artifact from private Git repository, even when putting base64 encoded, unencrypted private key. I'm getting errors like this:

failed to download artifact "git::ssh://[email protected]/org/repo.git": error downloading 'ssh://[email protected]/org/repo.git?sshkey=LS0t[...]Qo%3D': /usr/bin/git exited with 128: Cloning into '/var/nomad/data/alloc/f6ab43c4-c664-b448-c713-ebed342bc3ab/service/local/app'... Host key verification failed. fatal: Could not read from remote repository. Please make sure you have the correct access rights and the repository exists. 

I believe it's related to https://github.com/hashicorp/go-getter/issues/55 i.e. github.com is not a known host, so ssh client refuses to open connection. How can I work around it?

[tad-lispy]

The workaround is to call

ssh-keygen -R github.com
ssh-keyscan -t rsa github.com >> ~/.ssh/known_hosts

on every node running a Consul client. Not very pretty.

Also, putting literal key in a job file is not very pretty. Is there any way to avoid it? It seems that interpolations just don't work for sshkey in artifact:

failed to download artifact "git::ssh://[email protected]/org/repo.git": error downloading 'ssh://[email protected]/org/repo.git?sshkey=%24%7Bdeployment_key%7D': illegal base64 data at input byte 0 

Below is a feature request. It would be perfect if sshkey (or some new option, say sshkey_file) could take a path to a file containing key (not base64 encoded). Then we could simply use template to render SSH key stored in Vault and pass the path to artifact, like that:

template {
  destination = "secrets/deployment_rsa"
  change_mode = "noop"
  data = "{{with secret "secret/example-app/deployment-keys"}}
    {{.Data.private }}
  {{end}}"
  EOF
}

artifact {
  source = "git::ssh://[email protected]/example-org/example-app.git"
  destination = "local/app"
  options {
    sshkey_file = "${NOMAD_SECRETS_DIR}/deployment_rsa"
  }
}

The way it is right now is really cumbersome and insecure. I believe this could be implemented without changes to go-getter. Nomad could read the file and encode it with base64 before passing it to go-getter. But changing go-getter to accept the file in the URL would be even better, as the risk of leaking the key to logs would be reduced.

[kitotik]

ideally we could hit the vault pki backend to issue the private key. For existing keys outside of vault wouldn't it be even easier to skip rendering the template and just pass the vault secret directly to the options?

Something like:

artifact {
  source = "git::ssh://[email protected]/example-org/example-app.git"
  destination = "local/app"
  options {
    sshkey_file = "{{ with secret "secret/deploy_key }}"
  }
}

[stale[bot]]

Hey there

Since this issue hasn't had any activity in a while - we're going to automatically close it in 30 days. If you're still seeing this issue with the latest version of Nomad, please respond here and we'll keep this open and take another look at this.

Thanks!

[eagle1981]

+1

[bhanuc]

Couldn't get this to work for bitbucket and gitlab projects(Probably because they use dynamic IPs) . This seems like a major ux issue for new users and documentation around this is shallow. Giving up on using git with nomad for now, will try things with containers.

[mashintsev]

+1

[analytically]

We could use this as well :-)

[dbolack]

+1

[joshuaclausen]

Downloading a specific file from a private github repo has been problematic for me, especially on Windows hosts that run Nomad. Doing the "ssh-keyscan -H github.com" implies every jobspec that needs to pull files from a private github repo need to have an additional task in the taskgroups.

My workaround has been to run the Python project "githubdl" in a task. This works on both Windows and Linux Nomad hosts. One bonus is that it's a lot easier to see exactly what's going on when troubleshooting jobspecs that need to download something from a private Github repo. You have to add some additional consideration in the task that executes the file that's being downloaded.

I can see the difficulty in this particular kind of use case, so I figured I'd mention a workflow that might help.

Here's the githubdl project: https://pypi.org/project/githubdl/

[jimrazmus]

+1

[Luladjiev]

Same, trying to download from private github repo. Looking at node's logs there is the authenticity of host can't be established and asks if I want to continue or not.

[IceKickr]

+1

[anapsix]

on windows hosts, this is still troublesome..
if having to pre-connect to git is on every target host is annoying
the following isn't pretty, but works on windows hosts (unless I've messed up string interpolation with env variables)

task "update_scripts" {
  driver = "raw_exec"

  env {
    GIT_SSH_COMMAND = "ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no"
    NOMAD_SSH_KEY = "%NOMAD_SSH_KEY%"  # can replace these from CI
    ARTIFACT_SRC = "%ARTIFACT_SOURCE%" # or use parametrized job
    ARTIFACT_DST_BASE = "%ARTIFACT_DESTINATION_BASE%"
    ARTIFACT_DST = "%ARTIFACT_DESTINATION%"
  }

  config {
    command = "powershell"
    args    = [
      "-command",
      <<-EOS
      [System.Text.Encoding]::UTF8.GetString([System.Convert]::FromBase64String($env:NOMAD_SSH_KEY)) | Out-File -Encoding ASCII ~/.ssh/id_rsa;
      If (Test-Path "$($env:ARTIFACT_DST_BASE)/$($env:ARTIFACT_DST)" {
        cd "$($env:ARTIFACT_DST_BASE)/$($env:ARTIFACT_DST)";
        git fetch;
        git reset --hard origin/master;
      } Else {
        cd $env:ARTIFACT_DST_BASE;
        git clone $env:ARTIFACT_SRC $env:ARTIFACT_DST_BASE;
      }
      EOS
    ]
  }

  lifecycle {
    hook    = "prestart"
    sidecar = false
  }
}