Salt: State git.latest does not work with SSH

[craigafinch]

Description of Issue/Question

I cannot access a Git repository via SSH with the git.latest state. My issue seems to be identical to 32237.

Setup

Here is a fragment from my SLS file:

  git.latest:
    - name: "ssh://[email protected]:MyOrg/my-repo.git"
    - branch: feature/my-feature
    - target: /var/www/html/my-website
    - identity: "/root/.ssh/id_rsa"
    - require:
      - ssh_known_hosts: github.com
      - php
      - api-deploy-key

  cmd.run:
    - name: git ls-remote [email protected]:MyOrg/my-repo.git
    - runas: root

Steps to Reproduce Issue

Apply the states.

Results from these two states on Master are shown below. The results of cmd.run verify that this minion can actually access the repo with its credentials:

          ID: my-repo
    Function: git.latest
        Name: ssh://[email protected]:MyOrg/my-repo.git
      Result: False
     Comment: Failed to check remote refs: Unable to authenticate using identity file:

              ssh: Could not resolve hostname github.com:myorg: Name or service not known
              fatal: Could not read from remote repository.

              Please make sure you have the correct access rights
              and the repository exists.
     Started: 22:46:29.768565
    Duration: 31.747 ms
     Changes:   
----------
          ID: my-repo
    Function: cmd.run
        Name: git ls-remote [email protected]:MyOrg/my-repo.git
      Result: True
     Comment: Command "git ls-remote [email protected]:my-org/my-app.git" run
     Started: 22:46:29.800511
    Duration: 328.625 ms
     Changes:   
              ----------
              pid:
                  2483
              retcode:   
                  0
              stderr:
              stdout:

              <snipped>

Results from /var/log/salt/minion:

2016-11-09 22:46:29,799 [salt.loaded.int.module.cmdmod][ERROR   ][2350] Command '['git', 'ls-remote', 'ssh://[email protected]:MyOrg/my-repo.git']' failed with return code: 128
2016-11-09 22:46:29,799 [salt.loaded.int.module.cmdmod][ERROR   ][2350] stderr: ssh: Could not resolve hostname github.com:myorg: Name or service not known
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
2016-11-09 22:46:29,800 [salt.loaded.int.module.cmdmod][ERROR   ][2350] retcode: 128
2016-11-09 22:46:29,800 [salt.state       ][ERROR   ][2350] Failed to check remote refs: Unable to authenticate using identity file:

ssh: Could not resolve hostname github.com:myorg: Name or service not known
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

Versions Report

Salt Version:
           Salt: 2016.3.4

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.7.2
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.8
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
         pygit2: Not Installed
         Python: 2.7.5 (default, Sep 15 2016, 22:37:39)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 15.3.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.1.4

System Versions:
           dist: centos 7.2.1511 Core
        machine: x86_64
        release: 3.10.0-327.22.2.el7.x86_64
         system: Linux
        version: CentOS Linux 7.2.1511 Core

Salt minion:

# salt-minion --version
salt-minion 2016.3.4 (Boron)

[terminalmage]

It's doubtful that this is the same _exact_ issue, as the fix implemented for that issue is the last modification to the git ssh wrapper.

Do you happen to have a github.com entry in /root/.ssh/config or /etc/ssh/ssh_config?

[craigafinch]

I have tried both with, and without, /root/.ssh/config. Here is my current config, which really shouldn't be necessary:

Host github.com
  HostName github.com
  IdentityFile /root/.ssh/id_rsa

[terminalmage]

OK, I can't reproduce, both with and without a Host entry. The weird thing is that I was able to reproduce once, immediately after adding a new passphraseless key to my github account for testing, but then it started working. Maybe the new key hadn't propagated out yet?

Is /root/.ssh/id_rsa passphraseless? What are its permissions? SSH won't permit an identity file from being used, for instance, when the identity file can be read by a user other than the owner. If the private key is not chmod'ed to 600, that could be causing the problem.

# salt-call --version
salt-call 2016.3.4 (Boron)
# salt-call --local state.single git.latest [email protected]:terminalmage/git_pillar target=/tmp/foo user=erik identity=/home/erik/.ssh/id_rsa_nopass
local:
----------
          ID: [email protected]:terminalmage/git_pillar
    Function: git.latest
      Result: True
     Comment: [email protected]:terminalmage/git_pillar cloned to /tmp/foo
     Started: 23:18:44.005130
    Duration: 2004.45 ms
     Changes:
              ----------
              new:
                  [email protected]:terminalmage/git_pillar => /tmp/foo
              revision:
                  ----------
                  new:
                      cdb65a58aec34122c540728f74ce4e4078d333d6
                  old:
                      None

Summary for local
------------
Succeeded: 1 (changed=1)
Failed:    0
------------
Total states run:     1
Total run time:   2.004 s

[craigafinch]

The key has no passphrase. If there were an underlying problem with SSH, such as key file permissions or propagation, I shouldn't be able to run the commands via cmd.run, but that state is successful. Could it be something specific to my GitHub config? My repo is part of an organization, and I'v added a deploy key to a repo that belongs to the organization.

[terminalmage]

I think I found the issue. The difference between our git.latest states was the syntax used in the repo URL. When I add the ssh:// to the beginning I can reproduce, and when I remove it, it works:

# salt-call --local state.single git.latest "ssh://[email protected]:terminalmage/git_pillar" target=/tmp/foo user=erik identity=/home/erik/.ssh/id_rsa_nopass
08:53:14,390 [salt.loaded.int.module.cmdmod][1620][ERROR   ][30917] Command '['git', 'ls-remote', 'ssh://[email protected]:terminalmage/git_pillar']' failed with return code: 128
08:53:14,392 [salt.loaded.int.module.cmdmod][1624][ERROR   ][30917] stderr: ssh: Could not resolve hostname github.com:terminalmage: Name or service not known
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
08:53:14,393 [salt.loaded.int.module.cmdmod][1626][ERROR   ][30917] retcode: 128
08:53:14,395 [salt.state][ 245][ERROR   ][30917] Failed to check remote refs: Unable to authenticate using identity file:

ssh: Could not resolve hostname github.com:terminalmage: Name or service not known
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
local:
----------
          ID: ssh://[email protected]:terminalmage/git_pillar
    Function: git.latest
      Result: False
     Comment: Failed to check remote refs: Unable to authenticate using identity file:

              ssh: Could not resolve hostname github.com:terminalmage: Name or service not known
              fatal: Could not read from remote repository.

              Please make sure you have the correct access rights
              and the repository exists.
     Started: 08:53:14.201432
    Duration: 194.558 ms
     Changes:

Summary for local
------------
Succeeded: 0
Failed:    1
------------
Total states run:     1
Total run time: 194.558 ms
# salt-call --local state.single git.latest [email protected]:terminalmage/git_pillar target=/tmp/foo user=erik identity=/home/erik/.ssh/id_rsa_nopass
local:
----------
          ID: [email protected]:terminalmage/git_pillar
    Function: git.latest
      Result: True
     Comment: Repository /tmp/foo is up-to-date
     Started: 08:53:27.428196
    Duration: 1480.513 ms
     Changes:

Summary for local
------------
Succeeded: 1
Failed:    0
------------
Total states run:     1
Total run time:   1.481 s

For that matter, using the ssh:// on the CLI fails in the same way:

# git clone ssh://[email protected]:terminalmage/git_pillar.git
Cloning into 'git_pillar'...
ssh: Could not resolve hostname github.com:terminalmage: Name or service not known
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

This seems to be a simple matter of incorrect syntax in your git URL. You've mixed SCP-style syntax (in which the domain is followed by a colon), with ssh:// syntax (in which the domain must be followed by a slash). Indeed, replacing the colon with a slash works:

# salt-call --local state.single git.latest "ssh://[email protected]/terminalmage/git_pillar" target=/tmp/foo user=erik identity=/home/erik/.ssh/id_rsa_nopass
local:
----------
          ID: ssh://[email protected]/terminalmage/git_pillar
    Function: git.latest
      Result: True
     Comment: Repository /tmp/foo is up-to-date

              Changes made: Remote 'origin' changed from [email protected]:terminalmage/git_pillar to ssh://[email protected]/terminalmage/git_pillar
     Started: 08:58:31.946811
    Duration: 1846.885 ms
     Changes:

Summary for local
------------
Succeeded: 1
Failed:    0
------------
Total states run:     1
Total run time:   1.847 s

I'll also note that you are using SCP-like syntax for your cmd.run state, which explains why it works.

[craigafinch]

Great-I'm glad you found the problem, because this state is so useful. I think we have actually uncovered a documentation bug. All the examples use the invalid syntax that I was trying, so I assumed that Salt required different syntax than Git itself:

git-website-staging:
  git.latest:
    - name: ssh://[email protected]:user/website.git
    - rev: gh-pages
    - target: /usr/share/nginx/staging
    - identity: /root/.ssh/website_id_rsa
    - require:
      - pkg: git
      - ssh_known_hosts: gitlab.example.com

If you go back to 32237, at least one other user made this mistake, probably due to the incorrect examples in the docs.

[terminalmage]

Whoa! Good catch. I'll get this taken care of.

[craigafinch]

Thank you. The free version of Salt has better support than most enterprise ID products I've paid for.

[terminalmage]

Thanks! I just submitted https://github.com/saltstack/salt/pull/37604 which makes some clarifications to the git.latest documentation. They will show up in the 2016.3.4 documentation within a couple days. I'll go ahead and close this.