Salt: Salt fails to fetch from a git remote in VSTS

[mmisztal1980]

Description of Issue/Question

Salt fails to fetch from a git remote using ssh auth. The following logentries appear:
Interestingly It's possible to git clone manually using ssh auth. ssh -T also works.

[DEBUG   ] Set update lock for gitfs remote 'ssh://{path-to-repo-in-vsts}'
[DEBUG   ] Fetching gitfs remote 'ssh://ssh://{path-to-repo-in-vsts}'
[ERROR   ] Error occured fetching gitfs remote 'ssh://ssh://{path-to-repo-in-vsts}': Failed to start SSH session: Unable to exchange encryption keys
[DEBUG   ] Removed update lock for gitfs remote 'ssh://ssh://{path-to-repo-in-vsts}'

Setup

• In general, follow this video
• Install a vanilla salt-master and salt-minion on Ubuntu 16.04
• Create a VSTS (visualstudio.com) git repository
• Create a SSH keypair and upload the public key to VSTS
• Copy the keypair to /etc/salt/master.d
• Apply the following settings to /etc/salt/master

fileserver_backend:
  - git

gitfs_remotes:
  - ssh://{Your Repo}
gitfs-pubkey: /etc/salt/master.d/{pubkey}
gitfs_privkey: /etc/salt/master.d/{privkey}
gitfs_base: master
gitffs_env_whitelist:
  - base

pillar_roots:
  base:
    - /srv/pillar
file_roots:
  base:
    - /srv/salt

Steps to Reproduce Issue

Start the salt-master as a process with --log-level=debug

Versions Report

Salt Version:
           Salt: 2015.8.8

Dependency Versions:
         Jinja2: 2.8
       M2Crypto: Not Installed
           Mako: 1.0.6
         PyYAML: 3.12
          PyZMQ: 15.2.0
         Python: 2.7.12 (default, Nov 19 2016, 06:48:10)
           RAET: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.1.4
           cffi: 1.9.1
       cherrypy: Not Installed
       dateutil: 2.6.0
          gitdb: 2.0.0
      gitpython: 2.1.0
          ioflo: Not Installed
        libgit2: 0.24.0
        libnacl: 1.5.0
   msgpack-pure: Not Installed
 msgpack-python: 0.4.8
   mysql-python: Not Installed
      pycparser: 2.17
       pycrypto: 2.6.1
         pygit2: 0.24.0
   python-gnupg: Not Installed
          smmap: 2.0.1
        timelib: Not Installed

System Versions:
           dist: Ubuntu 16.04 xenial
        machine: x86_64
        release: 4.4.0-47-generic
         system: Ubuntu 16.04 xenial

[Ch3LL]

@mmisztal1980 would you mind sharing the ssh url format you have. you do not need to specify the exact name but just something like ssh://[email protected]:2*

This will just help me to attempt to replicate it. I've tired multiple urls and am getting invalid url errors, since this is my first time interacting with vsts.

[mmisztal1980]

Hi there,

Here you go,
https://{subdomain}.visualstudio.com/DefaultCollection/{project-name}/_git/{repo-name}

In ssh:

ssh://{user-name}@{sub-domain}.visualstudio.com:22/DefaultCollection/{project-name}/_git/{repo-name}

If you like, you can reach me on skype under the same name as my github handle.

[mmisztal1980]

Updated the above answer to include ssh, sorry it's been a long day.

[mmisztal1980]

@Ch3LL is this the info that you needed? :)

[Ch3LL]

Yes thats perfect. Seems my url I was using was indeed is the same. I think my issue might be with my username [email protected]. Doesn't seem to like that. Do you happen to know how to handle that? Thanks for any help, i'm just not familiar with vsts but definitely want to see if we can replicate this or help you out :)

[mmisztal1980]

@Ch3LL

As far as I know VSTS:

• you can use your email to setup a MS "Live" account
• you use that account to sign in to VSTS
• you create a team in VSTS - In my case this has mapped directly onto the username in the ssh url.
• you create a project/repo in VSTS

If you like, we can have a call, and I'll try to help with VSTS.

[mmisztal1980]

@Ch3LL Any luck so far? did you manage to reproduce the issue?

[blakemartin]

@Ch3LL - I'm seeing exactly the same issue with VSTS and gitfs remotes. Any updates?

Thanks!

[Ch3LL]

Thanks for bumping thsi for me. @mmisztal1980 sorry for missing this.

I can now replicate this thanks to @mmisztal1980's help on this.

[ERROR   ] Error occurred fetching gitfs remote 'ssh://[email protected]:22/_git/MyFirstProject': Failed to start SSH session: Unable to exchange encryption keys
Traceback (most recent call last):
  File "/home/ch3ll/git/salt/salt/utils/gitfs.py", line 1406, in _fetch
    fetch_results = origin.fetch(**fetch_kwargs)
  File "/usr/lib64/python2.7/site-packages/pygit2/remote.py", line 221, in fetch
    check_error(err)
  File "/usr/lib64/python2.7/site-packages/pygit2/errors.py", line 56, in check_error
    raise GitError(message)
GitError: Failed to start SSH session: Unable to exchange encryption keys

Here is my config:

fileserver_backend:
  - git

gitfs_remotes:
  - ssh://[email protected]:22/_git/MyFirstProject
gitfs_pubkey: /home/ch3ll/.ssh/key.pub
gitfs_privkey: /home/ch3ll/.ssh/key
gitfs_base: master
gitffs_env_whitelist:
  - base

pillar_roots:
  base:
    - /srv/pillar
file_roots:
  base:
    - /srv/salt

I also verified that I could do a git clone with that repo and key just fine.

@terminalmage maybe you have any quick ideas as to why we are getting this error? If you need I can give you access to my repo so you can use that exact config and test.

[terminalmage]

I've @Ch3LL add my pubkey to her VSTS repo, and will look at this when time allows (currently tied up making Python 3 compatibility changes).

[mmisztal1980]

Just wondering if there have been any development in here

[dpdeputy]

Yes, I"m hitting this issue as well.

$ salt --versions-report
Salt Version:
Salt: 2016.11.3

Dependency Versions:
cffi: 1.9.1
cherrypy: Not Installed
dateutil: 2.4.2
gitdb: 0.6.4
gitpython: 1.0.1
ioflo: Not Installed
Jinja2: 2.8
libgit2: 0.25.1
libnacl: Not Installed
M2Crypto: Not Installed
Mako: 1.0.3
msgpack-pure: Not Installed
msgpack-python: 0.4.6
mysql-python: Not Installed
pycparser: 2.17
pycrypto: 2.6.1
pygit2: 0.25.1
Python: 2.7.12 (default, Nov 19 2016, 06:48:10)
python-gnupg: Not Installed
PyYAML: 3.11
PyZMQ: 15.2.0
RAET: Not Installed
smmap: 0.9.0
timelib: Not Installed
Tornado: 4.2.1
ZMQ: 4.1.4

System Versions:
dist: Ubuntu 16.04 xenial
machine: x86_64
release: 4.4.0-71-generic
system: Linux
version: Ubuntu 16.04 xenial

[terminalmage]

I've put this on my priority list for this week so I should have more information in the next few days.

[terminalmage]

After looking into this issue, I have found that it is an upstream issue in libssh2, which libgit2 uses to connect via SSH. Neither pygit2 nor libgit2 handle anything with regard to SSH directly, it's all done via libssh2.

In my testing I found that CentOS 7 and Ubuntu 16.04 (libssh2 1.4.3 and 1.5.0, respectively) both fail, but on an Arch Linux box with libssh2 1.8.0, pygit2 is able to successfully authenticate to the repository.

According to debug logging on ssh CLI usage, the remote is using diffie-hellman-group14-sha1 (which is supported in libssh2):

debug1: kex: algorithm: diffie-hellman-group14-sha1

I don't see anything in the commits to src/kex.c between libssh2 version 1.5.0 and 1.8.0 which jump out at me as an explanation, but I'm in a little over my head when it comes to the SSH protocol to be honest.

Regardless of this though, this is not something that Salt has any control over. My best recommendation at this point is to use GitPython. To do so, you will need to do the following:

1. Install GitPython
2. Set gitfs_provider: gitpython in your master config.
3. Remove/comment out the gitfs_pubkey and gitfs_privkey lines from the master config.
4. Add the following in /root/.ssh/config:

Host visualstudio.com
    User yourusername
    PreferredAuthentications publickey
    IdentityFile /path/to/privkey

Since GitPython wraps the git CLI, the permission requirements for key auth apply. Make sure that the directory which contains the private key is chmod'ed to 0700 and that the private key itself is chmod'ed to 0600.

[terminalmage]

Just as another data point, I tested just now on Fedora 24 (libssh 1.7.0) and it worked with that version as well.

[terminalmage]

I accidentally closed this issue, so I have reopened it. But I honestly think it should be closed, if @mmisztal1980 agrees.

[mmisztal1980]

I agree. I'll proceed with your suggestion(s).

[terminalmage]

@mmisztal1980 Sorry I couldn't give you better news. I've updated the GitFS documentation with a note about this libssh2 incompatibility. See https://github.com/saltstack/salt/pull/41123.

[mikeadamz]

@terminalmage I notice similar problems if enforcing strong KexAlgorithms, MACs, or Ciphers. Which is weird. The system ssh supports these but whatever PyGit is using barfs unless you enable sha1 algorithms.

[deuscapturus]

I just ran into this error across all of my salt-masters trying to connect to github.com today. Anybody else?

[mmisztal1980]

possibly related to: https://githubengineering.com/crypto-removal-notice/ ?

On Thu, Feb 22, 2018 at 10:39 PM, Theodore Cowan [e-mail]
wrote:

I just ran into this error across all of my salt-masters trying to connect
to github.com today. Anybody else?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/saltstack/salt/issues/38066#issuecomment-367831044,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABS1AULNvzLK7tn0hlvLME261evrscxRks5tXd6RgaJpZM4LDlXK
.

[liftedkilt]

@deuscapturus I'm also hitting this on my salt-master trying to connect to github.com. Have you found a way around it yet?

[deuscapturus]

The problem is as @mmisztal1980 posted.

So far we have found that libgit2 and pygit2 need to be updated to versions 0.26. Sadly libgit2 is a platform package and 0.26 isn't available on all OS's. Fedora 27 was good. Ubuntu 16.04 and Centos 7 were no good.

Another option is to switch to gitfs_provider: gitpython

[liftedkilt]

@deuscapturus thanks for the heads up.

Note: I upgraded from Ubuntu 16.04 to 17.10, which is using libgit2 version 0.24.6, and that has restored connectivity without resorting to gitpython

[deuscapturus]

After further investigation, we resolved the problem by just updating libssh2.