Pkg.jl: SSH auth keys, just very painful outside of ssh-agent.

[oxinabox]

This is a meta-issue for just how problematic it is to try and do anything with SSH auth outside of ssh-agent.
ssh-agent really shouldn't be the only way to do this, or even the standard way.
This is kinda an expansion upon https://github.com/JuliaLang/julia/issues/29851

In Pkg3 it shows up in 2 places

1. If you have a personal registry in ~/.julia/registries that uses ssh auth
2. if you dev a package passing in its URL as a [email protected]

I used to be able to produce all these same issues in julia 0.6 if I had a package git repo that was using ssh auth/transport. It is generally less bad in Pkg3, since seperate eviroments and dev --local and also update not doing git pull cuts down on it a lot.

I am just going to list problems,
and this can be broken up into sub-issues
(I feel like some of these already have issues, and those can be cross-refed).

To be clear I have not managed to successfully use SSH auth since at least julia 0.5.
But I did largely stop trying at that point and only started again today.
For now, excluding ssh-agent, because for other reasons I don't use that.
everything works fine for my keys when using the git commandline client
I have both public key and private key in same folder.
all are chmod +400

1. Here are a list of ways of providing the key that it does not work

and just cause the query message "Private key location for '[email protected]' : " to be asked again

I believe this is probably related to https://github.com/libgit2/pygit2/issues/836
(but see point 4, no error messages so IDK)

• [ ] 1a. Providing any answer to the query message the query for private key location including all combinations of

  • relative path, absolute path, or path including ~
  • path to id_rsa or a path to a key with a less standard name (eg. id_github_rsa)
  • Basically, that input never seems to do anythign useful

• [ ] 1b. having your key in ~/.ssh/id_rsa. This is the SSH default, so it should just work.

  • Having that key does seem like it sets the default for the query (in that the message changes) but that doesn't help since any input the the query for location never works

• [ ] 1c. Setting SSH_PUB_KEY_PATH and SSH_KEY_PATH doesn't work.

• [ ] 1d. specifying the key for the URL .ssh/config

---

Other issues

• [ ] 2. SSH_PUB_KEY_PATH and SSH_KEY_PATH are not documented.

  • they are mentioned occationally on Discourse etc.
  • given they don't work AFAICT, documenting them seems low priority.
  • This is tracked in https://github.com/JuliaLang/julia/issues/29851

• [ ] 3. the private key location query line does not support readline functionality. It is a raw text input and so pressing up inserts control characters.

  • I assume this is becasue it is inside LibSSH, rather than being a julia input.
• [ ] 4. No error messages are displayed ever.
  
  
  
  • I don't really have any good handle on where to start debugging this.
  
  
  • odds are there is something like LibSSH hates something about the format of my key files or some such. Or maybe I make typos in the path. or ... but I can't know
  
  
• [ ] 5. .ssh/config is not respected
  
  
  
  • more breadly that 1d. there are a lot more things you can do in your .ssh/config than specify the keys. Like aliasing hostnames and such
    
    
    
    
    
    • This was tracked in https://github.com/libgit2/libgit2/issues/4338 which has been closed.
    
    
    
    • SSH config should be the canonical source for how to communicate on SSH.
    
    
    
    • commandline git respects it.
    
    
    
  
  

---

I would appreciate it if people could point me at corresponding sub-issues already open in Pkg3/Julia/LibSSH/LibGit2 etc.

[omus]

Thanks for reporting these. Most of these are LibGit2.jl problems or possibly libgit2 problems.

[ExpandingMan]

More of a Julia issue than a Pkg issue, but I just wanted to point out that in light of the above and how quickly libgit2 moves, it seems pretty important to upgrade libgit2 as much as possible on each Julia release.

[omus]

I've been wanting to work on this but I haven't had the time lately

[codykrieger]

Per https://github.com/libgit2/pygit2/issues/836#issuecomment-501620594, generating a new SSH key passing -m PEM to ssh-keygen fixed the Private key location for '[email protected]' issue for me on macOS.

[oxinabox]

Per libgit2/pygit2#836 (comment), generating a new SSH key passing -m PEM to ssh-keygen fixed the Private key location for '[email protected]' issue for me on macOS.

We should detect if a key doesn't have that right and if so give a better message

[bliang26]

It is quite inconvenient when working with registries and repos sitting in an Enterprise GitHub which always requires SSH keys. For now I need to launch julia with SSH_PUB_KEY_PATH and SSH_KEY_PATH but this can't be a long term solution. Would this be fixed any time soon?

[GunnarFarneback]

For the record, repeated queries for "Private key location" does not necessarily indicate a failure of providing the key as such, only that something, anything, went wrong during authentication. Debugging is tricky but see https://github.com/JuliaLang/Pkg.jl/issues/1516#issuecomment-560794389 for a way of forcing libssh2 to emit debug messages. Those are not necessarily easy to understand either but at least they may give some hints.

[GunnarFarneback]

I have found -m PEM to be necessary also on Windows recently.

Edit:
Naturally this is a question of version. Specifically ssh-keygen is provided by OpenSSH, which in their release notes for version 7.8 (2018-08-24) write

• ssh-keygen(1): write OpenSSH format private keys by default
  instead of using OpenSSL's PEM format. The OpenSSH format,
  supported in OpenSSH releases since 2014 and described in the
  PROTOCOL.key file in the source distribution, offers substantially
  better protection against offline password guessing and supports
  key comments in private keys. If necessary, it is possible to write
  old PEM-style keys by adding "-m PEM" to ssh-keygen's arguments
  when generating or updating a key.

A key stored in OpenSSL PEM format, which libssh2 understands, can be recognized by starting with

-----BEGIN RSA PRIVATE KEY-----

whereas a key stored in OpenSSH format begins with

-----BEGIN OPENSSH PRIVATE KEY-----

[ExpandingMan]

Perhaps not directly relevant, but I have found that keychain (available on most distros) makes dealing with SSH agents drastically simpler. So much so, in fact, that this has no longer seemed like much of an issue to me.

Still, it would be great if Pkg could use command-line git which just respects the ~/.ssh/config, in which case most users should have to do literally nothing.

[tbenst]

I'm just a regular old user with my ssh key sitting in ~/.ssh/id_rsa. I have no need for ssh-agent and no interest in using it. All other programs on my computer have no issues using this (git, ssh, rsync, VScode, etc.).

I'm sure it's nice for ssh-agent users to have support out-of-the-box but if anything that should be the opt-in behavior. At least, would be nice to fall back to ~/.ssh/id_rsa.

Just a small gripe on an otherwise best-in-class package manager!

[ExpandingMan]

There are plenty of "regular old users" who will either have multiple keys in ~/.ssh or a key there under a different name. I don't see this as an edge case but an extremely common one for private keys. Of course, if libgit2 somehow respected the .ssh/config, your .ssh/id_rsa would still work (assuming you did not go in and set something wrong in the config).

It's been quite a long time and I still don't see any solution to this that's any better than using an agent manager like keychain. Certainly it seems totally crazy to expect Pkg to parse .ssh/config. In all likelihood we are just going to be waiting on someone to do something about it in libgit2, which would probably involve changing how they call the ssh back-end. The fact that command line git works as expected in these cases suggests there must be something that can be done there.

[StefanKarpinski]

Folks, if you have complaints, please open issues on libgit2. If libgit2 does whacky stuff when talking over SSH or doesn't respect your SSH key setups, there's nothing we can do about it. Venting here has zero effect since we are not about to start adding major features to the libgit2 project. These hands are already quite full.

[tbenst]

For what it’s worth, Rust’s package manager can use system git to avoid this issue: https://github.com/rust-lang/cargo/issues/2078#issuecomment-434388584. Cargo has a thread tracking this issue here: https://github.com/rust-lang/cargo/issues/2078

Falling back to system git would seem the most user friendly. It does seem better to have this fixed in libgit2 rather than have everyone write a workaround.

[StefanKarpinski]

I'm working on adding a system git fallback for 1.6, so that might address this.

[ExpandingMan]

Yeah, I don't understand what's going on over in libgit2 that they don't seem to have the will to do something about this. It was discussed in this issue but they seem to have closed it despite not coming to any kind of resolution. Out of curiosity, does anyone know how git itself handles this? I guess from what they're saying in libgit2 they use libssh instead of libssh2. So maybe the can is kicked further down the road and this is actually a libssh2 issue...

[StefanKarpinski]

My interpretation is that libgit2 is essentially developed by GitHub for GitHub and they don't care about anything that doesn't affect their use cases. In particular, anything regarding end-users using libgit2 on their personal systems is a non-priority for the project. Which is in stark contrast with the git project, where that is the only priority. Perhaps an uncharitable view, but in the years that we've used the library, that seems to be the way things pan out.

[ExpandingMan]

If that's a correct interpretation it sounds like a good reason to abandon it entirely in favor of command line git, if that's practical.

[StefanKarpinski]

I've made the case for that on pkg-dev calls, but there are very legitimate concerns in the other direction:

1. There are many users who many not have any git command installed and they should still be able to use the package manager, even for unregistered packages. Shipping Julia with git on Windows would be a very heavy-weight, awkward dependency since it requires shipping bash and other stuff that git uses.

2. Faulty though it is, libgit2 is what we've been using for a while now and there are lots of people that it is working for. If we drop it entirely, then we may break their workflows for the sake of fixing that of others.

So I think the best approach is probably to use both:

• Try using libgit2 first and if that works, great.
• If libgit2 fails, look for a git command and try that instead.
• Allow people to choose whether to use one or both and in what order.

I'm working on this, but I'm stuck in incredibly tedious stdlibs plumbing work that's necessary just to be able to replace the current ramshackle download functionality with something sane: https://github.com/JuliaLang/julia/pull/37686, https://github.com/JuliaLang/julia/pull/37340, https://github.com/JuliaLang/julia/pull/37611, https://github.com/JuliaLang/julia/pull/37763. All that noise is just to be able to add Downloads and Tar as stdlibs so that we can download and extract things consistently everywhere. Every time someone proposes that they want to add a new stdlib, I just laugh a world-wearied jaded little laugh and think "if only they had any idea how awful adding stdlibs actually is, they would not think this was a good idea".

[KristofferC]

Having the system git as a fallback (or as an option via some kind of configuration) for fetching things from the internet is perfectly fine. However, we use LibGit2 for much more than just that. In those cases, having a proper Julian API with types having nice show methods etc makes things much much easier than having to go via system git and using string output from a process to communicate. Constantly launching git executables is also bad for performance (especially on Windows) and IIRC was one of the major reasons for starting using LibGit2 in the package manager at all (back in Pkg2 time).

[StefanKarpinski]

However, we use LibGit2 for much more than just that.

Going forward there are only three things we will be using git for in Pkg as far as I can tell:

1. Acquiring package tarballs from git repos.
2. Checking out a dev copy of a package.
3. Updating registries that are git repos.

The first requires only two very high-level git operations: git clone --bare and git archive. The second only requires git clone and seems fine to just use CLI git for since the user needs git to interact with the dev'd package anyway; in fact it seems far more likely to cause problems to use libgit2 for this since it will potentially set up the clone differently than git would, causing confusing as soon as the user tries to interact with it via git. The third case we could frankly get rid of as soon as I've implemented loading registries directly from tarballs. Having multiple different ways to acquire and install registries doesn't seem ideal.

[StefanKarpinski]

In particular, there are no more situations where we would have to do git operations that don't hit the network (which, slow as exec is on Windows, it's still faster than a network RTT). Yes, we _used_ to do lots of little local git operations, but Pkg3 got rid of all of that. Where do you see us doing that sort of thing anymore?

[KristofferC]

In particular, there are no more situations where we would have to do git operations that don't hit the network

Some examples: we use it to check if a certain tree hash is in a cached git repo when we add by a branch or commit. We use it for diff support in status output. We use it to checkout a subdir of a repo (https://github.com/JuliaLang/Pkg.jl/blob/f1430b5dbbdeb09236a77faf755275a611690a93/src/Types.jl#L623-L626)

Those lines above are something that I think would be quite a lot harder using the git command line.

Updating registries that are git repos.

The third case we could frankly get rid of as soon as I've implemented loading registries directly from tarballs. Having multiple different ways to acquire and install registries doesn't seem ideal.

People host their own registries on GitHub or other git host services and I don't think that is a feature we can just break. Having to implement your own Pkg server is a significant step up in complexity over just setting up LocalRegistry.jl, especially since you then have to serve all packages.

[StefanKarpinski]

People host their own registries on GitHub or other git host services and I don't think that is a feature we can just break. Having to implement your own Pkg server is a significant step up in complexity over just setting up LocalRegistry.jl, especially since you then have to serve all packages.

That's fine and the tarball can still be acquired from a git repo. The thing I don't want to support is having the registry be a git clone or a tarball locally. I'd prefer to only support it being a tarball.

[GunnarFarneback]

Those lines above are something that I think would be quite a lot harder using the git command line.

Do they do something significantly different from https://github.com/GunnarFarneback/LocalRegistry.jl/blob/7f26935a56cfc16e9cbed8fe316df5276280cd06/src/LocalRegistry.jl#L339 ?

[msavael]

I'm working on adding a system git fallback for 1.6, so that might address this.

Hi @StefanKarpinski - do you think this is something that could still get into 1.6?

[StefanKarpinski]

No: The 1.6 release has been feature frozen for months and is entering the release candidate phase. Using SSH without using ssh-agent is just a world of pain in general, even outside of libgit2. Fortunately, ssh-agent exists, is excellent, solves all of these problems, and there's no good reason not to use it.