Git: Git 2.27 reports "invalid format" with the supplied ssh client

[brezanac]

• [x] I was not able to find an open or closed issue matching what I'm seeing

Setup

• Which version of Git for Windows are you using? Is it 32-bit or 64-bit?

$ git --version --build-options

$ git --version --build-options
git version 2.27.0.windows.1
cpu: x86_64
built from commit: 907ab1011dce9112700498e034b974ba60f8b407
sizeof-long: 4
sizeof-size_t: 8

• Which version of Windows are you running? Vista, 7, 8, 10? Is it 32-bit or 64-bit?

$ cmd.exe /c ver

Microsoft Windows [Version 10.0.19041.329]

• What options did you set as part of the installation? Or did you choose the
  defaults?

# One of the following:
> type "C:\Program Files\Git\etc\install-options.txt"
> type "C:\Program Files (x86)\Git\etc\install-options.txt"
> type "%USERPROFILE%\AppData\Local\Programs\Git\etc\install-options.txt"
$ cat /etc/install-options.txt

Editor Option: VIM
Custom Editor Path:
Path Option: Cmd
SSH Option: OpenSSH
Tortoise Option: false
CURL Option: OpenSSL
CRLF Option: CRLFAlways
Bash Terminal Option: ConHost
Git Pull Behavior Option: Merge
Performance Tweaks FSCache: Enabled
Use Credential Manager: Enabled
Enable Symlinks: Disabled
Enable Pseudo Console Support: Disabled

• Any other interesting things about your environment that might be related
  to the issue you're seeing?

Nothing specific about the environment. Pretty much a fresh installation.

Details

• Which terminal/shell are you running Git from? e.g Bash/CMD/PowerShell/other

Mintty, Cmder etc. They all return an invalid format error after upgrading to Git 2.27.

• What commands did you run to trigger this issue? If you can provide a
  Minimal, Complete, and Verifiable example
  this will help us understand the issue.

ssh domain.tld

• What did you expect to occur after running these commands?

A successful login to the SSH server without a invalid format error.

• What actually happened instead?

The SSH client, suplied with Git 2.27, reports invalid format and the SSH session pretty much refuses to execute any commands on the server.

• If the problem was occurring with a specific repository, can you provide the
  URL to that repository to help us with testing?

The problem appears with all repositories and all SSH servers after upgrading to Git 2.27.

Reverting back to Git 2.26 solves the issue but it's obviously not a solution.

[brezanac]

A side note.

I understand that this is not directly tied to Git itself but probably to the OpenSSH version that is shipped with Git 2.27.

However, given that in most setups the supplied ssh client overrides all other clients it could prove to be a problem for a lot of installations.

The only difference I could find between OpenSSH 8.2 and OpenSSH 8.3 is the announced depeciation of the ssh-rsa key type, but as far as I know those keys should still be usable in OpenSSH 8.3.

[rimrul]

does ssh -o "HostKeyAlgorithms ssh-rsa" hostname work? (replacing hostname with the actual host you want to connect to)

[brezanac]

does ssh -o "HostKeyAlgorithms ssh-rsa" hostname work? (replacing hostname with the actual host you want to connect to)

Unfortunately no, it does not resolve the issue.

I will try to find some time to investigate a potetntial workaround but honestly for now just staying on 2.26 seems like a better solution.

[landstander668]

Just a shot in the dark, but I've seen some references to this occurring with OpenSSH 8.3 when the public key isn't present in the same location as the private key. Can you verify whether or not your public key is also available?

[bohanyang]

Same problem here

[brezanac]

UPDATE 2

There are still some weird issues happening with the ssh client even after a successful connection. I really have no time to look for the cause anymore. I am reverting back to Git for Windows 2.6.2 for now.

UPDATE

The public key trick actually seems to work, however there is a quirk.

Please see my post bellow for details.

Just a shot in the dark, but I've seen some references to this occurring with OpenSSH 8.3 when the public key isn't present in the same location as the private key. Can you verify whether or not your public key is also available?

For security reasons I only generate public keys when they are needed and never keep my private and public keys in the same place.

However, I tried generating public keys and using them from the same location where their private counterparts were and the issue remains.

I did some more investigating and I seem to now understand what's actually going on.

On to the cause

Unlike previous versions, ssh-keygen and ssh, which are supplied with Git for Windows 2.7 default to the new OpenSSH private key format, which was apparently a thing since OpenSSH 7.8.

 * ssh-keygen(1): write OpenSSH format private keys by default
   instead of using OpenSSL's PEM format. The OpenSSH format,
   supported in OpenSSH releases since 2014 and described in the
   PROTOCOL.key file in the source distribution, offers substantially
   better protection against offline password guessing and supports
   key comments in private keys. If necessary, it is possible to write
   old PEM-style keys by adding "-m PEM" to ssh-keygen's arguments
   when generating or updating a key.

In other words Git for Windows 2.26.2 and older default to the so called PEM format which looks like this.

-----BEGIN RSA PRIVATE KEY-----
[actual key]
-----END RSA PRIVATE KEY-----

Git for Windows 2.27 uses the new format.

-----BEGIN OPENSSH PRIVATE KEY-----
[actual key]
-----END OPENSSH PRIVATE KEY-----

But that still doesn't explain why there were no issues up until Git for Windows 2.27, because even version 2.26.2 was using a way newer version than 7.8 (OpenSSH 8.2p to be exact).

Solutions (in a way)

One way to resolve the issue is to convert the old private keys into the new format.

In Putty, there is a dedicated option for that (see image bellow).

You load the existing key and select the marked option to export it into the new format.

If you want to use ssh-keygen for conversion then you can just type the following and replace keyfile with the actual path to the keyfile.

ssh-keygen -p -f keyfile -N '' -P ''

WARNING: this will replace the original file with the new one.

If you accidently converted the original file to the new format without backing the old one you can reverse the process with

ssh-keygen -p -f keyfile -N '' -P '' -m pem

Conclusion

All the tests have been performed on freshly installed Windows machines, so conflicts with other software are pretty much out of the question.

It seems like the ssh client, which is bundled with Git for Windows 2.7 and later, is just acting weirdly, especially considering that even after establishing connection with the newly converted keys (PEM to OpenSSH) there are still issues like connection losses, garbled text in MC (Midnight Commander etc.)

Considering that version 2.26.2 still works flawlesly and that even the new 2.28 RC versions show signs of the same issue I am pretty sure that staying on version 2.26.2 for now is the way to go.

At least until the issues have been identified and resolved.

[bohanyang]

Added the corresponding .pub key and the error has gone.

[brezanac]

Added the corresponding .pub key and the error gone.

Suprisingly, this works.

However, there is a quirk.

If you convert the PEM files to the new format and then put them along the generated .pub files it WON'T work.

However, if you use the old PEM format and add or create corresponding .pub files in the same location it WORKS.

If this is not a bug I'd really love to see why having a public key next to a private key is now required.

If it's not an intended feature then it's definitely a bug.

[KaiWalter]

Added the corresponding .pub key and the error has gone.

I removed .pub file some days ago to clean up my .ssh folder but was not able to relate this to the problem.

Thanks @brentybh

[KaiWalter]

Right @brezanac this notification also tricked me into updating

[brezanac]

Same issues with the newly released Git for Windows 2.28.

I've run out of patience so I decided to just convert all existing PEM based format keys into the new OpenSSH format.

The invalid format message is now gone but I am still absolutely confident that something is wrong with either the bundled ssh client or possibly with some of the shipped libraries.

Sadly I do not have any more time for Gremlin hunts and the conversion must be enough for now.

But I am really not comfortable using the Git for Windows bundle knowing that something might be very wrong with it.

[brezanac]

Due to additional issues with 2.28 like all of the sudden garbled Midnight Commander (no trick I know of helped resolve this) and weird shell behavior during established SSH sessions I have decided to permanently downgrade all my Git for Windows installations to 2.24.1.2.

Absolutely zero issues after that.

[dscho]

@brezanac I am sorry that you have so many troubles, and I am even more sorry to see that you seem to have waited for somebody else to investigate further (thus, you've "run out of patience"). I give you that: in Open Source often somebody else shares your problem and finds a solution for your problem and you don't have to work all that hard. Sometimes, however, you yourself need to be that somebody for other people, and it appears to me that you simply fold on that challenge.

[bohanyang]

emm. I think that he IS the somebody... I respect what he did. Look that count of words and Markdown format and details and clues he posted, he's definitely not someone impatient. Everyone has his limit so may be not able to solve it directly. This is open source (and if the issue is not urgent) so it's totally FINE to wait for other people who care about this issue.

[brezanac]

@dscho With all due respect, but I hardly consider spending weeks on a problem, prior to even reporting it here, to be "_waiting for somebody else to investigate further_". Especially since in the meantime I also had to actually use Git for Windows for productive purposes.

I've been contributing to open source for almost two decades. I understand that it's about what people are willing or able to give, not what they are expected to give. But even if someone is willing to contribute, they can't if they lack the expertise or knowledge required to do so. I personally lack the knowledge about the fine nuances of cryptography to contribute in any other way here except by thoroughly documenting my experience in trying to fix or at least investigate the issue. In other words, you can't expect from people to be fully knowledgeable in metalurgy and woodworking if they merely use hammers and it just so happens that one of them gets broken.

What I am trying to say is that in the end my ticket was not raised with intention to make people jump straight away and burn the midnight oil to fix it, just to raise awarness that something, which to me personally feels like a serious issue, is potentially broken.

[dscho]

@brezanac okay, I understand now where you're coming from. My apologies for being harsh on you.

[dscho]

Oh, BTW on a hunch, I tried to look for the error message in OpenSSH's source code, but unfortunately, there are many matches: https://github.com/openssh/openssh-portable/search?q=invalid+format&unscoped_q=invalid+format

Then, I looked at OpenSSH's bug tracker, and found this: https://bugzilla.mindrot.org/show_bug.cgi?id=3173

The latest comment on that ticket mentions this blog post, which seems relevant: https://blog.hqcodeshop.fi/archives/482-OpenSSH-8.3-client-fails-with-load-pubkey-invalid-format.html

The blog post ends with the good news that a fix was committed to OpenSSH: https://github.com/openssh/openssh-portable/commit/c514f3c0522855b4d548286eaa113e209051a6d2

Sadly, no new version has been released since: https://github.com/openssh/openssh-portable/releases

So now the big question is: should we wait until OpenSSH v8.4, or should we build a patched OpenSSH for use in Git for Windows?

I did not find any public roadmap for OpenSSH, and it does not seem as if there is any fixed release cadence: May 27th, February 14th, before that October 9th and April 18th 2019, looks like _roughly_ 3-6 months between releases. Judging by https://lists.mindrot.org/pipermail/openssh-unix-dev/2020-February/thread.html and https://lists.mindrot.org/pipermail/openssh-unix-dev/2020-May/thread.html, a "Call for testing" mail goes out some 1-2 weeks before release time, but I haven't seen any during this month: https://lists.mindrot.org/pipermail/openssh-unix-dev/2020-August/thread.html.

Of course, we have the luxury of waiting a bit because Git for Windows v2.29.0 (i.e. the next release) will happen only on October 19th or soon thereafter (following Git's release cycle: https://tinyurl.com/gitCal).

In the meantime, it would be good, of course, if we could get confirmation that the fix _does_ work for us. To that end, I prepared a branch for easy testing. This is how anyone who experienced the "invalid format" error can verify the fix:

1. install Git for Windows' SDK,
2. sdk cd openssh,
3. git pull https://github.com/dscho/MSYS2-packages/ openssh-pubkey-fix
4. sdk build
5. install via pacman -U openssh*.pkg.tar.xz
6. Run the ssh command that produced the error message

[brezanac]

@dscho Thank you! I'll go through all the linked resources and test the fix as soon as I find some time for it.

[dscho]

I verified that this fixes the issue over here (I moved the public key out of the way, and saw the warning with the current OpenSSH, then verified that the warning does not show up with the patched OpenSSH).

[rimrul]

Shouldn't the release notes mention OpenSSH instead of GPG?

[dscho]

Shouldn't the release notes mention OpenSSH instead of GPG?

Uh, yes, of course. Let me try to find a hole to hide in.

[FeodorFitsner]

@dscho when are you guys planning for the next release?

[dscho]

On October 19th (when Git v2.29.0 is due), or soon thereafter.

FWIW I just integrated OpenSSH v8.4 into Git for Windows' SDK, and confirmed that the patch is already applied in that version.