Packer: deprecation/removal of ssh-rsa from PubKeyAcceptedTypes

[lsm5]

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request.
If you are interested in working on this issue or have submitted a pull request, please leave a comment.

Description

The podman team recently tried to build Fedora 33 Beta images for use in our CI and we noticed SSH refused to connect and the daemon logs complained about not finding ssh-rsa in PubKeyAcceptedTypes. And this went away after making the relevant change in /etc/crypto-policies/back-ends/opensshserver.config.

Looks like Fedora 33 has removed ssh-rsa by default, and it's quite likely that other distros will as well, if not already.

It would be nice to not depend on ssh-rsa going forward.

@cevich Please correct me if I got anything wrong ^ :smile:

Potential References

https://github.com/containers/automation_images/pull/26#issuecomment-705780840

[cevich]

Suggestion: Add an ssh_keytype option to allow overriding the temporary key that packer generates. e.g. for the googlecompute builder or globally.

[SwampDragons]

Hi, thanks for reaching out! I agree that it's a good idea to add an option for telling Packer to generate a different key type.

We could probably do this now-ish for Packer with non-rsa type keys, for example ecdsa, and that may actually provide a workaround for users currently bitten by https://github.com/hashicorp/packer/issues/8609 while we wait for the golang team find the time to look at https://github.com/golang/go/issues/39885.

Meanwhile, I think you can work around this yourself by generating your own key and providing it to your builder using the ssh_private_key option

[cevich]

Great, thank for picking this up, and thanks for the suggested work-around. Unfortunately because of the way our specific setup works, it's more difficult to use a custom key. Allowing the less-secure ssh-rsa key was the simpler solution, and security is a relatively low-concern for our images. I doubt that's acceptable for other people though.

[azr]

Hello there, just to say we have binaries that handles this for GCP here.

You can now set the temporary_key_pair_type/temporary_key_pair_bits fields.

I'll leave this issue open until it's fixed for all builders.

[cevich]

Thanks!

[azr]

Hello there, I updated the binaries and the latest ones are here. Seems like to me we are going to keep these. 🙂

[cevich]

FWIW: I just encountered what appears to be the ssh-rsa problem on Ubuntu 20.10 (recently released).

We're not currently setup to install packer from other than the official site, and it seems circle-ci demands an oauth2 token to access the file. Any chance of having this change included in an official alpha/beta release I can point my scripts at?

[azr]

Hey @cevich, #10111 adds that feature for GCP, once it is merged you will be able to use the nightly build the day after it is merged, the PR looks like it's about to be merged so stay tuned ! 🙂

Also the upcoming release should be soon-ish.

[hswong3i]

The key is PubkeyAcceptedKeyTypes, see:

• https://gitlab.com/redhat-crypto/fedora-crypto-policies/-/commit/b298a9e1#49303b123e8d502a8711bccf3ba657fd55abe97c_5_3

[defanator]

Meanwhile, I think you can work around this yourself by generating your own key and providing it to your builder using the ssh_private_key option

Well, this doesn't work for me with packer 1.6.5 and Fedora 33 Cloud Base image (ami-04c4b559df358b45a in eu-central-1). I'm using 2048-bit key with SHA256 encryption, and it allows me to log in EC2 instance launched from the above AMI, but when I'm trying to either use ssh_private_key or put the key to my local ssh agent and enable ssh_agent_auth, I'm getting handshake failures from packer logs with the following log entries at the other side (Fedora 33 instance):

Nov 16 12:43:14 ip-10-1-18-46 sshd[6965]: error: kex_exchange_identification: Connection closed by remote host
Nov 16 12:43:14 ip-10-1-18-46 sshd[6965]: Connection closed by 10.1.2.32 port 43088
Nov 16 12:43:14 ip-10-1-18-46 sshd[6966]: userauth_pubkey: key type ssh-rsa not in PubkeyAcceptedKeyTypes [preauth]
Nov 16 12:43:14 ip-10-1-18-46 sshd[6966]: Connection closed by authenticating user fedora 10.1.2.32 port 43090 [preauth]

I'm currently looking for at least temporary solution (without modifying original Fedora Cloud Base AMIs) - appreciate any input here.

[azr]

@defanator That PubkeyAcceptedKeyTypes setting needs to be on in client and server side. If that still doesn't work I'd try with other ssh key types, ( packer can generate RSA, DSA, ECDSA and ED25519 temporary keys types for GCP currently ) . There should be at least one that works !

[defanator]

@azr I'm using pre-generated (existing) keypair which works fine on the same client (OS X 10.14) when I'm using system ssh client, but it doesn't work with the one bundled into packer.

We are using original AMI images published by OS vendors to build our custom ones, and currently we can't use Fedora 33 Cloud Base image due to this restriction in packer.

[cevich]

@defanator FWIW, @lsm5 originally discovered the workaround to this problem by switching to an ED25519 key. Just to confirm, are you using the new temporary_key_pair_type option to the googlecompute builder?

[cevich]

Woops...seems you're using AWS not GCP....my bad. I don't see a corresponding option for the amazon-instance builder (but I'm also no familiar with using it).

[defanator]

@cevich yes, I'm using EC2 builder, and the funny fact is that there's no way to import non-RSA key to EC2 via keypair API. But this shouldn't be an issue - as I mentioned previously, my SHA256-encoded key works just fine from anywhere except packer built-in ssh client.

[SwampDragons]

@defanator sounds like you're facing https://github.com/hashicorp/packer/issues/8609. It's an upstream bug and I wrote a patch for the golang crypto library months ago to try to get it fixed and it's waiting for review. You can track it here: https://github.com/golang/go/issues/39885

There's a very messy workaround you can do while we wait: https://github.com/hashicorp/packer/issues/8609#issuecomment-649128385

[defanator]

@SwampDragons appreciate your input here! Yes, it looks more like #8609 but the bad thing is that we can't do the proposed workaround as Fedora 33 official (Cloud Base) images in EC2 are coming with ssh-rsa disabled already. I was thinking of another approach where we would add some bootstrap bits via userdata while launching an instance from AMI and use that for either putting another key for the default user, or even adding another user to let packer log in, but it seems like currently there's no way to specify _existing_ instance ID instead of launching a new one from specified image (AMI). That is, I hope you'll sort this out in #8609 - thanks for your efforts!

[SwampDragons]

Got it. This is starting to affect enough people that we're gonna have to figure something out instead of waiting for Golang to fix it. I'll talk to our security team about options.