Teleport: gpg-agent does not support SSH certificates, tsh should be aware of this

[webvictim]

What happened:

• Yubikeys use gpg-agent to hold keys/certificates
• gpg-agent does not support SSH certificates (https://dev.gnupg.org/T1756)
• When gpg-agent is running and you log into a Teleport cluster, tsh adds stuff to the agent that it doesn't understand
• This causes issues for people using Yubikeys

What you expected to happen:
1)tsh should have a command-line flag which disables any attempt to write to a running agent
2) Ideally, tsh would be able to detect when gpg-agent is running rather than ssh-agent and deliberately avoid writing its keys there.

This would enable much smoother Teleport operations for Yubikey users. tsh still writes its certificates to ~/.tsh/keys/<cluster>, which means that people can use these for their Teleport operations instead.

How to reproduce it (as minimally and precisely as possible): Run gpg-agent, log into a Teleport cluster using tsh then observe that what's in the agent is not a valid certificate (error fetching identities: Invalid key length)

Environment:

• Teleport version (use teleport version): Teleport Enterprise v4.1.4git:v4.1.4-0-gc487a75c go1.13.2
• Tsh version (use tsh version): Teleport v4.1.4 git:v4.1.4-0-gc487a75c go1.13.2
• OS (e.g. from /etc/os-release): Fedora 30

[russjones]

In a recent discussion with a customer:

Sure. gpg-agent works with our YubiKeys so when we have individual users SSH keys,
they would be stored on a yubikey. gpg-agent is the agent that connects between the
YubiKey and acts as an ssh agent. The normal ssh-agent doesn't have such a capability.
So, we would switch ssh-agent out for gpg-agent. When we do that, when tsh adds the
ssh certificate to gpg-agent, it doesn't know how to handle it (It adds the cert, but
then will no longer serve up any identities, effectively killing the gpg-agent)

[thardie]

Interestingly, on the YubiKey side, at least 1 other user claims there is a way to make this work: https://github.com/drduh/YubiKey-Guide/issues/43

[benarent]

This came up again with Reliability Improvements for E- Adding other customer to the issue.

[aelkugia]

Had another conversation with a customer interested in this, who has some engineers using gpg-agent/yubikey.

[cpanato]

+1

[cnelson]

Also ran into this. currently working around it with alias tsh='SSH_AUTH_SOCK= tsh'

[awly]

Fix is in review.

FWIW, it appears that OpenSSH 8.2+ supports security keys now: https://buttondown.email/cryptography-dispatches/archive/cryptography-dispatches-openssh-82-just-works/.
I haven't tried it myself, but might be interesting for folks in here.

[eddiewang]

Tried using the --use-local-ssh-agent flag with no avail on Teleport v4.4.0-dev git:v4.2.0-alpha.5-478-gb1280007e go1.14.2

Still getting error fetching identities: Invalid key length after a successful login. Using gpg-agent with Yubikey.

[webvictim]

@eddiewang You want --no-use-local-ssh-agent

[eddiewang]

@webvictim lol thanks. that deserves a :man_facepalming:

[webvictim]

No worries - it's a bit of a confusing flag. We've written some documentation explaining it, but that won't be coming out until Teleport 4.3 does (which will be very soon)

[klizhentas]

Let's give it another go in 5.1

1. Can we detect gpg-agent and not load the certs?
2. Can we reduce the amount of flags? There are other requests, for example load agent, so there are 4 states possible. Can we use one flag with values or may be there is another, more creative way. The less flags the better.

[klizhentas]

Can we patch gpg-agent upstream to support certs?

[Chili-Man]

Is there anyway to disable tsh login from generating an SSH key when we only use teleport for kubernetes access?

And for those wondering how to resolve the error by removing the offending keys; they can be manually removed from within the ~/.gnupg/private-keys-v1.d directory

[webvictim]

There's not currently a way to stop SSH certs from being generated completely, but you can prevent tsh from attempting to add them to the agent by using tsh login --no-use-local-ssh-agent or setting the TELEPORT_USE_LOCAL_SSH_AGENT environment variable to false.

[Chili-Man]

Even using tsh login --no-use-local-ssh-agent doesn't seem to help; after I login, and then I try to use the loaded kubeconfig provided through that command, by invoking any kubectl command, its added back into the ssh agent when kubectl tryies to get the credentials with the command in kubeconfig tsh kube credentials --kube-cluster=teleport.example.com --teleport-cluster=teleport.example.com

[Chili-Man]

After further testing, it appears that if you manually add the --no-use-local-ssh-agent as part of the command it uses in the kubeconfig to get the credentials, it won't add the ssh key to your gpg-agent. Is there a way to have that flag automatically added?

it appears that you can actually export TELEPORT_USE_LOCAL_SSH_AGENT=false in your shell's profile or rc file, where it will preserve the effect of not adding the tsh ssh keys to your gpg-agent

[webvictim]

@Chili-Man What happens if you run export TELEPORT_USE_LOCAL_SSH_AGENT=false (or whatever is appropriate for your shell) and then try? Does the embedded tsh command honour that? (it should)

[Chili-Man]

@webvictim thanks, that works! I've updated my comment above to reflect that.

[klizhentas]

@awly please look into the issue when working on U2F support for tsh. In addition to that we have requests for people who don't want the keys to be written to disk and stored only in memory in agent.

I think we have to build a matrix of all possible combinations we are going to support and provide some UX that integrates well with U2f support you are adding.

[klizhentas]

https://github.com/gravitational/teleport/issues/4863

[nategadzhi]

+1. Just shot myself in the foot with this, too.

[thardie]

The only reason I was using gpg-agent was to use a yubikey GPG based auth for SSH. You can instead use PIV with Yubikey to achieve similar functionality, then use the regular SSH Agent instead of gpg-agent. See here for details on how to set this up: https://developers.yubico.com/PIV/Guides/SSH_with_PIV_and_PKCS11.html

Using this method then lets teleport and yubikey coexist properly with a regular SSH Agent.

[Chili-Man]

@thardie unfortunately, it appears that PIV only works if your gpg key was an RSA 2048-bit key 😞

https://github.com/Yubico/yubico-piv-tool/issues/58

Seems to be an underlying issue with the specification for PIV

[webvictim]

Related to #5487