Streisand: Server creation blocked by GPG key import

[jansendotsh]

Expected behavior:

Complete install of services on DigitalOcean.

Actual Behavior:

Stalls during GPG key refresh
https://github.com/StreisandEffect/streisand/blob/fcfb0d560cd4c8fa20e55c5a49ac111bf80b175b/playbooks/roles/gpg/tasks/main.yml#L78

Manually running GPG refresh on target host shows following error of signatures being too large for keyring.

# gpg2 --no-default-keyring --keyring /root/.gnupg/streisand/pubring.gpg --keyserver-options timeout=120 --refresh
gpg: refreshing 10 keys from hkps://hkps.pool.sks-keyservers.net
gpg: key 4D4D5458: "Corban Raun <[email protected]>" not changed
gpg: key F67DA905: "Jay Carlson <[email protected]>" not changed
gpg: key A7A356D6: "Joshua Lund <[email protected]>" not changed
gpg: key BF0F6049: "Daniel McCarney <[email protected]>" not changed
gpg: key DD3AAAA3: "Michał Trojnara <[email protected]>" not changed
gpg: key B43434E4: "PuTTY Releases <[email protected]>" not changed
gpg: error writing keyring '/root/.gnupg/streisand/pubring.gpg': Provided object is too large
gpg: key 93298290: "Tor Browser Developers (signing key) <[email protected]>" 100103 new signatures
gpg: key 93298290: "Tor Browser Developers (signing key) <[email protected]>" 1 new subkey
gpg: key 96865171: "Nikos Mavrogiannopoulos <[email protected]>" 22 new signatures
gpg: key 7F343FA7: "Nikos Mavrogiannopoulos <[email protected]>" not changed
gpg: key 2F2B01E7: "OpenVPN - Security Mailing List <[email protected]>" 39 new signatures
gpg: key 2F2B01E7: "OpenVPN - Security Mailing List <[email protected]>" 2 new subkeys
gpg: Total number processed: 10
gpg:              unchanged: 7
gpg:            new subkeys: 3
gpg:         new signatures: 100164
gpg:           not imported: 1

Steps to Reproduce:

1. Start Streisand script (run on my own device or via localhost advanced install)
2. Enable all services and fill out information for Let's Encrypt cert.
3. During GPG key refresh, output shared above is observed in Ansible errors.

Ansible Information

• Ansible version: 2.8.0
• Ansible system: Linux
• Host OS: Fedora
• Host OS version: 30
• Python interpreter: python
• Python version: 2.7.16

Streisand Information

• Streisand Git revision: dae1832c9722f98f0f2bd173cab0979ea0acd5b4
• Streisand Git clone has untracked changes: no
• Genesis role: genesis-digitalocean
• Custom SSH key: False

Enabled Roles

• Shadowsocks enabled: True
• Wireguard enabled: True
• OpenVPN enabled: True
• stunnel enabled: True
• Tor enabled: True
• Openconnect enabled: True
• TinyProxy enabled: True
• SSH forward user enabled: True
• Configured number of VPN clients: 5

[maxthomas]

also seeing this on a non-DO host, trying several different keyservers

[plattrap]

The 100,000 signatures on the TOR key may be related to the SKS Keyserver attack reported elsewhere: https://gist.github.com/rjhansen/67ab921ffb4084c865b3618d6955275f

[jpmontez]

This is pretty annoying (at least on the Vagrant VMs) to troubleshoot.

In @plattrap's link, the recommended alternative key server is keys.openpgp.org, however if you replace the streisand_gpg_keyserver_address variable in the gpg role default, you'll get another error:

2019-07-07 04:02:08 dirmngr[24101.1] resolve_dns_addr for 'keys.openpgp.org': 'keys.openpgp.org' [already known]
2019-07-07 04:02:08 dirmngr[24101.1] TLS verification of peer failed: status=0x0042
2019-07-07 04:02:08 dirmngr[24101.1] TLS verification of peer failed: The certificate is NOT trusted. The certificate issuer is unknown.
2019-07-07 04:02:08 dirmngr[24101.1] DBG: expected hostname: keys.openpgp.org
2019-07-07 04:02:08 dirmngr[24101.1] DBG: BEGIN Certificate 'server[0]':
2019-07-07 04:02:08 dirmngr[24101.1] DBG:      serial: 03D804FE5B5614E157F04E714A7BEBC64E91
2019-07-07 04:02:08 dirmngr[24101.1] DBG:   notBefore: 2019-06-07 13:24:06
2019-07-07 04:02:08 dirmngr[24101.1] DBG:    notAfter: 2019-09-05 13:24:06
2019-07-07 04:02:08 dirmngr[24101.1] DBG:      issuer: CN=Let's Encrypt Authority X3,O=Let's Encrypt,C=US
2019-07-07 04:02:08 dirmngr[24101.1] DBG:     subject: CN=keys.openpgp.org
2019-07-07 04:02:08 dirmngr[24101.1] DBG:   hash algo: 1.2.840.113549.1.1.11
2019-07-07 04:02:08 dirmngr[24101.1] DBG:   SHA1 fingerprint: 163C12F44D3D2597904C2A86C091B9A7464E8BC4
2019-07-07 04:02:08 dirmngr[24101.1] DBG: END Certificate
2019-07-07 04:02:08 dirmngr[24101.1] DBG: BEGIN Certificate 'server[1]':
2019-07-07 04:02:08 dirmngr[24101.1] DBG:      serial: 0A0141420000015385736A0B85ECA708
2019-07-07 04:02:08 dirmngr[24101.1] DBG:   notBefore: 2016-03-17 16:40:46
2019-07-07 04:02:08 dirmngr[24101.1] DBG:    notAfter: 2021-03-17 16:40:46
2019-07-07 04:02:08 dirmngr[24101.1] DBG:      issuer: CN=DST Root CA X3,O=Digital Signature Trust Co.
2019-07-07 04:02:08 dirmngr[24101.1] DBG:     subject: CN=Let's Encrypt Authority X3,O=Let's Encrypt,C=US
2019-07-07 04:02:08 dirmngr[24101.1] DBG:   hash algo: 1.2.840.113549.1.1.11
2019-07-07 04:02:08 dirmngr[24101.1] DBG:   SHA1 fingerprint: E6A3B45B062D509B3382282D196EFE97D5956CCB
2019-07-07 04:02:08 dirmngr[24101.1] DBG: END Certificate
2019-07-07 04:02:08 dirmngr[24101.1] TLS connection authentication failed: General error
2019-07-07 04:02:08 dirmngr[24101.1] error connecting to 'https://keys.openpgp.org:443': General error
2019-07-07 04:02:08 dirmngr[24101.1] TLS verification of peer failed: status=0x0042
2019-07-07 04:02:08 dirmngr[24101.1] TLS verification of peer failed: The certificate is NOT trusted. The certificate issuer is unknown.

The LetsEncrypt CA isn't loaded by GnuPG's dirmngr service by default. So, you'd need to download the certificates from LetsEncrypt's certificates page and specify them in the dirmngr.conf file, like so:

hkp-cacert /path/to/isrgrootx1.pem
keyserver hkps://keys.openpgp.org

However, there's two problems with this approach. One, there's hidden binary data within the file, so you'd need to strip that beforehand. Second, there's a long running bug in the OpenSSL parser (https://github.com/openssl/openssl/issues/1381) that prevents certificates from being read correctly. It's supposedly resolved, but that brings us to another issue…

So, I had the thought of simply mitigating the need to use the alternative keys.openpgp.org at all. What would be a better alternative than keyserver.ubuntu.com? Well, the administrators misconfigured a server behind it, cassava.canonical.com's SSL certificate targets don't match.

root@streisand-host:/etc/ssl/certs# curl https://cassava.canonical.com
curl: (51) SSL: certificate subject name (keyserver.ubuntu.com) does not match target host name 'cassava.canonical.com'

Them fixing this wouldn't necessary solve the issue either because of another issue. In short, dirmngr doesn't read system certificates in the current published version of GnuPG 2. The feature was added in 2.2.x: https://dev.gnupg.org/rG7c1613d41566f7d8db116790087de323621205fe

While we have…

root@streisand-host:/etc/ssl/certs# gpg2 --version
gpg (GnuPG) 2.1.11

It's almost 2 am, so I'm gonna stop digging for the night. I hope these notes help anyone else curious about this.

[chechulnikov]

Same problem. Are there any temp solutions?

[maxthomas]

i changed playbooks/roles/gpg/vars/main.yml to the following:

streisand_gpg_keyserver_address: "hkp://pgp.surfnet.nl"

which seemed to work OK. of course, you should do due diligence if your thread model involves compromised key servers.

[raul-gracia]

Looks like the problem is not with the key server but the ssh keys.

The Key 96865171 <[email protected]> has been recently revoked.
Also seems like some of the other ones are too big (4096R and 3072R vs 2048R ) and the gpg2 has problems with it ( or that's what I've read ).

I've fixed it by commenting out all those keys from the vars file (playbooks/roles/gpg/vars/main.yml)

# Which keys from files/ do we add to the keyring in first setup? These keys are
# trusted ultimately to verify software signatures during Streisand provisioning.
streisand_bootstrap_gpg_keys:
  # OpenVPN release signing key
  # - [email protected]
  # Openconnect
  # - [email protected]
  # - [email protected]
  # Tor browser release signing key
  # - [email protected]
  # PuTTY release signing key
  - [email protected]
  # Stunnel release signing key
  - [email protected]
  # Streisand maintainer - Github @cpu
  - [email protected]
  # Streisand maintainer - Github @jlund
  - [email protected]
  # Streisand maintainer - Github @nopdotcom
  - [email protected]
  # Streisand maintainer - Github @CorbanR
  - [email protected]
  # Streisand maintainer - Github @alimakki (Commented out for now because it
  # breaks CI)
  #- [email protected]

In my case I'm only configuring WireGuard so it works just fine. I guess they will have to figure out which are the correct keys to add so it knows how to refresh them from the server.

[lucasff]

What worked for me is changing streisand_gpg_keyserver_address to hkp://pgp.mit.edu:80

[DrMattChristian]

I don't use streisand but, related to @jpmontez 's troubleshooting above, I ran across similar hkps: connectivity issues trying to connect to the new keyserver hkps://keys.openpgp.org in the (older) GnuPG 2.0.14, which does not have the dirmngr component.

In my case, the secret sauce was to manually specify the ca-cert-file key server option in my ~/.gnupg/gpg.conf file. This option sets the CAfile setting for the NSS SSL/TLS library, which is used by curl to connect to the secure key server.

keyserver hkps://keys.openpgp.org
keyserver-options ca-cert-file=/etc/pki/tls/certs/ca-bundle.crt

On another server with slightly newer GnuPG 2.0.22 (but still no dirmngr), that gpg was able to connect to hkps://keys.openpgp.org without any ca-cert-file option explicitly set. However while debugging, I noticed that NSS' CAfile was already correctly set to the system-wide certificate authority bundle file.

The correct ca-cert-file path will differ depending on the underlying operating system.

The alternative hkp: based key servers that @maxthomas and @lucasff mentioned above also bypass these SSL/TLS connectivity issues, by avoiding key server encryption/security entirely. The hkp: method is like plain-text HTTP vs. hkps: is more equivalent to encrypted HTTPS.

[nickgnazzo]

Here's what worked for me (git patch file included below). I tested this by enabling all Streisand features/services (including Tor). I put the changes into a git patch file which you can grab from the gist URL below, and apply with git am < 0001-Move-to-Mozilla-GPG-Keyserver-fix-a-few-GPG-verifica.patch:

https://gist.github.com/nickgnazzo/e7204e35c281e30c8fbdde08f72af1c4#file-0001-move-to-mozilla-gpg-keyserver-fix-a-few-gpg-verifica-patch

The tl;dr steps:

1. cd /path/to/Streisand/repo
2. curl -O https://gist.githubusercontent.com/nickgnazzo/e7204e35c281e30c8fbdde08f72af1c4/raw/b9e4066c85e315774134d8721d916022d15e718e/0001-Move-to-Mozilla-GPG-Keyserver-fix-a-few-GPG-verifica.patch
3. git am < 0001-Move-to-Mozilla-GPG-Keyserver-fix-a-few-GPG-verifica.patch
4. Re-run your Streisand script.

Summary of changes made:

• Change GPG keyserver to hkps://gpg.mozilla.org
  
  
  
  • For whatever reason, this keyserver doesn't seem affected by the key poisoning attack (yet). I think using this keyserver is likely only a temporary workaround. It might not be part of the SKS Keyserver Pool, but could still be vulnerable to the poisoning attack.
  
  
  • Unfortunately, using the new keys.openpgp.org (which has some mitigations against the SKS Keyserver attack) won't work _yet_. There's an issue causing gpg2 to fail to refresh keys when no "user ID" information is available. The problem here's two-fold, keys.openpgp.org requires email verification in order to allow publishing keys with their email address attached, and gpg2 refuses to process keys that have no user ID when refreshing. So by default, if you don't opt-in/verify your email address with OpenPGP, they won't publish the email associated with a public key (so when you run gpg2 refresh, you get the public keys without any user ID info), and then gpg2 refuses to process/import the keys. Meaning in order for this to work, you either have to wait for every key owner in Streisand's keyring to verfiy their email/identity with OpenPGP, or wait for GnuPG to fix the issue with the gpg2 client (or some other workaround I haven't thought of). The people running the OpenPGP keyserver are "working with" the GnuPG maintainers to address the problem (they mention it here).
  
  
  • If you try using keys.openpgp.org yourself, you'll likely see this error somewhere when Streisand runs gpg2 refresh: gpg: key XXXXXXXXXXXXXXXX: no user ID
  
  
• Add Mozilla Keyserver Root CA to dirmngr (using "hkp-cacert" option)
  
  
  
  • Since we want to use hkps, we have to tell dirmngr to trust Mozilla's Root CA to verify TLS. This is currently an Amazon Root CA, which is cross-signed by Starfield. So I just pointed dirmngr to the Starfield PEM file located in /etc/ssl/certs which should be available by default.
  
  
• Modify GPG Key ID used to verify Tor download signatures
  
  
  
  • Apparently the Key ID used to sign Tor downloads has changed recently. From what I can tell, the old key ID used by Streisand was C3C07136 (which seems to be Georg Koppen's pubkey), but they have switched to using the pubkey with user ID "The Tor Browser Developers" (D9FF06E2). Both of these key IDs are on Tor's signing keys page.
  
  
  • I needed to make this change for the "Verify" step of the Download/Mirror Tor Browser task to work. Otherwise, the signatures would technically show as "Good", but Streisand was looking for the old key ID in the verification command output to test if the verify worked, and thought it was failing.
  
  
• Modify GPG Key ID used to verify OpenVPN download signatures
  
  
  
  • Same deal as the Tor GPG change I mentioned above, the OpenVPN signing key in Streisand seems to also be outdated. I changed this from AF131CAE to 5ACFEAC6 within Streisand's variables and that made the verification pass for me.
  
  
  • Source of new key ID can be found on OpenVPN's page. It is technically a subkey of the key found on that page (I was able to find the new subkey 5ACFEAC6 in a few keyservers).
  
  
  • This seems like it's being tracked in a few other issues on Streisand's Github.
  
  
• Add new PuTTY Signing Key to Streisand's "Bootstrap GPG Keys"
  
  
  
  • Since the new Key ID used for PuTTY signatures is a new standalone key (and not a subkey of the current master key), I had to add it to Streisand's "Bootstrap GPG Key" setup tasks in order to get the PuTTY verification to work. Since it's not a subkey of the current signing key, it won't be downloaded when we run gpg refresh.
  
  
  • These keys live under playbooks/roles/gpg/files/ and are imported using the variable streisand_bootstrap_gpg_keys (referenced by another task).
  
  
• Modify PuTTY Key ID used to verify PuTTY download signatures
  
  
  
  • Change key ID from B43434E4 to 4AE8DA82 within Streisand variables.
  
  
  • Source of new key ID can be found here
  
  
  • Seems this is being tracked in another issue.
  
  

[chopraaa]

Here's what worked for me:

# - name: "Refresh the Streisand GPG keyring with keyserver information"
#   command: "gpg2 {{ streisand_default_gpg_flags }} {{ streisand_default_key_import_flags }} --refresh"
#   register: gpg2_refresh_result
#   until: "gpg2_refresh_result is success"
#   retries: 10
#   delay: 5
#   # NOTE(@cpu): We skip the keyring refresh in CI so that when the static keys
#   # in the repo become too stale to be used without successsful refresh the
#   # maintainers will notice failed builds and fix them by refreshing their own
#   # keyrings and updating the static repo keys until the build passes again.
#   when: not streisand_ci

[bonesoul]

same issue here.

[nopdotcom]

Fixed in #1604; reopen if it's still an issue.