Go: crypto/x509: Go 1.15 will break default connections to AWS RDS

[kevinburkemeter]

Amazon Relational Database Storage (RDS) is a popular database storage engine. RDS offers a method to securely connect to your database instance, described here: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/UsingWithRDS.SSL.html.

RDS databases - Postgres ones, anyway, have not tested the other SQL flavors - send back a certificate that includes a common name field. Our own databases are inside of a VPN so it's a little trickier to share access for you to test them that way, but let me know if you need to and we can work something out.

/usr/local/opt/[email protected]/bin/openssl s_client -starttls postgres secretdatabasename.us-west-2.rds.amazonaws.com:5432
CONNECTED(00000005)
depth=2 C = US, L = Seattle, ST = Washington, O = "Amazon Web Services, Inc.", OU = Amazon RDS, CN = Amazon RDS Root 2019 CA
verify error:num=19:self signed certificate in certificate chain
verify return:1
depth=2 C = US, L = Seattle, ST = Washington, O = "Amazon Web Services, Inc.", OU = Amazon RDS, CN = Amazon RDS Root 2019 CA
verify return:1
depth=1 C = US, ST = Washington, L = Seattle, O = "Amazon Web Services, Inc.", OU = Amazon RDS, CN = Amazon RDS us-west-2 2019 CA
verify return:1
depth=0 CN = secretdatabasename.us-west-2.rds.amazonaws.com, OU = RDS, O = Amazon.com, L = Seattle, ST = Washington, C = US
verify return:1
---
Certificate chain
 0 s:CN = secretdatabasename.us-west-2.rds.amazonaws.com, OU = RDS, O = Amazon.com, L = Seattle, ST = Washington, C = US
   i:C = US, ST = Washington, L = Seattle, O = "Amazon Web Services, Inc.", OU = Amazon RDS, CN = Amazon RDS us-west-2 2019 CA
 1 s:C = US, ST = Washington, L = Seattle, O = "Amazon Web Services, Inc.", OU = Amazon RDS, CN = Amazon RDS us-west-2 2019 CA
   i:C = US, L = Seattle, ST = Washington, O = "Amazon Web Services, Inc.", OU = Amazon RDS, CN = Amazon RDS Root 2019 CA
 2 s:C = US, L = Seattle, ST = Washington, O = "Amazon Web Services, Inc.", OU = Amazon RDS, CN = Amazon RDS Root 2019 CA
   i:C = US, L = Seattle, ST = Washington, O = "Amazon Web Services, Inc.", OU = Amazon RDS, CN = Amazon RDS Root 2019 CA

There are no problems connecting using Go up through 1.14. However, if you try to connect to a RDS database using TLS and the Go 1.15 beta, this is the message that you get:

x509: certificate relies on legacy Common Name field, use SANs or temporarily enable Common Name matching with GODEBUG=x509ignoreCN=0

I suspect that a lot of people are going to get caught out by this. (Well, maybe not, maybe people don't encrypt connections to the database.)

To be clear I think it is good that we are trying to get code (and certificates) to do the right thing. But it's not great that a bunch of code that previously worked will now _not_ work.

I don't have the juice with AWS to ask them to update the certs to use SAN's. Perhaps someone reading this thread does have the juice, and can explain the problem and ask them to upgrade their certificates?

Could we call this out more clearly in the docs? For example, we could move the description of the change closer to the top of go1.15.html, or indicate that we expect that this change is going to break a lot of existing deployments, including RDS and possibly others.

[ianlancetaylor]

CC @FiloSottile @katiehockman

[ianlancetaylor]

Marking as a 1.15 release blocker to make sure that we make some sort of fix or decision.

[kevinburkemeter]

Amazon's response may be something along the lines of "this doesn't really matter if you are using our VPC since packets in a VPC are encrypted and authenticated" but you'd still need it if you were connecting to the database from outside of AWS.

[graywolf-at-work]

Doesn't this kinda break https://golang.org/doc/go1compat? Or this this (not using common name) classify as a matter of security?

[mholt]

@graywolf-at-work No, that's only a language guarantee. Old Go programs still compile and run. This is like phasing out support for SSLv3 and Windows XP, and is necessary to keep the standard library current with external changes.

[kaelanfouwels]

CN verification is still specified by RFC 6125 as a fallback if SAN isn’t present.

Breaking external compatibility to prove a point seems excessive.

https://tools.ietf.org/html/rfc6125#section-6.4.4

[ulikunitz]

CL 231379 made "ignoreCN" the default. Reverting the first line of the diff, should bring back the old behavior.

[rolandshoemaker]

It should be noted that this will only break non-publicly trusted, non-web PKI certificates since the CN is mandated in the CABF BRs to be, if present at all, one of the identities in the SAN, making this change non-breaking for conforming CAs.

CN was deprecated in _2000_, and RFC 6125 made the fallback to it optional, rather a requirement (which was initially suggested in the RFC that deprecated it, RFC 2818).

[alex]

It seems like the most efficient path forward here is if someone from AWS can indicate their plans around adding SANs to RDS certificates.

[kevinburkemeter]

There's some indication on Twitter that the AWS team is looking at it but I'm not sure it will be ready by the release date. I don't know people in the right places at AWS to say otherwise. Filippo asked them for a timeline when they have one. https://twitter.com/seakoz/status/1271681101149270016

[FiloSottile]

The ideal resolution here is for AWS to use the correct field in their certificates. Let's wait to hear from them, and then we can decide whether to back off this change to Go 1.16.

Note that this change doesn't make it impossible to (manually) use CN for verification, it simply aligns with the spec in not considering it a hostname or an IP for VerifyHostname or VerifyOptions.DNSName purposes.

CN has been deprecated for 20 years, the specs do not recommend supporting it anymore, the browsers successfully dropped it, the WebPKI CAs are prohibited from setting it to anything that is not also in the SANs, and its behavior is completely unspecified.

The last point is a security issue with Name Constraint validation (which we support), because CN is not constrained (because it's not a hostname), so if we consider it a hostname we allow NCs to be bypassed. We have workarounds but they are complex unspecified hacks (see #24151).

[kevinburkemeter]

I'm just gonna say - I caught this because I deployed the 1.15 beta to our staging environment shortly after it was released. This gave us and AWS months of lead time between detection and the deadline for the release.

Please try to run the betas/RC's in your staging environments so that the entire community can benefit from the issues you find there, _before_ they go live to production. Thanks!

[kse-clearhaus]

We run a 3-D Secure Server. The certificates used by the scheme HTTPS endpoints
are universally issued by internal CAs.

This change will also break our connections to several major card schemes,
since they do not include SANs in certificates.

I'll grant you, that we will be able to work around it.
My point is, there are likely many areas where this causes issues.
Since there's actually an RFC that allows for a fallback, I would appriciate
if this was allowed.

Though I applaud your attempt to enforce a 20 year old deprecation :)

On a sidenote:

These CA's universally also encode nearly everything as ASN.1
PrintableString. This causes us heaps of trouble, much more than this would.

There's been several issues regarding this:
https://github.com/golang/go/issues?q=is%3Aissue+is%3Aclosed+printablestring

[cagedmantis]

@FiloSottile If it's decided that this is going to be rolled back then it would be best to do it before RC1 is released.

[FiloSottile]

@FiloSottile If it's decided that this is going to be rolled back then it would be best to do it before RC1 is released.

I don't think this needs to be rolled back based only on the AWS RDS breakage.

I do think (hopefully incorrectly) that we'll see more breakage as more people test this, but if we roll it back based on my hunch of what will come out of RC1, then we're not actually learning what breaks and we'll never get to land this, because the same argument will apply unchanged for Go 1.16rc1.

[gopherbot]

Change https://golang.org/cl/243221 mentions this issue: doc/go1.15: surface the crypto/x509 Common Name deprecation note

[toothrot]

As an update, we intend on releasing the RC with this change. There will be documentation in the release notes on how to get the old behavior while people work through issues. We're keeping this issue labeled as a release-blocker for Go 1.15, pending feedback from more people testing.

[luxifer]

For the record I'm experiencing the same issue as described but with Cisco ISE. Certificates generated by ISE contains CN but no SAN thus the certificate is not validated by the go tls.

[danp]

AWS have added a note to https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/UsingWithRDS.SSL-certificate-rotation.html about this issue:

If you are using a Go version 1.15 application with a DB instance that was created or updated to the rds-ca-2019 certificate prior to July 28, 2020, you must update the certificate again. Run the modify-db-instance command shown in the AWS CLI section using rds-ca-2019 as the CA certificate identifier. In this case, it isn't possible to update the certificate using the AWS Management Console. If you created your DB instance or updated its certificate after July 28, 2020, no action is required. For more information, see Go GitHub issue #39568

[kevinburkemeter]

Confirmed that the modify-db-instance fix described above works for AWS Postgres RDS and Go 1.15rc1.

[FiloSottile]

Amazon fixed this on their side (💙) so I'm going to close this.

Here are some details for anyone else encountering the same x509: certificate relies on legacy Common Name field error.

This error means that the certificate has the requested hostname or IP in the Common Name field, and not in the Subject Alternative Names extension. Verifying Common Name is unspecified, and has been deprecated for 20 years. Go 1.15 dropped support for it now, most browsers did it a long time ago.

You can fix this in a few different ways.

1. If the certificate is operated by a third-party, you should ask them to fix their certificates, like Amazon did. (If you are using AWS RDS there are instructions here.)
2. If you operate the certificate, you should make sure it has SANs, for example by using code like in generate_certificate.go or a tool like mkcert. Let's Encrypt has an example of how to use SANs with openssl.
3. You can temporarily re-enable the legacy behavior by setting the GODEBUG=x509ignoreCN=0 environment variable. This will be removed in Go 1.16, let us know if and why you need it.
4. If you actually need to check the Common Name field, you can use the new VerifyConnection callback to customize the verification process, not setting VerifyOptions.DNSName and doing a manual check instead.

[psanford]

@FiloSottile can we add note in the release notes pointing to the instructions from AWS on how to generate a new cert? I suspect there will be a bunch of RDS users who don't realize that they need to do this and will be sad when they upgrade go 1.15 and things stop working.

[FiloSottile]

@psanford I prefer not to call out specific third-party services in the release notes, but hopefully a web search leads them here easily enough.

I dropped a worked out example of VerifyConnection for Common Name checking in https://github.com/golang/go/issues/40748#issuecomment-673612108, by the way.

[cmars]

Having to set InsecureSkipVerify in order to verify the common name isn't great.

[FiloSottile]

Having to set InsecureSkipVerify in order to verify the common name isn't great.

You are opting out of hostname verification because the certificate does not have hostnames (as the Common Name is not a hostname), I think the "hic sunt leones" is justified. If you want to check Common Name _on top of_ hostname verification you don't have to set InsecureSkipVerify.

[cmars]

For applications that have no choice but to work around this new limitation (no clout to change service certificates, interoperability with legacy systems, stuck in some circle of enterprise hell, etc), I wonder if it's too easy to get it wrong? It just feels weird to simultaneously say "I'm skipping verification" but also "verify the connection like this". Which one wins? You can't tell unless you read the implementation.

Maybe there's no other good option without breaking APIs, and I get that legacy behavior can't be supported forever, but it gives an uneasy feeling.

[jdorothy]

Relating to what @cmars mentioned - as far as I can tell the AWS cert fix doesn't apply to, for example, GovCloud, which is still on rds-ca-2017. I don't know if that affects the decision to drop the workaround completely in Go 1.16, but it does put a lot of government projects in a bind.

[alex]

@jdorothy Have you confirmed that GovCloud still doesn't issue certs with SANs for RDS instances? It seems possible to me they use a different CA, but updated the issuing code to include SANs for it.

(I'm reasonably certain I recognize your name, which means we have some former colleagues we can probably ping to help nudge AWS along here.)

[jdorothy]

@alex certainly didn't expect to see your name chiming in! Based on the output of openssl s_client -starttls postgres database:5432 < /dev/null | openssl x509 -noout -text it turns out GovCloud is indeed including SANs on new DBs:

        X509v3 extensions:
            X509v3 Authority Key Identifier:
                keyid:41:BB:F9:BA:FC:2A:4B:F0:3D:DE:60:EF:54:6F:87:6B:2D:72:52:5B
                DirName:/C=US/L=Seattle/ST=Washington/O=Amazon Web Services, Inc./OU=Amazon RDS/CN=Amazon RDS GovCloud Root CA
                serial:10:00

            X509v3 Subject Key Identifier:
                09:9A:13:D6:08:99:E5:DD:3E:7F:CA:A6:B5:F3:71:AD:79:E3:1F:16
            X509v3 Subject Alternative Name:
                DNS:dbname.us-gov-west-1.rds.amazonaws.com

Then I ran the same command against an older DB and sure enough, no SAN.

Solution
The solution is to run the exact same command linked in https://github.com/golang/go/issues/39568#issuecomment-665863345, but replacing rds-ca-2019 with rds-ca-2017, even if you're already using rds-ca-2017. There appears to be a similar date cutoff as described in the AWS link, where new DBs are fine but DBs older than a certain date need to be updated (the DB I had to update was created 07-22-2020).

[alex]

:wave: :-) Wonderful!

[cjcjameson]

@jdorothy thanks for the command to verify if we would be vulnerable to this. We are encountering this issue with a (custom, non-AWS) Elasticsearch cluster; for that we are just using a command like openssl s_client -connect localhost:9200 | openssl x509 -noout -text to check. Seems like it's similar enough, just wanted to confirm and give breadcrumbs for other folks who might encounter this in non-AWS-RDS situations. cc @Sparkbean