Gsutil: "Host ... returned an invalid certificate" when using gsutil rsync with a s3 bucket that has dots in its name.

[7heo]

Using gsutil rsync with a bucket whose name contains dots causes the following error:

$ gsutil rsync s3://xxxxxx.xxxxxx.xxxxxxxx.xxxxxxxxxxx/ gs://xxxxxxx-xxxxxxx-x/
Building synchronization state...
Caught non-retryable exception while listing s3://xxxxxx.xxxxxx.xxxxxxxx.xxxxxxxxxxx/: Host
xxxxxx.xxxxxx.xxxxxxxx.xxxxxxxxxxx.s3.amazonaws.com returned an invalid certificate (remote
hostname "xxxxxx.xxxxxx.xxxxxxxx.xxxxxxxxxxx.s3.amazonaws.com" does not match certificate):
{
  'notAfter': 'Jun  5 23:59:59 2015 GMT',
  'subjectAltName': (
                      (u'DNS', '*.s3.amazonaws.com'),
                      (u'DNS', 's3.amazonaws.com')
                    ),
  'subject': (
               (('countryName', u'US'),),
               (('stateOrProvinceName', u'Washington'),),
               (('localityName', u'Seattle'),),
               (('organizationName', u'Amazon.com Inc.'),),
               (('organizationalUnitName', u'S3-A'),),
               (('commonName', u'*.s3.amazonaws.com'),)
             )
}

It is possible to circumvent that problem by using https://s3-<region>.amazonaws.com/xxxxxx.xxxxxx.xxxxxxxx.xxxxxxxxxxx as the URL instead; but gsutil does not construct endpoints with that form even in the case of dotted names, causing the error.

[Capstan]

Is there a corresponding S3 service issue report for them not presenting the correct certificate? Or known issue link?

[Capstan]

Or is this a matter of Python not supporting SNI and Amazon would yield the right certificate if it did?

[7heo]

Is there a corresponding S3 service issue report for them not presenting the correct certificate?

Not as far as I imagine, but I didn't check: they provide an endpoint (the second form in my bug report) that works with the certificate they provide; so they probably deprecated (at least they do in practice) the subdomain form.

Is this a matter of Python not supporting SNI and Amazon would yield the right certificate if it did?

@Capstan I have connected to both https://xxxxxx.xxxxxx.xxxxxxxx.xxxxxxxxxxx.s3.amazonaws.com and https://s3-us-west-1.amazonaws.com/xxxxxx.xxxxxx.xxxxxxxx.xxxxxxxxxxx using my browser (firefox-latest) for testing, and only the last one has a valid certificate (I think amazon only supports one level of sub-domains below s3.amazonaws.com, as it would be rather annoying for them to support more; while they provide another, working, endpoint). I believe you can try that in your browser to get a confirmation of that behavior.

[7heo]

@Capstan In any case, after digging in the code, it seems like it is more of a boto issue than of a gsutil/gcloud one. I'll check boto then (I've tracked the error until the _CloudUrl class, but then it seems that boto constructs the bucket name).

[7heo]

After further investigation, this issue is a known boto issue (boto/boto#2836). So I guess it's safe to close it.

[7heo]

@thobrla @Capstan Is there a plan on the Google side (gcloud/gsutil) to switch from boto to boto3? If yes, can you communicate on it? Thanks!

[7heo]

@thobrla @Capstan this issue has been fixed by a PR, please see https://github.com/boto/boto/issues/2836#issuecomment-101336788

[elliotthill]

When will this fix be available in a gsutil release? Im on version 4.18 and it still having the same issue.

[s3] calling_format = boto.s3.connection.OrdinaryCallingFormat

Does not work for me. It then returns bucket not found errors.

[thobrla]

Before we can include a fix in gsutil, we need:

1. An actual committed fix in boto (as of today there are only uncommitted pull requests)
2. A boto release that includes the fix

[bisol]

Is there any update on this? I'm getting this error with gsutil 4.27

$ gsutil cp s3://xxx.yyy.zzz gs://xxx-xxx-xxx-xxx
Failure: Host xxx.yyy.zzz.s3.amazonaws.com returned an invalid certificate (remote hostname "xxx.yyy.zzz.s3.amazonaws.com" does not match certificate):
{
'crlDistributionPoints': (
u'http://crl3.digicert.com/DigiCertBaltimoreCA-2G2.crl',
u'http://crl4.digicert.com/DigiCertBaltimoreCA-2G2.crl'
),
'subjectAltName': (
('DNS', '.s3.amazonaws.com'),
('DNS', 's3.amazonaws.com')
),
'notBefore': u'Sep 22 00:00:00 2017 GMT',
'caIssuers': (u'http://cacerts.digicert.com/DigiCertBaltimoreCA-2G2.crt',),
'OCSP': (u'http://ocsp.digicert.com',),
'serialNumber': u'0551B592FA53CF2052B8B70F275CC159',
'notAfter': 'Jan 3 12:00:00 2019 GMT',
'version': 3L,
'subject': (
(('countryName', u'US'),),
(('stateOrProvinceName', u'Washington'),),
(('localityName', u'Seattle'),),
(('organizationName', u'Amazon.com Inc.'),),
(('commonName', u'.s3.amazonaws.com'),)
),
'issuer': (
(('countryName', u'US'),),
(('organizationName', u'DigiCert Inc'),),
(('organizationalUnitName', u'www.digicert.com'),),
(('commonName', u'DigiCert Baltimore CA-2 G2'),)
)
}.

[houglum]

I'd like to provide positive news, but it looks like the PR mentioned above containing the fix for Boto isn't getting merged :(

[carlthome]

Got caught up on this today. Any fix available?

[7heo]

Nope. Never will.

[kuzaxak]

Append the next section to ~/.boto file:

[s3] 
calling_format = boto.s3.connection.OrdinaryCallingFormat
host = s3-eu-west-1.amazonaws.com

It will fix the issue. Replace region if you need.