Urllib3: Usage of cryptography + idna breaks on wildcard certs

[sigmavirus24]

The changes in #930 seem to have broken with some wildcard certificates: https://stackoverflow.com/questions/39521147/why-is-urllib3-idna-complaining-about-a-wildcard-in-an-x509-cert-how-do-i-fix-i

I haven't attempted to reproduce this, but I thought recording it here would be of use.

[Lukasa]

So we're using idna basically because cryptography does. @reaperhulk, any insight here?

[reaperhulk]

Besides that idna was a terrible mistake?

I need to know what version of cryptography. I was just able to load the cert in question without problems with both current master and version 1.5.

[marc-swingler]

title = "cryptography"
...
version = "1.5"

This is running under python 2.7 on an Ubuntu 12.04 VM.

I'm running under a virtual env with the following:

setuptools==26.1.1
td-client==0.5.0
urllib3[secure]==1.17

I also have the required system packages listed in the user guide:
https://cryptography.io/en/latest/installation/#building-cryptography-on-linux

dpkg -l | grep build-essential
ii build-essential 11.6ubuntu6 amd64 Informational list of build-essential packages
dpkg -l | grep libssl-dev
ii libssl-dev:amd64 1.0.1f-1ubuntu2.7 amd64 Secure Sockets Layer toolkit - development files
dpkg -l | grep libffi-dev
ii libffi-dev:amd64 3.1~rc1+r3.0.13-12 amd64 Foreign Function Interface library (development files)
dpkg -l | grep "\spython-dev\s"
ii python-dev 2.7.5-5ubuntu3 amd64 header files and a static library for Python (default)

[reaperhulk]

What happens if you run this script:

from cryptography import x509
from cryptography.hazmat.backends.openssl.backend import backend


pem = b"""-----BEGIN CERTIFICATE-----
MIIFMjCCBBqgAwIBAgIHK427mrAroDANBgkqhkiG9w0BAQsFADCBtDELMAkGA1UE
BhMCVVMxEDAOBgNVBAgTB0FyaXpvbmExEzARBgNVBAcTClNjb3R0c2RhbGUxGjAY
BgNVBAoTEUdvRGFkZHkuY29tLCBJbmMuMS0wKwYDVQQLEyRodHRwOi8vY2VydHMu
Z29kYWRkeS5jb20vcmVwb3NpdG9yeS8xMzAxBgNVBAMTKkdvIERhZGR5IFNlY3Vy
ZSBDZXJ0aWZpY2F0ZSBBdXRob3JpdHkgLSBHMjAeFw0xNDA0MTAwNjA2MDNaFw0x
NzAxMjAxMDA3MDRaMEAxITAfBgNVBAsTGERvbWFpbiBDb250cm9sIFZhbGlkYXRl
ZDEbMBkGA1UEAwwSKi50cmVhc3VyZWRhdGEuY29tMIIBIjANBgkqhkiG9w0BAQEF
AAOCAQ8AMIIBCgKCAQEAzmjm0Ogr4IPN2YQAWFcxJGzCegewnT0E3hW5tD7kgFBj
bGpLvHFdqnhd1nGhxYF/CufioUxKSgWtvrw5sL0zVyO1oeURjk5+FFwPMTKd3scW
1v7u2FJrs2uMoYL+Y2DlqWqUQikbX9U3flXcol1XSWD9MZbZLMNeZdKHMRtTi5ew
FVXQusYWG+eCovYtetutZ+qhoaGChirTKwg4lNXeID79/XuAleNQ0rUD+HgD6B1K
ZBXxomMibV48VerPQLzrp3BztXgjroSIqqtZGEcQHloMYfcn1zq7rRmhR5268RqP
sUc6cvijI537IsBvlNaByb2WW5DeLgQKraPt3v2YIQIDAQABo4IBujCCAbYwDwYD
VR0TAQH/BAUwAwEBADAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwDgYD
VR0PAQH/BAQDAgWgMDYGA1UdHwQvMC0wK6ApoCeGJWh0dHA6Ly9jcmwuZ29kYWRk
eS5jb20vZ2RpZzJzMS0zOS5jcmwwUwYDVR0gBEwwSjBIBgtghkgBhv1tAQcXATA5
MDcGCCsGAQUFBwIBFitodHRwOi8vY2VydGlmaWNhdGVzLmdvZGFkZHkuY29tL3Jl
cG9zaXRvcnkvMHYGCCsGAQUFBwEBBGowaDAkBggrBgEFBQcwAYYYaHR0cDovL29j
c3AuZ29kYWRkeS5jb20vMEAGCCsGAQUFBzAChjRodHRwOi8vY2VydGlmaWNhdGVz
LmdvZGFkZHkuY29tL3JlcG9zaXRvcnkvZ2RpZzIuY3J0MB8GA1UdIwQYMBaAFEDC
vSeOzDSDMKIz1/tss/C0LIDOMC8GA1UdEQQoMCaCEioudHJlYXN1cmVkYXRhLmNv
bYIQdHJlYXN1cmVkYXRhLmNvbTAdBgNVHQ4EFgQU3oO+gpNB3WYMVywxSM2p2E6f
etMwDQYJKoZIhvcNAQELBQADggEBAJLvOqYh7tiPbT+qKyNHuuSjetfUYHWBx29A
t+hXI7Khn1waKxbgUa/QSZiUUD9C097LMhydx/Z8p6bHlkFqIfSOKkxWS736N17C
Tldrz8RSSTByp2OfCdA9Ts9ANToR9R33vR40aOQDL29NyoyTAjNPRCHKmLz4F8sE
1tlKJKysO9p6qA1+B5ZEHOY6McK4tm3L2Jhz0DgMzk3QbBh8aJ/cGfezQQoC+RT4
L3Nx0+yhclrOY9L1PEQ4hBjJCsUKd7XcKfUNx+iXTNpTfmTiTfsBmfNx+V79cu+v
RoSAetF+rkP0MTReO7DD9jybxn+2ttsC1x8HCw6Nc+NK7GqEHok=
-----END CERTIFICATE-----"""

cert = x509.load_pem_x509_certificate(pem, backend)
print(cert.subject)
print(cert.extensions.get_extension_for_class(x509.SubjectAlternativeName))

That should load and print info about the certificate you linked to on SO.

[marc-swingler]

Running with urllib3 v1.7

~/repos/akamai_pusher/current_version$ ./akamai_pusher.sh                                 
<Name([<NameAttribute(oid=<ObjectIdentifier(oid=2.5.4.11, name=organizationalUnitName)>, value=u'Domain Control Validated')>, <NameAttribute(oid=<ObjectIdentifier(oid=2.5.4.3, name=commonName)>, value=u'*.treasuredata.com')>])>
<Extension(oid=<ObjectIdentifier(oid=2.5.29.17, name=subjectAltName)>, critical=False, value=<SubjectAlternativeName(<GeneralNames([<DNSName(value=*.treasuredata.com)>, <DNSName(value=treasuredata.com)>])>)>)>

[reaperhulk]

Okay, I understand this bug now.

https://github.com/shazow/urllib3/blob/0319846b13ad44e1a2a6b3985369cdc15cefe914/urllib3/contrib/pyopenssl.py#L141 is way too simplistic. It assumes any name given to it will be IDNA encodable, which is untrue. In reality this code needs to look like what we do in cryptography (https://github.com/pyca/cryptography/blob/master/src/cryptography/hazmat/backends/openssl/encode_asn1.py#L365-L371).

[Lukasa]

Ok fab, I'll give that a shot and see if it helps.