Cryptography: "Could not deserialize key data" for ES256 on Alpine 3.8.0

[mikob]

On alpine 3.8.0, python 3.7. Using the python:3.7-alpine docker image a5f497d596f5, I tried installing cryptography 3 different ways, it installs successfully but then I can't do an ES256 encoding. This works fine on my ubuntu machine, but doesn't work in alpine -- hence I know it's not an issue with the key format -- which has proper spacing and everything.

I tried with openssl-dev instead of libressl-devusing the compilation method, and apk add/apk del method detailed here: https://github.com/pyca/cryptography/issues/4264.

I also tried installing with regular old pip install cryptography too of course.

cryptography==2.3.1
cffi==1.11.5
pipenv==2018.7.1

>>> jwt.encode({'hi': 'there'}, secret, algorithm='ES256')
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/jwt/algorithms.py", line 349, in prepare_key
    key = load_pem_public_key(key, backend=default_backend())
  File "/usr/local/lib/python3.7/site-packages/cryptography/hazmat/primitives/serialization.py", line 24, in load_pem_
public_key
    return backend.load_pem_public_key(data)
  File "/usr/local/lib/python3.7/site-packages/cryptography/hazmat/backends/openssl/backend.py", line 1040, in load_pe
m_public_key
    self._handle_key_loading_error()
  File "/usr/local/lib/python3.7/site-packages/cryptography/hazmat/backends/openssl/backend.py", line 1291, in _handle
_key_loading_error
    raise ValueError("Could not deserialize key data.")
ValueError: Could not deserialize key data.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.7/site-packages/jwt/api_jwt.py", line 66, in encode
    json_payload, key, algorithm, headers, json_encoder
  File "/usr/local/lib/python3.7/site-packages/jwt/api_jws.py", line 114, in encode
    key = alg_obj.prepare_key(key)
  File "/usr/local/lib/python3.7/site-packages/jwt/algorithms.py", line 351, in prepare_key
    key = load_pem_private_key(key, password=None, backend=default_backend())
  File "/usr/local/lib/python3.7/site-packages/cryptography/hazmat/primitives/serialization.py", line 20, in load_pem_
private_key
    return backend.load_pem_private_key(data, password)
  File "/usr/local/lib/python3.7/site-packages/cryptography/hazmat/backends/openssl/backend.py", line 1014, in load_pe
m_private_key
    password,
  File "/usr/local/lib/python3.7/site-packages/cryptography/hazmat/backends/openssl/backend.py", line 1233, in _load_k
ey
    self._handle_key_loading_error()
  File "/usr/local/lib/python3.7/site-packages/cryptography/hazmat/backends/openssl/backend.py", line 1291, in _handle
_key_loading_error
    raise ValueError("Could not deserialize key data.")
ValueError: Could not deserialize key data.

[reaperhulk]

I can't replicate this. Here's what I did:

docker run --rm -ti python:3.7-alpine /bin/sh
apk add libressl-dev gcc libffi-dev musl-dev
pip install pyjwt cryptography

Then I ran this in the Python REPL:

import jwt

from cryptography.hazmat.primitives.asymmetric import ec
key = ec.generate_private_key(ec.SECP256R1(), backend)
jwt.encode({'hi': 'there'}, key, algorithm='ES256')

That works. But you mentioned it couldn't deserialize so I tried:

from cryptography.hazmat.primitives import serialization
pk = key.private_bytes(serialization.Encoding.PEM, serialization.PrivateFormat.PKCS8, serialization.NoEncryption())
jwt.encode({'hi': 'there'}, pk, algorithm='ES256')

If those work for you then that suggests there's something unusual about the key you're attempting to use. If you parse it with openssl ec -text -in <filename> does it say that it's P-256?

[mikob]

weird! Doing those same steps do work for me when using the generated key as you did. But with my own key, they don't work... same error as in the OP. Here's an excerpt of the output of the openssl command you asked for:

ASN1 OID: prime256v1
NIST CURVE: P-256
writing EC key
-----BEGIN EC PRIVATE KEY-----
...

Also, I think my own key works fine when I install openssl-dev instead of libressl-dev, but I would need to verify that because I've tried so many things I may be mixing them up :P

[mikob]

Yes, I've confirmed it does work fine if I use openssl-dev instead of libressl-dev with my key. But that workaround gives me other problems. This is a p8 key for apple push notifications. I can perhaps get you a key and revoke it if it's useful for your testing?

[reaperhulk]

It would definitely be useful to know if there's a difference in behavior between openssl and libressl. Right now this looks like it could potentially be an issue with one of those libraries rather than a problem specific to cryptography.

[mikob]

Very interesting! A new key I generated does work just fine with the normal means and libressl-dev. So I've double checked things, and in summary:
1) Old key works with openssl, not libressl
2) New key of the same format works with libressl (assuming it works fine with openssl)

Thanks for looking into this. What do you propose we do next? Would you recommend I file an issue with libressl instead?

[reaperhulk]

Two questions: Is the key is no longer used for anything sensitive and can therefore be disclosed? And does the libressl command line (confusingly called openssl of course) fail to parse the key as well?

[mikob]

openssl parses it just fine. Hence I could give you some of the output here: https://github.com/pyca/cryptography/issues/4417#issuecomment-416635644

I can disclose it soon as I'm using the new key. I will disclose tomorrow.

[mikob]

sorry, just read your comment more thoroughly today. I see that you said with the libressl command line utility now. Actually the command line utility fails to parse the new key:

/ # openssl ec -text -in /tmp/keys/AuthKey_H8VQ597Q33.p8 
read EC key
unable to load Key
140667078503372:error:09FFF064:PEM routines:CRYPTO_internal:bad base64 decode:pem/pem_lib.c:800:

And here is the problematic key which now isn't useful for anything sensitive:

-----BEGIN PRIVATE KEY-----
MIGTAgEAMBMGByqGSM49AgEGCCqGSM49AwEHBHkwdwIBAQQgsegINAr5xcE48BiDyfXjsfQmEk1ReGtD7bSuKsKx04CgCgYIKoZIzj0DAQehRANCAASauMCp36D8FOF15OGI1+fe5oeRoCbY5yGQ2Jk0Gi9P92ksyvC8LK7JDqtzKfEf18UsScYc+NWffEttv413G73q
-----END PRIVATE KEY-----

[reaperhulk]

Oh, the problem is that newer OpenSSL is apparently tolerant of non-linewrapped PEM while libre is not. This will parse in libre (the key you provided but line wrapped):

-----BEGIN PRIVATE KEY-----
MIGTAgEAMBMGByqGSM49AgEGCCqGSM49AwEHBHkwdwIBAQQgsegINAr5xcE48BiD
yfXjsfQmEk1ReGtD7bSuKsKx04CgCgYIKoZIzj0DAQehRANCAASauMCp36D8FOF1
5OGI1+fe5oeRoCbY5yGQ2Jk0Gi9P92ksyvC8LK7JDqtzKfEf18UsScYc+NWffEtt
v413G73q
-----END PRIVATE KEY-----

Mystery solved!

[mikob]

Thanks @reaperhulk glad we got to the bottom of it. Hope other ppl with the same issue find this, spent a lot of time on it :)