Cryptography: Hex encoding issues with resulting PKCS12 private RSA decrypted bytes

[spiritedfang]

System Information
Databricks Runtime Version 5.2 (Apache Spark 2.4.0, Scala 2.11)
Python 3.5 (pip version 18.1)
pyOpenSSL 16.1.0, cryptography (stable)

Description
Hi there. I have cryptography installed using pyOpenSSL in Databricks. I'm using a command in pyOpenSSL called to_cryptography_key, which returns the cryptography package's RSAPrivateKey interface. The problem I'm having is calling the decrypt method, followed by bytearray.fromhex. It looks like an encoding issue, but I'm not sure where - the hex encoder throws the error below. If I add characters to the front of the hex_aes_decrypted_key string, the position in the error shifts to the beginning character of the original output from decrypt. Perhaps it's the UTF-8 decode I'm doing after the fact? If I copy the output of the first print statement and past it into a string for fromhex, it works. It's really confusing me. Any help would be appreciated. Thanks!

ValueError: non-hexadecimal number found in fromhex() arg at position 0
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<command-2550337372484884> in <module>()
     24 hex_aes_decrypted_key = aes_decrypted_key.decode("utf8")
     25 print("test:", hex_aes_decrypted_key)
---> 26 print("test:", bytearray.fromhex(hex_aes_decrypted_key))
     27 aes_key = bytearray.fromhex(hex_aes_decrypted_key)
     28 aes_iv  = bytearray(dbutils.widgets.get("BlobAesIV"), encoding="utf8")

Sample

from Crypto.Cipher import AES
from OpenSSL       import crypto
from cryptography.hazmat.primitives.asymmetric import padding
from cryptography.hazmat.primitives            import hashes
import base64

# Load in the certificate from Databricks storage
pfx_certpath = dbutils.widgets.get("PfxCertPath")
pfx_password = dbutils.widgets.get("PfxPassword")
pfx = open(pfx_certpath, 'rb').read()
p12 = crypto.load_pkcs12(pfx, pfx_password)

# Decrypt the AES key
p12_private_key = p12.get_privatekey()
p12_private_rsa = p12_private_key.to_cryptography_key()
aes_encrypted_key = base64.b64decode(dbutils.widgets.get("BlobAesKey"))
aes_decrypted_key = p12_private_rsa.decrypt(aes_encrypted_key, padding.OAEP(
    mgf=padding.MGF1(algorithm=hashes.SHA1()),
    algorithm=hashes.SHA1(),
    label=None
))

# AES Configuration Settings
hex_aes_decrypted_key = aes_decrypted_key.decode("utf8")
print("test:", hex_aes_decrypted_key)
print("test:", bytearray.fromhex(hex_aes_decrypted_key))
aes_key = bytearray.fromhex(hex_aes_decrypted_key)
aes_iv  = bytearray(dbutils.widgets.get("BlobAesIV"), encoding="utf8")

[alex]

Your problem is that hex_aes_decrypted_key is not actually hex. Without seeing yoru encryption code, it's hard for us to know why that is. I recommend you print it out and see what format it _is_ in.

[spiritedfang]

Sorry, here's the output from the first print:

test: 6B25DFBDB6292816E5EC22CA369EBAFB715C5A9A24CC3937942F866D8F75985F

[alex]

bytearray.fromhex("6B25DFBDB6292816E5EC22CA369EBA FB715C5A9A24CC3937942F866D8F75985F") works, so there must be something
else going on here.

On Fri, Mar 29, 2019 at 8:19 PM Spirited [e-mail] wrote:

System Information
Databricks Runtime Version 5.2 (Apache Spark 2.4.0, Scala 2.11)
Python 3.5 (pip version 18.1)
pyOpenSSL 16.1.0, cryptography (stable)

Description
Hi there. I have cryptography installed using pyOpenSSL in Databricks. I'm
using a command in pyOpenSSL called to_cryptography_key
https://pyopenssl.org/en/stable/api/crypto.html#OpenSSL.crypto.PKey.to_cryptography_key,
which returns the cryptography package's RSAPrivateKey
https://cryptography.io/en/latest/hazmat/primitives/asymmetric/rsa/#cryptography.hazmat.primitives.asymmetric.rsa.RSAPrivateKey
interface. The problem I'm having is calling the decrypt
https://cryptography.io/en/latest/hazmat/primitives/asymmetric/rsa/#cryptography.hazmat.primitives.asymmetric.rsa.RSAPrivateKey.decrypt
method, followed by bytearray.fromhex
https://python-reference.readthedocs.io/en/latest/docs/bytearray/fromhex.html.
It looks like an encoding issue, but I'm not sure where - the hex encoder
throws the error below. If I add characters to the front of the
hex_aes_decrypted_key string, the position in the error shifts to the
beginning character of the original output from decrypt. Perhaps it's the
UTF-8 decode I'm doing after the fact? If I copy the output of the first
print statement and past it into a string for fromhex, it works. It's
really confusing me. Any help would be appreciated. Thanks!

ValueError: non-hexadecimal number found in fromhex() arg at position 0

ValueError Traceback (most recent call last)
in ()
24 hex_aes_decrypted_key = aes_decrypted_key.decode("utf8")
25 print("test:", hex_aes_decrypted_key)
---> 26 print("test:", bytearray.fromhex(hex_aes_decrypted_key))
27 aes_key = bytearray.fromhex(hex_aes_decrypted_key)
28 aes_iv = bytearray(dbutils.widgets.get("BlobAesIV"), encoding="utf8")

Sample

from Crypto.Cipher import AES
from OpenSSL import crypto
from cryptography.hazmat.primitives.asymmetric import padding
from cryptography.hazmat.primitives import hashes
import base64

Load in the certificate from Databricks storage

pfx_certpath = dbutils.widgets.get("PfxCertPath")
pfx_password = dbutils.widgets.get("PfxPassword")
pfx = open(pfx_certpath, 'rb').read()
p12 = crypto.load_pkcs12(pfx, pfx_password)

Decrypt the AES key

p12_private_key = p12.get_privatekey()
p12_private_rsa = p12_private_key.to_cryptography_key()
aes_encrypted_key = base64.b64decode(dbutils.widgets.get("BlobAesKey"))
aes_decrypted_key = p12_private_rsa.decrypt(aes_encrypted_key, padding.OAEP(
mgf=padding.MGF1(algorithm=hashes.SHA1()),
algorithm=hashes.SHA1(),
label=None
))

AES Configuration Settings

hex_aes_decrypted_key = aes_decrypted_key.decode("utf8")
print("test:", hex_aes_decrypted_key)
print("test:", bytearray.fromhex(hex_aes_decrypted_key))
aes_key = bytearray.fromhex(hex_aes_decrypted_key)
aes_iv = bytearray(dbutils.widgets.get("BlobAesIV"), encoding="utf8")

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/pyca/cryptography/issues/4825, or mute the thread
https://github.com/notifications/unsubscribe-auth/AAADBB_kWMWXuIrqDHlWYtIzjz7awQR5ks5vbq2hgaJpZM4cTbmJ
.

--
All that is necessary for evil to succeed is for good people to do nothing.

[spiritedfang]

I tried another test with print("test:", aes_decrypted_key) and got the following:

test: b'6\x00B\x002\x005\x00D\x00F\x00B\x00D\x00B\x006\x002\x009\x002\x008\x001\x006\x00E\x005\x00E\x00C\x002\x002\x00C\x00A\x003\x006\x009\x00E\x00B\x00A\x00F\x00B\x007\x001\x005\x00C\x005\x00A\x009\x00A\x002\x004\x00C\x00C\x003\x009\x003\x007\x009\x004\x002\x00F\x008\x006\x006\x00D\x008\x00F\x007\x005\x009\x008\x005\x00F\x00'

[alex]

Well, that is not hex. So whatever you're encrypting is not hex.

On Fri, Mar 29, 2019 at 8:31 PM Spirited [e-mail] wrote:

I tried another test with print("test:", aes_decrypted_key) and got the
following:

test: b'6\x00B\x002\x005\x00D\x00F\x00B\x00D\x00B\x006\x002\x009\x002\x008\x001\x006\x00E\x005\x00E\x00C\x002\x002\x00C\x00A\x003\x006\x009\x00E\x00B\x00A\x00F\x00B\x007\x001\x005\x00C\x005\x00A\x009\x00A\x002\x004\x00C\x00C\x003\x009\x003\x007\x009\x004\x002\x00F\x008\x006\x006\x00D\x008\x00F\x007\x005\x009\x008\x005\x00F\x00'

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/pyca/cryptography/issues/4825#issuecomment-478187892,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAADBBYu1LcNAr2NaxWHC-qagy-HDJKUks5vbrBJgaJpZM4cTbmJ
.

--
All that is necessary for evil to succeed is for good people to do nothing.

[spiritedfang]

What's interesting is that this totally works in other languages like C#. It's the same certificate and key that's been proven working. I wonder if it's a different string encoding that I just can't figure out, or something with my padding use for the RSA decrypt. Thoughts?

[alex]

Without seeing your encryption code, it's really hard for us to guess what the problem is here.

[spiritedfang]

Figured it out. Me saying it worked in C# immediately made me think of Microsoft's dumb UTF-16 garbage, and sure enough, that was the issue. It's string encoded in UTF-16. Sorry for the troubles, and thank you so much for your time. Appreciate the rubber duckying.