Cryptography: Implement (or document) streaming API

[Nikratio]

I'm looking at https://cryptography.io/en/latest/fernet/, and I'm wondering how to use this API if I want to encrypt _a lot_ of data.

I hope that I can simply call encrypt in a loop:

with open('secret.txt', 'rb') as ifh:
    with open('encrypted.txt', 'wb') as ofh:
        f = Fernet(key)
        while True:
            buf = ifh.read(BUFSIZE)
            if not buf:
                break
            ofh.write(f.encrypt(buf)))

..but it would be nice to have this offically blessed (and implemented, if it doesn't work like that at the moment).

[reaperhulk]

Fernet is unfortunately unsuitable for streaming encryption due to the requirement that the entirety of the payload be processed by HMAC before you know if any data can be used. We have some ideas for an authenticated encryption recipe that allows streaming (utilizing some concepts from Tahoe-LAFS's merkle tree authentication of ciphertext) but no spec has yet been written.

[Nikratio]

Ah, I see. I also just noticed that the data returned by Fernet is base64 encoded - probably also not a good idea if there's a lot of it :-).

In case you are interested (just throwing it out there): I was interested in the cryptography module because I was thinking it could replace my own stream-encryption code in https://bitbucket.org/nikratio/s3ql/src/407ece4bfb515c98f973c2ecb0eab1d3a13fb766/src/s3ql/backends/comprenc.py?at=default&fileviewer=file-view-default#comprenc.py-453. Maybe this can give you some ideas for API design (or what mistakes people do when implementing it themselves from the primitives) :-).

[nils-werner]

Another streaming AES implementation I wrote over the last days: https://github.com/nils-werner/zget/blob/crypto/zget/crypto.py

• Implemented as a simple generator
• hashes password using PBKDF2 and salt
• uses AES CTR (so we can accept arbitrarily sized blocks to encrypt)
• uses SHA256 HMAC for message Encrypt-then-MAC authentication
• prepends IV and salt to message
• automatically appends HMAC signature
• automatically appends cipher.finalize() call
• Python 2 and 3 compatible and tested

Because it is implemented as a generator you can use it very easily, e.g.

from zget import crypto

cipher = crypto.aes.encrypt("secret_key")

with open("README.md", "r") as fh:
    for line in cipher(fh):
        print line

[chadwhitacre]

We have some ideas

FTR: https://mail.python.org/pipermail/cryptography-dev/2015-January/000371.html

[nils-werner]

We have some ideas

did it lead anywhere?

[drkjam]

Also interested in this.

[nils-werner]

I have updated my streaming AES implementation to now also do AES-CTR+HMAC.

[nils-werner]

And also, an implementation that does AES-CTR+HMAC+SPAKE2. None of this code has been reviewed yet (but is about 130 lines long, plus a couple of lines for SPAKE), so any feedback very welcome. It can be used like

from zget import crypto

cipher = crypto.aes_spake.encrypt("secret_key")
key_a = cipher.start()    # send key_a to the recipient
cipher.finish(key_b)      # received key_b from recipient

with open("README.md", "r") as fh:
    for line in cipher(fh):
        print line

[pquentin]

We would also be really happy to see this officially supported.

[reaperhulk]

I'm still interested in this and have a potential spec but haven't had the bandwidth to get it reviewed by enough cryptographers to feel comfortable implementing it in a public API yet. It's always useful to know about more people who could use an authenticated encryption streaming API though!

[Darkheir]

I would also love to see an authenticated encryption streaming API!