Cryptography: Feature Request: SMIME Support

[abhishek-ram]

I am working on a tool that implements the AS2 protocol for file transfers, a major requirement for it is SMIME support. I am currently using m2crypto to implement this but that project is no longer supported and also does not support Python 3.x and lacks support for PKCS7 signatures using the latest algorithms. Are there any plans to add SMIME support in your library?

[reaperhulk]

We do plan to eventually support SMIME, yes. The spec is pretty broad (and we're still working on some dependencies like X509), but we'll get there.

[Ayrx]

IIRC SMIME requires CMS? So we need to get #924 done as well.

[davidism]

Consider prioritizing this feature. The only other Python library providing SMIME is m2crypto, which is not available for Python 3, provides a very "raw" experience, and does not seem to be actively maintained.

This library is in a position to provide a feature that the Python ecosystem sorely lacks.

[reaperhulk]

@davidism Do you have some code written against M2Crypto right now that demonstrates your needs? The entirety of SMIME/CMS/X509 is very large and complex so supporting them generically is a long term project. However, if there's a way we can start supporting a subset of the complete specification and cover most use cases that'd be great.

[davidism]

I currently use it to sign (not encrypt) emails.

# load keys
smime = SMIME.SMIME()
smime.load_key('signer.key', 'signer.crt')

# generate signature from original message
# msg is an email object with no mail headers (just the multipart structure)
data = BIO.MemoryBuffer(msg.as_string())
p7 = smime.sign(data)
data.reset()
p7.write(data)

# signature attachment, remove the 'begin' and 'end' delimiters
sig = MIMEApplication(
    '\n'.join(data.read().split('\n')[1:-2]),
    'pkcs7-signature', encode_noop, name='smime.p7s'
)
sig.add_header('Content-Disposition', 'attachment;filename=smime.p7s')
sig.add_header('Content-Transfer-Encoding', 'base64')

# pack original message and signature
signed = MIMEMultipart('signed', protocol='application/pkcs7-signature', micalg='sha1')
signed.attach(msg)
signed.attach(sig)

# now set the mail headers on the new signed multipart structure

I don't have much experience with email, cryptography, or SMIME, this was all put together by comparing what this built to existing signed emails.

It would be great if there was better integration with Python's email library.

[abhishek-ram]

I think if ASN.1 support(parse/generate) can be provided then that will be a great starting place. As you have rightly said the entire SMIME/CMS/X509 set is too large to implement currently. If we have ASN.1 and a method to create specs(Classes similar to pyasn1) for different X509 types then each dev can contribute to these specs based on their needs and thus speeded up developement of CMS/X509 and inturn SMIME.

[davidism]

While I _use_ SMIME to sign emails, I would not know where to start to create a "spec" for signing emails.

[abhishek-ram]

Yeah I can understand, but to be able to start to sign as per SMIME, we need a way to generate PKCS7 signed/enveloped data and to do that we need ASN.1. So if we can have ASN.1 then we can create the specfication for creating/reading PKCS7 type data as per the RFC. Once this is in place you will be able to pass conent to a function that will build the signature as per the previously defined specs. So this is a long process that starts with ASN.1.

[ttanner]

https://github.com/ttanner/kryptomime supports S/MIME

[davidism]

It appears to use subprocess to call openssl (there are a few of these libs, I just didn't mention them all). That's less ideal than having an actual Python wrapper.

[ttanner]

I was referring to your comment, M2Crypto would be the only Python library providing S/MIME support.
Why do you think having a native wrapper would be ideal?

[davidism]

Speed, as well as a native interface rather than trying to build command line args. As well as not accomplishing many of cryptography's goals, such as "high level APIs", "introspectability and testability", and avoiding "extremely error prone APIs, and bad defaults".

[alex]

subprocess is also problematic when used from large processes, because of
how fork/exec interacts wtih the virtual memory system (tl;dr; COW is not
sufficient)

On Thu, Mar 12, 2015 at 10:49 AM, David Lord [email protected]
wrote:

Speed, as well as a native interface rather than trying to build command
line args.

—
Reply to this email directly or view it on GitHub
https://github.com/pyca/cryptography/issues/1621#issuecomment-78494518.

"I disapprove of what you say, but I will defend to the death your right to
say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero
GPG Key fingerprint: 125F 5C67 DFE9 4084

[timdawborn]

+1 support for prioritising S/MIME support for the reasons @davidism stated.

Consider prioritizing this feature. The only other Python library providing SMIME is m2crypto, which is not available for Python 3, provides a very "raw" experience, and does not seem to be actively maintained.

My particular use case is generating PayPal signed forms, à la what the django-paypal project does: https://github.com/spookylukey/django-paypal/blob/master/paypal/standard/forms.py#L155

[davidism]

What is the current status of SMIME in the library? I see that some PKCS7 support was added, is that enough to sign messages properly? Sorry if that question sounds silly, but I don't know enough about cryptography to evaluate this on my own. If enough is available to sign messages now, it would be nice to see a small explanation of how to go about it. If not, it would be nice to see a checklist of what still needs to be done.

[alex]

SMIME status is unchanged. We dont' really have PKCS7 support either, just
the OpenSSL bindings for it.

On Fri, Aug 21, 2015 at 11:55 AM, David Lord [email protected]
wrote:

What is the current status of SMIME in the library? I see that some PKCS7
support was added, is that enough to sign messages properly? I can create
the mail envelopes myself, I just need to be able to sign the messages
correctly.

—
Reply to this email directly or view it on GitHub
https://github.com/pyca/cryptography/issues/1621#issuecomment-133471446.

"I disapprove of what you say, but I will defend to the death your right to
say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero
GPG Key fingerprint: 125F 5C67 DFE9 4084

[davidism]

Thanks for the update. Is progress being made? What does "just the OpenSSL bindings" mean, is it possible to use them?

[alex]

If you want to directly call the OpenSSL API's from Python, you can. We
don't have any sane APIs for it though. I don't bleieve anyone is actively
workign on this.

On Fri, Aug 21, 2015 at 11:59 AM, David Lord [email protected]
wrote:

Thanks for the update. Is progress being made? What does "just the OpenSSL
bindings mean", is it possible to use them?

—
Reply to this email directly or view it on GitHub
https://github.com/pyca/cryptography/issues/1621#issuecomment-133473251.

"I disapprove of what you say, but I will defend to the death your right to
say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero
GPG Key fingerprint: 125F 5C67 DFE9 4084

[croessner]

I am also interested in S/MIME support. I used M2Crypto to sign and encrypt mails. See "smilla" and "automx" projects here on Github. In smilla, I encrypt mails (right now with subprocess that is ugly; old M2Crypto sample is in the comments) and in automx, I sign iOS MDM profiles. So I really miss S/MIME in Python 3 :(

[reaperhulk]

For those who need this -- we need an API proposal before we can implement it. Just the methods, what their arguments would be, and what they'd return would be sufficient, although bear in mind that we're going to ask a lot of questions about what is possible to be sure the API encompasses most use cases (e.g. RFC 5652 is the latest CMS RFC and CMS is the successor to PKCS7 v1.5, so can we implement it all together or do we need to do SMIME first and CMS later, etc).

[abhishek-ram]

I had to implement SMIME compression/decompression for one of my projects and I did it using pyASN1 so I have some idea regarding this functionality.
We would need to first develop the CMS module which would have functions to generate/parse the signed/compressed/encrypted ASN.1 messages as per the RFC. We would then use these functions in the SMIME module which would then envelope/de-envelope the messages in the email MIME format.

[sim0nx]

Not sure if it helps but there is this project:
https://github.com/balena/python-smime

which implements SMIME support fully in python using both:
asn1crypto
pycrypto

[waynew]

From an API perspective, I think I would love to see something like this:

import smtplib
from email.mime.text import MIMEText
from cryptography.email import SMIME

msg = MIMEText('This is some email that will get signed and encrypted')

msg['Subject'] = 'A signed an encrypted email for you!'
msg['From'] = '[email protected]'
msg['To'] = '[email protected]'

smime = SMIME(cert='/path/to/cert', key='path/to/key')
msg = smime.sign(smime.encrypt(msg))
s = smtplib.SMTP('localhost')
s.send_message(msg)
s.quit()

The result being a MIMEMultipart message (or subclass) with the proper headers. That would at least follow the usage that I had in the past, and seems like it would support davidism's use case

[ruffsl]

Not sure if it helps but there is this project:
https://github.com/balena/python-smime

which implements SMIME support fully in python using both:
asn1crypto
pycrypto

Looks like python-smime only supports encrypting, but not singing. I'd like to get the same openssl funcilatiy of openssl smime -sign -text ... in pythonic form. Additionally python-smime also seem to have some issues with python3 and is dependent on pycrypto; now deprecated but seemingly replaced by pycryptodome .

For those who need this -- we need an API proposal before we can implement it. Just the methods, what their arguments would be, and what they'd return would be sufficient

Is there a specific past example of such I could memic from? I'd be interested in drafting something, but have never done something the like for this project.

[reaperhulk]

@ruffsl thanks for offering to help! We probably don't have a specific example of how it's done, but you can look at, for example, our current X509 API to see how we generally approach APIs in this project.

The main thing would be to define a set of objects with interfaces and describe how they're going to work. Questions you'll want to answer:

• Where do the sign and verify methods live? Are they on the private/public key or should they actually be top level functions?
• Is there a separate SMIME package?
• What is the set of options that you can pass to SMIME? The API needs to be sufficiently expressive to encompass pretty much the entire spec, although we can potentially only implement a subset at first.
• Do we need to add other features first (e.g. PKCS7 envelope reading) before we can add SMIME?
• How does CMS complicate this? CMS is effectively a newer superset of SMIME so do we need to actually design the API around that?
• Where do we get test vectors for the set of permutations we intend to support. Do they already exist or do we need to generate them. If we need to generate them what other implementation do we use to verify that OpenSSL is producing interoperable output?

[tuck1s]

@ruffsl you may have seen that https://github.com/balena/python-smime has recently switched to use this library. I've found that it works for the _encryption_ case on plain text bodies, and I got it to work for encryption of multipart bodies too with a small change (made a PR).
Next I'm trying to get signing to work with cryptography library instead of M2Crypto.

[tuck1s]

I ended up using OpenSSL.Crypto instead for signing, based on this answer: https://stackoverflow.com/a/47098879/8545455

[ruffsl]

Hi @tuck1s , thanks for the pointers. I tried out that stack overflow example, but the buffered out data after signing doesn't look like the email S/MIME signature I'm expecting, even after setting the SMIME.PKCS7_DETACHED and SMIME.PKCS7_TEXT flags. I think that example is missing the add_signer step that m2crypto appears to be doing:

https://gitlab.com/m2crypto/m2crypto/blob/6ecc61456ffbfe89dfd48b689c9224ec66b4b816/SWIG/_m2crypto_wrap.c#L8726-8739

Additionally, it looks like pyOpenSSL has a note discouraging its use for anything other than TLS:

Note: The Python Cryptographic Authority strongly suggests the use of pyca/cryptography where possible. If you are using pyOpenSSL for anything other than making a TLS connection you should move to cryptography and drop your pyOpenSSL dependency.

@reaperhulk

Where do the sign and verify methods live? Are they on the private/public key or should they actually be top level functions?
Is there a separate SMIME package?

I'm not sure. I would think this would warrant its own package, like already exists for Fernet. I'm not sure SMIME related methods like sign, encrypt, would fit well in the x509 package.

What is the set of options that you can pass to SMIME? The API needs to be sufficiently expressive to encompass pretty much the entire spec, although we can potentially only implement a subset at first.

I'm not sure to what extent the spec reaches, but I'd be fine with at least supporting ietf's example how to:
https://tools.ietf.org/doc/python-m2crypto/howto.smime.html

Do we need to add other features first (e.g. PKCS7 envelope reading) before we can add SMIME?
How does CMS complicate this? CMS is effectively a newer superset of SMIME so do we need to actually design the API around that?

From what I understand, S/MIME uses CMS for the cryptographic parts, while S/MIME itself is the glue between CMS and emails. Perhaps SMIME should be a submodule to CMS (PKCS#7)?
https://security.stackexchange.com/questions/41399/openssl-pkcs7-vs-s-mime

Where do we get test vectors for the set of permutations we intend to support. Do they already exist or do we need to generate them. If we need to generate them what other implementation do we use to verify that OpenSSL is producing interoperable output?

I checked to see how openSSL constructed their tests for SMIME, but I didn't see them while searching. M2crypto , however has some test examples we could pull from.

https://github.com/eventbrite/m2crypto/blob/0f61819722c07ecffd7cf47c25095b516bf2d85c/tests/test_smime.py

It looks like RFC4134 provides some Examples of S/MIME Messages.
Looks like some folks using golang have a gist consolidating the test cases reddit post: signing_mails_smime_pkcs7.

https://gist.github.com/egonelbre/123c5cbbcca2fefbf0e3

[reaperhulk]

@ruffsl thanks for the comprehensive comment! I think we can work something out here as a first pass for the next release so I'm going to go ahead and put this in the 2.5 milestone. I think, much like some of our other older bugs, that we've let perfect become the enemy of good with regard to supporting some of these features. Our goal should be to come up with a signing API that covers the use cases people have without becoming bogged down forever with concerns about future API expandability.

[ruffsl]

I think, much like some of our other older bugs, that we've let perfect become the enemy of good with regard to supporting some of these features. Our goal should be to come up with a signing API that covers the use cases people have without becoming bogged down forever with concerns about future API expandability.

To be honest, after losing my way down the SMIME rabbit hole, I didn't feel certain enough to address the questions you posed above. As an end user, I appreciate how cryptography can provide pythonic API for even the hairiest libraries like OpenSSL, but am not sure what would best fit for this case. However, it seems the absence of this feature has lead some of us to reimplement basic primitives or patch older workarounds, encouraging greater cryptographic implementation errors out in the wild. A modern, safe and sane API for working with CMS would be great to have, even if its not perfect just out of the gate.

Other than borrowing verbs and arguments from openssl's cms/smime commands, I'm not sure what else to sugest; but I'd be happey to test out candidate APIs with my own use case dependacies.

[ruffsl]

I looked arround for that ZSmime project mentioned by Pheng Siong Ng; not much left, but some more examples use of python and SMIME I guess:
https://gitlab.com/m2crypto/m2crypto/tree/4d85fbca2573735f0e93cef5a6005421e48a47e2/demo/smime
https://gitlab.com/m2crypto/m2crypto/tree/4d85fbca2573735f0e93cef5a6005421e48a47e2/demo/Zope/lib/python/Products/ZSmime

[alex]

We just landed basic SMIME signing.

[frennkie]

It's great that Sign has been implemented. But as far as I see there is no documented interface for Verify.

And what about Decrypt and Encrypt? Any chance that could also be included too (after all, the title of this issue is SMIME Support.. Not only signing)?

[reaperhulk]

@frennkie I'm happy to look at additional use cases! The historical obstacles we've had are understanding exactly what people actually need. Ideally with what they're currently doing in OpenSSL so I can make sure we build an API that allows it. SMIME is a crazy mess of a specification so this is a real challenge. If you can articulate your needs please file a new issue and we can discuss it there (see #5433 for how I'd like future SMIME discussions to go).

[hughsie]

understanding exactly what people actually need

If it helps, lvfs-website does a huge number of crazy things (involving gnutls, flatpak and command line scraping) to generate a PKCS#7 detached signature for a binary blob for a specific certificate. It does most of the same crazy things to verify that a specific PKCS#7 detached signature verifies a binary blob, for a list of given PKCS#7 certificates.

If cryptography could do those things I could probably drop a thousand lines of async python and tests from lvfs-website :)

If that's a useful use case @reaperhulk then I'd happily create an issue, but I don't know if the LVFS is a big enough use case, or if you care about this specific PKCS#7 detached key generation.

[reaperhulk]

@hughsie Please do! It may or may not be too specific a case, but we won't know unless we walk through your needs. I'd be curious if the generation is covered by the new API we just landed.

[hughsie]

@reaperhulk done, thanks: https://github.com/pyca/cryptography/issues/5471