x/crypto is meant to follow the same philosophy as the rest of the Go cryptography standard libraries: safe, useful subsets of widely used and robust primitives and protocols. Some packages in x/crypto don't clear that bar anymore, and I propose we freeze and deprecate (but not remove) them.
The intended outcomes are
All open and future issues and CLs that are not security relevant will be closed linking to this issue.
/cc @agl with whom we discussed this at RWC, and @golang/proposal-review for approval
There are different reasons for deprecating different packages, and I give a short explanation for each of them below, but here's a summary.
I was also hoping to kill pkcs12/ because it's a legacy file format that relies on primitives even worse than the ones listed above, and the package only does decoding while a better fork on GitHub also does encoding, but it has way too many users so that's only been frozen instead.
I would prefer to see ripemd160
maintained as it is used by more cryptocurrencies than just Bitcoin. Honestly, Bitcoin is a pretty big deal on its own and would be enough reason to maintain it in my opinion.
Additionally, I performed a quick survey via godoc.org importers and it shows that the ripemd160
package is imported by at least 511 packages at the current time, including many widely known pieces of software such as Btcsuite, Discord, Decred, Ethereum, and Ripple.
@davecgh I am not aware of any reasons to pick RIPEMD-160 for a new design, so the fact that cryptocurrencies inherited it from the Bitcoin paper is not a good enough reason not to deprecate it.
I don't expect it will require any maintenance anyway—it's a hash function, the package is almost certainly fine as is—but deprecating it will send the message to users not to look there if they just need a hash function, and communicates that we would for example not take an optimized assembly implementation for it.
Just picking two examples to illustrate my opinion:
md4 is used in the NTLM protocol.
pkcs12 is used to store certificates in a lot of real world applications.
Both are dated, one is a PITA. Yet I think x/crypto should not be reduced to state of the art ciphers/protocols since we are not always developing self contained systems where we can safely choose the best available tech, we also have to connect to legacy systems or simply not-that-well-developed third-party systems.
Also I think that especially the pretty self contained and matured algorithms like MD4 don't suffer much from "being unmaintained". I mean: what would you maintain? The algorithm is stable (although "broken"). We cannot do anything about attacks on the algorithm (apart from refraining from future use).
I understand your point with OpenPGP. Though it might also be an option to try and get one of the forks to be reintegrated and actually maintain that.
If a living protocol (that still receives updates; has a moving spec) rots ... yeah, throw it out. The sophisticated implementations however, that are simply not "maintained" because there is nothing to maintain there ... I would like to keep them.
- otr/ — a messaging security protocol from 2000's, mostly obsoleted by the Signal double ratchet and other messaging protocols like OMEMO and MLP, and a very rarely used package
The reasons given for obsoleting the otr
package are perfectly sensible, but if it's not too much of a burden I would argue for nevertheless keeping this package: this is because in the XMPP world, the OTR protocol is still used as the standard secure messaging protocol and Signal has virtually no adoption there.
Aside from this argument I admittedly have nothing else with which to claim that keeping otr
is relevant.
blowfish can not be compared with AES since the former has a fixed block size of 64 bits; the later, just like Serpent and Twofish, have block sizes of 128 bits.
The fact of that a algorithm been used into a public or known project doesn't mean that it doesn't been used at private projects. At my case, I've used mainly Ripemd160 instead of both MD5 and SHA1 due to greater security, and I use it instead of SHA2 according to the requirements about security and perfomance that my project needs.
Twofish and also Serpent have greater security than Rijndael, but the later was the AES winner due to greater performance. A lot of people not only chooses an algorithm due to performance else by greater security.
The block mode XTS is a block mode very safe, only recommended for full-disk encryption. So, do you want to remove it because you didn't know about it, and the coders of open source are not using it, don't you?
They are coders that build commercial and/or private projects.
At my case, I'm using it like alternative to cryptsetup.
I'd rather keep openpgp (and would consider making it compatible with GnuPg2 ).
Still the problems there are quite surprising
Would be cool to have someone to take a look.
I'd similarly be very disappointed to see openpgp removed. Frankly, the implementation even as it exists is much higher-quality than the upstream GnuPG one, incomplete as it may be; removing it would thus be a loss to the community as a whole.
Admittedly, I'm one of those folks with a private fork, but I have approval from my employer's legal team to submit our changes upstream; if getting off my arse and doing so is the cost of keeping it in-tree (and thus being sure that the existing functionality continues to work with future versions of Go), I'm happy to do so.
@tredoe @charles-dyfis-net, this issue is about freezing+deprecating them, not removing them.
@bradfitz To be fair, deprecating them has almost the same effect as removing them; people will look elsewhere for alternatives that aren't marked as deprecated. Which I imagine is the intended effect. Unfortunately, the alternatives may not always be of higher quality. For example, I'd much rather see people use x/crypto/md4 over some other, random md4 implementation.
I don't think the "these algorithms are broken" holds as an argument. A lot of legacy still relies on them, unfortunately. As such, we should rather look at the cost of not deprecating a package. One can of course argue that another well-organized party that isn't Go should maintain legacy packages.
Have we explored why these packages made it into x/crypto in the first place?
To be fair, deprecating them has almost the same effect as removing them
I disagree. The effects are quite different: removing them breaks users, and deprecating them does not. Deprecating them is just adding a comment saying, "Yo, new applications shouldn't use this, and also we're not adding new features so don't send PRs."
people will look elsewhere for alternatives that aren't marked as deprecated. Which I imagine is the intended effect.
Sounds like the intended effect is reducing maintenance cost by not accepting changes to add features or optimize the performance.
I think the "deprecated" marker would move the same thinking you have here down the line. Your argument for deprecation (or removal) is that the code is old and the algorithms should not be used. While the code is actually most likely still fine and just doesn't need any maintenance (if it does, someone will report it and or create a PR).
If it's marked "deprecated" it will lead to managers/decision makers/etc. thinking "oh, that code is not maintained, we cannot use it. go find something else".
I had discussions like that myself. A library for a microcontroller we were working with was over 7 years old and did not have a single change. It took a considerate amount of time to convince my superior that this did not matter since a) the code was well documented b) had good test coverage c) the microcontroller in question was that old as well - there was simply nothing that could have changed in the meantime.
The same holds for the algorithms in question. MD4, cast5, blowfish, twofish, etc. will not change. They may not be a good choice for cryptographic strength ... but they may still be the right choice to implement a specific interface. And then it's better to have a central source that does not imply "we will get rid of this soon" instead of a bunch of third party sources.
If it's marked "deprecated" it will lead to managers/decision makers/etc. thinking "oh, that code is not maintained, we cannot use it. go find something else".
Case in point: https://github.com/dominikh/go-tools/issues/413
Everywhere in the standard library, deprecated means "stop using this, move to an alternative".
Remember that deprecation comes with a message, so we can explain the reasons and implications for each package individually, and I'll make sure to provide as much context as possible.
There will be packages that say "this algorithm is broken [or superfluous] because X and should not be used except for compatibility with legacy systems". This does not call for an alternative implementation, but it warns people off using it, and recommends moving to non-broken things.
Other packages will say "this implementation is incomplete and unmantained, consider an alternative protocol like X, or a community maintained package". This hopefully leads to a centralized target for maintenance efforts which does not involve our resources.
I am replying to a few points individually below, before commenting please check if your point was already made above, to prevent this from becoming a giant thread nobody reads or feels like participating in.
I think that especially the pretty self contained and matured algorithms like MD4 don't suffer much from "being unmaintained".
Indeed, deprecating them comes at near-zero cost because there are not going to be new features and bug fixes. But the message to developers about not using them is important.
in the XMPP world, the OTR protocol is still used as the standard secure messaging protocol and Signal has virtually no adoption there
Ack, but I'm afraid that's not enough to justify the Go project maintaining an implementation. I would probably not even accept a Signal protocol implementation in x/crypto.
blowfish can not be compared with AES since the former has a fixed block size of 64 bits;
Good point! That makes it fundamentally vulnerable to Sweet32, updated the proposal.
The fact of that a algorithm been used into a public or known project doesn't mean that it doesn't been used at private projects.
I use usage in public projects as an indicator of popularity and therefore usefulness to the ecosystem, not as proof that nobody uses the package at all, and that's one part of the tradeoff decision on whether it's worth the resources to maintain.
Twofish and also Serpent have greater security than Rijndael
AES-256 is secure enough for Top Secret data in NSA's Suite B, I don't feel the need to carry anything else to fill a security gap.
The block mode XTS is a block mode very safe, only recommended for full-disk encryption.
XTS is quite bad in fact. See also a better tweakable cipher.
Admittedly, I'm one of those folks with a private fork [of openpgp], but I have approval from my employer's legal team to submit our changes upstream
Almost half of all issues open against x/crypto packages are for x/crypto/openpgp, and a good chunk of all CLs are for it as well. I'm not blaming the community for not contributing back here, I blame the protocol for requiring too much maintenance to properly implement. It just doesn't make sense for us to put in the effort to do the review and design work to bring this package to a state in which we are happy about it. Hopefully the community will centralize on a fork.
I'd much rather see people use x/crypto/md4 over some other, random md4 implementation.
MD4 is a bad example, because it's literally as broken as a hash function can be. Go is a memory-safe language, so a random implementation off the street can't introduce an RCE without really trying, and the next worse thing a hash function from the last 20 years can have is a collision, and well, I might still have some Python to generate one on a laptop.
I don't think the "these algorithms are broken" holds as an argument. A lot of legacy still relies on them, unfortunately. As such, we should rather look at the cost of not deprecating a package.
The cost includes developers being misled into thinking that a primitive or protocol can serve their needs because it's in x/crypto, diluting the safety properties of using the standard library.
Have we explored why these packages made it into x/crypto in the first place?
Some to support OpenPGP, some have always been there from before there was a conscious effort to limit what was in x/crypto, some were reasonable at the time and aren't anymore, some were a mistake.
Everywhere in the standard library, deprecated means "stop using this, move to an alternative".
If you are using any of the above, I would like you to stop using it and move to an alternative. Sometimes an alternative implementation, sometimes an alternative algorithm. If you know you can't because of legacy compatibility reasons, at least I made you aware of it, and made sure the question got asked at review time, and the answer hopefully documented.
Having seen cryptography mismanaged in the standard libraries of many other languages I use, this seems like a great move to me, and a breath of fresh air by comparison. Cryptography in the standard library should be held to a high bar, and I'm glad to see that bar enforced.
In languages with a "batteries included" standard library like Go's, the ecosystem is largely shaped by what's available in the standard library. Go has pretty much the best cryptography ever seen in a language's standard library with a few exceptions, and this proposal covers most of the exceptions.
For implementations of legacy protocols, I think users are almost certainly better off with something out-of-tree. This not only discourages accidental use in new projects, but also provides greater agility in the event of a security incident, as the response does not involve a new release of the language/standard library. Deprecating these algorithms from the standard library isn't a value judgment on their overall utility, just a signal that people should move to out-of-tree implementations. The remaining concern there is that they will lose the reluctant maintainers these algorithm implementations previously had. If the algorithms are truly useful and important, I think you can find new maintainers, and ones who aren't reluctant. Otherwise, it's entirely possible they're not as useful as they're claimed to be.
The remaining arguments seem to fall into a category I call "unusual requirements". For things like symmetric ciphers and hash functions, there are huge reasons to prefer algorithms like AES and SHA-2 over things like Serpent, Twofish, or RIPEMD-160, namely AES has hardware acceleration through a number of different platform-specific instructions like AES-NI which provide better performance and sidechannel resistance. In the case of RIPEMD-160 it has only 80-bits preimage resistance which is considered rather weak by today's standards. The specific security and performance requirements (much less threat model) weren't stated in any of these cases, but in my opinion as a cryptography professional specializing in symmetric encryption I think the claims made by people alleging they selected these algorithms due to "unusual requirements" are, to put it bluntly, factually inaccurate.
tl;dr: eliminating junk cryptography from the Go stdlib will help improve the overall security of the ecosystem, so I say go for it.
... as the response does not involve a new release of the language/standard library.
@tarcieri, this bug is about x/crypto, which is not the standard library.
@bradfitz well don’t I look foolish. Mea culpa
Hey!
Sorry for jumping into the discussion.
- otr/ — a messaging security protocol from 2000's, mostly obsoleted by the Signal double ratchet and other messaging protocols like OMEMO and MLP, and a very rarely used package
Well, with the upcoming OTRv4 protocol, I don't how true this will still hold. There is a lot of people now interested in using OTRv4 again and implementing it on different clients; so I don't if it will be a shame to loose it here. That said, for the most people who are actually using OTR in golang, they actually use this library instead: https://github.com/coyim/otr3
I also don't know the state of OTR in golang (if it is fully implemented). I can check if needed.
I don't know what MLP means, is that MLS?
Btw, here some useful links around OTRv4, in case someone wants to check its interest/proposal:
Hope this information helps :)
That said, for the most people who are actually using OTR in golang, they actually use this library instead: https://github.com/coyim/otr3
I'm happy to hear there is work on the protocol, and that there is a community package for the previous version. That suggests deprecating the x/crypto one would be good for the ecosystem, letting it focus on modern and maintained implementations.
I don't know what MLP means, is that MLS?
Yes, thanks, fixed.
blowfish can not be compared with AES since the former has a fixed block size of 64 bits;
Good point! That makes it fundamentally vulnerable to Sweet32, updated the proposal.
Simply because the attack is possible doesn't mean it is particularly easy to carry out. For Sweet32, the attacker needs to be able to monitor traffic passing between the user and a vulnerable website, as well as control JavaScript on a web page loaded by the user's browser. It would take about 38 hours to collect hundreds of gigabytes of data necessary to decrypt the authentication cookie.
https://blog.securityevaluators.com/what-is-sweet32-4a62dca90296
The fact of that a algorithm been used into a public or known project doesn't mean that it doesn't been used at private projects.
I use usage in public projects as an indicator of popularity and therefore usefulness to the ecosystem, not as proof that nobody uses the package at all, and that's one part of the tradeoff decision on whether it's worth the resources to maintain.
Fair point
Twofish and also Serpent have greater security than Rijndael
AES-256 is secure enough for Top Secret data in NSA's Suite B, I don't feel the need to carry anything else to fill a security gap.
Although NSA has categorized this in Suite B, they have also recommended using higher than 128-bit keys for encryption. So, should be also removed or deprecated AES-128?
Besides of that, and althought I wouldn't assume that NSA has cracked AES ciphers, I would assume that most crypto systems that use AES have implementation flaws that the NSA could exploit when they want throught its own data center, because the NSA are actively subverting security on the standard level or in collusion with software developers.
The block mode XTS is a block mode very safe, only recommended for full-disk encryption.
XTS is quite bad in fact. See also a better tweakable cipher.
At the first, in practice, many encrypted filesystems already use encryption that's strictly worse than XTS, and they generally use the same block-based model as full disk encryption for exactly the same performance reasons.
Besides, the title of such article is confusing because many people will only ever encounter XTS when setting up full disk encryption with dm-crypt, where they'll be presented with the choice between XTS and CBC-ESSIV.
And finally, it's also not saying that you shouldn't use XTS for full disk encryption. In fact it seems to say it's probably OK for full disk encryption: "It’s certainly better than ECB, CBC, and CTR for FDE. For the crappy job we ask it to do, XTS is probably up to the task."
I would agree that getting rid of otr(which I believe is v2?) from x/crypto makes sense, there's an otrv3 implementation(https://github.com/coyim/otr3) in Golang and a partial implementation of otrv4 in Golang(https://github.com/otrv4/otr4) underway.
As we've been discussing internally in some calls with Ian Goldberg, we'd like to get rid of v2 in favor of v3 and especially v4, so, keeping the otrv2 implementation in x/crypto doesn't make much sense.
Yes, I agree with @FiloSottile and @DrWhax . We will surely develop OTRv4 in golang, and it will maintained; so deprecating the old version in golang libs (which might be in between v2 and v3) makes sense. In the future, if wanted, we can implement OTRv4 in golang libs itself. Thanks!
I can confirm that well-meaning but uninformed developers choose equally randomly among the options in crypto/
and x/crypto
, and therefore I strongly support the proposal. In fact, I gave a GopherCon talk once about exactly this problem.
Deprecating an algorithm in x/crypto isn't the same as banishing it from the Earth, it just means Go's very few crypto reviewers will have more time to focus on things that are/should be widely used.
This proposal should be put on hold for now.
We shouldn't remove anything from x/ until (1) modules are enabled by default and (2) we have started tagging versions. Once we have started tagging versions, we can discuss removing things; modules will mean users don't automatically get the latest copy, and the tagged version will make it easy to record that what exactly they depend on.
@rsc perhaps I'm missing something, but it appears that removal isn't part of the proposed plan here:
and I propose we freeze and deprecate (but not remove) them
@bradfitz and @andybons point out that I misunderstood and that this proposal is only about marking things _deprecated_, not removing them. Sorry.
Taking md4 as an example, obviously that should be marked as completely broken and something no one should use except under extreme duress. It's funny that crypto/md5 says
r$ go doc md5
package md5 // import "crypto/md5"
Package md5 implements the MD5 hash algorithm as defined in RFC 1321.
MD5 is cryptographically broken and should not be used for secure
applications.
but md4 does not. Obviously it should. Adding that kind of documentation note doesn't require a proposal.
Similarly, x/net/websocket says:
r$ go doc golang.org/x/net/websocket
package websocket // import "golang.org/x/net/websocket"
Package websocket implements a client and server for the WebSocket protocol
as specified in RFC 6455.
This package currently lacks some features found in an alternative and more
actively maintained WebSocket package:
https://godoc.org/github.com/gorilla/websocket
That's another very clear message that seems fine.
But it seems like a few are not quite in that camp. OpenPGP for example, isn't cryptographically broken, right? I'm less sure about marking that package as deprecated. Is there an established fork that we should point to instead? Or should we invite them to own x/crypto/openpgp?
@rsc I want to send a stronger message than just adding a line to some package-level docs that nobody will pay attention to (at the very least for existing dependencies). I want developers to make an active, hopefully justified, choice to ignore a tooling-provided flag.
If you need some WebSocket feature you'll know, and if it's not in golang.org/x/net/websocket, you might look at the docs and learn where to go look for it. If you are using a bad hash, it will look and feel like a perfectly good hash.
But it seems like a few are not quite in that camp. OpenPGP for example, isn't cryptographically broken, right? I'm less sure about marking that package as deprecated. Is there an established fork that we should point to instead? Or should we invite them to own x/crypto/openpgp?
OpenPGP is legacy, brittle, and very high maintenance to implement. I discourage any new system from using it, and I don't think it's worth any of our limited resources to maintain. If it were only for lack of resources, I would be seeking an owner for x/crypto/openpgp, but given that it's not something people should be encouraged to use, I don't think that would be a service to our users.
I expand on this in the "Why" section above, under the "openpgp/" bullet, and at https://github.com/golang/go/issues/30141#issuecomment-462067035.
I want to send a stronger message than just adding a line to some package-level docs that nobody will pay attention to (at the very least for existing dependencies). I want developers to make an active, hopefully justified, choice to ignore a tooling-provided flag.
Wait, I thought this whole bug was just about adding a "Deprecated: ..." comment, but you refer to an "active choice".
Are you speaking of existing or hypothetical future tooling which would prevent use of deprecated things?
@bradfitz The "tooling" is staticcheck
. There's an open issue for packages (which @dominikh linked in https://github.com/golang/go/issues/30141#issuecomment-462066837 above) but it already warns on the use of deprecated symbols.
Wait, I thought this whole bug was just about adding a "Deprecated: ..." comment, but you refer to an "active choice".
I consider ignoring or silencing a new linting warning an active choice, as opposed to simply not re-reading the package docs for something you already use.
Not everyone uses tooling like staticcheck tough.
I just wanted to briefly pointed out that while it is true that there are vulnerabilities surrounding PGP (specifically, the CBC/CFB Gadget Attack -although, the current draft of OpenPGP RFC4880 (bis05) reflect the recommendation of the authors to deprecate the SE packet type and that modified ciphertexts should not be displayed-, as Direct Exfiltration is more around abusing vulnerabilities in Apple Mail, iOS Mail and Mozilla Thunderbird), some libraries might depend on this pgp to work. Take the case of gopass, which is widely used. I agree on that if this is an unmaintained package, it should be good to deprecate it; but maybe giving a solution to projects that use it will be in place.
- the protocol predates modern cryptography engineering, which is still causing severe vulnerabilities, and any new project should rely on alternative modern solutions
I'm not aware of any alternative modern solutions to pgp that are widely deployed.
I think that deprecating is a right approach; but maybe with giving alternatives as well :)
My two cents :)
I don't think there is a direct modern alternative to PGP, because the modern alternative is to use more specific solutions to your problem. For example, here's two common uses of PGP and some modern, maintained alternatives:
This is the wrong place to be litigating the future of PGP, so I'm going to stop doing it. If you're the maintainer of a widely used Go package that depends on x/crypto/openpgp, feel free to message me off-thread! I will be happy to look at your use case to see if we can think of something that will get you away from it.
It's sad but there isn't much alternatives to PGP out there. I personally use this tiny script I made to encrypt content to people and give them the key in another channel.
Polite nudge not to litigate the state of PGP in this issue. Only comment if you think your remark has not been made above, and it might change the decision to deprecate even if there are multiple reasons to deprecate openpgp/.
I intend to add OpenPGP support to my email client, written in Go, and had considered using this library for that purpose. The remarks in this thread about the dangers and obsolecene of PGP are obscenely exagerrated.
Efail has nothing to do with the openpgp and a lot to do with programs that used the GPG api incorrectly.
The OpenPGP specification predates authenticated encryption, and retroactively added bespoke authentication schemes. There have been attempts to modernize OpenPGP by adding support for authenticated encryption modes, however these proposals never made it to the RFC stage.
The lack of authenticated encryption is the root cause of Efail's CBC/CFB gadget attack as CBC and CFB are both unauthenticated encryption modes. The recommended protocol changes to avoid similar attacks in the future were as follows (emphasis mine):
How did EFAIL influence the developments in the current standards?
There is an ongoing work on two new email security standards. Both considered countermeasures presented in our paper.
The S/MIME standard draft references our EFAIL paper and recommends the usage of authenticated encryption with AES-GCM. Furthermore, it warns that different parts in multipart/mixed emails should be treated as being of different origins.
The OpenPGP standard draft deprecates Symmetrically Encrypted (SE) data packets which are not protected by MDCs. It proposes AEAD protected data packets and mentions that the implementation should not allow users to access erroneous data.
regarding @Mikotochan's comment:
...used the GPG api incorrectly.
Blaming the users of cryptographic APIs for their failure to use them correctly has shown itself to be a particularly bad strategy for developing secure software. The alternative many developers of cryptographic libraries have pursued as a result is designing hard-to-misuse (a.k.a. "misuse resistant") APIs.
That seems to be the direction @FiloSottile would like to go with x/crypto
.
x/crypto doesn't implement the GPG API. It implements PGP.
@ddevault as I just mentioned, it implements OpenPGP. Do you have a specific concern about what I just said regarding the deficiencies of the OpenPGP protocol?
Yes. You quoted:
...used the GPG api incorrectly.
The GPG API is a porcelain CLI tool. OpenPGP is a standard. By my measure x/crypto is already doing a damn good job of avoiding the problems which led to EFAIL: shitty usage of the GPG command line tool by applications.
The GPG API is a porcelain CLI tool. OpenPGP is a standard.
That is why I specifically asked you:
Do you have a specific concern about what I just said regarding the deficiencies of the OpenPGP protocol?
It would appear you are the one colluding the CLI tool with the protocol. So I'll ask you again: is there something about my analysis of the OpenPGP protocol you take issue with?
You're conflating EFAIL, GPG, and OpenPGP into a single incoherent argument. I'm trying to disentangle it.
If your problem is collusion of terms, could you please respond to my questions about the OpenPGP protocol without colluding terms yourself?
As far as I can tell, you haven't actually said anything at all about OpenPGP.
I will quote https://efail.de/ again:
How did EFAIL influence the developments in the current standards?
[...]
The OpenPGP standard draft deprecates Symmetrically Encrypted (SE) data packets which are not protected by MDCs. It proposes AEAD protected data packets and mentions that the implementation should not allow users to access erroneous data.
The larger point is that Efail was recognized by the relevant authors of the OpenPGP and S/MIME standards as a protocol-level failure, which there is ongoing work to address.
"EFAIL" is an alarmist puff piece written by morons to slander PGP and inflate their egos. The standards don't need to change to fix the problems it mentions. The proposals help... marginally. The problem is not and was never with OpenPGP, it's with poorly written email clients (e.g. all email clients).
I know this is a discussion and note a vote, but if I could vote on this, I'd vote to not deprecate OpenPGP. It has RFCs (2440, 4880) and is widely deployed in corporations and security firms. I use it a lot (in and out of Go) to encrypt files to multiple keys and it works well. I do agree with you that it is complicated and I can imagine it is hard to maintain (I've opened issues that your guys have fixed)... Thanks for doing that!
I don't feel as strongly about the other algos/methods, but I like blowfish a lot, too, it's pretty decent. And if you care at all about Windows Active Directory, then you need MD4 which is the heart of NTLM hashes and still widely used in corporate environments. I know, it's a weak hash, but that's the reality we have to deal with as developers in the trenches where Go is a god send.
Thanks for the consideration.
Please keep the conversation polite and respectful. Thanks.
@tarcieri
You could argue that the reason that programs used the GPG api is that the GPG api is crap, but this is absolutely irrelevant to the issue at hand which concerns OpenPGP as a standard. I also never blamed anyone, so please do not put words in my mouth.
The OpenPGP standard right now supports something called MDS (Modification Detection Code) which prevents the attack described in Efail, even though it uses CFB.
One major concern I have about deprecating OpenPGP is its use in signing git commits. It doesn't seem unreasonable to agree that is important to many here, but git only supports PGP right now. Deprecating OpenPGP will remove a very useful programmatic alternative to the gpg command line tool. If this goes away, what is another _good_ alternative to manage and generate keys other than the gpg tool?
@Mikotochan attacks like Efail are strategically eliminated through the use of authenticated encryption, as authenticated encryption primitives don't disclose ciphertexts when decrypting unless MAC verification is successful.
It's true a supplementary MAC can be and has been added to several protocols including OpenPGP, however this approach of trying to shoehorn authentication into protocols which were originally designed without it as opposed to moving to an authenticated encryption mode is brittle and has failed repeatedly in practice. Efail is an example of such a failure, and one of the takeaways from OpenPGP and S/MIME standards authors was the need to update the relevant standards to support authenticated encryption modes.
It is very much a protocol-level problem with OpenPGP. Perhaps it will be addressed.
@FiloSottile
I think it's wrong to deprecate/freeze packages that implement protocols or algorithms just because those algorithms "should not be used in new developments". Go is not only used to implement new software and new protocols; it is also used to implement software that must interoperate with the existing world, imperfect as it might be.
For instance, I've used Go to download third-party files (not authored by me) through HTTP together with their OpenPGP signature and verify their signature after download. I think this is a perfectly good use-case for the OpenPGP package; I want to provide this feature that gives an additional layer of security to users of my software, irrespective of whether there would superior crypto alternatives to solve this problem; it is not my choice to use OpenPGP to sign those third party files, I don't and shouldn't start a crypto war with authors of these third-party files so that they switch to a new signature algorithm just because I don't have a maintained OpenPGP Go package anymore.
If you feel strongly that OpenPGP is not a good choice for new software that starts from scratch and doesn't need to interoperate with the world, than I'm sure a documentation note is good (or in fact, a cycle of good articles on your blog would do even better). But if I need to interoperate with the existing world, I just need an OpenPGP packaged that is actively maintained and compatible, irrespective on the merits of the algorithm.
If you want to retarget x/crypto
to a short group of "good algorithms that are good choices" instead of "comprehensive library to interoperate with existing usages of crypto protocols", it would be a serious pity for me, but it's your library and your choice as a maintainer. I really hope somebody will fork it because it provides serious benefits to Go users the way it is now; I personally don't need x/crypto directory list to school me about good choices of crypto algorithms; there's Google, conferences, and papers for that.
The argument that I buy is that you don't have enough resources. In that case, you should first evaluate asking the community for maintainers, before freezing the libraries for good.
@tarcieri
attacks like Efail are strategically eliminated through the use of authenticated encryption
The use of MDS in OpenPGP constitutes an instance of AE.
authenticated encryption primitives don't disclose ciphertexts when decrypting unless MAC verification is successful
That only has to do with the implementation of said primitives.
as opposed to moving to an authenticated encryption mode is brittle and has failed repeatedly in practice
Can you give examples of such failure? (and no, Efail is not one of these)
Efail is an example of such a failure
Again, Efail is caused only by the incorrect usage of the (potentially ill-designed) GPG api.
The use of MDS in OpenPGP constitutes an instance of AE.
When I say "authenticated encryption", I'm referring to IND-CCA3 constructions, e.g. AES-GCM, as presented through an AEAD interface. These constructions, unlike the CBC and CFB modes used by OpenPGP, will never disclose ciphertext until it has been authenticated. There is no way to implement a protocol using these modes that will be vulnerable to something like the Efail attack.
Again note that there ongoing work to add these modes to the OpenPGP spec (motivated by the Efail attack), which in and of itself should be a clear indication that the existing bespoke authentication scheme is considered insufficient by OpenPGP protocol spec authors.
Can you give examples of such failure?
This is getting somewhat off topic, but BEAST, Lucky13, and POODLE are all examples such attacks on TLS. Since that time TLS has specified ciphersuites which incorporate AES-GCM and ChaCha20Poly1305 (both AEAD modes) as mandatory to implement, and Internet traffic has largely switched to using these ciphersuites.
Note that SSH was also vulnerable to similar attacks. Here's a choice quote:
In this paper we show that, contrary to this consensus, the SSH BPP specification in [26, Section 6] has serious design flaws that lead directly to plaintext-recovering attacks against SSH.
The main takeaway from this is AEAD modes are well-studied and have security properties which are provable under the random oracle model. Authentication schemes like SSH's BPP and OpenPGP's MDS are amateur constructions which do not have security proofs. In the case of SSH's BPP, it failed in practice.
Final nudge not to debate the merits of OpenPGP as a protocol independently of the x/crypto/openpgp implementation in this thread anymore. Further comments about it will be hidden as off-topic. This thread is close to being too long to be useful, I'm afraid.
I am still convinced OpenPGP is not a modern protocol and ecosystem. But again, that's not alone the reason for deprecation. x/crypto/openpgp
is an incomplete implementation (and not in the useful way crypto/tls
is), can't be complete without being inconsistent with the rest of the Go standard and additional cryptographic libraries, and is a significant resource drain. The fact that the protocol is legacy is simply the reason not to ignore those facts.
@rasky Your argument is a very valid argument against removal, which is indeed not in scope. We disagree on the endorsement value of something being in x/crypto though: while you might vet your choice of algorithms further than "it's in x/crypto, it must be fine", I know for a fact that's not common. I think that justifies deprecation. As for freezing, note that all packages are either very fine as-is (hashes, ciphers), or a resource drain incommensurate to their usefulness (openpgp). As for finding a maintainer, I feel a level of personal responsibility over the quality of x/crypto packages, so while I do hope a community maintained package endures, I would like it to be elsewhere.
@FiloSottile OK I then hope somebody will fork and continue x/crypto
the way it is today as it does provide useful value. In fact, I'll think of doing it myself.
I'll just note that my argument is against deprecation (not removal), because I can't really use x/crypto/openpgp
anymore in production software, if I know for a fact that is unmaintained and deprecated, for reasons that I think are obvious. Both deprecating and removing has the net effect of forcing me to remove it from downstream maintained projects (removing is just harsher, though I always vendor third-party libraries...).
@FiloSottile
can't be complete without being inconsistent with the rest of the Go standard and additional cryptographic libraries
Can you elaborate on this? I might be overlooking something (long thread), but I can't find any argument why this would be the case. Also, FWIW, I don't think "completeness" is a useful criterion, because it can almost never be well-defined. Like, net/http
is not "complete".
FWIW, I acknowledge, that x/crypto/openpgp is costly to maintain. But TBH, to me that's pretty much the reason I don't want it to get deprecated. I thought about it for a couple of days and there really is no group or person I can think of that I'd trust to maintain it in the future. It also feels a bit dissonant to me, that its derided as badly maintained and it's pointed out that its the package in x/crypto with most open CLs and there is a subtext of "community-maintained vs. x/". It seems to me this implies that there are actually people willing to contribute to maintaining it. And that, if the cost thusly is not having enough reviewers, that adding more of them (like whomever we'd expect to maintain a fork) would be an alternative solution. And that, as far as I can see, the only thing speaking against that is some implicit notion of "x/ is not by the community".
Or, to put it another way: If the idea is to deprecate x/crypto/openpgp
in favor of a "community-maintained" package - what is preventing that community-maintained package from living at x/crypto/openpgp
?
OK I then hope somebody will fork and continue
x/crypto
the way it is today as it does provide useful value. In fact, I'll think of doing it myself.
@rasky I'd suggest forking packages that you need and that require maintenance (AFAICT, only openpgp) individually, rather than forking x/crypto
. Better yet, help centralize on one of the existing x/crypto/openpgp
forks.
@Merovius If you search the issue tracker and Gerrit you'll find x/crypto/openpgp
is missing a lot of features, and even more won't even show up because many people shared privately that they gave up long ago and just added things to their own fork. I believe pouring in the resources to add them all would make it a jack-of-all-trades, more aligned to the OpenSSL philosophy than to the Go one.
Or, to put it another way: If the idea is to deprecate
x/crypto/openpgp
in favor of a "community-maintained" package - what is preventing that community-maintained package from living atx/crypto/openpgp
?
I touched on it in https://github.com/golang/go/issues/30141#issuecomment-464639578:
As for finding a maintainer, I feel a level of personal responsibility over the quality of x/crypto packages, so while I _do_ hope a community maintained package endures, I would like it to be elsewhere.
If I'm not going to put in the design and review effort (because I'm going to redirect it to other Go crypto work, obviously), I don't want it living on in x/crypto. I'm not saying this is or should be the policy for any other part of the standard or additional libraries, but it feels like a reasonable position for cryptographic code, partially confirmed by your statement above:
I thought about it for a couple of days and there really is no group or person I can think of that I'd trust to maintain it in the future.
I feel a level of personal responsibility over the quality of x/crypto packages […]
If I'm not going to put in the design and review effort (because I'm going to redirect it to other Go crypto work, obviously), I don't want it living on in x/crypto.
I don't know a polite way to say this. But what makes you, as a person, so special here? Why is you feeling that way an argument for how the Go project as a whole should act? IMO it is a problem if you feel personal responsibility for x/crypto
to the degree that it's dictating policy - as it's a project repository, not a personal one.
After more thought, I think I could actually replace my main OpenPGP use case in Go (encrypting large files to multiple keys) with nacl/box. I'd just need to chunk the large files into smaller messages (16KB or so) maybe.
If users could as easily generate and manage nacl key pairs as they can OpenPGP key pairs, then I would say that nacl would be easier to maintain as it's simpler (by a lot) and much more specific in what it does.
Thanks again for the chance to discuss this.
@w8rbt you may find that https://saltpack.org (which has a solid Go implementation) already addresses some needs you'll have in applying nacl, like key generation, chunking, choosing an on-disk format, supporting multiple recipients, and authenticating your data to prevent tampering and truncation of your files.
@tarcieri
These constructions, unlike the CBC and CFB modes used by OpenPGP, will never disclose ciphertext until it has been authenticated
This is factually incorrect. GCM is literally CTR + a MAC tag, there is nothing stopping an implementation from just ignoring the MAC.
BEAST, Lucky13, and POODLE
All of which are examples of CBC being crap.
@FiloSottile
Final nudge not to debate the merits of OpenPGP as a protocol independently of the x/crypto/openpgp implementation in this thread anymore
Should not have claimed that Efail is a fault of the OpenPGP protocol (or should have at least acknowledged that your post was wrong) if you did not want people to inform you of your mistake - especially when you use it as an argument to claim that OpenPGP should be avoided.
I don't know a polite way to say this. But what makes you, as a person, so special here? Why is you feeling that way an argument for how the Go project as a whole should act? IMO it is a problem if you feel _personal_ responsibility for
x/crypto
to the degree that it's dictating policy - as it's a _project_ repository, not a personal one.
@Merovius Let me rephrase, “personal” was the wrong term. I feel an individual responsibility in my professional capacity as the code owner of the Go cryptographic libraries.
I feel like this degree of close ownership is justified for cryptographic code.
@FiloSottile Nice censorship there dude. Are you upset that people do not agree with your anti-openpgp agenda?
@FiloSottile Okay. But ISTM that's not a really good reason not to pass ownership of at least some of the crypto libraries over then, if you don't want to maintain them. If the Go project as a whole thinks the openpgp
package (sorry, that's just the one I'm actually interested in) shouldn't exist or be deprecated, then fair enough. But if the whole issue is that you don't want to maintain it, then it would seem finding someone else to own it would solve all the problems - you'd be relieved of your professional responsibility and would be free to focus your efforts elsewhere.
I just don't get why you are standing in the way of someone else doing what you don't want to. Personally, I want x/crypto/openpgp
to exist and be under the umbrella of the Go team as an organization - not necessarily you as a person. I care about the set of processes and the institutional commitment that come along with it, especially as it means the package gets some maintenance even if a particular person doesn't want to do it. IMO it says something disappointing about the Go project, if a package gets deprecated/dropped just because one specific person doesn't want to maintain it anymore.
(also, FTR: There are other arguments for dropping x/crypto/openpgp
in particular, but given the embargo on talking about the merits of OpenPGP as a concept itself, it doesn't seem particularly fair to take those arguments into account - as we apparently can't try to refute them)
@Merovius As I understand it, you want a 3rd-party maintained x/crypto/openpgp
to stay in x/crypto
even if no one in the Go team will be responsible for it. I want that package to live somewhere else, like github.com/go-openpgp/openpgp
(I made this up).
I think both are valid options, backed by a number of arguments. Like many decisions, this is about a trade-off, and I stand by my opinion on what is best for the ecosystem.
@FiloSottile Sure. But FTR, there is a third option, which is that the Go team stays responsible (and note that "being responsible for" isn't the same as "doing all the work") for it - just that it's not you specifically, if you don't want to do it. It'd my preferred option, which is why I'm making it explicit. So far it sounds as if you're categorically ruling out the existence of a package in x/crypto
that you are not personally owning and doing all the review- and design-work for.
As a user of OpenPGP, I'm happy to host the openpgp
packages under go4.org
, with the same CLA & code review policies, if it comes to that.
(Was asked to comment here.)
I see three broad categories of problematic packages here:
First are the simple packages that are just bad ideas: md4, cast5, etc. Here the issue is that inclusion in x/crypto may convey an inappropriate (and unintended) blessing, but they're basically zero maintenance. (BoringSSL calls them “decrepit”.)
Second are the barely used packages: xts and xtea, for example. Here the issue is that these packages are so obscure that they don't meet the bar for inclusion. This is my fault: there was a time when we (or at least I) were significantly more promiscuous in accepting things. These are also basically zero maintenance (because nobody cares) and there is a little overlap with the first category too. (For example, using twofish doesn't make sense now.)
Third, and most thorny, are the “Adam used to use x/crypto as a personal repository” packages: openpgp and otr being the obvious examples.
openpgp came about because Brad asked me years ago for something to do encryption-at-rest in Camlistore and OpenPGP was a) a standard, b) not terrible, and c) interoperable with other stuff, which seemed like it might be useful. But OpenPGP is a large standard and over the years some people have tried to extend the package to cover more modern additions. Several of them tried to rewrite large parts (poorly) and I declined the changes. More recently, I've been affirmatively ignoring Go stuff because Filippo exists. But the package should probably have lived in Camlistore.
otr exists because I had a friend who wanted to use OTR over XMPP and, after taking a look at the code quality of some XMPP clients, I decided that I should write my own. The OTR part was landed in x/crypto because it seemed relatively independent and, recall, x/crypto had a rather lower bar because Go usage was dramatically smaller back then.
But these packages are big and really need an active maintainer. Also, I suspect basically nobody is looking to find the horrendous security bugs that I've almost certainly botched in there. But being in x/crypto means that we have to maintain a high quality, but Filippo & I are not putting in the time to review and maintain them, so they're deadlocked.
bn256 is special in that it falls into all three categories!
For the first group, one cannot escape that interoperability does mean that, sometimes, they cannot be avoided. But I agree with Filippo that a warning in the package documentation seems insufficient and that including them at the top-level of x/crypto seems wrong now. Freezing and deprecating them seems like a minimal step.
For the second group, we should be on a path to removing them at some point and their obscurity hopefully means that the impact will be tiny. As a first step, freezing and deprecating seems fine to me.
The third group is the most nettlesome. The OpenPGP spec is problematic in several ways but there's not much else in the space and at least it's (much) better than CMS. However, they're already effectively frozen in the environment of x/crypto because our standards are too high and we're not willing to put the effort into maintaining them. I think they should move elsewhere: hopefully each to a single canonical location. I don't have a detailed prescription, but I do feel that their future does not lie in x/crypto and it would be good to establish that officially.
Third, and most thorny, are the “Adam used to use x/crypto as a personal repository” packages: openpgp and otr being the obvious examples.
Actually, openpgp was in the Go standard library before we even had the "go.crypto" mercurial subrepo: 5aeacadc5a17251aa21e8e4e44118b9f1070863d, 0bec484e6fc1553edce91eed523d9642f7870516, 364cb831f725dea61738171003600c76950e5448, 6ca009f58d0d183360cb05cbca71227a712dae53, etc. :smile:
We've created a golang-openpgp mailing list:
https://groups.google.com/group/golang-openpgp
Please join if you're an interested golang.org/x/crypto/openpgp user, contributor, or possibly as a maintainer. (Our goal is broaden its maintainership.)
We will be deprecating the packages listed above, except openpgp/
, otr/
and xts/
.
I'll update the proposal and send a CL shortly.
Hey!
Thanks @agl and @FiloSottile for all the very nice explanations.
Around OTR, as said, if you are willing, as the person right now leading the OTR project, go ahead deprecating it. The current code that is in the golang repo seems like a mix between v3 and v2, and we have decided with Ian Goldberg to completely deprecate "globally" v2 (https://bugs.otr.im/lib/libotr/issues/140). That said, we can also try to improve the current state of OTR in the golang library, but we will prefer to maintain our own library. I see that now there is a 'golang-openpgp mailing list', will there be on for OTR?
At the end of the day is your call :) But we will not like users using a very old version of OTR.
Thanks!
We don't plan to make a golang-otr mailing list, but you're welcome to if you think it'd be useful.
Change https://golang.org/cl/163537 mentions this issue: all: deprecate broken and legacy packages
Thanks @bradfitz ! I don't think it will be super useful :) Looking forward to the open-gpg list content.
in the XMPP world, the OTR protocol is still used as the standard secure messaging protocol
That's basically not true.
OTR support has been withdrawn from major clients such as Conversations and Gajim as OMEMO is widely supported.
As for OTRv4 no major client plans to implement it, I am aware of only one implementation in the wild (Coy.im, by authors of OTRv4).
Hey @wiktor-k !
Thanks for your comments here; but as @FiloSottile has widely said around OpenGPG, I don't think this is the place to litigate the state of something.
That said:
OTR support has been withdrawn from major clients such as Conversations and Gajim as OMEMO is widely supported.
As for OTRv4 no major client plans to implement it, I am aware of only one implementation in the wild (Coy.im, by authors of OTRv4).
I don't think that having been removed from two clients is bad in the OTR world (one by the authors of OMEMO). And, for OTRv4, I can tell you that a lot of clients (not only related to XMPP, as OMEMO is only for XMPP) are planning to implement it. :)
Thanks!
I don't think that having been removed from two clients is bad in the OTR world
I did comment only on the OTR being standard in XMPP world being grossly untrue. Even XMPP Wiki says that "OTR has widely been replaced by OMEMO in the XMPP network and is recommended to be used instead."
And, for OTRv4, I can tell you that a lot of clients (not only related to XMPP, as OMEMO is only for XMPP) are planning to implement it. :)
You may want to list these clients on OTR clients page. The current list that contains unmaintained projects (such as Adium or Chatsecure) doesn't inspire much confidence.
I did comment only on the OTR being standard in XMPP world being grossly untrue. Even XMPP Wiki says that "OTR has widely been replaced by OMEMO in the XMPP network and is recommended to be used instead."
Sure. But, as I read that, and correct me if I'm wrong, that seems to be a recommendation.
You may want to list these clients on OTR clients page. The current list that contains unmaintained projects (such as Adium or Chatsecure) doesn't inspire much confidence.
OTRv4 protocol design is still in development. Listing on a page which ones will support it seems dishonest and advertising-like. Once everything is done, we can put them here. So worry not. :)
But, as I read that, and correct me if I'm wrong, that seems to be a recommendation.
It seems to be some ambiguous wording out there but the history page says "recommend OMEMO as replacement for OTR". I'll get this sorted out in XSF. Thanks for bringing it!
OTRv4 protocol design is still in development. Listing on a page which ones will support it seems dishonest and advertising-like. Once everything is done, we can put them here. So worry not. :)
Good luck and godspeed :+1: Now we can finally put this thread to rest...
Most helpful comment
Remember that deprecation comes with a message, so we can explain the reasons and implications for each package individually, and I'll make sure to provide as much context as possible.
There will be packages that say "this algorithm is broken [or superfluous] because X and should not be used except for compatibility with legacy systems". This does not call for an alternative implementation, but it warns people off using it, and recommends moving to non-broken things.
Other packages will say "this implementation is incomplete and unmantained, consider an alternative protocol like X, or a community maintained package". This hopefully leads to a centralized target for maintenance efforts which does not involve our resources.
I am replying to a few points individually below, before commenting please check if your point was already made above, to prevent this from becoming a giant thread nobody reads or feels like participating in.
Indeed, deprecating them comes at near-zero cost because there are not going to be new features and bug fixes. But the message to developers about not using them is important.
Ack, but I'm afraid that's not enough to justify the Go project maintaining an implementation. I would probably not even accept a Signal protocol implementation in x/crypto.
Good point! That makes it fundamentally vulnerable to Sweet32, updated the proposal.
I use usage in public projects as an indicator of popularity and therefore usefulness to the ecosystem, not as proof that nobody uses the package at all, and that's one part of the tradeoff decision on whether it's worth the resources to maintain.
AES-256 is secure enough for Top Secret data in NSA's Suite B, I don't feel the need to carry anything else to fill a security gap.
XTS is quite bad in fact. See also a better tweakable cipher.
Almost half of all issues open against x/crypto packages are for x/crypto/openpgp, and a good chunk of all CLs are for it as well. I'm not blaming the community for not contributing back here, I blame the protocol for requiring too much maintenance to properly implement. It just doesn't make sense for us to put in the effort to do the review and design work to bring this package to a state in which we are happy about it. Hopefully the community will centralize on a fork.
MD4 is a bad example, because it's literally as broken as a hash function can be. Go is a memory-safe language, so a random implementation off the street can't introduce an RCE without really trying, and the next worse thing a hash function from the last 20 years can have is a collision, and well, I might still have some Python to generate one on a laptop.
The cost includes developers being misled into thinking that a primitive or protocol can serve their needs because it's in x/crypto, diluting the safety properties of using the standard library.
Some to support OpenPGP, some have always been there from before there was a conscious effort to limit what was in x/crypto, some were reasonable at the time and aren't anymore, some were a mistake.
If you are using any of the above, I would like you to stop using it and move to an alternative. Sometimes an alternative implementation, sometimes an alternative algorithm. If you know you can't because of legacy compatibility reasons, at least I made you aware of it, and made sure the question got asked at review time, and the answer hopefully documented.