Node: Distrust Symantec root certs

[ChALkeR]

Similarly to #9434, I think that we should follow Google/Mozilla on this one, too.

I'm opening this in advance as it deserves some messaging and announcements in advance, after the decision would be made.

Refs:

• https://wiki.mozilla.org/CA:Symantec_Issues (includes issuing fake google.com and gmail.com certs, issuing certs after deadline, vulnerabilities, audition failures, etc.).
• https://git.cryto.net/joepie91/ca-incidents#symantec — more Symantec CA issues.
• https://groups.google.com/a/chromium.org/d/msg/blink-dev/eUAKwjihhBs/rpxMXjZHCQAJ (some of the story and proposed schedule for Chrome back in March).
• https://groups.google.com/a/chromium.org/d/msg/blink-dev/eUAKwjihhBs/El1mH8S6AwAJ — final proposal by Google
• https://groups.google.com/d/msg/mozilla.dev.security.policy/Oaeqtddo_Cw/Vl7bJvQKCQAJ (Mozilla thread, citing the final decision by Google, some proposed dates for Firefox)
• https://groups.google.com/d/msg/mozilla.dev.security.policy/gn1i2JNVCnc/CcTfqVcrBgAJ (another Mozilla thread)

Not yet listed on the CA:Symantec_Issues page: https://blog.hboeck.de/archives/888-How-I-tricked-Symantec-with-a-Fake-Private-Key.html (also I believe that this came up _after_ Google/Mozilla drafted the current plan).

The final roadmap for Chrome is (dates are approximate as usual per Chrome release schedule):

• October 24, 2017 (Chrome 62) — start printing warnings to DevTools console for certs that will be rejected in Chrome 66.
  _That's around Node.js 9.0 release._
• April 17, 2018 (Chrome 66) — reject Symantec-signed certs issued before June 1, 2016.
  _That's around Node.js 10.0 release._
• October 23, 2018 (Chrome 70) — reject all Symantec-signed certs.
  _That's around Node.js 11.0 release._

The roadmap for Mozilla is not final yet, but it's expected that they will reject all Symantec-signed certs starting from October 16, 2018 (Firefox 63) or November 27, 2018 (Firefox 64). There also is an intermediate step, that they are considering to place somewhere between December 1, 2017 and April 2018, as I understood things.

So, for Node.js, I would propose to:

• Distrust Symantec-signed certs, issued before June 1, 2016, in Node.js 10.0 (April 2018).
• Distrust all Symantec-signed certs in Node.js 11.0 (October 2018).
• Message that in advance through regular channels.
• It could be useful to print deprecation messages for Symantec-signed certs issued before June 1, 2016 under --pending-deprecation for Node.js 9.0 release, ref: #11968, cc @jasnell.
  _Chrome will start logging those to DevTools console in October 2017._
  I don't think that we should print those deprecation messages by default at all, as those will most likely be just thirdparty website issues that will just confuse users in most cases, and they will be forced to fix that by browsers.

/cc @shigeki, @bnoordhuis, @indutny specifically and @nodejs/ctc in general.

_This also affects Thawte, GeoTrust and RapidSSL certs_.

[ChALkeR]

New updates:

• https://www.reuters.com/article/us-symantec-divestiture-thoma-bravo-idUSKBN1AI2N6
• https://www.symantec.com/connect/blogs/symantec-ca-acquisition

Symantec is selling its CA business to Thoma Bravo, the company behind DigiCert.

[shigeki]

Mozilla has decided to follow Google. https://groups.google.com/d/msg/mozilla.dev.security.policy/Oaeqtddo_Cw/lN-Pp6h1AAAJ

I would like to see how they deprecate and remove them from certdata.txt if we can make it by just updating root certs.

[bnoordhuis]

I was going to suggest the same thing: just wait until NSS removes them from certdata.txt, then upgrade.

[ChALkeR]

@bnoordhuis I have two questions about that approach.

The main one is: do I underdtand correctly that the removal from certdata.txt won't happen for the intermediate step of distrusting certs issued before June 1, 2016 (which browsers are going to ship the coming spring) and will happen only with full Symantec certs drop the next autumn, so we won't have the intermediate step and will have to wait one more major release to catch up on that?

The second one, time-constained question is — do we want to emit deprecation messages under --print-deprecation in 9.0 (following Chrome which would print those messages in developers console) or not? I'm not sure if we should, so I'm actually +0 on those deprecation messages in 9.0.

[bnoordhuis]

we won't have the intermediate step and will have to wait one more major release to catch up on that?

That doesn't have to be an issue. We can (and do) upgrade root certificates in minor releases.

do we want to emit deprecation messages under --print-deprecation in 9.0

Dunno. It's probably only an annoyance for node.js TLS clients because they don't control what certificate is served to them.

For node.js TLS servers, I don't know if it's worth the effort. I expect more users will find out through their browser than through --print-deprecation.

[ChALkeR]

@nodejs/tsc is this something we want to do for 10.0?

[Trott]

Adding tsc-agenda label, although hoping we can come to consensus here in the issue tracker instead of taking this up at a meeting.

[mcollina]

I'm 👍 in removing the affected root certs for 10. We should probably think doing it in 8 as well.

[BridgeAR]

+1 from me for removing the certs in 10.x and for backporting.

[mhdawson]

+1 for removing in 10.x

[fhinkel]

+1

[fhinkel]

Since there's multiple 👍 and no objections, does this still need to be on the TSC agenda?

[fhinkel]

Summoning @Trott as he was the one who added the label 😆

[Trott]

@fhinkel Happy to remove it from the agenda. If anyone (members of TSC or not) have concerns or objections, they can add it back to the agenda. (Our governance rules are that anyone can add anything to the TSC agenda.)

[rvagg]

+1 from me, and backporting to all the things

[jasnell]

@nodejs/tsc @nodejs/crypto ... If this is going to make it in time for 10.0.0 it really needs to get landed today.

[bnoordhuis]

I only identified what certificates would need to be distrusted, I didn't write any code. What about you, @ChALkeR?

(FWIW, I also don't think any action is needed right now but maybe that's just me.)

[jasnell]

If that's the case, then awesome :-) Just want to make sure nothing slips through. I'll take this off the 10.0.0 milestone. Please add it back on and give me a heads up if it definitely needs to be added back.

[richardlau]

Is this likely to happen for v11 given the dates in https://github.com/nodejs/node/issues/22180?

[apapirovski]

I take it this didn't happen? @ChALkeR

[richardlau]

This didn't happen in 10 nor 11. Is this likely to happen for 12 or should we just close this? If we do want it to happen, what needs to be done?

@nodejs/tsc @nodejs/crypto

[shigeki]

Mozilla delayed to distrust Symantec Cert to Firefox64, probably mid of Dec.
https://blog.mozilla.org/security/2018/10/10/delaying-further-symantec-tls-certificate-distrust/
Google started to distrust it but it is gradually deployed under the control of Google. We do not know whether it is finished it or not.
https://security.googleblog.com/2018/03/distrust-of-symantec-pki-immediate.html

I think it is better to wait for it until Mozilla does it.

[bnoordhuis]

I'm going to close this per @shigeki's comment. Symantec's certificates are still present upstream.

[ChALkeR]

@bnoordhuis, um, @shigeki's comment was from last year and it was about December 2018.

The distrust happened in Firefox 64, according to the release notes:

• https://hacks.mozilla.org/2018/12/firefox-64-released/
• https://www.mozilla.org/en-US/firefox/64.0/releasenotes/

[bnoordhuis]

@ChALkeR They're still present in upstream NSS (v3.42.1 as of this writing), which is where we get our root certificates from.

I thought we were taking the approach I mentioned in https://github.com/nodejs/node/issues/14537#issuecomment-319967213 but if we're going to perform active blacklisting, then someone needs to write the code and be quick about it too because the release is next month.

[sam-github]

/to @nodejs/security @nodejs/security-wg

As I understand it, the question is whether we should deviate from the standard root update process, or not?

If Node.js is going to start overriding the Mozilla trust decisions, we should have a pretty good reason. They have a well-defined process for deciding trust, I don't think we do.

[ChALkeR]

More info about Mozilla impl of the distrust: https://wiki.mozilla.org/CA/Additional_Trust_Changes#Symantec

The opt-out flag security.pki.distrust_ca_policy is still there (it looks it wasn't removed in 65 yet), as could be seen on https://github.com/mozilla/gecko-dev/blob/b2d35912da5b2acecb0274eb113777d344ffb75e/security/manager/ssl/nsNSSComponent.cpp#L1229-L1253

This is how the distrust is applied:

https://github.com/mozilla/gecko-dev/blob/b2d35912da5b2acecb0274eb113777d344ffb75e/security/certverifier/NSSCertDBTrustDomain.cpp#L857-L898

The default value is DistrustSymantecRootsRegardlessOfDate.

[rvagg]

So Mozilla is not trusting Symantec by default but are still shipping root certs in NSS .. I suppose this is to be maximally compatible with users who still need that option .. I think it would be entirely consistent of us to apply a similar rule and maybe that simply means editing our update process to explicitly strip out Symantec (some sed or perl after the curl would suffice). If Node users need it back there are multiple ways to put them back into play for your own implementation.

There's plenty of +1's here to support this. Removing _during_ 12 is going to be trickier than adding it back in if we find out we made a terrible mistake.

[ChALkeR]

@rvagg, looking at the code -- it's not going to be that simple, I suppose.

There is a set of whitelisted intermediates, listed on https://wiki.mozilla.org/CA/Additional_Trust_Changes#Symantec, and applied here:

• https://github.com/mozilla/gecko-dev/blob/b2d35912da5b2acecb0274eb113777d344ffb75e/security/certverifier/NSSCertDBTrustDomain.cpp#L857-L898
• https://github.com/mozilla/gecko-dev/blob/265e6721798a455604328ed5262f430cfcc37c2f/security/certverifier/TrustOverrideUtils.h#L80-L131

Simply sedding out root Symantec certs would break those trusted intermediates, wouldn't it?

Perhaps it would make sense to postpone things till the next major release, because (at least as I currently understand things) the code for this particular distrust isn't going to be very nice.

On the other hand, we don't need e.g. permitAfterDate logic, a simple flag would be fine enough -- that reduces the code size up to twice, I think?

[rvagg]

Hah, yeah that whitelist does make it kind of complicated, I'm still inclined to just rip all Symantec out but that's not easy to justify without any data on how many certs were issued by those itermediates

So we wait then I guess, unless someone comes up with some code that can mirror that solution and graduate us toward full removal. We could backport a --distrust-symantec arg if we implemented this in code too.

[sam-github]

I'd be OK with ripping symantec certs out for 12.0.0, and seeing if there is an uproar. If not, fait accompli. If there is pushback, there is 6 months to stabilize, they can be added back (or the whitelist code added).

[ChALkeR]

@rvagg @sam-github Off-hand, I don't think that it's a good idea to block certs that are specifically whitelisted by Mozilla and Google. Exactly because of the "overriding the Mozilla trust decisions" point.

In https://github.com/nodejs/node/issues/14537#issuecomment-470622307, you mentioned "overriding the Mozilla trust decisions", but what I was proposing was the opposite of overriding Mozilla trust decisions -- it was __following__ them, instead.

NSS alone does not perfectly represent Mozilla trust decisions for Firefox, NSS + overrides inside FF do. We have NSS, but we don't have the overrides, this is where the difference comes from.

Blocking more than Mozilla does (by ignoring the whitelist) is most likely not a good solution, imo.
We didn't do a research on how many websites out there are using those whitelisted certs, Mozilla did.

---

It is still not clear what to do here, I'm conflicted. I'm going to briefly mention this tomorrow for more discussion and input, I think? There were some thumbs up earlier, but those people have not commented after the situation became more clear yet.

[sam-github]

@ChALkeR I also said "have a good reason". :-). The technical challenges of reproducing the Mozilla whitelist code is compelling (to me), but its uncomfortable having to make this decision, especially on little data.

We have three options:

1. use the NSS cert list, but since we don't have a whitelist we will allow more certs than Mozilla does
2. use the NSS cert list but remove symantec, which means we will allow less certs than Mozilla does (but our users are more technical than Firefox browser users -- well, some are -- and they have programmable access to add the symantec certs)
3. reverse engineering the Mozilla whitelist code, and reproduce it in node.js, so we allow exactly the certs that Mozilla allows

3. is clearly the best option, but someone has to step up and do the work.

2. has the advantage of being more secure than Mozilla, and being what Mozilla's long-term intention is (if I read correctly). It has the disadvantage that we don't know how many sites this will break.

If we try 2., and it doesn't work, its not the end of the world, IMO, because 1. is an easy fallback, and 3 is a less easy fallback. Also, if we hate 1, and it turns out there are barely any problems with 2, we avoid having to do the work of 3.

This discussion is mostly about Mozilla, because that is where we get our certs from, but what about Chrome or the other browsers? Does anybody know what they are doing about this?

[ChALkeR]

This was mentioned in the TSC Meeting today (https://github.com/nodejs/TSC/issues/673), @rvagg proposed to give this more time and to move the decision to a minor 12.x release instead (meaning a removal from 12.0 milestone), perhaps with a note in 12.0 release that certs trust could be updated in a minor release.

There were no objections to potentially landing the distrust in a minor 12.x release, if that happens before it goes into LTS mode.

[rvagg]

And for anyone in here that's especially keen on this happening—you could help do this work because there's nobody obvious stepping up at the moment to do it.

I'm imagining copying that mozilla logic such that we blacklist Symantec but whitelist the intermediates in question. That could go in as the default behaviour in 12 _before_ LTS (October this year) with a command-line argument to revert to original (full trust) behaviour to help people with breakage (--enable-symantec-root-certs? or we could give it a CVE and --revert=CVE...). We've given ourselves permission to bump minor on a semver-major _security_ fix in a release line and this could fall under that designation. Post-LTS, that becomes a bit harder to justify for this one I think.

[bnoordhuis]

Another five months have passed and no one has put up a pull request so far. The deadline is 2019-10-22, that's when v12.x LTS starts.

[jasnell]

We've gone nearly a full year on this and it hasn't happened. ping @nodejs/crypto

[bnoordhuis]

I'm going to close this out because it's unlikely to happen, not at this point. The cutoff was over a year ago.