Nixpkgs: Get representatives for NixOS assigned to the OSS-Security "distros" mailing list

[peti]

NixOS should have representatives subscribed to the "distros" mailing list at http://oss-security.openwall.org/wiki/mailing-lists/distros so that we are informed about embargoed security issues and have a chance to provide our users with fast updates on the coordinated release date.

[copumpkin]

Great idea! I'd love to do at least part of it. I have a strong background in security and would set aside some time to do the NixOS work needed for this. How many representatives can we get involved?

[domenkozar]

I have little background in security (coursera Crypto), but I'd also like to help on this front. I think we should start by having a team/herd that cares about security and make a roadmap for each NixOS release what to focus on.

[copumpkin]

I asked for more details on IRC (#oss-security on freenode):

<copumpkin> http://oss-security.openwall.org/wiki/mailing-lists/distros mentions that discussion of new distro membership should happen on the list. Is there a standard way to propose it, or does one just send an email to the list? I couldn't find any recent such discussions
<sarnold>   copumpkin: just send an email; I think the last successful "please add me to this list" came from amazon probably a year or two ago
<sarnold>   copumpkin: be sure to demonstrate that you've got users, that you've been publishing security updates for those users for a while
<copumpkin> cool, thanks. We were considering applying over in NixOS. Not sure what the threshold for users should be, and we don't have an easy way of gauging user count, but it's a reasonably notable project with lots of IRC users and github participation. Do you think that would be eligible?
<copumpkin> (we've definitely been pushing security updates, etc.)
<sarnold>   nixos seems pretty plausible to me, I think most people have a good opinion of your work anyway.
<copumpkin> sweet! How many representatives do distros typically propose? I'd like for it not to rely on one person for a few (probably obvious) reasons :)
<sarnold>   one or two, probably best if both are known in their own rights somehow..
<copumpkin> makes sense

[vcunat]

/participate

[peti]

The purpose of that mailing list is to coordinate important security updates behind-the-scenes, i.e. serious vulnerabilities are disclosed to the members of that list before the general public learns about them. This gives distributors a head start of, say, 2 weeks until the publication of the CVE so that they have a chance to prepare package updates ahead of time and to release them to their users at the same time as the CVE is published, thereby minimizing the window of exposure of their users.

Since the members of that list have access to extremely sensitive material, representatives we suggest should be reliable, long-term contributors of the project who are well connected within the NixOS community.

Furthermore, these representatives must have the ability to prepare package updates _secretly_. For example, suppose a serious vulnerability is discovered in glibc. Then the best possible solution would be for our representative to patch glibc, re-build most of the Nixpkgs package set for Linux/i686 and Linux/x86_64, and to upload all those builds to cache.nixos.org. At the time of the coordinated release date -- when the vulnerability becomes public --, the appropriate patch to the glibc Nix expression is then checked into our Git repository so that everyone can see it. The important bit it is that at the time that patch hits the Nixpkgs repository all the binary packages for the new state are already available for download.

Obviously, that procedure can be re-fined a lot, still, but the point I am trying to make is that our representatives need the ability to mess with hydra.nixos.org and cache.nixos.org in a way that's invisible to the rest of the world, i.e. without going through the public Git repository.

This means, IMHO, that our representative pretty much have to be @edolstra and @rbvermaa, because no-one else has the necessary privileges and technical skills to pull off the necessary updates.

[copumpkin]

@peti I buy your point today, but I'd like to move away from centralizing further on @edolstra and @rbvermaa. With @nbp's upcoming changes, security fixes don't necessarily need to rebuild the world, so the special tie-ins with Hydra become less necessary. Furthermore, this project is already buckling under the weight of our dependency on Eelco and Rob, and adding more seems like the wrong direction. If anything, I'd like to figure out ways to make access to Hydra less magical while still retaining the nice trust properties we have today.

[copumpkin]

Here's more that could simplify our decisions a bit:

<copumpkin> what's the expected level of secrecy here? are the two representatives expected to single-handedly prepare/coordinate the patch releases for their distros, or can they confer with others as needed on the distro security team
<copumpkin> (need-to-know, trusted, etc.)
<sarnold>   copumpkin: probably confer as needed, assuming they'd instruct the others in the appropriate protocols
<copumpkin> cool
<copumpkin> yeah, that sounds a lot more manageable than lumping all the responsibility onto two people

[vcunat]

ability to mess with hydra.nixos.org and cache.nixos.org in a way that's invisible to the rest of the world

It would be best if hydra supported jobs invisible to most users. A simpler option would be to create another parallel instance, running on the same HW, but only accessible to the selected individual(s) (and in effect also to anyone with SSH access to those machines).

[vcunat]

I suppose there will always be that information leak when build farm puts unusually little resources to the visible jobs, but I think that's perfectly acceptable, as it's even common to pre-announce security updates.

[nbp]

I totally agree, some well acknowledged, trust-able, and available community member should apply there.

On the other hand, I think this might be a bit premature to do that if we have no way to ship these security updates in a timely manner. (#10851 and the long tail of static analysis reports to fix)

[nbp]

@peti, do we have any idea of the volume of CVE that these persons would have to handle?

[grahamc]

According to sarnold,there is very little traffic on the mailing list.

[vcunat]

If one is given a week of heads up, even a full rebuild is feasible on our Hydra...

What is _very little traffic_? About a pair of threads with several e-mails in a month?

[grahamc]

Not sure. I asked, but haven't received a reply yet. I would suspect no more than what you said.

I believe it might also be helpful to have a team for following oss-security in general and make sure we handle those in a timely manner as well.

[grahamc]

Update: "it's kind of hard to quantify; there's 208 mails in the last year, over what feels like less than a dozen "actionable" things, a handful of CVE assignments."

[vcunat]

OK, I believe I could handle the _distros_ list and the related agenda without any money for it, especially if paired with someone else. I currently can't claim that about the _oss-security_ list. (But note that I'm rather cheap ;-)

[peti]

@nbp, the volume on _distros_ is small and the vulnerability reports usually come with ready-to-apply fixes attached. If we'd care only about master, then handling those issue would be almost no effort: you'd simply apply the patches (or update the package to the new version) and set off some kind of at job to push the update to Nixpkgs at the coordinated release date. Things become more interesting (and more effort) only when it comes to fixing older versions of the package which might still be in use in previous release branches since upstream may not provide a patch for a package version other than the latest one. In that case, you'll have to back-port the patch to the older code base -- or update the version in the release branch. That might be tricky. On such occasions, it's probably a good idea to involve someone else who has the necessary expertise. Anyhow, NixOS has no LTS branches where 3+ year old core packages are still in active use, so in our case dealing with such issues will be very straightfoward almost all of the time.

IMHO, the biggest problem is how to prepare binary packages to release on the CRD without revealing any details about the update beforehand.

[nbp]

Note, this mailing list does not change the fact that we would have to handle 0-days, without any rebuild.

So I do not see why we should enforce rebuilds when we have the option to not do that, as this also involve bandwidth that not all users can spare on a daily bases.

[copumpkin]

@nbp I'm treating it as a given that we'll merge your PR. I see this ticket more about organizational issues.

[vcunat]

Yes, I consider that rebuild-reducing work orthogonal to this discussion.

[grahamc]

What is the next step in resolving this issue? I'd hate to see this fall by the way side.

[vcunat]

I think now we "only" need to find _who_ would be on that list; _two_ seems the best number. As I wrote, I can be one if there's lack of interest from others.

[grahamc]

OK. I deleted my comment suggesting myself. Assuming they agree, perhaps @edolstra and @rbvermaa should pick a couple people.

[edolstra]

@nbp We can push out -small updates requiring a full rebuild in < 3 hours. And in any case Hydra would have to rebuild _something_ (e.g. for a Glibc patch it will need to rebuild Glibc). So regardless of #10851, we need either a private Hydra instance, or some access control in Hydra. There was some interest at work in having private jobsets, so I may be able to spend some time implementing that.

Regarding our representatives, how about @copumpkin and @domenkozar? (It wasn't clear to me if @peti was volunteering himself.)

[peti]

@edolstra, I didn't mean to nominate myself. I just felt like pointing out work that I'd like other people to do. :-)

[copumpkin]

For more information on the last addition to the linux-distros list, I dug up this old thread from when Amazon got a representative added. It might help clarify the sorts of things they look for in a distro and representative:

http://www.openwall.com/lists/oss-security/2014/04/10/5

[copumpkin]

In short, what they appear to be looking for:

• Representatives with past participation in the public oss-security list (probably not good at this)
• Evidence that the distro backs the representatives (this thread can be evidence)
• Evidence that the distro puts out timely security releases (I think we're generally good about this and will only get better)
• Evidence that the distro includes information about which CVEs are fixed (we're patchier on this)
• Evidence that the distro actively notifies users about available security updates (I don't think we do this today)

These all seem like reasonable things to look for. Perhaps we should put together a NixOS security mailing list for those of us who are interested, so we can further discuss the various finer points around this stuff? Or I guess we could do it all on GitHub, but it might get unwieldy to notify all interested parties.

[vcunat]

Heh, that list's not easy to fulfill.

[domenkozar]

Relevant: https://blog.flyingcircus.io/2016/05/20/introducing-vulnix-a-vulnerability-scanner-for-nixos/

[domenkozar]

Also relevant: https://github.com/NixOS/nixpkgs/issues/13515

[peti]

In my opinion, the community members who volunteer to represent NixOS on the distros mailing list should take matters into their own hands and initiate some kind of "poll for support" via nix-dev to legitimize their claim. Once a significant number of Nix'er has expressed their support for the candidates, our representatives should get in touch with distros and try to make things happen. As it is now, nobody feels responsible for achieving this goal, which means that we won't. That would be a shame.

[davidak]

I like how Ubuntu notifies it's users. They have a mailing list and webpages with the same informations, for example http://www.ubuntu.com/usn/usn-3089-1/

That is great to give your team leader a link to inform him about a serious update. A RSS feed on the NixOS homepage would also be great.

Thanks for taking the initiative. I really appreciate it as a user!

[grahamc]

CoreOS is trying to get on the distro list. Here are some requirements they've noted on the ML:

It looks like CoreOS is shipping Linux and respecting the various licenses
in a volume sufficient to make sense for them being given access to the
Linux distros list, and shipping security updates (I would say they could
benefit from shipping advisories, but they put the CVE's in the ChangeLog
so I really can't complain). Assuming they can handle embargoed issues (do
you have private bug tracking/code repos/CI/whatever else you need to ship
an update?) I would have no objections to them joining the Linux distros
list. Can you confirm you have infrastructure to handle embargoed issues?
If yes I guess it's up to Solar to add you.

[Mic92]

So a private hydra is required?

[grahamc]

Not sure. I could imagine this being The Most Enterprise:

I also think there is talk about private jobsets on the public hydra.

Note, this also requires a private nixpkgs repository and bug tracker.

[Moredread]

Has there been made progress on that front?

[ckauhaus]

FTR: We discussed this issue during the Hackday @ NixCon 2018 and decided that it is not about time to apply for oss-security now. We are currently not in a position to produce evidence in all required aspects.

I'd personally recommend to work hard towards meeting the criteria and reconsider the issue then. Perhaps this issue can be closed and we will open a new when it is really actionable.

[peti]

Yeah, let's close this issue as it's not actionable right now. We need to take other, smaller steps first before we can tackle this one.