Nixpkgs: There needs to be an automated tool to harvest the NVD vulnerability database and match it with the current nixpkgs tree.

[spacekitteh]

Currently, people like @grahamc are manually doing this in their vulnerability roundup sprints. This means it's error prone, and isn't reliable (no offence!). By having an automated tool, we could automatically open issues, have a counter on the readme showing how many CVEs currently apply, and keep it up to date.

This is, in my opinion, a necessary prerequisite for reliable security updates (#10851).

From NIST Special Publication 800-150: Guide to Cyber Threat Information Sharing (pdf):

Use secure, automated workflows to publish, consume, analyze, and act upon cyber threat information.
The use of standardized data formats and transport protocols to share cyber threat information makes it easier to automate threat information processing. The use of automation enables cyber threat information to be rapidly shared, transformed, enriched, analyzed, and acted upon with less need for manual intervention.

Current related tools that I know of:

• http://monitor.nixos.org/ (could be improved to create github tickets for each vuln match)
• https://github.com/flyingcircusio/vulnix (could be run on hydra, maybe?)

[NeQuissimus]

I am all for automating (parts of) this workflow.

I have tried using vulnix but it spit out a lot of false positives. Not sure it can be improved easily, I have not looked into that.

Not even sure what monitor.nix.org does, it seems to be way behind in its analysis...

[spacekitteh]

Trying out vulnix now, yeah, it seems to not be able to get the correct version of some of the installed software.

[vcunat]

Close, now that we have https://github.com/NixOS/security ?

[grahamc]

I think this issue was opened before security was open source, so that alone may make it closable. It is worth noting that security doesn't automatically match to the tree. This is a conscious decision as I believe this automating this process to be far more error prone than the manual method.

Hopefully the founding of the NixOS Security Team (http://lists.science.uu.nl/pipermail/nix-dev/2016-December/022276.html) will make this more reliable (by reducing my bus factor.)

With regards to the NIST guidelines, here is how I'd grade the tooling I've built:

Use secure, automated workflows to publish, consume, analyze, and act upon cyber threat information.

Use:

• [x] Secure: we use an open and auditable process preventing fraud, and use HTTPS to fetch updates.
• [x] Automated: The reports are compiled automatically from the datasource, promising thorough coverage.

To:

• [ ] Publish: I think this is half-done. The ported.sh script makes an effort, but requires manual verification and editing. I think manual verification is good and important, and I'm not looking to eliminate that, but the "half done" part is the tool isn't great :)
• [x] Consume: Again, the consumption and report generation is fully automated and thorough.
• [x] Analyze: The current tooling outputs URLs and information, and has some smarts to collate and split packages to increase the likelihood they will be matched to something in nixpkgs. Analysis in inherently a human-driven task.
• [x] Act Upon: Our weekly lists are promptly and thoroughly closed, with only a few of them ever having carry-over items.

These tools I've built are not perfect, nor the end goal of these developments. However: I do believe they fairly thoroughly cover these requirements.

[spacekitteh]

What's the lag time between a CVE with patch available and it being merged into nixpkgs now?

[grahamc]

Hard to know, and without either:

one, seriously bulking up the group of people involved in regular patching
or two, being paid to do this...

it isn't a metric I'm particularly interested in tracking. I'm thrilled enough that we're patching on a weekly basis. See also: https://github.com/NixOS/nixpkgs/issues/13515#issuecomment-261642222

update: I meant to say the first time around: I _would_ be thrilled to see someone else who was interested in tracking that figure, and put forth the time/effort/code to do so :)

[spacekitteh]

That's kinda half the reason why I opened the ticket - the point of the NVD after all is for the process to be automated! It is tedious and boring and repetitive and perfectly suited for an automated tool.

That way, the security team can focus on the more complex, interesting, non-trivial issues, instead of just "download and apply this patch and bob's your uncle"

[grahamc]

I appreciate the sentiment, and as soon as a proficient and effective tool exists which automatically applies patches to master and does the right thing with regards to backporting to stable, the security work roundups will quickly become unnecessary.

[spacekitteh]

Obviously the -applying- patches part is very complicated to get right 100% of the time, but:

1. The first step - automatic, immediate notification - is MUCH more achievable
2. Automatic notification is useful entirely by itself without any automatic patch application
3. If even a fraction of patches can be successfully applied automatically, that's better than nothing, and is still useful and likely worth the effort (couldn't resist)

[grahamc]

The application of patches is a separate process that can evolve separately from the roundups. The list of new issues still needs to be compared against nixpkgs to identify what has and has not been patched.

[spacekitteh]

Aye, that's exactly the point of this proposed tool - to match against nixpkgs.

[grahamc]

I would welcome this tool. It is not on my roadmap.

[dckc]

While I agree that more automation is in order, I don't think the lack of it is sufficient evidence of a vulnerability to merit a "1.severity: security" label. I prefer that you reserve the use of that label for issues that indicate more "clear and present danger".

[spacekitteh]

It absolutely is a clear and present danger, in my opinion.

On Fri, 6 Jan 2017 at 06:40 Dan Connolly [e-mail] wrote:

While I agree that more automation is in order, I don't think the lack of
it is sufficient evidence of a vulnerability to merit a "1.severity:
security" label. I prefer that you reserve the use of that label for issues
that indicate more "clear and present danger".

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/NixOS/nixpkgs/issues/20176#issuecomment-270751372,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AB2crO4PYFXJy8YqwA1cs-Jzcn02P7-Bks5rPVU-gaJpZM4KqIvy
.

[dckc]

I haven't seen any evidence that the current semi-automated process has missed anything.

But reviewing the other issues tagged "1.severity: security" I see it doesn't mean what I thought it meant.

My experience with the term "severity 1" is that everything else comes to an immediate, screeching halt until it's fixed, as in:

Critical situations may require ... personnel to be at their respective work locations on an around-the-clock basis. -- first google hit I found on severity levels

But I gather it has a more laid-back meaning here.

[grahamc]

I think with the developments and significant progress made in NixOS
security over the past few months makes this issue not worth the effort.

The "Publish" step is now more thoroughly automated (source code in the
security repo,) and so now I think we really hit all the NIST guidelines
out of the park.

Because of this, I'm closing this issue as WONTFIX.

FWIW The NVD database[0] is frequently very slow to update their
database. It is very common for issues to be sent through oss-security
ages before it arrives in MITRE / NVD / etc. I don't even click NVD
links from the LWN reports we look at, because they almost never have
information about the CVE yet.

This _is not_ a complaint about NVD. NVD is operated by people and
updated by people. The CVE databases and trackers need people to
contribute time and effort to keep it up to date. Looking back at older
issues, NVD is a hugely valuable resource. I would encourage anyone whe
is interested to look in to how to contribute and help in making this
database better.

---

Dan Connolly [e-mail] writes:

But I gather it has a more laid-back meaning here.

Instead it is in the more global scope of issues, not a "level 1
security issue". I agree it would be benefitial to be grading security
issues based on impact. Unfortunately, I can't commit to this effort
given my other commitments.