Security-wg: Optimize for impact (including downloads/month)

[ChALkeR]

Hi!

Given that the human/time resources for triaging and processing reports are limited, I think that there should be better guidelines that would optimize for overall impact on the ecosystem and increase the effectiveness here. E.g. for HackerOne issues, there might be more churn that impact for packages with small downloads number.

So, my suggestion would be to:

1. Add a lower limit for accepting the issues on H1, like 1000 or 5000 downloads/month. The exact value might need to be bikeshedded (see below) and might be made dependent on the severity (but that would overcomplicate things).
   For everything below the threshold — recommend contacting the author directly or even opening a public issue / pull request.
2. Accept pull requests to the DB for any packages, but for those that don't fit into the constraints above — only after author fixed the issue or proved unesponsive to a _public_ issue for example a month (could be bikeshedded).
   
   
   
   1. Add d/m count at the time of report or the fix (whichever is greater) to the future database entries.
   
   

Some stats:

• ~0.3% (about 2000) of the ecosystem packages are above 1 000 000 d/m
• ~0.9% (about 5000) of the ecosystem packages are above 100 000 d/m
• ~2.5% (about 15k) of the ecosystem packages are above 10 000 d/m
  
  
  
  • ~7% (about 40k) of the ecosystem packages are above 1 000 d/m.
  
  
  • ~27% (about 150k) of the ecosystem packages are above 100 d/m
  
  

[vdeturckheim]

I generaly agree here. The triage team is under very heavy load right now and reducing scope might make sense.

Also, I would temper item (1) byt: If someone in the triag team feel like a report is woth it, he can still triage it even if unpopular.

[vdeturckheim]

I'd like to have your opinions @lirantal @gergelyke and @reedloden :)

[gergelyke]

I love this - I'd argue, that if we do the lower limit we should even accept reports that do not meet that threshold. If we'd say, for those feel free to directly reach out to the author / open a PR, we could free up a lot of resources on our end, and we could focus more on the more important reports.

Of course, if we will ever run out of them, we can rethink those thresholds. What do you think about it?

[bl4de]

Maybe instead introducing thresholds, you could just add to program policy an information that reports about issues in less popular/old/not maintained modules will not be triagged immediately, only when triage team will have no other reports to verify/triage/resolve?

[lirantal]

1. I'm ok with this, and >1k/month seems reasonable.
2. we also propose that today in the process of vuln reports for anything that is not submitted through HackerOne so this seems ok to me and falls under current policy (with minor changes to the threshold thing).
3. is yet more work on triage process and we need to get that automated first IMHO.

That's a good suggestion considering the workload we're currently handling.

My only concern is that we don't want to "scare off" new bug hunters to the program just because they won't be able to find security issues in the top 7% npm modules. This is why I wouldn't want to set hard limits on threshold, and instead I'd prefer to update the policy to convey that we give more popular modules a higher attention priority and we encourage bug hunters to reach out via a repo issue to the maintainers to work with them on the fix while still having the report submitted to HackerOne so they can earn the reputation points etc.

[mhdawson]

I wonder if tagging reports based on the criteria to indicate a priority level would be useful? Instead of a cutoff the tagged priority would indicate how quickly it might be triaged (we might even document expectations). That would also allow us to adjust priority assigned to specific reports when the case can be made that it's more/less important than the metric we apply suggests.

[ChALkeR]

Perhaps it could be added as a mere recommendation for now to the h1 page?

E.g.:

• If the package has >200 downloads/week, report the issue here.
• If the package has an explicit opt-in to the security-wg program (e.g. a badge), stating that its vulns should be reported here, report the issue here.
• If you think that this issue is otherwise significant enough (has a noticeable impact) — report it here.
• In all other cases, consider reporting the issue to the package repo (or to a security contact if they have one). Filing PRs to the security-wg repo after the vuln was disclosed is preferrable in those cases.

[lirantal]

I wonder if it's not too confusing or would sound like an "extra work" for someone who will be reporting an issue and they'd just bail out of it? Maybe if we can say the same thing but a little bit more concise?

My other concern is that I wouldn't want to give a bad impression of the WG not handling modules triage of specific types. I think we can do a better job to quickly triage a module that has low impact in the sense of asking the reporter to actively contact the maintainer and for us inviting them to the conversation is pretty easy.

[sam-github]

It sounds like there is some consensus here on updating the triage process to be somewhat aware of the significance of the package affected.

Would someone be interested in updating the process description? Or, perhaps that has already happened and this can be closed?