Security-wg: HackerOne Support team ideas

[lirantal]

Some ideas on further improving the triage process with the help of the HackerOne team which has been valuable from the recent reports I've seen (thanks a lot Alek, Megan and Reed).

1. We currently have a Node.js WG triage team bucket for everything that has been triaged. Quite a bit of the reports are still for low downloads modules such as 5 or 30 weekly and I could get busy triaging those while there are others that are more important but I will only get to later (because I'm just doing FIFO from the bucket). My suggestion is then to have Low and High priority buckets which we'd push triaged reports into based on module popularity/criticality or some other criteria that works..

2. The HackerOne team summary is helpful. Is it possible to add to it the vulnerable versions as well?

cc @reedloden

[MarcinHoppe]

I think getting priorities other than FIFO would be amazingly helpful!

💯

[mralekzandr]

Hi team - Alek from H1 here.

Having priority queues is a great idea. In an email, Liran said we can base the priority queues on the number of downloads that the modules have. Whatever methodology y'all prefer is fine with us, as long as we can clearly outline and identify that methodology to our triage team. An important thing to remember is that our triage team is still learning the lay of the land when it comes to node.js and its ecosystem. So my initial questions for the SWG:

1. Are the number of downloads clearly identifiable from the triage perspective?
2. How many downloads constitutes low priority vs high priority?

[sam-github]

Easy to see. For example, https://www.npmjs.com/package/node-red, look on right side of page, see the 16,096 Weekly Downloads.

Not sure what the threshold is, but anything under 50 is barely used, and under 10 is essentially unused. Once its in the thousands a week, its actually being used, though not very much. 10s of thousands is respectable. https://www.npmjs.com/package/express is 16 _million_ a week... lodash is 25 million, debug is 47 million.... those are the kinds of projects that bounties might be worth paying out on.

[mralekzandr]

@sam-github - thanks for that overview! Definitely seems simple enough to implement for the triage team. Would it make sense to be even more granular with the priority queues?

• 0-1000 downloads = No priority
• 1k-10k = low priority
• 10k-999k = medium priority
• 1million+ = high priority?

Alternatively, to keep it simple, we could keep it as low and high priority:

• <10k? or <50k = low priority?
• 10k+ or 50k+ = high priority?

I understand that those distinctions may be made on a case-by-case basis. But any way we can take as much guessing work or speculation away from our Triage team, the better it will be for your team overall.

I'd like to see what the rest of the SWG thinks of the distinctions; we're definitely open to suggestions!

[MarcinHoppe]

I am assuming those are not set in stone and I'd prefer to go with the simple option for now and adjust if need be.

I also think there might be value in setting a minimum number of weekly downloads for a package to even be considered a valid submission. We could include this in a program brief and it could potentially steer researchers away from finding trivial vulns in packages serving static content that people seem to produce in great quantities.

[sam-github]

If a package execs "rm -rf /", or opens control port, its worth looking at no matter how many downloads. But XSS or some shell injection possible only via direct (mis)use of the API, on a package with < 100 downloads a week should perhaps be reported directly to the maintainer. Its hard to see the value in reproducing and reporting something like that, though value is somewhat up to the people doing triage. @lirantal and @vdeturckheim , for example, should weigh in here.

[mhdawson]

I agree with @MarcinHoppe that including the thresholds in a program brief would be useful if it steers researchers towards finding more important bugs.

[lirantal]

So far my take-aways are for prioritization are:

• Number of downloads threshold: 100 downloads a week
• If a package is malicious such as doing rm -rf then regardless of the threshold its regarded as high
• We advertise the threshold in the program brief

Let's roll with this and make changes as we go? we can always find-tune and I prefer starting quick with something as reports keep flowing in as we speak.

[mhdawson]

I'd probably go with a higher threshold, 100 or even 10000

[lirantal]

100 is what I said ;-)
I think that's enough because we get a lot of reports below that threshold so let's start with this, as I also don't want to cause starvation for the low priority issues

[vdeturckheim]

👍 on a higher threshold, we could base it on package ranking (by dl). Is the goal to cover the 50% most popular modules? 80?

[lirantal]

@vdeturckheim notice we're not currently discussing what scope fo handle, we aim to handle all reports and all of npm, it's just that triage team is swamped with reports and a quick first thing to do is to make sure we don't spend the little time we have on triaging reports for a 10 d/l week module when the bottom of the queue has a report for 100000 d/l week module

Let's fix that first by establishing a threshold which I think 100 should do, and we can then fine-tune as we go along.

[MarcinHoppe]

I agree with @lirantal that we should start with an arbitrary threshold that we intuitively feel will help us focus on reports that have higher impact, and possibly fine tune it later on.

I also had a thought that reports that are along the lines of "rm -rf /" or opening up a remote control port would be classified as malware and be directly raised with the package registry operator(s).

[vdeturckheim]

LGTM! :D

[mhdawson]

@lirantal, my bad it was a typo I meant to say 1000 or 10000

[mhdawson]

But if the consensus is to start with 100 then I'm ok with that too.

[MarcinHoppe]

I think with 100 we can rule out some abandoned packages that are (too) easy target for researchers.

[0xNA]

100 sounds reasonable, and honestly too low but a good place to start. The
good thing is that even “small footprint” packages will still have high
metrics since they are depended on by the huge packages.

-na

On Mon, Sep 16, 2019 at 6:48 PM Marcin Hoppe [e-mail]
wrote:

I think with 100 we can rule out some abandoned packages that are (too)
easy target for researchers.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/nodejs/security-wg/issues/570?email_source=notifications&email_token=AFI3Q5PZEREKV4HVJVZGB2TQJ6TC7A5CNFSM4ITPMLMKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD6ZTDSQ#issuecomment-531837386,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFI3Q5JPRE3Z22DAKFL4VBLQJ6TC7ANCNFSM4ITPMLMA
.

>

Sent from a tiny device while on the move.

[lirantal]

To add context on 100: in the 7 reports that were opened in the last 24 hours, the first 5 were under 100, the 6th was 500, and the 7th was under 100 (as in 2 downloads a week, like the rest of them). So again, the 100 thresholds automatically filters out 6 of them and prioritizes the 1 that was more important to look at.

I'll go with 100 for now.

[MarcinHoppe]

@lirantal are you going to edit the program brief?

[lirantal]

@MarcinHoppe can I ask you kindly to do that? currently without my mobile to go through 2FA for H1

[mralekzandr]

@lirantal @MarcinHoppe I can make these edits today! One of our triagers actually brought this up separately this morning, without being privy to this conversation :D

Once I get settled into the office, I'll review this thread and make the necessary edits. I'll ping you guys if I have any further questions.

[lirantal]

@mralekzandr awesome thank you. To summarize so you don't need to go through everything:

1. The threshold we decided on is 100 (in the future we'll increase it) to separate between low and high priority triaging buckets
2. Update the HackerOne Program Brief of the front page

Thanks! 🙏

[MarcinHoppe]

Thanks @mralekzandr!

@lirantal I don't think I am a program admin so cannot edit the program brief anyways.

[mralekzandr]

@lirantal thank you for the synopsis!

Final question - is there a threshold in which the team isn't concerned with a report at all? For example, if it has 0 downloads or less than 10 downloads - do we still want to know about it?

[lirantal]

Nope, we always care. 0 falls under 100 so that's low prio too.
I forgot to mention that except of the 100 threshold, if there's something malicious like a package that gets downloaded 5 times/week but does an 'rm -rf' then we want it high priority too. Not sure what other examples we can give, but I hope the point is conveyed to apply some basic judgement on this being something critical for us to review.

[mralekzandr]

Got it!

I'll update our internal triage notes and policy accordingly soon.

[sam-github]

Anyone with access to H1 can see there are a set of reports that fit the < 100 criteria, I'm not clear what we should do about them.

What should we do about issues that are reported currently against packages with less than 100 downloads? Actually, what should we do about new ones reported on packages below the threshold?

Should they be closed, and the reporter told to report it directly to the package author, and to npmjs.com if the author is not responsive?

Should the H1 report be simply closed, or should it be publicized as informative?

Do we simply not care enough to do anything other than close and leave next actions up to the reporter?

I think we should agree on some kind of boilerplate response.

[mralekzandr]

Team - I was going to update triage notes, but Sam brings up some good points. Without knowing of this thread here, a HackerOne triager messaged me:

I can instruct our team to continue triaging everything normally but assigning into different priority queues, but will issues with <100 downloads actually be remediated by this group? Or is it a better flow to close as informative and send the reporter to the package author instead?

[sam-github]

will issues with <100 downloads actually be remediated by this group?

Were you under the impression that anybody in the sec-wg would actually _fix_ any issues reported on H1, or do I misunderstand what you mean by "remediate"? If so, we might have to describe the program better.

To be clear, nobody here is actually going to fix any issues in thirdparty packages (and actually, we can't, we don't have that ability to access github or npm accounts of random people). We can help facilitate communication with the author of the package, if necessary, and perhaps help the package author fix the issue if they are unsure of how to proceed.

Right now, who, if anyone, does the work of finding the package author, and notifying them of the reported vulnerability? I would assume that work would be pushed back to the vulnerability reporter (because that is an approach that scales).

[mralekzandr]

@sam-github Thanks for your direct feedback. I apologize for the incorrect terminology. I understand that this group isn't remediating the issues themselves, and I also understand the purpose of the program. I should not have used the word "remediate".

I'll defer back to everyone on this team on how you'd like to proceed. If we have concluded to simply create two priority queues, one for <100 downloads and another for >100 downloads, that is easy enough and I'll make the updates accordingly. If there are still lingering conversations to have, I'll wait to be pinged for final confirmation.

[Mik317]

Hi team :-),
I've been invited to participate in this discussion :-)

I saw your points and are quite interesting, however I'd like to say

Should the H1 report be simply closed, or should it be publicized as informative?

I'd like to remember the effort spent in find issues is anyway consistent, and report issues on packages <100 downloads is possible because the policy (at least until now), doesn't present precise points to refer to... so it would be right against the reporters to close the reports as valid, in order to avoid informative reports (they make signal minor, and this could happen because a problem in the policy, not in the report).

Also, I understand it's not really interesting triage/asses those type of issues, so a limit could be useful also to avoid "report spamming" by reporters (like me lol ... 20 reports (valid) in 3 days :)).
In this case, I suggest to to add that limit, but at the same time to apply that rules only to new reports after policy change :-)

I like also the latest idea: create 2 queues could be a good way to handle also issues in minor repositories :).

Best, Mik

[sam-github]

I'm something of a bystander here, because I'm not active in ecosystem triage, but consider this:

If the problem is that issues in unused packages are getting dealt with before packages that have users, then two queues helps.

If the problem is total amount of work triaging, then 2 queues just moves the work around, it doesn't decrease it.

[MarcinHoppe]

Right now, who, if anyone, does the work of finding the package author, and notifying them of the reported vulnerability? I would assume that work would be pushed back to the vulnerability reporter (because that is an approach that scales).

The ecosystem triage team handles this, not reporters.

[0xNA]

Is this (notifying of authors on vulns) automated? I could help with the
tooling if need be.

-na

On Wed, Sep 18, 2019 at 4:25 PM Marcin Hoppe [e-mail]
wrote:

Right now, who, if anyone, does the work of finding the package author,
and notifying them of the reported vulnerability? I would assume that work
would be pushed back to the vulnerability reporter (because that is an
approach that scales).

The ecosystem triage team
https://github.com/nodejs/security-wg/blob/master/processes/third_party_vuln_process.md#the-triage-team
handles this, not reporters.

—
You are receiving this because you commented.

Reply to this email directly, view it on GitHub
https://github.com/nodejs/security-wg/issues/570?email_source=notifications&email_token=AFI3Q5MV2XGWNRQJDVYJTRLQKIT3FA5CNFSM4ITPMLMKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7ABPJQ#issuecomment-532682662,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFI3Q5MAGUIDMW5BP3KZ5R3QKIT3FANCNFSM4ITPMLMA
.

>

Sent from a tiny device while on the move.

[MarcinHoppe]

It's not automated. Sometimes findings contact info for maintainers is a bit challenging, i.e. it is not easily available on npm or GitHub.

[sam-github]

it could be easy on github if people fill in the sec policy, but most projects don't. example: https://github.com/nodejs/security-wg/security/policy (not that this wg needs a policy)

similarly, npm points to github, or can, and from there the policy is easy to find

But even with a policy, its not very automatable.

[sam-github]

With the help of the individual who reported the vulnerability, they contact the maintainers of the vulnerable package to make them aware of the vulnerability. The maintainers can be invited as participants to the reported issue.

• https://github.com/nodejs/security-wg/blob/master/processes/third_party_vuln_process.md#correction-follow-up

@lirantal Does the threshold of 100 affect this? If so, the docs will need updating.

[0xNA]

I can build some tooling that can get the emails through some “Github
mining”. Who should I follow up offline on this?

-na

On Wed, Sep 18, 2019 at 5:57 PM Sam Roberts [e-mail]
wrote:

With the help of the individual who reported the vulnerability, they
contact the maintainers of the vulnerable package to make them aware of the
vulnerability. The maintainers can be invited as participants to the
reported issue.

-
https://github.com/nodejs/security-wg/blob/master/processes/third_party_vuln_process.md#correction-follow-up

@lirantal https://github.com/lirantal Does the threshold of 100 affect
this? If so, the docs will need updating.

—
You are receiving this because you commented.

Reply to this email directly, view it on GitHub
https://github.com/nodejs/security-wg/issues/570?email_source=notifications&email_token=AFI3Q5I5CYH5IPM22DXH573QKI6VTA5CNFSM4ITPMLMKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7ALQGI#issuecomment-532723737,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFI3Q5L4ERVWLRTLZDTPPBDQKI6VTANCNFSM4ITPMLMA
.

>

Sent from a tiny device while on the move.

[mralekzandr]

Hi everyone,

Just circling back here - have we agreed to have any containers with >100 downloads to be "high priority", and anything below will be "low priority"?

[MarcinHoppe]

Yes, I think this was the agreement.

[mralekzandr]

Hi All,

I have created two inbox groups/views for our Triage team to begin assigning reports to. As new reports come in, H1 Triage will assign it to the respective 'group'. Once assigned, the report will live in the corresponding Inbox View

• High Priority = >100 Downloads
• Low Priority = <100 Downloads

The inbox view will show all OPEN reports assigned to each 'group'. Once a report is CLOSED (resolved, informative, n/a, etc), it will be removed from the queue.

H1 Triage has already been informed of this change, so all new reports should be filed accordingly. If you have any questions or concerns, please don't hesitate to reach out to me directly. Email is best and generally gets the fastest response- [email protected].

Thanks!
-Alek (H1 Program Manager)

[lirantal]

Thanks Alex, sounds great and I can confirm I see it on my H1 inbox.
Appreciate it!