Pnpm: Support audit

[jiripospisil]

There's a new command available called "audit" since NPM [email protected] and npm@6. It requires the presence of both package.json and package-lock.json meaning it cannot be just a pass through. It would be great if PNPM supported it as well.

This version of npm adds a new command, npm audit, which will run a security audit of your project's dependency tree and notify you about any actions you may need to take.

As part of this change, the npm CLI now sends scrubbed and cryptographically anonymized metadata about your dependency tree to your configured registry, to allow notifying you about the existence of critical security flaws.

https://blog.npmjs.org/post/173719309445/npm-audit-identify-and-fix-insecure

[croqaz]

One quick idea that pops into my mind from this: https://stackoverflow.com/questions/37078968/how-to-specify-the-path-of-package-json-to-npm

If the pnpm package would be converted into package.json into a temp file, npm can be launched pointing to that and let it do its thing. Not sure if this is the desired behaviour, or it should be included inside PNPM.

[ExE-Boss]

pnpm has its own lockfile format in the form of pnpm‑lock.yaml which is incompatible with npm’s package‑lock.json.

Meaning this will have to be implemented as part of pnpm.

[milaz]

You can perform an audit by running yarn audit in a package with pnpm directory structure and no package-lock.json. I hope this information might be helpful in reducing the amount of work needed to implement this feature in pnpm.

[ExE-Boss]

I’ve been working on understanding the npm audit algorithm.

2014 has been done as a pre‑requisite for this (so that I can have type information when working on this).

[shellscape]

Tossing in my support for this capability. npm audit is presently a major part of our CI pipeline. Nothing gets published when there are alerts for a dependency.

[zkochan]

@ExE-Boss do you work on this? (if you have PayPal, we can award you with some money from the OpenCollective)

[Bessonov]

Missing audit is a blocker for us too.

I don't know if it helps, but here is yarn audit.

[zkochan]

I was contacted by a developer from snyk. He says we can use their open API to do security audits. Any objections to use Snyk?

[shellscape]

I replied on twitter to their devrel on the thread. Here's the gist of my thoughts:

I think that'd be a good solution. However, the discrepancy between Snyk, npm audit, and Github Alerts is something to take a look at. I consistently get more alerts from the latter two than I do from Snyk. Lots of silly alerts, low level alerts that aren't a real threat to anyone not hacking themselves, but the alerts will still show for anyone using npm to install modules. there should be parity between Snyk's audit results and npms, if pnpm is to leverage the service.

[lirantal]

@shellscape I saw your tweet and replied to it just now. Like I replied, I highly doubt those discrepancies with regards to Snyk not having that data and majority of the time this is the other way around.

There are generally two reasons you may think you find less issues with Snyk:

1. Snyk by default doesn't scan devDependencies in order to reduce the noise. You can pass a flag to scan everything.
2. Snyk reports a specific vulnerability identified by a CVE only once, where-as npm audit reports any occurrence of it. Taking lodash for example, it is easily used within hundreds of nested modules in a typical JS project. npm audit might tell you found 491 vulnerabilities where-as Snyk might just report 1 of them because all of those 491 vulnerabilities are the same lodash vulnerability which is simply introduced via different modules, which snyk calls "vulnerable paths". For example:

On top of that, Snyk is actively performing security research and discloses security issues earlier than a CVE is obtained, which for open source may be often times very late or never. Some examples off the top of my head:

1. Sequelize, a very known and popular SQL ORM for Node.js in which Snyk's security research team found multiple issues and worked with the maintainers to responsibly disclosed them, however more importantly it took 16 days for npm audit to catch up with an SQL Injection Snyk published in August 2019.
2. Angular (1.x) has 21 vulnerabilities, and React has a couple too, none of which is being reported with npm audit. Angular 1.x btw still pulls about 30% of all of Angular (old and new) downloads so that's about 3 million of those.

full disclosure: as I was writing all of this now seems to be like a pitch so I want to explicitly make clear that while I'm involved with the Node.js Security WG I'm also a developer advocate at Snyk. None of this is of course meant to be any marketing but rather to share about the usefulness of Snyk's database in response to @shellescape's concerns.

[zkochan]

cc @pnpm/collaborators

[shellscape]

(I'm good with two channels, it can only be informative at the end of the day)

Snyk by default doesn't scan devDependencies in order to reduce the noise. You can pass a flag to scan everything.

I've been a Snyk subscriber since the service was announced, and I've never known about this. That could be my fault as a user, but I've only ever wanted the service to tell me what's wrong, so I've not deep dived the settings. Maybe that's somewhat telling, in the sense that it could probably be more widely mentioned so that users are in the know.

FWIW, I just went through every single setting I could find on snyk.io and could not find the setting to enable devDeps for any repos or orgs. Please point me in the right direction if I've missing something obvious.

Snyk reports a specific vulnerability identified by a CVE only once, where-as npm audit reports any occurrence of it. Taking lodash for example, it is easily used within hundreds of nested modules in a typical JS project. npm audit might tell you found 491 vulnerabilities where-as Snyk might just report 1 of them because all of those 491 vulnerabilities are the same lodash vulnerability which is simply introduced via different modules, which snyk calls "vulnerable paths".

These are also important distinctions that have gone unknown.

I think this is all very important information, but there is still the matter of perception. My passion are tools for the developer experience, and I can tell you without equivocation that after years in the space, public and private, that perception is reality for users. A large discrepency between numbers of the two tools will require _excellent_ UI explaining the differences, or a straight (close to) parity in displaying results. Leads and other decision makers will want to be comfortable that a change to pnpm's audit capabilities are on parity to that of npm's audit. And that will come down to the information displayed and how it's presented. We'll need the capability to say that there are 491 vulnerabilities, even if subsequently displaying only the 1 offending dependency.

The choice to skip devDeps by default is curious - my experience has been that maintainers are rather rabid about making sure that their dev env stays up to date and doesn't contain any vulnerabilities. I'm sure this has been thoroughly debated and vetted, but it would seem more prudent to provide maximum info and allow that to be trimmed or pruned (anyone who deals with process logs would agree). If/when pnpm starts utilizing Snyk, I do hope it goes in that direction by scanning devDeps by default.

I also want to make sure you all understand that I am not opposed to pnpm using Snyk whatsoever, nor am I attempting to be difficult. I'm only after the absolute best DX for pnpm.

[zkochan]

I created an exploratory poll

https://twitter.com/pnpmjs/status/1184439771998896128

But I will prefer the decision of the core contributors.

[etamponi]

I don't work for Glitch anymore, but I think pnpm audit would be a great addition for them (if they decide to upgrade to the latest pnpm...).

Personally, I think the same, pnpm audit would be a great addition :)

... and I am fine using Snyk :)

[nikoladev]

I agree with what @shellscape is saying. Switching to Snyk is great as long as we meet the following conditions:

• devDependencies should be scanned by default
• Perception is everything. The number of found vulnerabilities should be about the same as with npm audit (or even larger). So something like this would be the preferred output: found 100 vulnerabilities, of which 98 are duplicate occurences

[Bessonov]

Did I understand correctly, that snyk send ONLY packages and versions to the servers?

As I can see, Snyk offers limited api for non-Open Source projects. That's not suitable for CI.

[raihle]

Snyk's limitation of 200 monthly scans (for private projects) would kill it for us.

[shellscape]

I was operating on the assumption that @lirantal, as an employee and advocate for Snyk, was offering to grant pnpm some allowances around those restrictions. Before we assume that we'd be locked into individual-contributor limitations, let's allow them to weigh in.

[lirantal]

Replying to @shellscape (thanks for your great feedback, I shared with the team ❤️)

Snyk by default doesn't scan devDependencies in order to reduce the noise. You can pass a flag to scan everything.

I've been a Snyk subscriber since the service was announced, and I've never known about this.

Thanks, I've shared this internally with the team and we'll work on making this more obvious.
With the CLI, to test devDeps too you can use $ snyk test --dev. With the UI, if you're connecting your git repository then we scan everything by default and devDeps aren't missed. Plus, we're also able to open automatic fix pull requests, as well as generic package upgrades when they are relevant, so that's the more recommended and pro-active security fixing approach.

The choice to skip devDeps by default is curious - my experience has been that maintainers are rather rabid about making sure that their dev env stays up to date

I can understand your point of view, but many times developers and maintainers will see these as noise. For examples, a denial fo service, prototype pollution or other vulnerabilities reported in packages like tap, jest, prettier will just be ignored because they aren't shipped with the app and we found more often than not that maintainers wave these off.

--

With regards to vulnerable paths vs number of vulnerabilities:

We'll need the capability to say that there are 491 vulnerabilities, even if subsequently displaying only the 1 offending dependency.

It's a personal opinion of course, but it makes sense to me to count distinct vulnerabilities as unique, even if they show up via different paths. We report both numbers in the CLI and you can see how it looks in the UI too:

To me, it is often times absurd that npm audit will report 16,000 vulnerabilities for a project with heavy lodash usage, when the whole entirely of public vulnerabilities in the npm ecosystem is about 2k. Maybe it's just my point of view though.

--

Love your input @shellscape, thank you again!

[lirantal]

@etamponi @shellscape @nikoladev thank you all for your vote of confidence and input.

With regards to how pnpm will utilize Snyk's database - let's discuss and see what makes sense the most for pnpm and the community. We're happy to help and make available as much as we can (I'll need to iron out the details with the team internally but we'll figure it out).

If someone wants to take a stab at it - the most immediate action item pnpm could take is simply to bundle the snyk CLI as a library (https://www.npmjs.com/package/snyk) and invoke it to scan a given project with whatever flags required (i.e: the aforementioned --dev for devDeps and also --json so it can then be parsed and presented in a custom way). This integration would require a one-time setup for a user to get a snyk account from which they can also then share/create their tokens for use in CI etc. This is probably a couple of days worth of work that will get pnpm audit fully functional.

While that is a quick start, let me follow-up with a plan to share on how to further work this out with perhaps making an API available, although that would incur more work on the client side, to then build and associate the dep tree of a project with the vulnerabilities that were found. Happy of course to hear other ideas.

[aparajita]

Why not support snyk and npm pass through? One will be a default, one or the other can be specified with an option.

[ExE-Boss]

We can’t just do npm passthrough, as the npmjs.com API requires submitting a package‑lock.json‑like file.

This means that we’ll have to create something to convert pnpm-lock.yaml to a package‑lock.json compatible format.

After that, it’s just a matter of submitting the result to the REST API and we’re pretty much done.

[lirantal]

Maybe we can have a snyk passthrough so that the default will be based on the integration we create and if a switch is turned on it will use the full capabilities with the user's snyk account etc.

[lirantal]

What do we think about:

the most immediate action item pnpm could take is simply to bundle the snyk CLI as a library (https://www.npmjs.com/package/snyk) and invoke it to scan a given project with whatever flags required (i.e: the aforementioned --dev for devDeps and also --json so it can then be parsed and presented in a custom way). This integration would require a one-time setup for a user to get a snyk account from which they can also then share/create their tokens for use in CI etc. This is probably a couple of days worth of work that will get pnpm audit fully functional.

[shellscape]

I think that's reasonable as a bystander, though maintainers will have to weigh in on project impact. What limitations are imposed by snyk using the snyk CLI?

[lirantal]

With regards to vulnerability scanning: no limits really when you are using it for open source projects. If you'd say run a test on a private project we have a scan limit of 200 tests.

[zkochan]

I don't have a clear understanding of whether using snyk will give our users the same functionality that npm audit provides, for free.

I also don't know what npm audit currently provides for free. I did not use that feature and I did not find any page that describes the feature in detail.

[zkochan]

I think @verdaccio also implements the same API that npm audit uses.

Maybe the right solution would be for snyk to implement an API that expects the same input as the npm registry's API. In that case we could just make a config for a security audit endpoint. And if snyk provides more functionality, snyk's endpoint can be the default value.

[Bessonov]

I also don't know what npm audit currently provides for free. I did not use that feature and I did not find any page that describes the feature in detail.

Did you see this pages?

https://docs.npmjs.com/cli/audit

The audit command submits a description of the dependencies configured in your project to your default registry and asks for a report of known vulnerabilities.

https://blog.npmjs.org/post/173719309445/npm-audit-identify-and-fix-insecure

npm audit is a new command that performs a moment-in-time security review of your project’s dependency tree.

https://docs.npmjs.com/reporting-a-vulnerability-in-an-npm-package

If you find a security vulnerability in an npm package (either yours or someone else’s), you can report it to the npm Security team to help keep the Javascript ecosystem safe.

https://stackoverflow.com/questions/55569305/how-npm-audit-works

There is no algorithm. Only people.

What npm audit does is look at what package you are using and what version and compare it to npm's vulnerability database. Here's the web interface to that database: https://www.npmjs.com/advisories

Maybe the right solution would be for snyk to implement an API that expects the same input as the npm registry's API.

Full ack.

[nikoladev]

the most immediate action item pnpm could take is simply to bundle the snyk CLI as a library (npmjs.com/package/snyk) and invoke it to scan a given project with whatever flags required (i.e: the aforementioned --dev for devDeps and also --json so it can then be parsed and presented in a custom way). This integration would require a one-time setup for a user to get a snyk account from which they can also then share/create their tokens for use in CI etc

As an end user I'd want to have the process similar to npm audit or yarn audit, ie. it should be a simple command that I can immediately run. Making the user sign up for an account and set up tokens (even if it only has to be done once) makes the process more complex and makes it less likely for the user to complete process (and therefore prevents the user from being able to run audits). This would be a drawback for pnpm's implementation compared to yarn's or npm's.

Though between the choice of having no audit or a complex audit, I would still prefer to have audit functionality.

[aparajita]

Making the user sign up for an account and set up tokens (even if it only has to be done once) makes the process more complex

Javascript development is already way too complicated. pnpm should make life easier, not harder.

[octogonz]

How does Yarn do it, given that their lockfile is a propriety format?

[zkochan]

Yarn creates a npm-compatible lockfile and posts it to the audit endpoint. This is what we will also do.

[zkochan]

A PR is in progress #2153

(funny, the PR ID is the issue ID + 1000 :smile:)

[zkochan]

:ship: 4.3.0-0

[zkochan]

In case of issues or feature requests, please create new issues.

[abhijit-hota]

Is there any way to fix the audit vulnerabilities?
Maybe something like pnpm audit fix ?

(P.S. Reading this thread was really insightful! 😄)

[zkochan]

There is no such feature at the moment. Probably running pnpm update --depth Infinity should fix most of the issues. And with pnpm update --interactive --latest, you can update the direct deps to newer major versions

[abhijit-hota]

@zkochan Thanks! Fixed them. (Updated my direct dependencies)