Amphtml: I2I: Consider switching AMP package management from Yarn to NPM

[rsimha]

Background

AMP currently uses yarn for package management, and lists it as a requirement in the developer docs. Yarn was originally introduced in 2016 because it provided a means for version pinning, which is critical to ensuring that the same environment is used by all developers and during CI.

Since then, NPM has introduced its own built-in mechanism for version pinning. Initial efforts to eliminate the additional tool (yarn) were abandoned because npm's lockfile was found to be inconsistent across OSs.

Meanwhile, Yarn's installation has become less reliable over time, with frequent failures on Travis and during local development (#15913, #23461). More recently, the latest version of yarn had issues like an invalid GPG signature and a version skew between github and npmjs.

Actions

This issue is to study and track a potential switch back to NPM.

• [x] Investigate npm to see if past OS inconsistencies still exist
• [x] List the steps involved in switching back to npm
• [x] Understand the impact on the developer workflow (Will it just work? Or do devs need to uninstall stuff?)
• [x] Discuss this during design review and get approval from wg-approvers
• [x] Send out a PR to implement the switch
• [x] Co-ordinate PR changes with changes to AMP's internal release automation with help from wg-infra
• [x] Merge all changes in the correct order

/cc @ampproject/wg-approvers @ampproject/wg-infra @kristoferbaxter

[kristoferbaxter]

Another factor to consider is yarn 1 is now EOL. 1.22.10 is the final version according to its developers, and yarn 2 is a significant departure from the original design.

We're looking at a likely forced migration to either yarn 2 or npm if another breaking issue is found with yarn 1 for our repositories.

[rsimha]

Here are the results of initial investigations into the transition path from Yarn 1 (a.k.a. Yarn Classic) to NPM:

• OS level inconsistencies: As of today, the AMPHTML repo's package.json and package-lock.json files remain consistent across OSs (tested with macOS and Linux). The OS-level back-and-forth for optional dependencies appears to have been fixed.
• In-place migration: It is possible to upgrade node_modules/ for a local checkout from yarn to NPM without having to blow away the directory. Pulling down the latest changes will automatically switch everyone from yarn.lock to package-lock.json, and the subsequent first-time invocation of npm i will update the contents of node_modules/. Desired settings (e.g. exact version pinning) can be enforced by checking in an .npmrc file to the repo.
• Globally installed tooling changes: Developers who have globally installed yarn will not be required to uninstall it. However, we will need to update our pre-install check to enforce the use of npm install instead of yarn install. This can be turned on after letting the community know that this change is upcoming.
• Integrity checks: We currently use yarn check --integrity to ensure that everyone is running with the desired package versions after pulling down upstream changes. While NPM doesn't have the exact same command, it does have npm outdated, which lists "current" and "wanted" versions of all packages. It should be simple enough to write a check to make sure they match up, and if not, do an npm install.
• Installation warnings / errors: Yarn and NPM surface installation warnings and errors in different ways. By default, npm is more verbose with its errors. For example, this error while installing node-gyp, a transitive dependency, on macOS. However, harmless errors can be silenced by setting loglevel=error in .npmrc.

At this point, here are the options in front of us, with tradeoffs:

Option A: Do nothing. This means we will be stuck with the currently version of yarn v1, which has reached end-of-life. There is a non-zero risk that any incompatibility with a future package version will break overall development.

Option B: Migrate from yarn v1 to v2. While possible, this is not a straightforward migration. The OSS community is split about whether the upgrade is worth it. There is a significant deviation from the set of commands in v1, which will require a learning curve. In addition, at least one command we rely on has been deprecated.

Option C: Migrate from yarn v1 to npm. This is the plan proposed by this I2I. There is a non-trivial migration involved, but it can be done at the infrastructure level (see above points). There will be a learning curve, where developers will need to be reminded to run npm install instead of yarn install during first-time install. (Subsequent installations are automatic.) A new integrity check will have to be written in order to ensure that developers have installed the desired package versions after pulling down upstream changes. However, there is the huge advantage of no longer having to maintain, check, and warn for versioning of two separate tools (npm and yarn), since npm is built-in to node, and pinning development to the latest active LTS version of node will automatically guarantee versioning.

This I2I can be implemented once 3 members of @ampproject/wg-approvers reply in the affirmative.

[kristoferbaxter]

Could we also discuss this during the next design review?

[rsimha]

This was discussed during design review. Option C received a thumbs up from @kristoferbaxter, @dvoytenko, and @jridgewell among others. (Please confirm with a 👍 reaction to this comment.) Implementation has begun and is expected to land in the next couple of days.

[dvoytenko]

LGTM from me.

[rsimha]

With #30694, this transition is complete.