Build: Automatic installation of plugins makes build much slower

[ehmicky]

When a plugin is used in netlify.toml but is not inside package.json, we automatically install it during the build. This creates two performance issues:

1. When user first install a plugin, they just look at their total build time and see that it increased. They then remove that plugin due to that increase, without trying another build. For caching plugin for netlify-plugin-gatsby-cache, they think caching is not working properly since it increases build time.
2. Since we cache node_modules and we only install plugins when calling require() fails, the next build does not have that additional install time. However each time the user changes their dependencies, devDependencies or lock file, the node_modules get re-generated, and the additional automatic install time gets added to the build.

Another issue suggested by @fool on Slack was that npm gets outages from time to time, and that would make builds fail.

Note that this does not mean we should not automatically install plugins, but more likely that we should have a better caching strategy that's both fast and resilient to npm outages.

[ehmicky]

One possible solution: at the moment:

• first, the buildbot is running npm install (if a package.json is found)
• then, Netlify Build runs npm install netlify-plugin-exampleOne netlify-plugin-exampleTwo etc. for every plugin that cannot be require()'d (i.e. that needs to be automatically installed)

We could group those into a single npm install to optimize both network parallelism and caching. We could do this by modifying the package.json to add plugins as devDependencies before running npm install in the buildbot. The list of plugins to add would be returned by @netlify/config. The package.json modification would not be git-committed. But since it would happen consistently on every build, it will always be cached and restored with the other node_modules.

We could do either do this inside the buildbot, or we could move the npm install logic from the buildbot to Netlify Build. We do want to do this eventually for the whole install_dependencies function since it allows us to provide onPreInstall logic to plugin authors, which some have been asking for in Netlify Community.

Note: we would need to make sure npm install is using user's preferred Node.js version.

When using Netlify CLI, we would still use the current way. We probably should consider adding the plugins to the package.json with a * version range (to match current production behavior) so that local builds are not affected by the same speed issue mentioned above?

Implementation note: this would need to work with different use cases (yarn, no package.json, etc.).

So I suggest the following action points:

1. Move the npm install logic from the buildbot to Netlify Build.
2. Before running npm install, add all plugins as devDependencies (unless already there)

What do you think?

[mheffner]

Move the npm install logic from the buildbot to Netlify Build.

Probably a good idea in general to do this to remove more logic from buildbot?

Before running npm install, add all plugins as devDependencies (unless already there)

If there's an error running the npm install, will we be able to differentiate the error from user error (bad package.json?) vs. a plugin install error?

[fool]

this sounds great from a performance point of view, and somewhat horrifying from a debugging/troubleshooting point of view: making invisible changes to customer files seems like it could be quite problematic.

"here's my package.json, it works, but not in netlify build. WTF?".
"huh, well, I can't see exactly what we use for the build since we change it at build time, so, it's hard to advise you".

I'm not solidly opposed to it, but I would like some debugging/transparency help, for instance:

• explicit loglines showing "added to the end of your package json: package: ^ver" (and different lines for yarn.lock)
• very thorough docs that explain that we do it.
• maybe even saving the file as part of the deploy, but not serving it, like we do with netlify.toml or _redirects? This way we/acustomer could download and examine it.

[ehmicky]

The fact that this is added to package.json would be more of an implementation quirk to get around the following issue: it is not possible to run both npm install (install all package.json dependencies) + npm install dependency... (install new dependencies) at the same time. At the moment, we run two separate commands, which leads to the issues above. Changing the package.json would be ephemeral, we could (should?) even undo it after npm install is done.

Ideally, if this quirk worked fine, users would not need to know about it. For example, if npm was adding a new option so we don't need to do this workaround anymore, we could use it without users noticing the difference.

That being said, I agree with @mheffner that plugin installs errors might be confusing to users. And with @fool that if things don't go as planned in a specific build, this might create debugging problems and confusion.

The alternative would be to install plugins as a separate step (which is what we do now) but optimizing it to prevent running into the caching issue mentioned above. For example, installing them in a different node_modules which would be separately cached. The buildbot could pass the path to the plugins's node_modules directory to Netlify Build so it knows where to find them.

One potential issue with this last solution would be for plugins that require dependencies that are in the repository's package.json. For example a ESLint plugin that calls the user's ESLint as a peer dependency, it would need to find it in parent directories' node_modules. This could be fixed though by ensuring this separate node_modules is located within the cloned repository, for example inside /opt/build/repo/.netlify/plugins/node_modules/?

This last solution might end up being faster in some cases actually since this would allow npm ci (instead of npm install) to be used when plugins changed but not dependencies, or vice-versa. Please let me know if you think this is a better solution! :)

[verythorough]

One small thing to consider if going the route of adding to an existing package.json to install plugins: not all repos will have a package.json, so the process would have to account for creating one (or just running the npm install xyz command) if one doesn't already exist.

[erquhart]

We could pass the path to a plugin's node_modules to require.resolve(), so we wouldn't need to mess with package.json, and peer dependency resolution would still work.

One caveat would be that this should not happen when requiring modules from the main project, only when resolving modules imported by the plugins themselves. This would help ensure against weird edge cases that could result from having an unexpected second set of dependencies available.

Not certain if this approach introduces other problems. None come to mind, but we should think it through.

[ehmicky]

@erquhart, yes I think this is what I was suggested in the second part of this comment:

• install all plugins inside .netlify/plugins/node_modules
• use resolve to locate the plugin main file
• always cache .netlify/plugins/node_modules, using the same logic we already use to cache node_modules

When the plugin itself calls require():

• it will receive the dependencies declared in its package.json (like any regular Node module)
• it will receive the repository's dependencies when requiring a peerDependency (like any regular Node module)

[mheffner]

Related discussion of trying to cache NPM installs internal to our build infra: https://github.com/netlify/infra/issues/55

[ehmicky]

I just realized the following additional problem: we want users to always use the latest version of plugins. This would imply every build should run npm install to check for new versions of all the plugins a build uses. Even if the plugins are already cached in a node_modules directory and their version has not changed, doing that npm install to check for new versions can add up to 30 seconds on every build.

[ehmicky]

Something to keep in mind about plugin automatic installs.
We seem to have quite many errors happening on npm install. Check the number of issues already: #1202, #1197, #1198, #1199, #1178, #1175.

I am currently investigating it as some of those might be bugs, but it seems to me, generally speaking, running npm install on behalf of users is quite tricky:

• Users have different configurations: npm vs yarn vs pnpm vs the next version manager. Different versions of those (NPM_VERSION). Different configurations (NPM_FLAGS).
• npm network is not completely reliable. @netlify/build tests randomly fail due to that actually, but also user builds.
• Generally speaking dependencies installation is a rather complex issue due to the number of use cases

We should find a solution that is very simple. Otherwise we will have to maintain lots of future bugs related to plugin automatic installs.

[ehmicky]

Added a summary of this issue in Notion.

[kaganjd]

Adding reports from Community here:

• https://community.netlify.com/t/build-stuck-at-start-saving-pip-cache/13948
• https://community.netlify.com/t/netlify-plugins-with-netlify-plugin-gatsby-cache-increases-build-time-not-reduce/14538

[mheffner]

Looks like some of those are having trouble saving the pip cache.

[ehmicky]

This issue should be fixed by #1280 and #1281.

[slorber]

Hi,

As @kaganjd reported my "saving pip cache issue" here https://community.netlify.com/t/build-stuck-at-start-saving-pip-cache/13948

I gave it another try just now, here's the deployment: https://app.netlify.com/sites/sebastienlorber/deploys/5ebe9b893ef93ea49b5d10d7

First the cache plugin takes 4min to install, that seems huge to me.

Then, it's still stuck at the "saving pip cache" step at the end (generally until 15min timeout), despite not using any Python.

My site works fine with build plugins disabled. It's a simple Gatsby site with 4 blog posts.

Edit: ho, actually it didn't timeout this time! But it was close to. I don't think it's normal to save pip cache for 5 minutes on a site not using Python:

3:48:21 PM: Started saving pip cache
3:53:48 PM: Finished saving pip cache

I'm now a paid Netlify user, because I'm unable to deploy my 4 posts blog efficiently :D

[ehmicky]

Hi @slorber,
We have identified the source of the problem and are working on solving it (see comments above).

[slorber]

Thanks, I actually thought the fix was live ;) tell me when to test again then

[ehmicky]

More work on this done in #1292 and #1282.

[erezrokah]

Another issue with automatic installations related to https://github.com/netlify/build/issues/1137#issuecomment-620060028 is that always installing the latest version prevents reproducible builds.

[ehmicky]

Good point @erezrokah, I have added an issue in #1294.

[ehmicky]

Closing this: plugins should now be pre-installed before builds start in production, meaning they should add 0s installation time in builds.

If the plugin is not one of the officially supported plugins, it won't be pre-installed though. In that case, it should be added to the Site's package.json.