Build-image: Support for monorepos, including lerna or yarn workspaces

[herkulano]

Currently, you're only caching the node_modules at the root level.
With lerna or yarn workspaces, there are sub-folders with node_modules in them that are not cached. This causes a failure to build. It then builds correctly if it's retried with a clean cache.

[fool]

related internal issue: https://github.com/netlify/buildbot/issues/163

[zanona]

For my use-case (yarn workspaces) I have been able to work around this issue by using:

netlify.toml

[build]
  command = "npm i [email protected] && yarn && yarn build"

Of course, this is not ideal as it takes additional time to run npm install and yarn install again, however, the cache dir which the npm installed version of yarn uses is the same so it should be all ready for the next time a build is triggered. Bottom line, it has increased building times by about 1 minute for me.

Hopefully, we will be able to revert to the expected behaviour soon, though :)

Btw, thanks for the help @fool

PS. Perhaps it's good practice to call yarn install with yarn $YARN_FLAGS as it should carry netlify or user defaults for installing the dependencies, such as --production

[eckdanny-osi]

I had a frustrating afternoon playing guess-and-check with netlify builds on my yarn workspaces + lerna repo. I'm 99% sure its todo with netlify's caching scheme for node_modules since my build goes from failing to working if i just clear the cache. I'd don't want to have to do this _everytime_ tho... looking for a config option or ENV var to the effect of NO_CACHE or lerna clean and re-install as part of the build process for now :\

@zanona you can set a build env var YARN_VERSION.

I think why its working for you is actually the && yarn part where you reinstall deps.

_UPDATE_:

can confirm. I had a good deploy. Then I did a no code change (updated a README). Build failed. Redeployed with the "clear cache" btn and it worked. For now, ~I guess I'll put a lerna clean && lerna bootstrap into my build routine~ i dunno what todo

[eckdanny-osi]

Strange. I have a YARN_VERSION env var in my web config (set to 1.9.4) and the build logs seem to indicate its working...

but i changed my package.json#scripts#build to "build": "yarn -v && exit 1" and the result is not expected:

...
6:32:06 PM: Executing user command: yarn build
6:32:06 PM: yarn run v1.9.4
6:32:06 PM: $ yarn -v && exit 1
6:32:06 PM: 1.3.2

The issue I'm seeing in my dead builds:

error An unexpected error occurred: "patterns.map is not a function".

was resolved by https://github.com/yarnpkg/yarn/pull/5114 and should work since yarn v1.4

~I'll try @zanona's workaround.~

Why do i not have the yarn version i think i should have?

_UPDATE_:

@zanona's workaround solved for now. I added a toml:

[build]
  command = "npm i -gs yarn && yarn build"

and my pkgJson#scripts:

{
    ...
    "prebuild": "lerna bootstrap",
    "build": "lerna run --stream build"
}

[iddan]

I expect Netlify to be able to look for different Netlify.toml files: So for deploying a monorepo with multiple sites I would create multiple Netlify sites each of them with a different Netlify.toml.

[tsaiDavid]

Just curious if anyone has [since] had luck with caching non-root node_modules?

[praneybehl]

I am in the same boat, even though I try to bootstrap lerna before the build command it still not able to...

[lucashfreitas]

+1

[jaredh159]

I also would really like to drop a netlify.toml file into my base dir, and have multiple of them for my monorepo. I was a bit surprised it didn't work like this already. :(

[marcosscriven]

Any news on this please @fool? In the meantime, please could you update the documentation to make it clear you can only have one netlify.toml for the whole repo, even though most other things work fine by per-site config in the UI.

The big thing missing for me is the greater redirect control one gets with the netlify.toml config, but that's not common to my sites in each of the sub directories.

I don't really even need to share anything, so presumably you could make each site build only aware of a given sub-dir, as if that's the root of the whole thing - then everything else would work as-is?

[backspace]

I’m working on a PR that builds both a static documentation Middleman site and an Ember UI. Part of the script involves checking what branch is being deployed and using a different _redirects file if it’s a UI PR, so maybe that would be useful to people.

But the caching is my concern, it doesn’t seem like it can be hacked around in this manner 😞

[chrfritsch]

Netlify now supports netlify.toml files per subdirectory in a monorepo.
See https://www.netlify.com/docs/continuous-deployment/#build-settings

But still, I am not able to set it up that the npm/yarn cache is working properly.

[seanemmer]

Congrats on netlify.toml support.

I am having an issue where, when using yarn workspaces, yarn installs all of the monorepo packages. This is problematic because yarn also ends up running package-specific postinstall scripts that break my build. I would like to do something like this to take out the offending package:

rm -rf packages/api && yarn install && yarn workspace provider build

But Netlify does not seem to be honoring the rm -rf command.

[rajington]

for monorepo support where there's a single root yarn.lock, we're currently making our site-specific netlify.toml build command be cd ../; yarn; cd -; yarn build

it might be easy to let us also specify the install command in netlify.toml so we don't double install deps

[build]
  install = "cd ../; yarn"

or if we could have access to some site-specific preset env variables (or the site name) in netlify.toml. not really toml anymore but, toml might be getting to m...
netlify.toml

[build]
  command = "yarn build-$SITE_NAME"
  publish = "apps/$SITE_NAME/"

[context.master]
  command = "yarn build-$SITE_NAME-dev"

or something like context. but for sites instead of branches, so we can use a single all-knowing root

[sites.foo]
  command = "yarn build-foo"
  publish = "apps/foo/"

  [sites.foo.context.master]
    command = "yarn build-foo-dev"

[dmgawel]

I experimented with Yarn Workspaces and Netlify's recently released better support for monorepos (netlify.toml per site in subdirectory). I discovered a setup that might solve some of the above mentioned problems.

Assumptions:

• You use Yarn Workspaces with more than one site (or one site in subdirectory),
• You set your base dir in Netlify UI to e.g. packages/website to use per-site netlify.toml file
• You want Netlify to install dependencies using yarn (without additional commands, changing directory or installing first with NPM and then with Yarn)
• You want Netlify to cache dependencies between builds

So, here's the fun part. Based on facts that:

• You can run yarn install in a Workspace directory instead of project root
• Netlify requires yarn.lock to exists in a base directory to use Yarn instead of NPM

You can set up your project like this:

├── packages
│   ├── frontend
│   │   └── index.js
│   └── website
│       ├── index.html
│       ├── main.js
│       ├── netlify.toml
│       ├── package.json
│       └── yarn.lock    # Empty file to trigger yarn install
├── package.json
└── yarn.lock            # Real yarn.lock

And Netlify will pick up your netlify.toml and install dependencies with Yarn 😎

Unfortunately, this does not eliminate the problem with proper caching of node_modules/. Fortunately, Yarn uses local cache directory that Netlify persists between builds. That speeds up things a bit. But for the issue to be fully resolved we have to wait for a native solution in build image. Alternatively, we could use Netlify Build Plugins to hook into getCache and saveCache and use /opt/build/cache/ but it's just an idea ;-) I'm happy to test it, I just signed up for Plugins Beta.

I hope this will help someone :-)

[rajington]

neat trick! trying it now and will update when i get it working!

[slawekkolodziej]

FWIW I managed to properly cache lerna packages between the builds. I created a minimal example here: https://github.com/slawekkolodziej/netlify-lerna-caching-demo As for Netlify Build Plugins, it seems that they are executed AFTER the yarn install, plugin won't help here (at least with currently available API)

[iddan]

Can this be integrated to the default image?

[jaredh159]

@slawekkolodziej really appreciate you creating that minimal demo. really clever work there, seems super promising. I'm going to give it a shot shortly, as I'm still struggling with long build times and churning through lots of Netlify build minutes.

[vincerubinetti]

There should be a note about this in the My site deploy fails unless Netlify’s build cache is cleared guide.

I struggled with this issue for far to long until a helpful Netlify support person mentioned it recently on the forums.

[fool]

Thanks for the suggestion @vincerubinetti I added - could you verify that it's in terms you would have perked up about, or suggest an improvement?

You might be running into our failure to handle yarn workspaces or other monorepo-based development flows in the expected way. This issue on GitHub has some more details.

[vincerubinetti]

Hi @fool , thanks for adding that. Looks good. I might move the keywords to the front of the sentence like this:

1. You're using a monorepo, like lerna or yarn workspaces. Currently, we don't support this in the expected way.

While we're on the subject, maybe reword 3. to be consistent with the other bullet points, which all start with some form of "you have a ___" or "you've done ___":

1. You have both a package.json and a yarn.lock file. We key off our process by detecting either a package.json OR yarn.lock ,so having both is an antipattern - we won’t run both for you automatically (see the logic we use in the build script 25 for more details).

Side note, shouldn't that say package-lock.json instead of package.json

[ehmicky]

Please note a workaround for some of the problems in this issue is ongoing at #440. It allows detecting Yarn using the environment variable NETLIFY_USE_YARN=true instead of only checking for the presence of a yarn.lock. Any feedback welcome on that PR :)

[dwilt]

FWIW, we tried endless ideas to solve this and even tried @slawekkolodziej's super clever idea but couldn't get that to work either as no folder was able to be cache manually by us.

We have switched over to doing our builds on Github Actions via Netlify Actions and have cut our build times in half (via being able to cache properly) and are deploying the build immediately to Netlify via the Netlify CLI. Wasn't hard to setup either and saves us a lot of money!

[chadly]

For anyone else frustrated with the lack of support for caching yarn workspaces, I also went @dwilt's route and implemented a GH action to set env variables based on what branch name is building. This is similar to how you can define variables for different contexts in netlify.toml. Check it out here: production-environment-variables.

[Plummat]

FWIW, we tried endless ideas and implemented @slawekkolodziej's super clever idea but couldn't get that to work either as no folder was able to be cache manually by us.

We have switched over to doing our builds on Github Actions via Netlify Actions and have cut our build times in half (via being able to cache properly) and are deploying the build immediately to Netlify via the Netlify CLI. Wasn't hard to setup either and saves us a lot of money!

How were you able to cache properly via this action?

[JGAntunes]

Hey folks 👋

Since this ended up working as a "catch-all" issue for many different problems with monorepo support on our build system over time, I wanted to clarify what is supported already and what's missing.

Supported:

• Currently, you should be able to either rely on a root netlify.toml with a base directory config on your root project to deploy a single site from your monorepo, or have multiple netlify.toml configs per monorepo package, each with its own netlify site.
• If you want to, you can force yarn usage even in the absence of a yarn.lock file in the sub-directory via the NETLIFY_USE_YARN variable - https://docs.netlify.com/configure-builds/environment-variables/#netlify-configuration-variables.
• If you're using npm and lerna or the newest npm v7 workspaces, the npm install process of your package and the respective node_modules caching should work as usual. Given npm cannot detect it's running under a monorepo within a sub-directory of the project, no dependencies are hoisted, and the module is installed as if it was a regular JS project.

Missing:

• The persisting problem currently is the inability to properly cache yarn projects using workspaces. This, however, is being discussed in #479 #432 and #526

Let me know if I missed anything in here, else I believe we can close this issue and move any remaining discussions around yarn workspaces to the respective issues 👍

[AoDev]

@JGAntunes Is there currently a full guide about using monorepos with Netlify?

Our case is a bit extreme because we have dependencies that need to be compiled and it takes a lot of time. Each git push leads to recompiling the exact same dependencies 4 - 5 times (once per site) which is very slow and costly.

So we are not sure of what's the correct way to go about it with the current capabilities of Netlify.

[tylerbrostrom]

If you're using npm and lerna or the newest npm v7 workspaces, the npm install process of your package and the respective node_modules caching should work as usual.

@JGAntunes thank you for the clarification!

I need to test out the following (haven’t yet because I should be working). It could be a promising pattern (assuming it works) for deploying monorepo projects containing multiple Netlify sites with shared local dependencies. It assumes the use of npm v7 workspaces.

Say we have a monorepo project using npm v7 workspaces. It contains multiple Netlify sites (each with its respective subfolder and netlify.toml).

Say we also have some local “lib” packages within the monorepo. The apps both depend on these libs. However, the apps themselves are independent from one another.

The top-level package.json looks like this:

{
    "name": "some-monorepo",
    "private": true,
    "workspaces": [
        "apps/*",
        "libs/*"
    ]
}

The folder structure looks something like this:

.
├── package-lock.json
├── package.json
├── libs/
│   ├── shared-1/
│   │   ├── index.js
│   │   └── package.json
│   └── shared-2/
│       ├── index.js
│       └── package.json
└── apps/
    ├── cms/
    │   ├── netlify.toml
    │   ├── package.json
    │   └── public/
    │       └── index.html
    └── www/
        ├── netlify.toml
        ├── package.json
        └── public/
            └── index.html

In this scenario, the expectation is that upon deployment of either Netlify site, npm install (or npm ci) could run from the root of the repository. We want to install from the root because npm v7 will hoist dependencies into a top-level node_modules folder (which could be then be cached on subsequent deployts).

Here’s what netlify.toml for one of those sites might look like:

```netlify.toml

apps/cms/netlify.toml

[build]
# set the repo root as the base folder so that npm install runs in that context
# we want npm v7 workspaces install behavior and this would theoretically do that
base = "../../"
# since the base folder is the root of the repo, we first need to change back down into apps/cms before building the app
command = "cd apps/cms/ && npm run build"
# since the www and cms apps are independent of one another, we don’t need to build the cms app just because some stuff changed in apps/www
# if _only_ something inside apps/www changed, we skip the build! wowzers!
# if there are changes elsewhere (e.g. in the cms package, or in a shared package) the build runs per usual
# the following git diff asks, “what changed in this latest commit, ignoring the apps/www folder”
ignore = "git diff HEAD^ HEAD ':(exclude)apps/www'"
```

TIL about the git diff :exclude() pattern. @AoDev maybe that will be helpful to curb excess builds?

Anyhoo, I’m supposed to be working, so I’ll come back and test this type of configuration later if I remember.

[JGAntunes]

@JGAntunes Is there currently a full guide about using monorepos with Netlify?

@AoDev we've got some base documentation here - https://docs.netlify.com/configure-builds/common-configurations/#monorepos. We have something more thorough being worked on. Also, not sure if it helps, but while working on this I've setup a couple of example test cases in this repo - https://github.com/JGAntunes/gatsby-yarn-netlify-test/blob/multiple-netlify-sties/packages/blog-1/netlify.toml - under different branches. Feel free to take a look and see if something could be of use for you.

Our case is a bit extreme because we have dependencies that need to be compiled and it takes a lot of time. Each git push leads to recompiling the exact same dependencies 4 - 5 times (once per site) which is very slow and costly.

Assuming you're using yarn I believe that should be addressed via #526

I need to test out the following (haven’t yet because I should be working). It could be a promising pattern (assuming it works) for deploying monorepo projects containing multiple Netlify sites with shared local dependencies. It assumes the use of npm v7 workspaces.

@tylerbrostrom can't say for sure that will work, would need to test it, but could be interesting 👍 . It's true though that one of the limitations for npm v7 workspaces right now is that we're unable to resolve local workspace dependencies, as we're only performing a npm install in the sub-package essentially.

[johncastorina]

@tylerbrostrom just tried a similar setup, although using yarn workspaces and no dice -

netlify build is throwing an error regarding the base directory -

Configuration property build.base must be inside the root directory.

Invalid syntax

  [build]
  base = "../../"

Valid syntax

  [build]
  base = ""

: exit status 1

still looking for a solution to this

[kaganjd]

Exciting update about Yarn workspaces: we’ve made improvements to how we cache dependencies for these projects. You can try out our new caching by creating an environment variable called NETLIFY_YARN_WORKSPACES and setting it to "true"

We want your feedback before we roll this out more widely- please let us know here or in the forum what your experience is like! You can also read more about this change here: https://answers.netlify.com/t/improved-caching-for-yarn-workspaces/36066

[gigamesh]

@kaganjd Hello :)

I'm trying to use NETLIFY_YARN_WORKSPACES but its not showing up in the build logs even when I try the clear cache & deploy option. The build fails because I'm using a sibling package as a dependency, but netlify is trying to find it on NPM. 🤔

[kaganjd]

Hey @gigamesh,
Could you share your site name and link to a deploy log so we can take a look?

[Brett-Best]

@johncastorina were you able to get it working? I’m facing the same error as you

[kaganjd]

Update: the caching improvements we shipped for Yarn workspaces have rolled out to all users, so the environment variable NETLIFY_YARN_WORKSPACES is no longer necessary. We've updated our announcement too: https://answers.netlify.com/t/improved-caching-for-yarn-workspaces/36066

Adding a suggestion from that thread for folks who are running into issues: try adding NETLIFY_USE_YARN=true so our build system knows to use Yarn for your build!

[rstavchansky]

Chiming in here for future reference: Our docs now have a more complete section on monorepos and the ignore command for controlling site builds:
https://docs.netlify.com/configure-builds/common-configurations/monorepos/
https://docs.netlify.com/configure-builds/common-configurations/ignore-builds/