Rules_nodejs: How can I support multiple `package.json` files with local dependencies?

[Jaspaul]

Description

I am attempting to setup a monorepo using Bazel, TypeScript, and Jasmine. I have my TypeScript projects divided into monorepo/services and monorepo/packages where services leverage various packages.

The goal is to have different applications depend on remote and local dependencies, with any of the services declaring different versions. When doing this, so long as there are no interdependencies between the services and packages, everything appears to work fine. However when I attempt to use one of my packages within a service I run into errors.

For example:

monorepo/
|-- BUILD
|-- WORKSPACE
|-- package.json // contains only @bazel and typescript dependencies
|-- packages/
|-- |-- permissions/
|-- |-- |-- BUILD
|-- |-- |-- package.json // contains a reference to some third party package (i.e. casbin)
|-- services/
|-- |-- pokedex/
|-- |-- |-- BUILD
|-- |-- |-- package.json // contains a reference to some third party package (i.e. express)
|-- |-- |-- src/
|-- |-- |-- |-- BUILD

monorepo/WORKSPACE

workspace(
    name = "monorepo",
    managed_directories = {
        "@npm": ["node_modules"],
        "@services_pokedex_npm": ["services/pokedex/node_modules"],
        "@packages_permissions_npm": ["packages/permissions/node_modules"],
    },
)

load("@bazel_tools//tools/build_defs/repo:git.bzl", "git_repository")
load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")

################################################################################
# Configure TypeScript                                                         #
################################################################################

# Provides the basic tools for running, building, and packaging nodejs programs in Bazel.

http_archive(
    name = "build_bazel_rules_nodejs",
    sha256 = "9abd649b74317c9c135f4810636aaa838d5bea4913bfa93a85c2f52a919fdaf3",
    urls = ["https://github.com/bazelbuild/rules_nodejs/releases/download/0.36.0/rules_nodejs-0.36.0.tar.gz"],
)

# The yarn_install rule.

load("@build_bazel_rules_nodejs//:defs.bzl", "yarn_install")

# Install our node dependencies

yarn_install(
    name = "npm",
    package_json = "//:package.json",
    yarn_lock = "//:yarn.lock",
)

yarn_install(
    name = "services_pokedex_npm",
    package_json = "//services/pokedex:package.json",
    yarn_lock = "//services/pokedex:yarn.lock",
)

yarn_install(
    name = "packages_permissions_npm",
    package_json = "//packages/permissions:package.json",
    yarn_lock = "//packages/permissions:yarn.lock",
)

load("@npm//:install_bazel_dependencies.bzl", "install_bazel_dependencies")

install_bazel_dependencies()

In the above if I attempt to reference //packages/permissions within the BUILD files on //services/pokedex I run into errors. These errors appear to stem from permissions and pokedex having different folders for their npm modules. I see the following error:

monorepo/services/pokedex/src/BUILD

package(default_visibility = ["//services/pokedex/src:__pkg__"])

load("@npm_bazel_typescript//:defs.bzl", "ts_library")

ts_library(
    name = "pokedex",
    srcs = glob(["*.ts"]),
    node_modules = "@services_pokedex_npm//:node_modules",
    deps = [
        "//packages/permissions/src:permissions",
    ],
)

All npm dependencies need to come from a single workspace. Found 'external/services_pokedex_npm/node_modules' and 'external/packages_permissions_npm/node_modules'.

When I attempted to use yarn to add the directory monorepo/packages/permissions to the pokedex service, to avoid the issue above, I ran into further issues claiming that Jasmine has previously been configured making my test runner fail. The issue here appears to be that I declare the @bazel/jasmine dependency in multiple package.json files.

Is there some documentation available that I can reference to accomplish what I'm looking for here, or is this something that simply isn't supported?

This may be related to https://github.com/bazelbuild/rules_nodejs/issues/964 however it's unclear if they're attempting to do exactly what I am trying to do here.

[pauldraper]

The Bazel approach is to generally have single versions of packages. In a large monorepo, this makes sense because the vast majority of your dependencies are in the monorepo, i.e. already having a single version.

---

Package resolution is a non-trivial problem with version ranges and deduplication.

Suppose: A depends on B and C. B depends on external@~1.0.0. C depends on external@~1.1.0. Then there need to be two different versions of external.

Suppose: A depends on B and C. B depends on external@^1.0.0. C depends on external@~1.1.0.
Then there can be a single version of external.

Currently, those problems are delegated to npm/yarn. For Bazel targets to participate in these calculations would require more integration with them than is currently present.

---

Also, such an approach does not scale well for hundreds of targets, since there would be many copies of dependencies on disk (unless a pnpm-like tool is used.)

[Jaspaul]

@pauldraper Thank you for the detailed response. I think I'll try to embrace the single versions of packages and move to using a single package.json file.

[ccate]

it would be good to have some guidance for monorepos that DO have multiple typescript packages with differing versions of dependencies.

[chrislloyd]

@pauldraper I think yarn (with workspaces) already does package resolution for bazel. There can be lots of package.jsons but only a single yarn.lock file. I think this is still a valid issue that fits well with Bazel/Monorepo approaches.

[kylecordes]

@chrislloyd This seems very workable - possibly the only need would be a way to give the yarn_install rule access to the additional package.json files referenced in the top level package's workspace list.

[jjangga0214]

Guys, we still can use yarn workspace with Bazel.
yarn workspace allows multiple package.json, and solves duplicated storage and dependency version problems.
Still, there is one single yarn.lock. What's more, yarn_install automatically installs all dependencies.
Thus, it make sense to use yarn workspace along with Bazel.

[baldurh]

@jjangga0214 I’ve been trying to add JS projects to our current bazel setup using yarn workspaces without much luck.
How did you get around the issue the OP had?

All npm dependencies need to come from a single workspace.

I’m running into the same thing as a JS project needs to access two node_modules folders. Its own and the root one. How did you make this work? Could you maybe elaborate on your configuration? 🙂