Rules_nodejs: Building nodejs_image on macOS

[Globegitter]

Currently it is not possible to build the nodejs_image on macOS, mostly because nodejs for mac is downloaded and copied into the container by bazel. See: https://github.com/bazelbuild/rules_docker/issues/458 and https://github.com/bazelbuild/rules_docker/issues/466.

I know bazel has some ways of specifying platforms for cross-compilation like --host_platform, platforms and others. I wonder if it was possible to make use of whatever the right flag/setting is to tell bazel to download the nodejs for the specified platform? I would say the number 1 use-case really is to build docker containers on macOS.

I also know this alone would not be a 100% bulletproof, e.g. when using native/compiled dependencies, but I do think it would already help quite a few people, it would definitely help our devs on macOS to be able to verify and use the docker image locally.

[chrislovecnm]

Can we use a container like planter as a workaround?
https://github.com/kubernetes/test-infra/blob/master/planter/README.md

[chrislovecnm]

Also this issue should be documented!!

[Globegitter]

Yep using something like planter or https://github.com/nadirizr/dazel should do the trick as the host platform is then linux based so these rules would download nodejs for linux - I do think however it would not be too difficult to support this use-case (at least ignoring native dependencies to start off with), rules_go has already support for this use-case.

[alexeagle]

@irengrig from the Bazel team is going to help us switch the nodejs binary to use Bazel toolchain support. That means the nodejs binary won't be a regular action input. I think it would let us solve this issue too.

[dicarlo2]

I also know this alone would not be a 100% bulletproof, e.g. when using native/compiled dependencies, but I do think it would already help quite a few people

Most native dependencies use node-pre-gyp which allows specifying a platform to download native modules (https://stackoverflow.com/questions/24961623/installing-node-js-packages-for-different-architecture):

npm i --target_arch=x64 --target_platform=linux

Might be able to leverage this to have full cross-platform support in the case where all native dependencies use node-pre-gyp. (I'm not sure if yarn has an equivalent set of options, so that may be a blocker)

[chrislovecnm]

I think I got planter working on a Mac btw

[dicarlo2]

@Globegitter @alexeagle FWIW - I managed to get nodejs images built on macOS with the following high level approach:
1/ Define the nodejs binary as a toolchain and use it in nodejs_binary
2/ Add environment variable support to yarn_install and use it to create a second workspace @npm_linux_amd64 where yarn is run with env variables set to download pre-compiled linux binaries for native dependencies[1]
3/ Modify nodejs_image to use the linux version of the nodejs binaries in the separate layer
4/ Setup a second/parallel set of build rules (in our case, ts_library rules) that use @npm_linux_amd64 for fine-grained dependencies rather than @npm.

Theoretically, we can run nodejs_binary on any mac/linux/windows machine and build linux based images on mac/linux/windows. I have no prior experience with bazel so I could be doing this totally wrong, particularly with the parallel dependencies/rules for linux specifically, but it works for our use-case.

[1] Environment variables that worked for the set of dependencies we have:

• For prebuild: npm_config_platform=linux, npm_config_arch=x64
• For node-pre-gyp: npm_config_target_platform=linux, npm_config_target_arch=x64, npm_config_target_libc=glibc

[Globegitter]

@dicarlo2 that is amazing progress - as far as I am aware what is usually done, rather then having two seperate repositories, have both linux und macOS deps in the same repository but then use a select statement that at runtime can then pick the right dependency for the right platform. That select can then also be used to select the right nodejs.

Here is an example from rules_go: https://github.com/bazelbuild/rules_go/blob/00adf001c27a5ef82c90e740e1d63509ead69ec4/tests/legacy/cgo_select/BUILD.bazel#L5

[chrislovecnm]

cc @milesmatthias

[chrislovecnm]

@dicarlo2 can we get an example or more details?

@irengrig thoughts?

[rohansingh]

@Globegitter I've implemented the first step of what @dicarlo2 mentioned in his last comment. But yeah, without a second set of rules, I'm no longer able to directly bazel run //app:image.binary since it's now a Linux binary. That's by design, since I always want a Linux binary to be dropped into the the Docker image.

To deal with this, I have a second nodejs_binary rule for the same sources, which builds for my platform (macOS).

I'm pretty new to Bazel as well, but would a possible solution here be to have nodejs_binary have multiple outputs, one for each platform? The nodejs_image rule from rules_docker could then select the Linux output every time, but the "default" output could be platform-dependent so that bazel run //app:image.binary continues to work as expected.

Apologies if I'm completely missing the point here.

[Globegitter]

@rohansingh I think the way of how to idiomatically do cross-compilation or produce fat binaries is not fully settled yet / the API for that is still being worked on. See https://bazel.build/roadmaps/configuration.html, there is a point Flagless multiplatform builds (unoptimized) as well as Starlark supports multi-architecture (“fat”) binaries and they are still a few months away from completion.

I would imagine for now the easiest approach would be to have two separate targets, but one can also have a look at rules_go for the options that they offer, one is having one target but then using the --platforms=@io_bazel_rules_go//go/toolchain:linux_amd64 flag to compile the target for the specified platform. That has the downside though that you have to choose one platform for that whole build, so it would not be possible to have one target build for mac and linux in the same invocation. But I guess that is not so important for this use-case.

If you have anything to share that would be great - I have not had any time yet to actually produce any code for that use-case as it has not been our top priority.

[rohansingh]

@Globegitter Alright, I really need to read through the roadmap. Right now I'm just experimenting with a closed-source repo, but will see if I can extract some of it to an open-source example.

[rohansingh]

Here's a partial example of how to rebuild node_modules on macOS to target Linux:
https://gist.github.com/rohansingh/62d363d4da47f0173a7243f520c0ffe4

Note that I'm not endorsing this, it's just an approach that "works" without changes to rules_nodejs itself.

[Globegitter]

If anyone is interested I started making some early progress on toolchain support: https://github.com/ecosia/rules_nodejs/tree/toolchain-support, it is far from complete but I believe I have the basics mapped out on how to get cross-compilation to work. If everything does work out the way I have planned it will just be one ǹodejs_binary that can be compiled for a chosen platform and it should also be possible for them to choose the right npm dependency without having to change anything. Thanks @dicarlo2 and @rohansingh for some pointers that has been useful for some of the node/npm specifics so far.

[chrislovecnm]

Any progress on this? Been open for awhile :(

[Globegitter]

@chrislovecnm Yeah sorry it has been taking longer than planned. The first part is now functional complete: https://github.com/bazelbuild/rules_nodejs/pull/645 and would allow you to build docker containers of nodejs images without any native dependencies. My plans are to finish the last bits (documentation & testing) up this week.

[chrislovecnm]

Completely understand. Thanks for the update!

[siberex]

@Globegitter I think this could be closed now?

[Globegitter]

@siberex Well, nodejs_image can indeed be built on macOS now but only without native dependencies. But maybe it is better to close this issue and open up a new, more specific one.

[alexeagle]

yes we have other issues for native deps

Thanks @Globegitter !

[marmos91]

@alexeagle can you please point me out to the native deps issues? I am interested in following them but cannot find them

[Toxicable]

@marmos91 I don't think we have one at the moment.
But the issue is that yarn/npm will install native deps for the host, the native deps are just copied accross to the docker image.