Rules_nodejs: Make integration with create-react-app easier

[pward123]

I was originally going to post this as a response to #97, but figured this deserved its own issue since create-react-app is fairly heavily used in the community.

While researching these problems, I believe I ran across some mention of plans to restructure how the sandbox used by these rules works so that you don't have to monkey-patch require. I'm creating this issue in the hopes that you may find ways to incorporate fixes for these problems into that effort.

I've created an example repo that gets create-react-apps minimally working with this set of rules. If you look at the commits, I think I've broken them up into clean steps I had to take.

Here are the issues I ran into while getting this working. It's a block copy of the readme from the repo above.

• nodejs_binary uses a different working directory than npm when launching the build/start/test scripts. I worked around this by injecting code into the top of these scripts that changes the directory based on __dirname

• cra resolves symlinks when executing webpack, but uses working directory when building the webpack config. This results in name mismatches. I worked around this by adding the resolved paths to some of the webpack config 'includes' entries

• bazel monkey-patches require, but babel uses the resolve npm package to load packages which bypasses the require patches. I got around this by adding the bazel sandbox npm folder to the resolve/modules section of the webpack config.

• cra uses a babel section in the package.json, but this also has problems (I believe due to the resolve issue above). To get around this, I explicitly added a require('babel-preset-react-app') to the babel-loader section of the webpack config that handles the cra app code.

Thanks for all your hard work on this project. Your efforts are greatly appreciated!

[olegsmetanin]

I tried to run the project in mac os x without docker.

$ cd packages
$ bazel run //cra-example

and I got problem with compilation of jsx

bazel-cra-example/packages/cra-example/src/index.js
SyntaxError: bazel-cra-example/packages/cra-example/src/index.js: Unexpected token (7:16)

   5 | import * as serviceWorker from './serviceWorker';
   6 |
>  7 | ReactDOM.render(<App />, document.getElementById('root'));

$ bazel version
Build label: 0.18.0

Is there a chance to solve the problem?

[alexeagle]

Thanks for raising this, @pward123

Let's back up a step - create-react-app sets up a build toolchain for your react app. Under Bazel we expect the toolchain to be quite different. For example, to run babel I'd expect a js_library rule that just runs the babel process (after https://github.com/bazelbuild/rules_nodejs/pull/217 lands)

What you've done here is just use Bazel to launch the prior toolchain, which might be a step in migrating from one to another, but by itself this gives you no incrementality/customizability of your build, so you get none of the Bazel benefits.

What are you trying to accomplish? I imagine the right formulation of the question is "how could we patch/plugin to create-react-app so that it stamps out a Bazel-based toolchain for the new app rather than the default one"

[pward123]

In my case, I'm fighting with a build environment that (due to some tools we're working hard to eliminate) uses Lerna but we're unable to use yarn workspaces. The resulting node_modules trees are in the realm of 2.1+gb (500k+ files).

Our environment has reached a level of complexity where we've implemented docker-compose for our development environment. Since we're using OSX machines for development, we have to use a virtual machine for docker. The synchronization of node_modules files between host and guest vm is crushing the development machines.

I've tried moving everything into a full vm and running docker-compose, git, etc from there, but the devs aren't keen on that development experience. I've also tried using docker-sync-stack, but it gets pretty rickety once you throw this many files at it.

At a future date, I could see us going through the create-react-apps we have and replacing their toolchains with a bazel-centric system. However at this point, I'm just interested in moving node_modules out of the development source file structure as quickly as I can.

[pward123]

@olegsmetanin I'm not surprised you ran into problems trying to run it outside docker. There's a lot of stuff in the cra setup that is not very friendly to pathing changes. I'd start by fixing the paths here and here. To really change the paths properly, you'd need to do a fs.realPath based conversion of process.cwd. Since I'm only interested in running this from docker, it's easier just to hard code the path.

You may also need to play around with this. I would console.log that value somewhere and bazel run //cra-example > /tmp/temp.txt then ctrl-c out of the app and take a look at the file. You may need to search around till you find the folder containing the node_modules and update the relative path.

[pward123]

@olegsmetanin You may also want to just look at the individual commits in that repo. I've broken my own customizations of cra code out into their own commits.

@alexeagle Even though I agree that migrating toolchains is ideal, you may find better js community adoption if you give new users a way to reuse their existing webpack/cra toolchains. Subjectively, it feels like the rollup support still requires me to do starlark scripting for anything non-trivial.

If you want me to close this issue, I'm cool with it. I've managed to get my cra apps converted to at least launch/build from bazel now. I'm excited about being able to use your rules to resolve the issues I've been fighting for a while now. Thanks!

[alexeagle]

So you're just using Bazel as a means to dynamically install npm dependencies outside of the project tree, but otherwise run the same tools as today? Seems like it could be a lot simpler to just run the yarn/npm install with some option to put the output in a different directory (like bazel does) and then change the $NODE_PATH variable so your nodejs process knows how to find them in that place.

[pward123]

Our use of webpack is limited to ui applications, so using webpack-dev-server is fine for us. We don't need to take the output of webpack as a js_library to feed into anything else.

Our back end is mostly node. Some of those backend npm packages have browser code used by the webpack apps. We're currently using lerna to run a watch process on each package that babelifies. My next step is to swap lerna for bazel.

We also have some golang, so the hope is to integrate that in with Bazel down the road.

[bugzpodder]

@pward123 so I want to do the exact same thing for an almost identical setup: webpack ui + node backend + golang server + lerna/babel. Is your example repo https://github.com/pward123/bazel-cra-example something I can look as a starting point?

[pward123]

For my own project, I plan to take bits and pieces (but mostly knowledge) from that example. The WORKSPACE file is useful as well as the scripts/merge file and its associated templates. I won't be using scripts/setup, nor templates/webpack.config.*. Those only exist because I was scripting the commits for the example and it was too difficult to sed the ejected cra scripts.

For now, I've worked around the original issue that brought me to Bazel by tossing a couple dozen named docker volumes into my docker-compose.yml. As it turns out, named volumes are only mounted from the containers to the Docker VM and not the Mac host. Each time I create a new monorepo package, I'll be creating a named volume to hold the node_modules that aren't hoisted by yarn.

I still think there are significant benefits to switching to Bazel. For example, I am currently wasting a lot of CI cycles rebuilding stuff that hasn't changed. For now, I've just been throwing hardware at the problem since its cheap. However, the feedback loop between qa/support and development is still limited by the time it takes for an individual build to finish.

I still plan on moving forward with Bazel, but the learning/adoption curve is so steep that I'm going to take a slow burn route getting there.

One thing that I will experiment with as I move forward is whether I really want to merge all package.json files together and force the devs to use the same version of npm packages in every monorepo package.

This line tells Bazel that it should install npm packages into its own sandbox. However, as far as I can tell, it doesn't support multiple package.json files. My guess is because it probably doesn't handle version conflicts.

If you run bazel run @nodejs//:yarn Bazel uses the multiple package.json files on this line, but copies the packages directly into the node_modules folders where the respective package.json resides (equivalent to lerna+npm with hoisting off).

I'm not really sure yet what the implications are for incrementally building items by using the latter, but now that I know about the Docker named volume workaround, it is a possibility. Forcing devs to upgrade all packages in the monorepo when we want to upgrade an npm package creates development friction that I'd rather do without.

Sorry for the long post, but I figured since you're in the same boat, the extra info might be useful.

[ayinlaaji]

I tried to run the project in mac os x without docker.

$ cd packages
$ bazel run //cra-example

and I got problem with compilation of jsx

bazel-cra-example/packages/cra-example/src/index.js
SyntaxError: bazel-cra-example/packages/cra-example/src/index.js: Unexpected token (7:16)

   5 | import * as serviceWorker from './serviceWorker';
   6 |
>  7 | ReactDOM.render(<App />, document.getElementById('root'));

$ bazel version Build label: 0.18.0
Is there a chance to solve the problem?

Any solution to this ?
Stuck on this for days

[bugzpodder]

I actually got this working today with help of a colleague.
What I used in particular:
yarn workspaces
create-react-app 2.1.1
If you don't use workspaces you can just skip the relevant steps
1) in main package.json, make sure relevant paths to yarn workspaces is fixed, since the directory structure in bazel sandbox is likely different. its ok if they point to non-existent paths.
2) in yarn_install for bazel WORKSPACE, add relevant yarn workspace package.json to data field
3) fix your hoisting in yarn workspaces, eg make sure your other external webpack versions match react-scripts
4) rewire your create-react-app config as necessary (I used create-app-rewired) to work with yarn workspaces. In particular:

config.resolve.symlinks = false; // this seem to help
config.resolve.plugins = [];  // get rid of ModuleScopePlugin
config.resolve.alias = {...} // your yarn workspace dirs
const srcPath = resolveApp("src"); // check out the path.js in react-scripts config
 const pkgPaths = [srcPath, ...];
  for (rule of config.module.rules) {
    if (rule.include === srcPath) {
      rule.include = pkgPaths;
    } else if (rule.oneOf) {
     for (oneOfRule of rule.oneOf) {
       if (oneOfRule.include === srcPath) {
          oneOfRule.include = pkgPaths;
        }
      }
    }
  }

6)
```nodejs_binary(
name = "client_build",
data = [
... // your src files here
],
entry_point = "react-app-rewired/bin/index.js",
node_modules = "@nodejs_deps//:node_modules",
templated_args = [
"--node_options=--preserve-symlinks",
"build",
],
visibility = ["//visibility:public"],
)
genrule(
name = "client",
outs = ["client.tar"],
cmd = """
ORIG_WD=$$PWD

    cd $(location :client_build).runfiles/
    ln -s nodejs_deps/node_modules ./ // this seem to be a quirk from react-scripts

    NODE_PATH=src $$ORIG_WD/$(location :client_build)
    tar -C build -cf $$ORIG_WD/$@ ./
""",
tools = [":client_build"],
visibility = ["//visibility:public"],

)

```
You can probably change the ln -s command by using NODE_PATH. react-scripts must use relative node_path.
NODE_PATH=src:../node_modules:$../node_modules/package1/node_modules $$ORIG_DIR/$(location :lims_ui_client_build)
Also, bazel docker images have all static files having the timestamp 0 (1970-1-1) this breaks caching scheme when these files are served with headers that correspond to the modified time. Suggest you run touch all on files in client build/

[ritchieanesco]

@ayinlaaji were u able to resolve ur issue? i'm facing the same and i dont really want to eject from CRA.

[teddyknox]

Bump, this would be very useful for a project I'm working on.

[bugzpodder]

I have been building using bazel in production (dev environment still uses yarn) for the last three months and it's working out so far. See my post above.

[Globegitter]

The idiomatic way btw, would be to write a custom react_component rule which can integrate with the rest of the eco-system here, like ts_library, ts_devserver, rollup_bundle, etc.

[Globegitter]

In fact if you are using typescript there might even be a way to integrate it directly with ts_library: https://www.typescriptlang.org/docs/handbook/jsx.html

Or one could even have a two-step compilation of ts_library -> react_component, i.e. tsx -> jsx -> js. But not sure if that would make a lot of sense if ts can already do the compilation of tsx -> js.

The other way, if you want to have something working today it would probably also be possible to use babel_library and just provide a custom config: https://github.com/ecosia/bazel_rules_nodejs_contrib#babel_library

[alexeagle]

We have a jsx_factory argument where we setup the typescript compiler, we should wire this up so ts_library can produce the right .js output for tsx files using React... https://github.com/bazelbuild/rules_nodejs/search?q=jsx_factory&unscoped_q=jsx_factory

[ghost]

Hi, I'm currently looking to integrate some JSX into our Bazel build, is JSX syntax supported already by the ts_* rules? If not, can I pass some assortment of flags to the ts_* rules to get JSX compilation working? Thank you!

I'm in a similar situation where we started with create-react-app to get something working, and are migrating it over to Bazel for production use.

EDIT: also happy to file a separate issue, or a PR if needed.

EDIT 2: I am seeing some success with using typescript and setting a jsx:react flag in the tsconfig.json.

[alexeagle]

@rustacian yes you can include .tsx files as srcs to ts_library. I'm working on getting this react example green: https://github.com/bazelbuild/rules_nodejs/pull/1076

[bweston92]

Does anyone know the actual reason this fails?

The following loader I believe handles this part and both the include and test key should pass as the values are correct.

https://github.com/facebook/create-react-app/blob/05f7924398d175429825c4b680d18e6c253435a9/packages/react-scripts/config/webpack.config.js#L388-L393

[soraliu]

you needn't eject react-scripts, you can use react-app-rewired to override the config.

These are my configurations, it works for me.

// configs/overrides.js
const getYarnWorkspaces = require('get-yarn-workspaces');
const { override, babelInclude } = require('customize-cra');

module.exports = override(
  babelInclude(getYarnWorkspaces())
);

// scripts/start.js
const path = require('path')
const spawn = require('cross-spawn')
const fs = require('fs')

const getAppRoot = (root = __dirname) => {
  if (fs.existsSync(path.resolve(root, 'package.json'))) {
    return root
  }

  return getAppRoot(path.dirname(root))
}
const getWorkspaceRoot = () => path.resolve(process.env['RUNFILES'], process.env['BAZEL_WORKSPACE'])

const relativeAppPath = path.relative(getWorkspaceRoot(), getAppRoot())
const rewiredBinPath = require.resolve('react-app-rewired/bin/index.js')
const overridesPath = './configs/overrides.js'

const start = spawn('bash', [
  '-c',
  `cd ${relativeAppPath} && node --preserve-symlinks ${rewiredBinPath} start --config-overrides ${overridesPath}`
], {stdio: 'inherit'})

# BUILD.bazel
load("@build_bazel_rules_nodejs//:index.bzl", "nodejs_binary")

deps = [
    "//:package.json",
    "@npm//react",
    "@npm//react-dom",
    "@npm//react-scripts",
    "@npm//react-app-rewired",
    "@npm//customize-cra",
    "@npm//get-yarn-workspaces",
]

srcs = glob([
    "package.json",
    "src/**",
    "public/**",
    "configs/**",
    "scripts/**",
])

nodejs_binary(
    name = "start",
    data = srcs + deps,
    entry_point = ":scripts/start.js",
)

# Root BUILD.bazel
package(default_visibility = ["//visibility:public"])

exports_files([
    "package.json",
])

[mattem]

There's a create-react-app example now within the examples directory, and some more extended docs on the different approaches that could be taken.

There is going to be further work on what an ejected and custom toolchain for CRA would look like, but I think the initial examples are there.