Jest: babel-jest issues with monorepo and jest multi project runner

[sibelius]

🐛 Bug Report

We are moving to a monorepo structure, and we'd like to use jest multi-project-runner structure.
We are using yarn workspaces and lerna to manage dependencies.
Here is a repro https://github.com/entria/entria-fullstack/pull/12, where you can reproduce some of the issues.

To Reproduce

Neither of this transforms works:

'^.+\\.(js|ts|tsx)?$': 'babel-jest',

^.+\\.(js|ts|tsx)?$': '<rootDir>/node_modules/babel-jest',

I think the problem here is that babel-jest resolution strategy is failing to find the correct babel.config.js file

Only creating a custom babel transformer the transform works, like the below:

const config = require('../babel.config');
 const { createTransformer } = require('babel-jest');
 module.exports = createTransformer({
  ...config,
});

Steps to reproduce the behavior:
Run jest on root of this project https://github.com/entria/entria-fullstack/pull/12

modify transform config from custom transformer to babel-jest

Another problem is that when using only projects option on jest.config.js root, it won't use a different jest config per project, it is looks like all the config should be on root jest.config.js instead of jest.config.js inside each project

Expected behavior

• [ ] it should find correct babel.config.js

• [ ] it should transpile all files inside packages/*

• [ ] it should use the jest.config.js for each project (transform options and so on)

Link to repl or repo (highly encouraged)

https://github.com/entria/entria-fullstack/pull/12

Run npx envinfo --preset jest

Paste the results here:

System:
    OS: macOS 10.14.1
    CPU: (4) x64 Intel(R) Core(TM) i7-5557U CPU @ 3.10GHz
  Binaries:
    Node: 10.12.0 - ~/.nvm/versions/node/v10.12.0/bin/node
    Yarn: 1.12.3 - /usr/local/bin/yarn
    npm: 6.4.1 - ~/.nvm/versions/node/v10.12.0/bin/npm
  npmPackages:
    jest: ^23.6.0 => 23.6.0 

[ide]

The Babel issue looks like it is because the transformer runs with cwd set to the repo root instead of cwd set to each package's directory when Jest runs tests for that package.

Reading through the runner code (https://github.com/facebook/jest/blob/9822231fba8ef6a8558700c61682751f397c1cd0/packages/jest-runner/src/index.js) it looks like we could include the desired cwd with each test's config and call process.chdir() for each test. In parallel mode, each worker VM would have a different cwd. In in-band mode, we run the tests serially (as chained promises) so setting the global cwd would work as long as we restored it at the end.

Alternatively -- for this specific issue with Babel -- we could have babel-jest take options like configFile and let people explicitly provide an absolute path to babel.config.js at runtime.

[haraldrudell]

I created a package jest-esnext that comes with babel7-ES.Next transpilation pre-packaged
Pre-packaging makes it reliable

basically one would do:
yarn add --dev jest jest-esnext
then, next to package.json, create a file jest.config.js:
module.exports = require('jest-esnext')
 the bare-bones config can be modified in this file
Now, yarn jest would test .js .mjs, … with ES.Next source code like phase 2 decorators and for await

This would avoid some troubles:

• Mysterious error messages from babel-jest, like not understanding import, or Root-related messages
• Unreliable transpilation by babel-jest and babel7: some files yes, other files no

[haraldrudell]

for @sibelius it might be more convenient to put your jest configuration in a fake sibling package than up at the workspace root, possibly. I used one [email protected] with yarn workspaces before I moved it to jest-esnext

[bradfordlemley]

I did some analysis of this issue on https://github.com/facebook/jest/issues/7771, before I found this issue.

Summary:
Assuming multi project runner is supposed to behave the same as running each project individually, the current behavior is not correct.

Analysis:
In multi project mode, babel's cwd is always set to the top-level project dir (as described above), instead of being set to each project dir. Since babel's cwd entirely controls babel config lookup, the behavior is completely different when running a project in multi project vs. individually.

Also worth noting that the jest repo itself demonstrates the issue in examples/react, but works around it by adding babelrcRoots: ['examples/*'] in the top-level babel.config.js.

Solution:
I suggest that the proper solution is to set babel's cwd to each project dir (i.e., here and here) so that the behavior is the same when running the project individually and via multi project runner.

What if users want top-level babel config?:
If individual projects want to include top-level babel config, it can be achieved by configuring babel to rootMode upward or whatever is desired. Currently, that configuration would need to be set in a custom jest transform module, but https://github.com/facebook/jest/pull/7288 adds the ability to configure babel-jest within jest's config, so that will hopefully become easier. This way, the top-level babel.config.js would apply if the project is run individually or via multi-project.

Aside:
There's a related issue for transforming files outside of project root, e.g.:

/monorepo
  babel.config.js
  globalSetup.js
  /prj-1
  /prj-2

Currently, babel.config.js would not apply to globalSetup.js when running jest in prj-1, but it does apply with running jest --projects prj-1 in monorepo. One might expect it to "just work", but it doesn't because babel cwd is prj-1.

[Globegitter]

There is another issue in a monorepo setup, where we are using bazel (https://bazel.build/) it, runs jest in the repository root and passes the test files directly to jest using --runTestsByPath (https://jestjs.io/docs/en/cli#runtestsbypath) but we have the project specific babel config in the projectdir, so:

/monorepo
  /prj-1
    babel.config.js

and apart from setting up a custom transformer like so:

const babelJest = require('babel-jest');

module.exports = babelJest.createTransformer({
  root: 'prj-1',
});

I do not see an easier way on telling babel-jest where to find the config. vue-jest for example has a config option that can be set in the jest.config.js, it would be nice if that can also be passed in as an option. But I will also test #7794, right now we are not using the --projects mode but maybe we can somehow utilise that with bazel also.

[sibelius]

@Globegitter do you have a sample repo of this bazel + jest config?

[bobbybobby]

@bradfordlemley said

What if users want top-level babel config?:
If individual projects want to include top-level babel config, it can be achieved by configuring babel to rootMode upward or whatever is desired. Currently, that configuration would need to be set in a custom jest transform module, but #7288 adds the ability to configure babel-jest within jest's config, so that will hopefully become easier. This way, the top-level babel.config.js would apply if the project is run individually or via multi-project.

Here's how I configured it in my monorepo project (that uses yarn workspaces for dependency management, and lerna for running commands on sub packages).

Project structure :

/monorepo
  babel.config.js
    /packages/
      /pkg1
        jest.config.js
        jestBabelTransform.js
        ...

/babel.config.js :

module.exports = {
  presets: [ "@babel/env", "@babel/react" ],
  babelrcRoots: ["./packages/*"]
};

/packages/pkg1/jest.config.js :

const path = require('path');

module.exports = {
  transform: {
    "^.+\\.js$": path.resolve(__dirname, "./jestBabelTransform.js")
  },
};

/packages/pkg1/jestBabelTransform.js :

const babelJest = require('babel-jest');

module.exports = babelJest.createTransformer({
  rootMode: 'upward'
});

I'm already using the rootMode: upward option with my build process, this way I get the same behaviour running jest. I can run jest from the pkg1 directory with yarn test or from the monorepo root with lerna run test --scope pkg1.

[codepunkt]

I can't get this to work at all, can someone have a look?
I have a yarn workspaces and lerna based monorepo with a typescript api gateway and a typescript create-react-app as the main projects.

yarn test in each of the packages/* directories runs fine, yarn test in root directory fails with
SyntaxError: Unexpected string" on an import, which is supposedly a babel issue.

Repository is here:
https://github.com/paderbornjs/site/tree/009a7cc98966511c3534d43231f4b3f6cb93237c

[bradfordlemley]

The quick and dirty general work-around for this problem is to add babel.config.js in your root:

// babel.config.js
module.exports = {
  // jest: https://github.com/facebook/jest/issues/7359
  babelrcRoots: ['packages/*'],
}

Use jest --clearCache (or --no-cache) while you're working through transpilation issues.

That solution makes your api tests work from root, but not the client.

The client has an issue with the transform config in its jest.config.js ... you'll see the same error if you run ../../node_modules/.bin/jest --no-cache directly in the client. If you can run ../../node_modules/.bin/jest --no-cache successfully in the client, it should work when run from root.

[codepunkt]

@bradfordlemley Thanks, your --no-cache tip is really helpful. Seems the client is missing a babel config. Gladly, everything is abstracted away into the react-app babel preset in create-react-app, so it's enough to add a .babelrc.js like this to the client folder:

module.exports = {
  presets: ['react-app'],
}

With this setup and the babel.config.js in root, things work as expected. That configuration and especially getting there on your own without expert help such as yours seems really hard and convoluted, though.

What i'm still missing is a way to add a displayName to the projects of my combined jest --watch task in order to differentiate between the projects. String-based projects configuration "/packages/*" works fine, but when i switch to an object i'm not sure how to configure it to basically reflect what the string-based packages option had + displayNames.

Edit: Doesn't work on a second computer - regardless of --no-cache is used or not. Back to the drawing board 😩

[sibelius]

@haraldrudell where is the code for this package?

[sibelius]

@bobbybobby do you have a top level jest.config.js in your example https://github.com/facebook/jest/issues/7359#issuecomment-471509996?

do you have a repo with the full example?

[dandv]

@sibelius: does my repo in #8238 help?

[jayarjo]

Adding babel.config.js to the workspace itself makes jest be able to test modules (through babel-jest). But then some of the modules use absolute imports, so I have to add jest.config.js in order to tell it where to look for modules (via moduleDirectories). However as soon as I do that jest stops to consult babel.config.js and transpilation breaks again. Specifying: transform: { '^.+\\.jsx?$': 'babel-jest' } directly doesn't have any effect.

[jhogendorn]

I found to get jest/lerna/yarn workspace monorepo testing working I have to symlink (or duplicate) .babelrc.js to babel.config.js in each package (but not at the root). If I miss one in a package, the whole test suite fails in a misleading manner (cant transform import in various files). This is using just jest's packages config, no specific config around transforms. It's not a case of one file or the other being resolvable, it seems the various moving parts each refer to one style or the other, so if you don't have both it fails.

jest 24.8.0
@babel/core 7.4.5

[ofhouse]

I am using jest multi project runner with babel-jest and it works just fine.
Each package in the monorepo has it's own local .babelrc.js and no .babelrc.js in the root is required.

Each of my packages has a jest.config.js with the following settings to make this work:

module.exports = {
  ...
  transform: {
    '\\.js$': ['babel-jest', { cwd: __dirname }],
  },
  ...
};

I made an example repo which contains two packages with different babel-configs, feel free to check it out: https://github.com/ofhouse/jest-project-babel-transformer

Edit: You need Jest >= 24.9.0 to pass options to transformers

[jayarjo]

I also have it working, although I can't say how. It's shame that it has to be so confusing. @ofhouse :+1:

Looked at my setup and yeah - I basically have everything everywhere. Guess that's the way.

[mtsmachado8]

Hi guys,
Node 12.12.0 LTS's finally released with better supports of mjs... so we really need the transform working
Any updates on this issue?
Whats the best workaround for now?

[GoMino]

@ofhouse solution worked for me thanks!