Most recent status: https://github.com/babel/babel/issues/6766#issuecomment-352225586
All below still applies, but also see more explanation in https://github.com/babel/babel/issues/6766#issuecomment-347966349
Not a fleshed out proposal, but filing for tracking and discussion.
Currently if you have:
.babelrc
src/
index.js
utils/
.babelrc
other.js
compiling index.js will use the root-level .babelrc, and other.js will use the src/utils/.babelrc. I assume this pattern was pulled from other utilities like ESLint, but not 100% sure.
We're at the point now where this approach has become a problem for us because:
.babelrc file to also apply to things that live in node_modules. Babel 6 was terrible at figuring out how to handle node_modules because it always assumes things are strict mode, and always rewrites this, but a lot of those issues aren't as big in Babel 7, or at least ideally won't be, because we plan to have better handling of the sourceType: script vs sourceType: module distinction. This _kind_ of works now, but see point (3)..babelrc files in their packages, or more troublingly, use the babel key in package.json (which you can't really get around publishing), users trying to use Webpack or anything like it to compile node_modules get extremely confusing errors about not being able to find plugins and presets referenced in those files. See https://github.com/facebookincubator/create-react-app/issues/1125 for a massive list of users that want to compile node_modules, but would _absolutely_ be bitten by this issue.So where does that leave us? The simplest thing I can think for us to do would be to change up Babel's API to, by default, accept a root directory as an argument where it would search for the config, defaulting to the working directory.
We're already at a point where I've thought about adding an argument for root directory to resolve the filename relative to, so that should be uncontroversial. Actually changing config resolution is a bigger question.
It would be _pretty_ easy for us to allow users to opt for the old behavior, so I think my proposal would probably be to add this new root-dir config resolution logic and make it the default, but keep the old behavior around as an opt in. So where now we have
babelrc: boolean
to toggle config resolution off and on, I could imagine us doing
babelrc: true | "root" => Look for config in root directory
babelrc: "relative" => Look for config, starting from compiled file, working upward
babelrc: false => Skip search entirely.
instead.
The primary loss of functionality here is that if users _do_ want to use .babelrc files inside nested folders to change Babel configs based on path, they don't have an easy way to do it anymore. I don't however have a good grasp of how many users that might be. It seems like we may want to consider https://github.com/babel/babel/issues/5451 as something to be included in this effort if we do want to change, which might end up blocked on https://github.com/babel/babel/issues/6765.
Seems totally reasonable to me
overrides for ESLint yeah we can consider https://github.com/babel/babel/issues/5451Thanks for all your work on babel folks! :clap:
Here are some thoughts:
root: true functionality works and just minutes ago was looking to see if babel had support for something like this. I'm not even joking.node_modules, but I think that it should be possible _and_ I would expect that the hierarchy should apply there as well. I can't speak too much to this though because I'm not big on the idea of transpiling node_modules. I just feel like there's probably a use case for it. But making it a little difficult to do isn't a bad thing IMO.I hope that's helpful! Thanks again!
@kentcdodds
There is a babelrc: false option which stops lookup as mentioned ^
Compiling node_modules is for when you want to consume a package from npm that is ES2015+. Maybe you support a different environment so you need to compile it down to a different level than ES5. Example: maybe a react package uses async but you don't want to use the "main" file which uses regenerator, so you want the package to output async. It may or may not be transpiled to async (no change), generators, or regenerator depending on the users preset-env targets
Ya it's interesting that eslint would never have this issue since there's no need to look at node_modules there 😛
There is a babelrc: false option which stops lookup as mentioned ^
:+1:
Solid use case example :+1: So long as it's possible that'd be good with me. And if I could disable the lookup using webpack/babel-cli etc. then that'd be sufficient I think
How about non-precompiled/non-distributable monorepos? This is currently what we do at @futuresimple, so e.g. we have packages/pkg-name, each with its own version of .babelrc that can vary across the packages in the monorepo. On the other hand, the root folder (repository root) doesn't have any babel config.
To handle such a case well, I think the default behavior should be to use the .babelrc next to the closest package.json, rather than closest to the file (as is now).
I've seen nested babelrc files in repos which contain server and client build.
I'm sure not everyone here will have read everything but we could possibly do https://github.com/babel/babel/issues/5451 which is an "overrides" option similar to eslint which gives you glob config + different config so you don't need nested .babelrcs.
I'm strongly in favor of only using the root babelrc, since config files in nested directories are really easy to miss.
If people want more control on their builds, they can use tools like gulp.
@hzoo #5451 won't help for monorepos (when used without a build step). You'd have to re-declare all the babelrc's in the root of the monorepo, which doesn't sound fun.
When you install packages in a monorepo, they end up symlinked as node_modules/MONOREPO_PACKAGE.
Since each package can have its own .babelrc, we depend on babel resolving those .babelrc's to transpile the code from those modules using the settings they declare.
I am all for this change, as long as it's possible to declare in the root .babelrc something like deepResolveBabelRc: true. Having this as a programmatic option in the API (as proposed in the issue) won't work for cases where we'd like to directly run: node -r babel-runtime node_modules/MONOREPO_PACKAGE/script.
@niieani Do you have example where you require different babel configs for different packages in a monorepo? I'd have thought they'd be very similar, and thus something like https://github.com/babel/babel/issues/5451 might be enough for your case.
@loganfsmyth I imagine when having both front end and back end code in your monorepo you can have different babel configs. Also personally I like to be explicit about my deps, even used transforms, and I do not include a jsx transform when transpiling utility library even though in other directory of the monorepo I have a React UI library
@loganfsmyth exactly as @Andarist says. We have both CLI and Frontend packages in the monorepo, at the same time we'd like to be able to open source some of the packages, so each package in the monorepo should be able to be run without the dependency of the root directory of the monorepo. #5451 introduces such a dependency, i.e. the package won't work when it's moved out of the monorepo into it's own repository via a git subtree.
Sounds good, thanks for the clarification. So it is not that there is are package-level .babelrcs that might merge with the one at the monorepo root, it's essentially that each inner package could be thought of as having its own root.
As for me - that would be desired. If I'd like to extend some other config I might do that easily with .babelrc.js, but in most cases I just want to keep specific babel config per package, even if they all live in monorepo at the moment.
Meteor supports nested .babelrc files, for essentially the same purpose that @sokra cited above: https://github.com/babel/babel/issues/6766#issuecomment-342706807
We do not attempt to merge multiple .babelrc files in ancestor directories. The first one found in an ancestor directory within the project root takes precedence.
I'm strongly in favor of only using the root babelrc, since config files in nested directories are really easy to miss.
@nicolo-ribaudo So don't use nested .babelrc files in your own projects. Taking that flexibility away from developers who want it is paternalistic and not helpful.
@benjamn Would https://github.com/babel/babel/issues/5451 address any of your concerns?
That would potentially allow you to do something like
{
overrides: [
{
test: "./client",
extends: "./client/.babelrc.js",
},
{
test: "./server",
extends: "./server/.babelrc.js",
},
]
}
so the root config could still easily delegate to another config.
I could also imagine the monorepo case being handled like
{
overrides: [
{
test: "./packages/*",
extends: "<TEST>/.babelrc.js",
},
]
}
for instance, so the root config would control everything, but still allow for flexibility.
Alternatively we could have the root config allow a relativeSearch: boolean flag, so if set, it'd load the root config, but then do search relative to the file.
Maybe we could even do overrideConfigs: "./packages/*" to automatically search in the given path for configs, and then essentially allow overrideConfigs: "**/*" to essentially say "search everywhere" like we do now.
I feel like I'd want the file to merge on top of the root config by default, but maybe we allow config files to say overwrite: true to overwrite existing config values when they load?
My point is, there are tons of options that would allow essentially the same flexibility. I just want it to be explicit instead of implicit in Babel's core behavior, because the implicit relative lookup is causing problems, as stated in the original issue.
AVA, which uses Babel to compile test files, can be directed to inherit from a user's Babel config. We only look for .babelrc files in the project directory (next to the package.json file), or in the package.json file itself. This was done so the Babel options only need to be resolved once, rather than for each test file.
In other words I'm 👍 on this proposal (as stated in @loganfsmyth's opening comment).
(I also agree with https://github.com/babel/babel/issues/6766#issuecomment-342595840)
I occasionally create a nested .babelrc file in my tests directory, so that certain transforms only apply to test files (not the same as "only transform in the test NODE_ENV"). Similarly, I might want certain transforms in a dev-only directory, and different ones in a production directory.
How would I do this with this proposal?
Just my 2 cents. I would like to explicitly pass the configuration file location and in addition I would disable completely the config lookup (not a BC since it's a new option).
This would solve many issues with "how can I use Babel from this dir? can I store my config there? ..." and probably some .babelrc config inheritance issues.
@ljharb What are your thoughts on my comments in https://github.com/babel/babel/issues/6766#issuecomment-346508545 I'm definitely not looking to break that usecase, though I am promoting changing the implementation and mental model behind how they would fit in.
If I had to summarize the goal of this issue, it's to try to clarify that Babel configs are configuring a project as a whole with compilation logic, rather than defining configs for specific files. That means files in node_modules shouldn't have their configs processed unless the root project asks for that, and project-root configs should be able to influence how node_modules are compiled. As Node gets more and more ES6 features, people publish more and more modules to npm with ES6 modules. We can tell people to compile to ES5 before publishing all we want, but modules containing ES6 are going to happen either way, and I think it is important that we position Babel in a place where it can address those concerns without it being extremely painful for users.
Additionally, I'm _very_ concerned about the pain users will feel if they run Babel 7 and some of their node_modules have published .babelrc files meant for Babel 6. Things will fail quite badly and they'll have no way to fix it.
Certainly, eslint 4+'s glob overrides approach would address the use case, but then it doesn't allow me to "hide" the overridden settings inside the directories that have different approaches. Perhaps this isn't a big deal.
It's a fair point that people who try to transpile node_modules (which I hope are still ignored by default), would run afoul of this issue simply by a module using different plugins/transforms than they have installed, and even more so with babel 6 vs 7.
Perhaps - tangential to this issue - babel configs for 7+ should be required to explicitly declare their version number?
Additionally, perhaps those who choose to transpile code they didn't write should be required to explicitly specify the location for the config (whether that's a specific .babelrc, or a specific package.json's "babel" section)?
Specifying parent config and override paths seems like heavy config for answering the question of should this config be treated as done or rolled into a parent config. I think something like a "root": true prop could be used to answer yes or no to whether to roll-up the config to the parent or not. So folks wanting separate configs for tests can just add a "root": true to those configs and the default behavior would be that it rolls-up to the parent.
I think something like a "root": true prop could be used to answer yes or no to whether to roll-up the config to the parent or not.
We already stop at the first .babelrc that is encountered, so root: true wouldn't have any effect. Users currently instead have to opt in with a parent like extends: "../../.babelrc" if they wanted to have a parent config continue applying.
I should clarify my main goal on this.
The primary issue at the core of this is that people often expect their project's .babelrc to be that, a project-level configuration. By having the files resolved relative to each individual file, it means we pick up entirely unrelated .babelrc files in node_modules. In Babel 6, we occasionally got reports early on because people had published Babel 5 .babelrc files. When we publish Babel 7, that's 100% going to happen again, and as we start publishing new versions of Babel more often, it's going to get even more confusing for users, especially if we decide to make more breaking changes in the future.
Even in the case of dependency modules with .babelrc _for_ Babel 7, even if it is for the same version, the config file could easily have been one that _already_ ran to create the code that is already published to npm anyway. The only place that actually knows what config a node module should actually use is the root application itself, not the node module itself.
An alternative to my original proposal would be that we check if a file is in node_modules, e.g.
project/
node_modules/mod/
.babelrc
src.js
.babelrc
If this project tries to compile project/node_modules/mod/src.js, we could immediately jump upward and start resolving above the node_modules folder, thus finding the project-level .babelrc
The primary edge case there is that I guess technically your project itself could for some reason be inside a folder called node_modules, but we could certainly accept that as something that will just not be supported by users.
To expand, right now we do:
const filepath = ...
let dirname = path.dirname(filepath);
while (true) {
if (exists(path.join(dirname), ".babelrc")) // ...
const parent = path.dirname(dirname);
if (parent === dirname) break;
dirname = parent;
}
but you could imagine us changing this to (as non-production code anyway)
const filepath = ...
let dirname = path.dirname(filepath);
// Chop off everything after the first "node_modules" in the path
let parts = dirname.split(path.sep);
let modIndex = parts.indexOf("node_modules");
if (modIndex !== -1) {
dirname = parts.slice(0, modIndex).join(path.sep);
}
// Resolve as usual.
while (true) {
if (exists(path.join(dirname), ".babelrc")) // ...
const parent = path.dirname(dirname);
if (parent === dirname) break;
dirname = parent;
}
but you could imagine us changing this to (as non-production code anyway)
I think that's how I thought it would work. Here's how I resolve config for std/esm:
I walk backwards in the directory structure, stopping at node_modules with a return of null if a config is not found. The mechanism is called getting package info (PkgInfo). So it's gated to packages. This means if I roll up config it stops at a package boundary. This also means I could add support for a "root":true to indicate not rolling up to the package boundary and stopping a that given config.
For folks that want to force a set of options across package boundaries we have a programmatic convention for that.
Update:
Clarified things a bit.
I walk backwards in the directory structure and stop at "nod_modules". The mechanism is called getting package info. So it's gated to packages.
Could you clarify that or point me at particular code?
In my description the config _inside_ node_modules be be explicitly ignored, which I think is the opposite of what you're describing? Though that may make more sense for std/esm.
I treat node_modules as the boundary (so ignored). If it walks back and hits node_modules it returns null to indicate not finding a config since it's gone outside its package boundary.
Gotcha. So for us I'm saying we'd explicitly never load configs a dependency package's own config, and then (maybe as an opt-in), the project-level .babelrc could apply to dependency packages.
I think we're saying the same thing. Treat packages as isolated independent bits that don't step on the toes of dependencies unless opted-in to doing so. It's very much the Node package way.
Cool. I think the edge case that concerns be is that technically someone could have
node_modules/
project/
node_modules/mod/
.babelrc
src.js
.babelrc
Babel doesn't really know that project/ is the root, so for instance of a user just does require("babel-register"), how does it know what directory it is supposed to be treating as the "root"?
I'm fine with defaulting to the cwd, and leaving it to users to pass an argument if that's not the case. Thoughts?
Babel doesn't really know that project/ is the root,
Have Babel do the Node thing and look for a package.json. The config at project/node_modules/.babelrc would essentially be ignored since it has nothing to apply to.
That has the same problem, since you need a directory to look for a package.json relative to.
I'm not seeing the problem. Follow the established Node way of resolving packages and don't apply configs across package boundaries. Seems totally doable (It's the Node way after all).
Let's take a step back. Given
node_modules/
project/
node_modules/mod/
.babelrc // some other config that the user accidentally published
lib.js
.babelrc // Babel 7 config for whole project
app.js
and I've got project/.babelrc that I want to apply to the whole project, including the mod/lib.js file.
Let's say Babel is called via babel-loader for instance, and the user wants their whole config. That will eventually call through to Babel along the lines of
babel.transform("code;", {
filename: "/some-root/node_modules/project/node_modules/mod/lib.js",
});
How does Babel know that I want it to use the project-root config /some-root/node_modules/project/.babelrc? We specifically _don't_ want it to use /some-root/node_modules/project/node_modules/mod/.babelrc because that config has nothing to do with the project that is actually being compiled, so Babel needs to know to start searching parent directories, starting from /some-root/node_modules/project.
What I meant with my comment above is we could use process.cwd() by default, and then allow users to do
babel.transform("code;", {
filename: "/some-root/node_modules/project/node_modules/mod/lib.js",
cwd: "/some-root/node_modules/project",
});
including the
mod/lib.jsfile.
You mean applying myproject/.babelrc to myproject/node_modules/mod/lib.js?
That crosses a package boundary so I think it shouldn't be allowed by default. I would not assume my project config affects unrelated third-party packages by default. I consider third-party packages to be their own separate independent thing.
To put it another way:
I have an app that uses Lodash.
I wouldn't want my application's linting rules to be applied to the third-party Lodash package.
Chances are Lodash has completely different linting styles than my application.
In the same way I wouldn't want my application's Babel config to be apply to unrelated third-party packages.
I wouldn't assume my project config affects third-party packages by default. I consider them their own separate-independent thing.
The core of this issue is that that _is_ I think what most people expect, for the case where users are running Babel on node_modules (which isn't most cases anyway). A sub-module's .babelrc is usually of _their_ build process. It is based on their dev dependencies that could be more an entirely different Babel version, that probably hasn't even been installed if it is a project's dependency. The plugins in a dependency module are build-time logic for that dependency only.
For instance, take a module cares about being compatible with Node 4 and thus leaves block scoping (as a simple example). Say a user is bundling an application for use on the web and they want to bundle a dependency that includes let and const, they would absolutely want their Babel config to be application-level, and apply to any file that they've loaded babel-loader for.
A dependency node_module's .babelrc should be explicitly ignored, because that is unrelated to the compilation target logic for your application as a whole. A given application could very well have entirely different compilation expectations from the dependency itself, while still caring very much how the dependency compiled itself originally.
for the case where users are running Babel on node_modules (which isn't most cases anyway)
Right, it's not most cases and goes against the standard Node way™ of handling packages. This is where an opt-in (so not the default) mechanism to force a config to be applied across package boundaries would be used.
To be clear, I care about changing Babel to automatically opt _out_ of processing .babelrc files in node_modules. I do not necessarily think Babel should automatically opt _in_ to having the project-level config act in its place. So
babel.transform("code;", {
filename: "/some-root/node_modules/project/node_modules/mod/lib.js",
});
could just default to not using _any_ config.
But even in that case, Babel needs to have a sense of "root" so it can know what qualifies as an ignorable config or not.
At the core of this, the .babelrc in a dependency module has nothing to do with the compilation requirements of the running application. Otherwise you're basically requiring that a .babelrc in a dependency module needs to be considered part of the public API of a module, which would essentially make any changes to a published .babelrc in your module qualify as a breaking change, and provides no way for you to run multiple Babel version in parallel since plugins and such have no guarantee to work across versions, or even be installed. No-one thinks of their Babel config as a public API of their module.
To be clear, I care about changing Babel to automatically opt out of processing
.babelrcfiles innode_modules. I do not necessarily think Babel should automatically opt in to having the project-level config act in its place.
Cool.
Stepping on third-party packages is tricky and should at least be given the proper disclaimers in documentation. A third-party package may use babel-plugins that your application does not, so if you were to want to force a config on them you'll have to be aware of those plugins and include them in your application too. I don't think, in those cases, that the extra leg work is unreasonable since you're wanting to apply a config across all code regardless of package boundary.
For std/esm we allow devs to opt-in to applying a config to all modules (even third-party) with a programmatic API. Folks can choose to supply their own config or pass true to apply their package level config to all other third-party modules. With that you get what you get though. If you specify all code be treated as ESM and there is CJS modules in third-party packages then you're gonna have a bad time.
The issue for me is API future-proofing. We've made breaking changes in the past and we'll make them in the future, and Babel has a sizable API surface area. A .babelrc is inextricably tied to a specific Babel version, and specific set of installed plugins. A .esmrc has a dramatically smaller area to cover.
Given those limitations, I personally think that with Babel's current approach, it would be entirely irresponsible to publish a .babelrc with the expectation that users of your dependency would actually run your specific plugins. The expectation is that any plugins you may rely on would have been run before your module was published. This is what motivates my goal of explicitly ignoring config files inside dependencies by default.
Stepping on third-party packages is tricky and should at least be given the proper disclaimers in documentation.
I honestly don't see it as stepping on them, because I just don't see publishing them with any kind of expectation that they will run to be a valid expectation.
A third-party package may use babel-plugins that your application does not
This is not a usecase that we've ever aimed to support, to the best of my knowledge. Plugins should have run before the package was published.
For std/esm we opt-in to forcing config on all modules (even third-party deps) with a programatic API.
I don't know that I'd want it to require programmatic use, but I would be totally fine with it being an opt-in behavior.
With that you get what you get though. If you specify all code be treated as ESM and there is CJS modules in third-party packages then you're gonna have a bad time.
Yeah totally fair, which does make be agree that using the project config on dependencies should be opt-in, at least until we've seen people using the opt-in behavior in the wild for a while and aren't getting spammed with requests that it be the default.
Alright, I'm going to explore leaving the folder-hierarchy search in place, but essentially skipping the resolution of .babelrc files entirely for things within node_modules. It's certainly a step in the right direction compared to the current behavior.
I will say, I think there is one case that I didn't mention in the original issue, which would be resolved by treating Babel's config as a project build config, rather than the current approach. That is for symlinked development dependencies. I'm 100% behind published modules being precompiled, but there's also very much a subset of users, either with a monorepo, or even just normal, where you'd want a single Babel config in the module doing for example a Webpack build.
I can absolutely see users doing
modules/
pkg-a/src.js
pkg-b/src.js
app/
node_modules/
pkg-a -> ../../pkg-a
pkg-b -> ../../pkg-b
.babelrc
index.js
webpack.config.js
and currently the _only_ way for users to configure compilation logic for pkg-a is to copy-paste some of your Babel config into your webpack config, because Babel has no way to know that the modules are symlinked, and the .babelrc is nowhere in the folder hierarchy of pkg-a.
Whereas, if babel-loader essentially said "load the local config, and apply it to anything that the configs should apply to", it would potentially be easier for people to handle cases like this.
@loganfsmyth If pkg-a and pkg-b are installed via npm link, then the npm link command should have run npm build, which should respect any .babelrc found in modules/pkg-{a,b}/, provided those packages compile themselves with Babel. It seems strange to override that logic with some other .babelrc file those packages never anticipated. If you control pkg-{a,b} as well as app, within some workspace, maybe you could have them all symlink to a shared .babelrc file?
As for .babelrc files inside node_modules, I agree it's hard to distinguish between an application with a .babelrc file that happens to be inside a node_modules directory, and the packages it depends on, if all you know is the absolute file path.
Suppose we don't ignore .babelrc files inside node_modules. Instead of having a .babelrc file in the root package directory, perhaps packages could have a src/.babelrc file that controls how src gets compiled into lib, so that src/.babelrc would apply to src/some/module.js when it gets compiled to lib/some/module.js. Other packages would not attempt to apply node_modules/<package>/src/.babelrc to node_modules/<package>/lib/some/module.js, because it's already been compiled, and no .babelrc file is found for node_modules/<package>/lib/some/module.js.
I realize that might not match what package authors do today, but I think it demonstrates the power of folder-hierarchical .babelrc lookup.
If pkg-a and pkg-b are installed via npm link, then the npm link command should have run npm build
What if the linked packages are just ES6 modules that don't have a build process at all because they are mostly targeted at running on Node? The _vast_ majority of the time that I'd expect people to be compiling node_modules is for packages that distribute ES6 for Node, and someone wants to package them up for a browser. This usecase is only going to get more common as time goes on.
which should respect any .babelrc found in modules/pkg-{a,b}/, provided those packages compile themselves with Babel.
What if that .babelrc is entirely unrelated to the browser-based build you are trying to do? Chances are quite hight that the developer published it entirely on accident, especially because that config might just be a "babel" field in their package.json.
I honestly don't see much of a distinguishing factor between a dependency that might be symlinked in, vs something that isn't, from the standpoint of someone who wants to run a transform on that file when they are trying to bundle it.
Suppose we don't ignore .babelrc files inside node_modules. Instead of having a .babelrc file in the root package directory, perhaps packages could have a src/.babelrc file that controls how src gets compiled into lib, so that src/.babelrc would apply to src/some/module.js when it gets compiled to lib/some/module.js.
That feels like a technical solution to what amounts to a babel user-level issue. People are always going to do the easy thing, which is putting the .babelrc at the root of the repo. Maybe not everyone, but a larger subset of people. Not to mention that there are already tons of packages published with this as an issue.
I also feel like I have yet to hear of a usecase where a developer would _want_ to publish a package with a .babelrc that the user is expected to run. If you need a source transform on your code for it to be minimally functional, that seems like something you need to have done before publishing anyway.
I realize that might not match what package authors do today, but I think it demonstrates the power of folder-hierarchical .babelrc lookup.
It absolutely demonstrates power, but I'm not at all convinced that it is power that people actually want, or power that the average user would use properly.
In what case would we want or expect a random dependency to be able to interfere with your build process? Like I mentioned, that's essentially making the .babelrc of a module part of said module's public API. What if one of your modules defines a .babelrc for one version of Babel and one for another? Who is responsible for installing the plugins that the dependency .babelrc references? None of these questions have clear answers.
Thanks for taking the time to respond to those ideas; I hope it was a useful thought process, even though you've got good reasons for disagreeing.
I also feel like I have yet to hear of a usecase where a developer would want to publish a package with a
.babelrcthat the user is expected to run.
This got me thinking about how package-lock.json is only supposed to apply to cloned source code, not packages installed from npm, which is why npm never publishes package-lock.json.
By analogy, the problems we've been discussing might be simpler if npm also did not publish .babelrc files, because then Babel wouldn't have to worry about distinguishing between .babelrc files in node_modules versus those in the application itself, because there wouldn't be any .babelrc files in published/installed node_modules packages.
Obviously the npm tool itself is never going to hard-code special treatment for .babelrc files the way it does for package-lock.json files, so that's not what I'm suggesting.
Somewhat more realistically, one might hope that package authors would put .babelrc in their .npmignore files, but I think I can guess your reaction to that idea: plenty of authors won't know/remember to do that.
So we're left with the question of how to tell if a .babelrc file belongs to an installed npm package, or belongs to the source application, keeping in mind that the source application could be in a node_modules directory (which shouldn't matter, but complicates the discrimination problem).
Edit: This won't work for packages installed by yarn, since yarn doesn't add any special metadata to package.json.
When a package is installed from npm, its package.json file gets a whole bunch of additional metadata properties:
{
"_from": "recast",
"_id": "[email protected]",
"_inBundle": false,
"_integrity": "sha512-OwBhvGNxrL0aP5xhuVM0ms3UDYVG/uSfhfZjbPklRzoKiXukDbDmLqkfM6CtY/hkTTtAGizcMCp3XkqagctWOQ==",
"_location": "/recast",
"_phantomChildren": {},
"_requested": {
"type": "tag",
"registry": true,
"raw": "recast",
"name": "recast",
"escapedName": "recast",
"rawSpec": "",
"saveSpec": null,
"fetchSpec": "latest"
},
"_requiredBy": [
"#USER",
"/"
],
"_resolved": "https://registry.npmjs.org/recast/-/recast-0.13.0.tgz",
"_shasum": "a343a394a37d24668d700f88ed04b8d2c314d40d",
"_spec": "recast",
"_where": "/Users/ben/exp/npm-link-build-test",
...
These underscored metadata properties could be used to tell the difference between a package.json file in a source application and one in an installed npm package.
tl;dr To decide whether to honor a certain .babelrc file:
package.json file, by searching ancestor directories, stopping at any node_modules directory (to avoid using a package.json file from a parent package).package.json file contains some reliable set of metadata properties (maybe just the _resolved property?), then ignore the .babelrc file, because it's in a published npm package..babelrc file.What do you think?
@benjamn yarn does not add this metadata, only npm does. In any case, I think this discrimination is too fragile.
@benjamn
Obviously the npm tool itself is never going to hard-code special treatment for .babelrc files the way it does for package-lock.json files, so that's not what I'm suggesting.
It couldn't hurt to open an issue on npm and yarn to ignore .babelrc and .babelrc.js files. Or open an issue on both for a standard way to detect an installed package.
@niieani Good point. Is there anything else that could reliably distinguish a package installed by yarn?
@jdalton Re: ignoring .babel* files, I suspect their answer will be "that's what .npmignore files are for." Re: standard way to detect installed packages, that's a promising idea, but I think @loganfsmyth wants a solution sooner than that would happen.
👆 cc'ing npm and yarn folks for visibility on this \@zkat @BYK
Wait, when would a source application be inside node_modules?
@ljharb https://github.com/babel/babel/issues/6766#issuecomment-347991494
This scenario (rare though it may be) would not be a problem if Babel knew which directory should be regarded as the application root, which is another possibility that we've been discussing here (e.g. in @loganfsmyth's original issue description). Knowing the root directory would also prevent Babel from accidentally using some stray .babelrc file outside the application root, when there are no intervening node_modules directories. And Babel could very easily ignore .babelrc files inside node_modules directories _inside_ the application root directory, if it knew which directory was the root. In short, I'm in favor of supporting (and strongly recommending) a babelOptions.root option, as @loganfsmyth has suggested, with process.cwd() as the default. It seems like that would make everything easier.
Is that contrived, or does that really happen?
It seems like it would be much, much easier if a) babel is ran only in projects, and b) projects never live inside node_modules. Then, the root directory for the project would be "the closest package.json", just like everything else in node.
Babel is extremely widely used, so even the most contrived edge cases will produce a steady trickle of bug reports.
If I had to guess, an application inside a node_modules directory is less common than a stray .babelrc file in some ancestor directory of the application, but both are edge cases worth considering for a tool like Babel.
There's a relatively easy solution here that happens to solve a lot of problems, including these contrived edge cases—specifically, adding a Babel option to specify the application root directory. I think that's what we should do.
I totally agree a root option makes sense - but I think that option should only be necessary in the edge case where a project is inside node_modules; I think the default should be the common case.
Since you mentioned running Babel on projects, it sounds like you're thinking of Babel as mostly a command-line tool where process.cwd() is usually the application directory. If that was all we had to consider, I agree users shouldn't have to specify the root directory, but we also have to consider the programmatic babel.transform(code, options) API.
From the perspective of the babel.transform function,
babel.transform(source, {
filename: "/some-root/node_modules/project/node_modules/mod/lib.js",
});
is simply not enough information, whereas
babel.transform(source, {
filename: "/some-root/node_modules/project/node_modules/mod/lib.js",
root: "/some-root/node_modules/project",
});
is just what we need. If the programmatic babel.transform API starts requiring babelOptions.root, the command-line tool can automatically provide process.cwd(), so the developer doesn't have to. If you're using the babel.transform API directly, though, I think it's pretty reasonable for babelOptions.root to be required, though it wouldn't be the end of the world if it just defaulted to process.cwd().
I would also expect babel.transform to only be run inside a project, or inside a binary ran inside a project.
Perhaps of some use: http://npm.im/find-npm-prefix
Dang this reply got long! AHHH
@zkat Cool, thanks! I'll keep that in mind if we decide to use semantics along those lines.
@ljharb
Is that contrived, or does that really happen?
Fair question. The the only case that comes to mind is someone publishing a CLI application and just including babel-register as part of the project expecting their own code to compile on the fly. The bug was that Babel didn't quite work, though I don't totally remember why honestly, but it was a case where someone was running Babel via babel-register on an installed CLI program, so the project calling babel was inside node_modules.
It seems like it would be much, much easier if a) babel is ran only in projects, and b) projects never live inside node_modules.
I'd also potentially be fine with basically chopping off everything from the first node_modules in the path. We could potentially make that the default for "root" instead of having the cwd.
The one edge case here that makes it an rough edge is that Babel, _most_ of the time, gets a realpath. That means that while using the cwd is _hopefully_ going to be the project root consistently, if we use your approach of using the filepath and expecting "project" to not be in node modules, linked modules end up being their own "root" since they are probably sibling modules that are linked into a main project, and thus the realpath does not include node_modules. That said, it may or may not matter depending on how we decide to use the "root" value. I'll expand more below.
Then, the root directory for the project would be "the closest package.json", just like everything else in node.
I think there's an interesting question here. A package's root is absolutely the closest package.json, but I think it's not clear for monorepo cases like ours, and kind of goes to the other question about how the root is used. While a package absolutely has a root, there isn't necessarily a direct expectation that the build process is rooted at the package boundary.
@benjamn Definitely appreciate the feedback and help thinking this through.
It seems there are a few ways that we could consider taking resolution of configs.
.babelrc for "root" packagesEntirely skip searching for config files that are not part of the "root" package. Note that this would requiring that users that wish to compile node_modules packages embed their config either in webpack.config (for example). To me, that seems like it would be extremely painful for users.
One thing I'll say is that users could work around that by doing
options: {
extends: "./.babelrc",
}
to essentially say override whatever you found with the root config, potentially alongside babelrc: false, since programmatic options are merged on top of any filesystem-level config.
That said, I personally feel that it is still pretty ugly to use, and is unintuitive for users.
If Babel is told to compile node modules or even specifically linked modules and "root" is calculated from the file realpath instead of from the cwd, linked modules would end up counting as their own "root", which means Babel would try to read the .babelrc file for the linked packages, but nothing else in node_modules. I find that to be a pretty weird difference between linked and non-linked things.
Search for config files starting from the first directory inside the root package, e.g. project/packages/some-thing/node_modules/mod/index.js would start searching for .babelrcs relative to project/packages/some-thing since it is the first parent that isn't in node_modules.
This would allow users to use a .babelrc in their package to configure plugins that should apply to their node_modules as well, since
project/
.babelrc
node_modules/
mod/
index.js
would jump up the package boundary, and then find the .babelrc.
I do want to clarify, we could _still_ make .babelrc files be able to filter what they apply to be default. So this would _not_ require that your project config applies to node modules by default, since people seemed to have an aversion to that above (and I do too). So you could imagine requiring users do like
{
test: "."
}
for the config to say "apply to everything inside this directory", with a default value of more like
{
test: [".", "!node_modules"],
}
so by default a .babelrc would still be restricted to applying to files in the package that it is inside of, but could opt in to applying to node modules.
This still has mostly the same limitations mentioned for the previous point.
Pretty much the original proposal. I'm including it because I think it still has some benefits compared to the above points and it is worth comparing, but I totally concede that it is a more dramatic departure from the current behavior than the other options.
If we don't search, and instead load the config explicitly based on some "root" config plus explicit config referenced from that config.
The main criticism is that people very much feel that config via directory hierarchy is a powerful approach. I'll also add that another downside of this is that, given an arbitrary filename, it is not possible to know which config is meant to apply to that file. I think that grants _flexibility_, but also complicates things.
The upside of this approach, mostly if the "root" was defaulting to the cwd, is that linked modules would be able to pick up the config automatically (potentially with a requirement that the config opt into that using a similar test: restriction along the lines of the above).
I'll also add that I like it because it means the logic is extremely easy to describe to users, and it can make config loading require fewer assumptions about configs. For instance right now, for performance, we assume that if a config was not found in a directory, it will _never_ be found, so if users add a new config to a directory, restarting your Babel process is the only want to reset that cache.
So the big questions then seem to be
I think we could support compilation of linked modules in an opt-in way, without relying on Webpack configuration, and without any additional effort for developers who don't need this behavior, by allowing the babelOptions.root option to specify multiple root directories (and then presumably it should be called babelOptions.roots).
Given this directory structure:
modules/
pkg-a/
.babelrc
src.js
pkg-b/src.js
app/
node_modules/
pkg-a -> ../../pkg-a
pkg-b -> ../../pkg-b
.babelrc
index.js
you could imagine calling babel.transform this way:
babel.transform(source, {
// Babel will call fs.realpath to normalize this filename, allowing us to look up any
// relevant .babelrc files using the real path.
filename: "/path/to/modules/app/node_modules/pkg-a/src.js",
// If we hit any of these directories while searching upward for a .babelrc file,
// stop the search. Also stop the search if we hit a node_modules directory before
// one of these directories.
roots: [
// Everything inside this directory that isn't inside a nested node_modules directory
// will be compiled with /path/to/modules/app/.babelrc.
"/path/to/modules/app",
// After Babel calls fs.realpath on the filename above, it will be able to determine
// that src.js is contained within this directory (with no intervening node_modules).
"/path/to/modules/pkg-a",
// Since Babel calls fs.realpath on all these paths anyway, this works too.
"/path/to/modules/app/node_modules/pkg-b",
]
});
Essentially, each additional root directory "activates" any .babelrc files contained under that directory, excluding any nested node_modules directories.
With this strategy, we would not use app/.babelrc to compile pkg-a/src.js, because that's not in any ancestor directory of the real path of src.js, but we would be able to opt into using pkg-a/.babelrc if desired.
I suspect the use case of compiling linked packages is rare enough to justify this level of extra effort by interested developers.
Is resolving configs based on folder hierarchy (by default, not by explicit opt-in), actually powerful enough to justify the fact that we need all this logic to decide where to start searching from?
I think it's hard to gauge the complexity of an approach without having some concrete implementation strategy to frame the evaluation. Otherwise "all this logic" is impossible to quantify.
As for how powerful this technique is, most Meteor apps have separate app/client/ and app/server/ directories, and I think it makes good sense to allow those directories to have their own .babelrc files, drawing from app/node_modules/. Granted, it's a bit easier for Meteor to determine automatically that app/ is the root directory, since it contains the app/.meteor/ directory, but I think this strategy remains appealing even if you have to specify the root(s) explicitly.
If the strategy I've proposed seems feasible, I think .babelrc should behave like other dot files (e.g. .gitignore), which tend to be nestable/hierarchical/closest-wins.
Babel will call fs.realpath to normalize this filename
I should clarify, it's not Babel that does this, but Webpack and Node themselves generally do this before babel-loader and babel-register call through to Babel.
Essentially, each additional root directory "activates" any .babelrc files contained under that directory, excluding any nested node_modules directories.
With this strategy, we would not use app/.babelrc to compile pkg-a/src.js, because that's not in any ancestor directory of the real path of src.js, but we would be able to opt into using pkg-a/.babelrc if desired.
To me, that seems like the opposite of what I'd expect a developer to want, because the linked module has no way to know what build targets the actual app might have. The linked module may not even have a build process if it is for instance a library that runs on both Node and is available for packaging in a browser.
I suspect the use case of compiling linked packages is rare enough to justify this level of extra effort by interested developers.
Interesting. I feel like npm link-based workflows of many sibling directories are pretty common. Has your experience been otherwise?
I think it's hard to gauge the complexity of an approach without having some concrete implementation strategy to frame the evaluation. Otherwise "all this logic" is impossible to quantify.
Yeah that's certainly fair.
As for how powerful this technique is, most Meteor apps have separate app/client/ and app/server/ directories, and I think it makes good sense to allow those directories to have their own .babelrc files
I'm 100% behind supporting allowing nested configs, I just kind of wanted to make people opt into that behavior via the root config, rather than have it be automatic.
If the strategy I've proposed seems feasible, I think .babelrc should behave like other dot files (e.g. .gitignore), which tend to be nestable/hierarchical/closest-wins.
I think part of my reasoning here is that I kind of wish Babel's config _wasn't_ a dotfile. The same way we have a webpack.config.js instead of a .webpackrc, I think that the default config for Babel is inherently focused on building a given project as a whole, rather than about defining hierarchical rules for files based on directory structure. Hierarchy can certainly be a side-effect of the configuration process, but I don't feel that the very idea of configuring a build tool implies hierarchy based on folder structure. Folder structures are just too brittle, as we're seeing with linked modules and node_modules here.
I'm sure you've considered this before, but what if the "babel" field of package.json could have a string value, identifying a local JS module that dynamically configures babel.transform options?
module.exports = function (configApi) {
configApi.enableDirectory(".", (filename, hash) => {
// Return babel.transform options for filename.
// The "." path is relative to the directory containing the control file.
// By default, files inside nested node_modules directories are ignored.
});
// Selectively enable compilation of some node_modules:
configApi.enableDirectory("./node_modules/pkg-*", (filename, hash) => {
// The implementation is free to use the same logic it used above.
});
};
Since this API is dynamic, automatic caching of the computed options might be more difficult, but the dynamism cuts both ways, because the control file can potentially do its own caching, so Babel doesn't have to manage that logic at all.
Having thought about this more over the last 2 weeks, I think where I've settled on the following:
root option that defaults to cwd. Babel will only search for .babelrc during compilation if the file being compiled is (a) inside the root (b) not inside a node_modules. This is already essentially the same logic we have in babel-register's automatic ignore. If we ended up defaulting sourceType to script we could even consider removing that from babel-register entirely, since core might be able to bail out fast enough to just run on every single file that is required.babel.config.js file inside the root directory, which will apply to _any_ file being compiled by Babel, just like programmatic options would.configFile option for users who want to set an explicit config file.babelrc: false. If users want local babelrc files, require them to set babelrc: true in their config file or programmatic options. This would for the current behavior while not doing a ton of extra work unless users opt in.I think this would address most of the current major concerns while keeping the existing behavior relatively consistent for the average case, at least inside your own package.
For other raised points:
node_modules, we can add a new symlinks option that would be allowed in config files and programmatic options, which Babel could resolve. If the file being compiled matches a symlink, it could essentially remap the passed-in filename to see if it matches any of the overrides from https://github.com/babel/babel/issues/5451.babel-eslint we probably want to find the closest package.json and use that as root, since you're almost never going to be linting something in node_modules, and users could always use a .eslintrc.js to explicitly pass in a root option in parserOptions.babelrc option to have a way to opt into the full current behavior of looking for .babelrc files, even in node_modules and outside root, but I'm not totally convinced it is a usecase that people actually want.1) Is there any existing documentation about the .babelrc.js file?
2) Is there any alternative to have specific env configuration overrides without external configuration files (but babel section in package.json)?
I'm currently using this (package.json):
"babel": {
"presets": [
[
"@babel/preset-env",
{
"modules": false,
"useBuiltIns": "usage",
"debug": true
}
],
"@babel/preset-stage-0",
"@babel/preset-react"
],
"plugins": [
"lodash",
"@babel/plugin-proposal-decorators",
[
"@babel/plugin-proposal-class-properties",
{
"loose": true
}
]
]
},
All I'd like to do is having "debug: false when NODE_ENV=production.
I just want to override @babel/preset-env configuration.
But I don't want to copy over everything again!
IMHO, compilation of node_modules is not a good pattern.
<script type="module"/> if you need Babel compilation of unreleased or non-spec JSnode_modules.Instead, library authors should just pick an arbitrary minimum supported Node.js version and compile down to that with the env preset and bundle as (CJS and MJS) with Rollup. When Node 10 is released, it will automatically pick up the .mjs version.
If they support browser usage, authors should pick which ECMAScript specs they support (or multiple specs).
babel-preset-envThen the package.json looks like this:
{
"files": [
"dist"
],
"main": "./dist/index",
"module": "./dist/index.mjs",
"browser": {
"./dist/index.js": "./dist/browser.es5.js",
"./dist/index.mjs": "./dist/browser.es5.mjs"
},
"es2015": {
"./dist/browser.es5.mjs": "./dist/browser.es2015.mjs"
},
"es2017": {
"./dist/browser.es5.mjs": "./dist/browser.es2017.mjs",
"./dist/browser.es2015.mjs": "./dist/browser.es2017.mjs"
},
"engines": {
"node": ">= 8"
}
}
dist/
├─ index.js // CJS for Node.js 8+
├─ index.mjs // MJS for Node.js 10+
├─ browser.es5.js // ES5 CJS for browserify and legacy consumers
├─ browser.es5.mjs // ES5 MJS for webpack and legacy browsers
├─ browser.es2015.mjs // ES2015 MJS for weppack and modern browsers
└─ browser.es2017.mjs // ES2017 MJS for webpack and evergreen
Then folks using webpack can just add extra enhanced-resolve fields and opt-into newer ECMAScript versions. Everything is backwards compatible with progressive enhancement.
It's not a big burden for authors to do this (I've made a helper CLI that does this out of the box)
FYI, we're starting the work to compile deps with babel-preset-env in Create React App: https://github.com/facebookincubator/create-react-app/pull/3776
Let us know if you have feedback about how this should work. Note we intentionally don’t plan to respect .babelrc, and will only compile “real” JS features without JSX/Flow/etc.
@loganfsmyth I wonder if it might be worth allowing the babelrc option to be a string if the API user wants to specify the exact file to use. I found myself wanting this feature today on a project that somehow wasn't picking up the right babelrc, and it might reduce cognitive load as well if the string value could also take the place of your suggested configFile property.
@guybedford I think that may just confuse things because I generally think users expect .babelrc files to behave strictly hierarchically. It also means people need to know what a babelrc is to search for the concept, and it's not really an obvious name for new users.
FYI there is a discussion about this in parcel as well. https://github.com/parcel-bundler/parcel/issues/13
We will also be supporting compilation of node_modules with babel-preset-env. We will use the package.json engines field, or browserslist field to determine the language of the source module, and the browserslist of the target app to determine what to compile to. Then we do a diff to include only the necessary plugins. That way modules will not be compiled all the way to ES5, but to the app’s browser target instead. https://github.com/parcel-bundler/parcel/pull/559
Most of the discussion now revolves around custom Babel plugins in node_modules, which is convenient for local development with linked modules. I proposed adding a field to package.json to selectively turn on custom babelrc support in node_modules. Would be interested to hear what you all think of that. I’d like to come up with something that can be used across tools, not just something parcel specific.
@devongovett how are you going to get the entrypoint of the untranspiled sources of your dependencies in node_modules?
Both main and module links to transpiled code today.
https://github.com/webpack/webpack/issues/5935
Just made an alternative suggestion to enable distributing both transpiled code and source code together. https://github.com/parcel-bundler/parcel/issues/13#issuecomment-362036014
Alright everyone, I've created https://github.com/babel/babel/pull/7358. Please look over what I've posted there and let me know what you think.
@sokra and others. For this of us searching for a solution to the problem now using Babel 6 how is this usually solved?
I've got a client side repo included in a server side repo via git submodules. I need to import some of the client code into the server side but the babelrc for the client application is screwing up the import due to the different configurations.
@phawxby Feel free to join Slack or post on our recommended support channels, but pinging everyone on this thread about unrelated stuff will distract from the goals of this issue so I'd rather not help here.
A little late to the party, just wanted to add some background for transforming/transpiling packages in node_modules (cc @ljharb).
For maximum performance of front-end bundles, some people (e.g., Phil Walton, myself and the team I work with) want to pull as many dependencies as possible in as uncompiled ES modules. This:
I got bit by the nested babelrc's this week, and would love a root: true or similar feature in Babel 7.
Hey @robwierzbowski thanks for your input! And yeah we have moved discussion a while back to https://github.com/babel/babel/pull/7358 and are going to be moving forward with that. And yes we understand compiling node_modules is a use-case we all want to shoot for, not really the reasoning but how it would work!
Most helpful comment
I've seen nested babelrc files in repos which contain server and client build.