Parcel: 🙋 Automatically install missing dependencies, part 2

[devongovett]

Now that #306 has landed, it would be awesome to take this even further. Currently we automatically install dependencies loaded by the internal localRequire helper. These are things like compilers (e.g. typescript, coffeescript), and plugins (e.g. postcss plugins, parcel plugins, etc.).

It would be super cool to take this even further and transparently install any npm dependencies you require in your code for you. e.g. if you add require('react') in your code, and it's not installed, Parcel can install it for you.

Thoughts?

[devongovett]

cc @DeMoorJasper

Also, looks like there is a webpack plugin that does something similar https://github.com/webpack-contrib/npm-install-webpack-plugin

[brandon93s]

So, to make sure I understand....

Instead of this:

$ git clone https://github.com/project/repo.git
$ yarn install
$ yarn start

All I would need is this:

$ git clone https://github.com/project/repo.git
$ yarn start

Assume yarn start points to parcel {entry} in these examples.

[devongovett]

Sure, or you have parcel already started, and change your code to import a new dependency. No need to ctrl+C and install it anymore, parcel will just install it automatically when it detects the import.

[shawwn]

I get the temptation to make Parcel be completely automatic, but you have
to be careful to weigh the risks.

Parcel will have a good run for a few years. Why will Parcel fail? Because
the codebase becomes too bloated and hard to change, just like Webpack.

I created a fork of Parcel called Parsley, and it was incredibly simple
because we have no configuration. We don't depend on any environment
variables or console settings that users would expect to maintain across
the fork.

Automatically installing dependencies is one of those things that seems
cool at first, but when you think about it, what's being saved? The users
are slightly less annoyed for a few seconds? Typing yarn add foo --dev is
too hard?

Worse, when it causes problems, it will cause problems in a huge way. Users
will run into strange mysterious errors that they'll have to google to get
answers for. "Why is parcel foo.html failing for this Vue project?" "Oh,
yeah, vue-hot-reload-api fails to install due to some bug in yarn. Really
annoying, just run yarn add vue-hot-reload-api --dev manually."

Combine this with the fact that we're spinning our wheels on problems that
don't really matter. We could be focusing on impactful work such as
extending the plugin system so that people have access to the core APIs
from plugins. They can't right now. There's no way to override the behavior
of the methods in Bundler. But we're focusing our thought and energy into
automatically installing dependencies instead, when no one really wants
that aside from a few enthusiasts. And those users are important, but
they're not the cream of our crop: disgruntled Webpack users.

The bigger our codebase becomes, the harder it will be to deal with
concerns that sprout up regarding all of these features. These features
have to be maintained, and they have to work, and we have to respond to
issues and PRs related to them. Codebase grows as O(n^2) with the number of
lines, because lines tend to communicate exponentially with other parts of
the codebase. It's not strictly O(n^2), but it certainly isn't O(n).

The cost of adding code outweighs the slight convenience of doing this.

That said, it's a team decision. :) If everyone on the team wants this,
then shrug go for it. But it'll be the #1 thing users will want to turn
off.

(If Parcel doesn't add a CLI flag to turn it off, I'd stop using Parcel.
But that just means I now have to remember to run parcel --no-install --no-cache index.html for development all the time now, which is quickly
becoming hilarious.)

On Thu, Dec 21, 2017 at 11:14 PM, Devon Govett [e-mail]
wrote:

Now that #306 https://github.com/parcel-bundler/parcel/pull/306 has
landed, it would be awesome to take this even further. Currently we
automatically install dependencies loaded by the internal localRequire
helper. These are things like compilers (e.g. typescript, coffeescript),
and plugins (e.g. postcss plugins, parcel plugins, etc.).

It would be super cool to take this even further and transparently install
any npm dependencies you require in your code for you. e.g. if you add
require('react') in your code, and it's not installed, Parcel can install
it for you.

Thoughts?

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/parcel-bundler/parcel/issues/376, or mute the thread
https://github.com/notifications/unsubscribe-auth/AADo8HEj0pmj6l0W9wUtnyqET15ybYD7ks5tCzqpgaJpZM4RKpxg
.

[shawwn]

That came out a bit more rant-y than I wanted, but the gist is that if someone wants to set up react, it takes them ten seconds to install the dependencies.

If someone wants to use react without Parcel, it takes them ten minutes to configure Webpack.

That's why Parcel is valuable. Saving ten seconds at the cost of maintainability and extensibility seems like a bad tradeoff, but maybe users feel differently. I don't know.

[devongovett]

Parcel is all about developer experience. Anything we can do, no matter how small, to save developers time is a win in my book. Developer hours add up quick!

We'll be working on things like improving plugins etc. in good time. Feel free to discuss how we can improve other aspects of Parcel in separate issues. There is room for everyone here.

[brandon93s]

@shawwn - It will be interesting to see how this plays out. The project is moving at breakneck pace at the moment, it's hard to say what things will look like tomorrow, nonetheless next week at the moment. I suspect if there is community outcry for or against a feature like this, the project will react accordingly. Let's see how the community feels about it.

From triaging issues that come in, I can say this will fix the occasional "what do I install" question.

I agree auto-install may feel a bit _gimmicky_ on the surface right now, but I'd like to give it a chance to see what it becomes and how the community adopts it.

[shawwn]

Mm, fair point.

But please ship a CLI option to turn it off. :)

On Thu, Dec 21, 2017 at 11:35 PM, Brandon Smith [e-mail]
wrote:

@shawwn https://github.com/shawwn - It will be interesting to see how
this plays out. The project is moving at breakneck pace at the moment, it's
hard to say what things will look like tomorrow, nonetheless next week at
the moment. I suspect if there is community outcry for or against a feature
like this, the project will react accordingly. Let's see how the community
feels about it.

From triaging issues that come in, I can say this will fix the occasional
"what do I install" question.

I agree auto-install may feel a bit gimmicky on the surface right now,
but I'd like to give it a chance to see what it becomes and how the
community adopts it.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/parcel-bundler/parcel/issues/376#issuecomment-353525830,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AADo8CTGZDLViD5duEFFJDHCmSanp1Dbks5tCz--gaJpZM4RKpxg
.

[brandon93s]

Sure, or you have parcel already started, and change your code to import a new dependency. No need to ctrl+C and install it anymore, parcel will just install it automatically when it detects the import.

Open question here would be whether or not it shows up in your package.json dependencies "automagically" or not. Would be pretty wild if I just add require('lodash') to a file and it installs it and adds the dependency for me. That sounds like a developer experience I can get behind. Your focus becomes the code you're writing, and nothing else at that point.

[devongovett]

@brandon93s That's exactly how it works now for compilers. e.g. import a .ts file and typescript is automatically installed and saved into devDependencies.

[shawwn]

If we do this, do we need to support letting the user write
require('coffeescript:dev') in their projects so that Parcel installs
coffeescript as a dev dependency?

We can get away with that with localRequire. But here, we're talking about
the general case where people want to avoid typing yarn add foo
altogether.

And if we want to do that, we might need a way to express "require this or
install it as a dev dependency" vs "require this or install it as a regular
dependency."

This ties in with updating require to be a bit more extensible, e.g.
passing query parameters like import "./foo.Vue?ssr" to do server-side
rendering, or import "./foo.css?scoped" to use CSS scoping.

That's "implicit configuration" -- the code defines the configuration --
but it's also handy for one-off scenarios. (Such as
require("coffeescript?dev"), actually.)

I don't know if this makes sense or not, but there you go.

On Thu, Dec 21, 2017 at 11:40 PM, Brandon Smith [e-mail]
wrote:

Sure, or you have parcel already started, and change your code to import a
new dependency. No need to ctrl+C and install it anymore, parcel will just
install it automatically when it detects the import.

Open question here would be whether or not it shows up in your
package.json dependencies "automagically" or not. Would be pretty wild if
I just add require('lodash') to a file and it installs it and adds the
dependency for me. That sounds like a developer experience I can get
behind. Your focus becomes the code you're writing, and nothing else at
that point.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/parcel-bundler/parcel/issues/376#issuecomment-353526334,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AADo8AWuPv8fzo6WPe4Th2oMLZ8WJs-0ks5tC0DJgaJpZM4RKpxg
.

[devongovett]

Not really a fan of overloading the require function or import syntax. but yeah, that's a valid concern.

[shawwn]

Meh, just make it install everything as regular dependencies then. It's not
strictly correct, but it will work in all cases, which is more important.
People can manually move dependencies to dev if they want to, later on.

On Fri, Dec 22, 2017 at 12:02 AM, Devon Govett [e-mail]
wrote:

Not really a fan of overloading the require function or import syntax.
but yeah, that's a valid concern.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/parcel-bundler/parcel/issues/376#issuecomment-353528787,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AADo8OIJbteQhRmAfL2P_pyVa7y8Db3Sks5tC0X_gaJpZM4RKpxg
.

[DeMoorJasper]

Why would there be a cli option to turn it off?
If you install the dependencies up front then it wouldn't even care to look to install them?
You can just still use it the same way @shawwn
And about the errors it's pretty straightforward it says it fails to autoinstall so than u just run the command in very restricted business environments or wathever, we could even extend this warning to literally give them the command.

[shawwn]

Because I don't want parcel doing magical things for me. I want to know
exactly which deps are added to my project and whether they're installed as
dev deps or real deps. And the only way to ensure that is to have control:
to tell parcel "please stop trying to hold my hand."

On Fri, Dec 22, 2017 at 3:18 AM, Jasper De Moor [e-mail]
wrote:

Why would there be a cli option to turn it off?
If you install the dependencies up front then it wouldn't even care to
look to install them?
You can just still use it the same way @shawwn https://github.com/shawwn
And about the errors it's pretty straightforward it says it fails to
autoinstall so than u just run the command in very restricted business
environments or wathever, we could even extend this warning to literally
give them the command.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/parcel-bundler/parcel/issues/376#issuecomment-353556881,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AADo8MoQWIxsgFX7mm6v-PyNnbP24no4ks5tC3PQgaJpZM4RKpxg
.

[shawwn]

(It's similar to the reason I run with --no-cache: The cache isn't perfect,
and there are many cases where it fails. Those cases cost more time than
the small convenience afforded.)

On Fri, Dec 22, 2017 at 3:31 AM, Shawn Presser [e-mail]
wrote:

Because I don't want parcel doing magical things for me. I want to know
exactly which deps are added to my project and whether they're installed as
dev deps or real deps. And the only way to ensure that is to have control:
to tell parcel "please stop trying to hold my hand."

On Fri, Dec 22, 2017 at 3:18 AM, Jasper De Moor [e-mail]
wrote:

Why would there be a cli option to turn it off?
If you install the dependencies up front then it wouldn't even care to
look to install them?
You can just still use it the same way @shawwn <
https://github.com/shawwn>
And about the errors it's pretty straightforward it says it fails to
autoinstall so than u just run the command in very restricted business
environments or wathever, we could even extend this warning to literally
give them the command.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
issuecomment-353556881>,
or mute the thread
PyNnbP24no4ks5tC3PQgaJpZM4RKpxg>
.

>

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/parcel-bundler/parcel/issues/376#issuecomment-353559341,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AADo8Pi5rXmhZavMpSQ34ZiDmCjyecD0ks5tC3b6gaJpZM4RKpxg
.

[DeMoorJasper]

'Those cases cost more time than the small convenience afforded.' Imo clearing out the cache once everytime u change a config doesn't cost much time. (Keeping in mind building speed goes down from a few secs to millisecs thanks to this...)
And same for dragging dependencies to dev-dependencies or vice versa, would save more time than it costs as far as i know.

But everyone has his own way of developing and u can't please everyone i guess...

[shawwn]

No, I'm saying caching straight-up fails a lot of the time. Like, a file
changes, and the change never shows up. This is especially true during
plugin development. If parcel had the mindset "Well, caching is so
wonderful, I'm going to force everyone to use it," then I wouldn't use it
because caching would interfere with my primary task: making new plugins.

Ditto for this automatic dependency installation stuff. I need to see
what's going on and be aware of all the changes to my projects. This is
both a personal quirk and a reason my
projects tend not to do unexpected things: if I know everything that's
happening, I can usually minimize the number of bugs I introduce. Meaning I
need to shut off the auto installer.

On Fri, Dec 22, 2017 at 3:52 AM, Jasper De Moor [e-mail]
wrote:

'Those cases cost more time than the small convenience afforded.' Imo
clearing out the cache once everytime u change a config doesn't cost much
time. (Keeping in mind building speed goes down from a few secs to
millisecs thanks to this...)
And same for dragging dependencies to dev-dependencies or vice versa,
would save more time than it costs as far as i know.

But everyone has his own way of developing and u can't please everyone i
guess...

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/parcel-bundler/parcel/issues/376#issuecomment-353563302,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AADo8DZo8IgVke_eis6yzy0VkaDoI7pwks5tC3vwgaJpZM4RKpxg
.

[rondonjon]

@shawwn is raising some very good points here IMO. I agree that this whole auto-installation thing may be nice at first sight but extends the scope of this project into fields that should be left to actual package managers. Users should be kept in control of their dependency management, especially in work environments. At the same time, the freed-up resources could be spent on key features of a better bundler.

[jdanyow]

I wonder if this will cause any issues in monorepos managed by lerna, yarn, workspaces, etc.

[devongovett]

@jdanyow it triggers the install at the nearest directory containing a package.json to the file where the import occurred.

[shawwn]

Won't this break yarn's lockfile then?

Consider importing node_modules/foo/index.js

Since node_modules/foo/package.json exists, it sounds like you're saying
parcel will run an install at node_modules/foo

Actually, maybe this will work fine – maybe yarn walks upwards until it
finds a lockfile. I'm not sure.

On Sat, Dec 30, 2017 at 10:16 PM, Devon Govett [e-mail]
wrote:

@jdanyow https://github.com/jdanyow it triggers the install at the
nearest directory containing a package.json to the file where the import
occurred.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/parcel-bundler/parcel/issues/376#issuecomment-354584402,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AADo8P7D7AOg8Cj-y4xVnmjx2TvqENOkks5tFwqegaJpZM4RKpxg
.

[DeMoorJasper]

@shawwn it'll prefer yarn over package.json so if a yarn file is present in the tree it'll take that one

[jamiebuilds]

Can we be sure to have an option to disable this behavior? Especially in CI environments

[davidnagli]

I’m a really big fan of this idea. I think we should go ahead and start thinking through how we can try implementing it.

Some questions we need to answer first:

1. Should we run this automagic install in production build mode (parcel build command)?
2. How are we implementing this!?? Here’s a few options:
   a) During the build process when we are resolving JS dependencies, if we come accross a dependency we can’t resolve (and it’s not a local file), then we attempt to npm install it (or use yarn if available)
   b) We run as usual, if we fail with an error, we go back and try to figure out which dependencies to install to make the error go away (dirty, but potentially necessary in some cases)
3. How do we tell apart imports of npm modules and local files?
   a) I’m pretty sure we can just check if the first characters are ./ since I think those are the required way to signify a local import, but we definitely need to research the proper way to detect this and cover any/all edge cases
4. Should we also cover installing dependencies that tools along the way need?
   a) For example, if babel needs to install a plugin/preset
   b) How would we detect this? Maybe we need to read the error output from Babel? Maybe we can figure out which installs it’ll need ahead of time by also parsing .babelrc?

The only SUPER important question here is number 2. Once we get that out of the way, we can begin implementing this feature and keep discussions about the rest of the points parallel to development.

[dpikt]

For future reference: flag to disable this behavior is --no-autoinstall

[DeMoorJasper]

@dpikt fyi this is documented in the docs: https://parceljs.org/cli.html#disable-autoinstall

[yumindeckard]

With all due respect, sir, you should try to suppress the impulse of making decisions for the users. You can give them the options and if they need it, they can just turn it on.