Uppy: Splitting into multiple packages

[goto-bus-stop]

We've been planning on splitting Uppy into multiple npm packages for a while. We're hoping to do it in the coming weeks. I'm opening this issue to explore some of the challenges and propose some ideas on how to do it! A bunch of these were discussed in our Slack call yest.

The easiest way to manage many packages seems to be using lerna. We can have a folder structure like this:

packages/uppy
packages/uppy-core
packages/uppy-dashboard
packages/uppy-tus
packages/uppy-util
packages/uppy-provider-views
packages/uppy-react

These could either be published on npm as uppy-core or as scoped packages like @uppy/core. I quite like scoped packages, they clearly show that packages are part of some bigger project. uppy-core would also work tho and put us on even footing with community packages. That could be good or bad :)

Here uppy would be a package that depends on all the other plugins, acting like the current uppy npm package. You could do import { Dashboard } from 'uppy' with it. We should provide an ES modules and a CJS entry point for this one so import syntax gets tree shaking by default when used with webpack.

uppy-core would contain the Core class.

• It could include the Plugin base class, too.
• We could have the Plugin base class in a separate package.
  Personally I like the aesthetics of having it in a separate package more, but it may be better to have the Core and Plugin classes tied to the same version, otherwise you might expect different uppy-plugin versions to work with a single uppy-core version. I'm not sure if we want to guarantee that.

uppy-dashboard and other UI plugins would just export their plugin class. Depending on the above decision, I think they would either have a peerDependency on uppy-core or a direct dependency on uppy-plugin.
The major issue with UI plugins is how to tackle CSS. Currently, I believe the team's consensus is leaning towards just sticking a separate CSS file in each plugin, that you have to import separately. Another option is to insert CSS automatically at runtime but this may step on some existing CSS's toes (where to insert? depending on that choice, we may override some existing rules, or not)

// browserify
const css = require('sheetify')
const Dashboard = require('uppy-dashboard')
css('uppy-dashboard/style.css')
// webpack
import Dashboard from 'uppy-dashboard'
import 'uppy-dashboard/style.css'

uppy-tus and other plugins that don't have UI would just export their plugin class, there is not much else to do here.

uppy-util would serve as our grab bag of utility functions. Instead of a single file like the current Utils.js, this could be a good opportunity to move functions into separate files. Then plugins can require uppy-util/lib/functionName. Plugins should just depend directly on uppy-util, I think. Even if you end up with multiple versions of it in your node_modules folder that would not be a big deal because only a few small functions would be duplicated.

uppy-provider-views and other Provider-related modules would export whatever they need to, doesn't really matter how or whatever since users would not include it manually anyway. Like uppy-util, it's just for sharing code internally. There is a question about how to deal with CSS here though. Should it expose its own CSS file or should providers like GoogleDrive do it instead? In the latter case, you might end up with lots of duplicate CSS by default. We could also include it in the Dashboard's CSS file. That would add some unnecessary weight for people who only use local uploads but it would be the easiest solution I think.

uppy-react would contain all the React components. We could also have separate packages for each component. I think having a single package is fine though. People could do require('uppy-react/lib/Dashboard') or import {Dashboard} from 'uppy-react'.

---

We would need to change the Babel build a little, currently we just run babel-cli to transform src to lib but if the src/ is split into packages/*/src that is a little harder. In munar I used gulp: https://github.com/welovekpop/munar/blob/master/gulpfile.js We could also write a custom script and stick it in ./bin/build-lib or somethin. That would be more in line with our current build steps, and npm would still be our task runner.

---

I think it would be cool to have the documentation from the website in each plugin's README file. You could use readme to read documentation for installed plugins offline. We could probably have a simple prepublish script that inserts the documentation into a readme template like

# uppy-tus

<!-- insert-docs ../../website/src/docs/tus.md -->

Have it strip the yaml front matter. We can do that separately after the fact tho.

[kvz]

I like all of this! Kudos for the write-up @goto-bus-stop!

uppy-util would serve as our grab bag of utility functions. Instead of a single file like the current Utils.js, this could be a good opportunity to move functions into separate files. Then plugins can require uppy-util/lib/functionName. Plugins should just depend directly on uppy-util, I think. Even if you end up with multiple versions of it in your node_modules folder that would not be a big deal because only a few small functions would be duplicated.

What about not having a module for that and, and requiring ../shared/utils/*, could symlinks/lerna/magic help us here? The functions used would then be duplicated across modules upon publishing. The advantage I see would be reducing dependency version hell (uppy-tus requires uppy-utils >2.0.0 as of 1.0.3 but before that 1.8 or 1.9 still work - but not 1.7!!!). Injecting shared libs that worked great at design times into the modules directly might speed up dev iterations as it saves us from guarding that interfacing surface like crazy. We'll already have to do that for plugins, but i think it might be more manageable there, and actually carry some benefits (want to update tus due to a bug, but really don't wanna update dashboard along with that)

Like uppy-util, it's just for sharing code internally

So perhaps this could apply to provider-views too?

[goto-bus-stop]

I don't think the uppy-util dependency would be a big problem. Users would not install it manually and npm would dedupe it, unless different plugins depend on incompatible versions, but even then you'd just get two uppy-util versions and a few duplicate lines in your final build.

[kvz]

That’s fair! You’d have to release two versions when you hack on the libs though right? But I guess we could automate away the nuisance? I still wonder, if users aren’t installing it, why we’d have to make this a module that’s installable separately tho?

Sent from mobile, pardon the brevity.

On 31 May 2018, at 13:14, Renée Kooi [e-mail] wrote:

I don't think the uppy-util dependency would be a big problem. Users would not install it manually and npm would dedupe it, unless different plugins depend on incompatible versions, but even then you'd just get two uppy-util versions and a few duplicate lines in your final build.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

[goto-bus-stop]

Lerna detects changes in packages and publishes new versions for all changed packages when you do lerna publish. I think a shared util package will be easier to do than writing custom scripts and a babel plugin to move shared files into each package and fix the require() calls

[kvz]

Okay that sounds great! 👌👌

Sent from mobile, pardon the brevity.

On 31 May 2018, at 13:46, Renée Kooi [e-mail] wrote:

Lerna detects changes in packages and publishes new versions for all changed packages when you do lerna publish. I think a shared util package will be easier to do than writing custom scripts and a babel plugin to move shared files into each package and fix the require() calls

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

[goto-bus-stop]

Reproducing some slack discussion about how to version these packages:

@goto-bus-stop

One more thing: unsure if we should use independent versioning or synced
With independent, you may be publishing [email protected] and [email protected] if one has a semver minor change and the other has a semver patch change
that's pretty reasonable but it has one downside: https://github.com/welovekpop/munar/releases basically makes the releases tab useless because there are so many git tags
with synced versioning you still get to make nice release tags like babel does https://github.com/babel/babel/releases

@kvz

+1 for synced :)

@goto-bus-stop

oh, just remembered the downside to synced: a breaking change in any plugin forces a major bump in every other package. would be solved by just not doing breaking changes :slightly_smiling_face:
synced is still ideal i think. the tough part would be when we add a new plugin that may require breaking changes based on user feedback

@kvz

What if the signature of a function changes in utils?

@goto-bus-stop

hah, that's a fair point
guess we could not do that

@kvz

Seems I might have to retract my +1 :)

@goto-bus-stop

wonder if we can do independent, but with a single release tag for each publish

---

Doing independent releases on a single tag is possible by just not letting lerna do anything git related (--skip-git), but then we'd have to decide on how to tag things. You can't use a version number because there are many different version numbers and they are not all bumped at the same time.

I think perhaps we could just do independent and forget about the releases page, instead direct people to the CHANGELOG.md and blog posts.

[kvz]

That sounds fair to me!

Sent from mobile, pardon the brevity.

On 4 Jun 2018, at 14:21, Renée Kooi [e-mail] wrote:

Reproducing some slack discussion about how to version these packages:

@goto-bus-stop

One more thing: unsure if we should use independent versioning or synced
With independent, you may be publishing [email protected] and [email protected] if one has a semver minor change and the other has a semver patch change
that's pretty reasonable but it has one downside: https://github.com/welovekpop/munar/releases basically makes the releases tab useless because there are so many git tags
with synced versioning you still get to make nice release tags like babel does https://github.com/babel/babel/releases

@kvz

+1 for synced :)

@goto-bus-stop

oh, just remembered the downside to synced: a breaking change in any plugin forces a major bump in every other package. would be solved by just not doing breaking changes 🙂
synced is still ideal i think. the tough part would be when we add a new plugin that may require breaking changes based on user feedback

@kvz

What if the signature of a function changes in utils?

@goto-bus-stop

hah, that's a fair point
guess we could not do that

@kvz

Seems I might have to retract my +1 :)

@goto-bus-stop

wonder if we can do independent, but with a single release tag for each publish

Doing independent releases on a single tag is possible by just not letting lerna do anything git related (--skip-git), but then we'd have to decide on how to tag things. You can't use a version number because there are many different version numbers and they are not all bumped at the same time.

I think perhaps we could just do independent and forget about the releases page, instead direct people to the CHANGELOG.md and blog posts.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

[arturi]

uppy-core would contain the Core class.
It could include the Plugin base class, too.
We could have the Plugin base class in a separate package.
Personally I like the aesthetics of having it in a separate package more, but it may be better to have the Core and Plugin classes tied to the same version, otherwise you might expect different uppy-plugin versions to work with a single uppy-core version. I'm not sure if we want to guarantee that.

Yeah, I think I agree with having Plugin in Core module.

Utils.js, this could be a good opportunity to move functions into separate files

👍I can try to take on that

[arturi]

There is a question about how to deal with CSS here though. Should it expose its own CSS file or should providers like GoogleDrive do it instead? In the latter case, you might end up with lots of duplicate CSS by default. We could also include it in the Dashboard's CSS file. That would add some unnecessary weight for people who only use local uploads but it would be the easiest solution I think.

Agreed, at least for starters let’s just bundle with Dashboard. That fiddles with a hypothetic mythical user who rocks GoogleDrive without Dashboard, but they can probably include the Dashboard CSS as well. Especially since they don’t actually exist.

[arturi]

uppy-react would contain all the React components. We could also have separate packages for each component. I think having a single package is fine though.

Single package is fine for now for sure!

We would need to change the Babel build a little

Agreed to custom script.

I think it would be cool to have the documentation from the website in each plugin's README file

Very much 👍

[arturi]

I think perhaps we could just do independent and forget about the releases page, instead direct people to the CHANGELOG.md and blog posts.

Seems to be the best tradeoff after all the discussion. We do keep a decent changelog, so :)

[goto-bus-stop]

UppySocket and Translator could get their own @uppy/server-socket, @uppy/translator packages too rather than being in core (not all plugins need those)

[hpvd]

nice!

tiny: link to lerna in first post is not working anymore. should be https://github.com/lerna/lerna

[hedgerh]

did you all decide to go with synced or independent versions? was there some downside to synced? it's really nice when packages that go hand in hand are synced.

what made you all decide to split things into multiple repos? reducing bundle size?

[kvz]

did you all decide to go with synced or independent versions? was there some downside to synced?
it's really nice when packages that go hand in hand are synced.

Synced seemed really convenient yes. But we were thinking about the case where we'd have an experimental plugin in the works, or wanted to change the interface of utils functions, and then we'd have to major version bump all-the-things. So a trade-off, where we felt independent might be more headachy for us, but also the lesser evil considering the consuming developer who might otherwise be seeing @uppy/core major-bumped on a monthly basis, having to take extreme caution even though they might not even use the experimental/internal thing we major-bumped. But maybe there’s a better solution for this that we don't know of(?)

what made you all decide to split things into multiple repos? reducing bundle size?

I guess bundle sizes wouldn't be affected by anyone bundling themselves, especially if they're able to use tree shaking, but by going multi-module we're trying to address the common concern of super heavy node_modules

:) Like, if you don't want any of the visual stuff, you could just install & require @uppy/core and that saves ~99% of files and weight. Although it might not be a serious issue on _all_ platforms (in many cases there's abundant diskspace/inodes/bandwidth), our thinking is there's enough of a case to be made for that efficiency on enough platforms. @uppy also gives us a scope on npm by which people can tell which modules are officially supported. This would have been harder seeing as how people are already shipping their own uppy plugins with e.g. uppy- prefixes.

multiple repos

I think you meant multiple modules here(?), but just in case, we are not splitting up repos, we'll actually keep all official modules under one big roof for some of the reasons outlined in babel's monorepo.md. After merging Lerna, we're even considering to move uppy-server into the same single repo. This way we can for instance add a provider(touching both client & server) in one PR, revert it as a single atomic unit, have issues reported that impact both components, and as an uppy dev, always have a compatible server & client no matter which commit hash you check out. And as a consuming dev, it's not gonna matter cause you're not going to receive the server when you npm install @uppy/core or other client-side modules.

While we've pretty much decided on this and are now in the executing stage, we still welcome questions/comments/concerns! We can always adapt/revert based on new insights

[goto-bus-stop]

In addition to the above I like the way multiple packages abstract over the filesystem; instead of having to reach into uppy/lib/plugins/Dashboard, which forces Uppy's internal organisation to be a public API, you can use @uppy/dashboard. With lerna changes in our file structure or build process will not affect the public API.

The way you use official plugins and third party plugins becomes more similar—the only difference being @uppy/transloadit vs uppy-competitor rather than uppy/lib/plugins/Transloadit vs uppy-competitor.

[goto-bus-stop]

https://github.com/transloadit/uppy/commit/0d3f1b6a650e80a6ce0a5f08a522cc7c864ae8f3 :tada:

[kvz]

😍😍😍

Sent from mobile, pardon the brevity.

On 28 Jun 2018, at 11:24, Renée Kooi [e-mail] wrote:

0d3f1b6 🎉

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

[hedgerh]

I agree that having to import using the internal file structure is not ideal.

And yeah, syncing isn't ideal, but it can potentially get messy if people have to figure out which versions of each package work together. You could declare peerDependencies in that case, I guess, but those are super easy to overlook and ignore :P Nonetheless, it'd be something to keep an eye out for and try to address if it becomes an issue.