P5.js: Update build process to use ES6 modules + Rollup

[aendrew]

Nature of issue?

• [x] New feature request

Most appropriate sub-area of p5.js?

• [x] Other (specify if possible)

New feature details:

See my comment here on #2244. Effectively, updating the build process to use ES6 modules would modernise the codebase, enabling smaller bundles and more flexibility in terms of how people use P5 in their own build process.

[Zalastax]

This is a solution to the problem in #522, right?

Rollup is nice in that it has support for multiple export types.
It creates small bundles and doesn't require any polyfills. Do we know how far away we are from a code base that could use ES6 modules?

For direct inclusion in the browser we would create a bundle using UMD. This bundle would just be included in the zip-file directly, and not referenced from package.json. I don't think there's any point in using a browser field.

For usage as a npm library we would get create two bundles: CommonJS and ES6-modules.
Doing this is very easy, we just need two different fields in package.json. This would be very similar to the rollup-starter-lib example.

[aendrew]

@Zalastax That would indeed help #522, though providing the sources in the npm package as per #2244 would also solve that — it's the difference between import something from 'p5/lib/something' versus const something = require('p5/lib/something') — you don't need to convert to ES6 modules to benefit from that, any build process supporting CJS would be able to already do that if you supply the sources in the npm package, but using ES6 modules would optimise the bundle more due to the static analysis requirement (thus enabling dead code removal/"tree shaking").

Also, as you've mentioned, yes, setting the module/js:next field in package.json enables tools like Rollup to get at the modular sources while still providing a UMD bundle for browser consumption via the main field (considering much of the docs for P5 use globals provided via UMD this is probably preferable to building a separate CJS bundle and setting that as main as per rollup-starter-lib, IMO — UMD can be consumed as CJS and it's likely most people wanting to use modular bits of the lib will be using tooling that can consume ES6 modules). Also, AFAICT, using ES6 modules when referenced via the module field is preferred over CJS.

[sakshamsaxena]

@aendrew Hi, apologies for the late arrival :smile:

but using ES6 modules would optimise the bundle more due to the static analysis requirement (thus enabling dead code removal/"tree shaking").

I'd suggest you checkout the "modularisation" task I set up last summer. It didn't really gain any traction then as it was a seldom discussed feature. However, your arguments gave me the impression that it might really help you out. You can find some background here and here. If you look at the task code closely, you can see that it prepares the necessary files to bundle and then essentially passes the list to Browserify, which is being used programmatically. I think Rollup can be used programmatically as well, and maybe a fork of this task can be potentially worked upon to include Rollup for building.

I might be terribly wrong/off point here, so kindly correct me if it is so :smile: Would love to take this discussion further :smile:

[meiamsome]

Are there any issues with the nature of the way p5 is built? What I mean is that all the sub-packages just add functionality onto the p5 object. I could see this being a bit weird to use as an import - would we want import { httpDo } from 'p5' or import { httpDo } from 'p5/io' or perhaps import httpDo from 'p5/io/httpDo'

[Zalastax]

@sakshamsaxena, yes rollup can be used programmatically :) The tricky thing is probably to restructure code to use the right module format. We will have to do all the restructuring first, and then when it's all done we can switch the bundler.

@meiamsome, the neatest solution to this problem that I've seen is the way RxJS does it. They have structured their project so that you can choose between patching the object or using free functions. They describe how it works on their website. I think they also do some re-exporting so that at least two of your suggestions both would work.

[meiamsome]

That looks like an excellent example of how to do it, @Zalastax! Do we think we can do the conversion in parts or will we have to rewrite the entire codebase at once?

[limzykenneth]

@meiamsome I believe you can include babelify as a transform in browserify so you can use es6 import in part of your codebase (babelify will transform them to commonjs require before passing to browserify), you won't get the benefit of tree shaking I think but you can start converting the codebase gradually that way.

[Zalastax]

Should we just add support for ES2015 modules or a full ES2015 transform? Is there a roadmap for the codebase?

[limzykenneth]

I don't think there's a roadmap yet. To be honest I'm a bit wary about a full convert as it means we will need to babel the whole codebase for compatibility, I much prefer to introduce new features as they become more supported worldwide (ie. less people using IE 😨 ). For example let is actually fully supported on IE11 so we should be able to introduce it with no problem and no conversion needed while retaining their benefits.

What do you think?

[Zalastax]

The env preset already makes sure that there are no conversions for features that are supported in all our target browsers. Since Babel only transforms syntax (like arrow functions), using newer features shouldn't inhibit a big cost. I think we should avoid babel-polyfill though, since it is quite large. http://babeljs.io/ has a brief list showing what you get from the transpilation and what you get from the polyfill.

[stephanmax]

@Zalastax I would love to see this coming to p5, please let me know if I can help.

[Spongman]

does anyone have a solution to the fact that what p5.js does - define a single class across many files - doesn't seem to be well supported by es2015 modules or ES6 language features?

[Zalastax]

I don't think there are any problems with augmenting the p5 "class" from multiple files - modules can handle it fine.
If we want to use classes, this is one method of doing so. I basically did exactly this in another project and as you see it's even compatible with TypeScript!

I suggest we start by just replacing browserify with rollup (no babel). YUIDoc already supports ES6 and modules so the change shouldn't break anything.

[Spongman]

yeah, that's kind of what i was worried about: every method definition requires at least two additional lines of boiler-plate just to shoehorn it into the prototype. i guess i'd like to see a complete example of a class defined in, say, 3 files with functions in each referencing properties & methods in the other files. i don't know that this is even possible in typescript without defining everything as interfaces due to issues with reference cycles.

[Zalastax]

TypeScript is only a consideration for the far future. I mentioned it just to illustrate that it's a well recognized pattern. ES6 modules shouldn't add any boilerplate at all.
If we eventually want to consider TypeScript, partial classes by naming an interface the same thing as the class + using the prototype is well supported. It might be the case that another structure will suit us better then but let's have that conversation when the time comes.

[Spongman]

sorry, yes i dind't mean to complicate things with typescript, i just know that typescript doesn't support partial classes (which is basically how p5.js is structured) and the reason usually given for this is that es2015 modules don't really support this kind of structure. but if you have some way of doing this nicely that's great!

[Zalastax]

I went and tried this out, both with and without babel.
Since we augment p5 not only as a class, but also as a module, some trickery is needed. E.g. if a file is converted from commonjs to modules we need to create a separate file that actually modifies the p5 module.
The end goal should however be that the p5 class is modified but that the p5 module is exported via export statements as it's done in rxjs. I tried that out but didn't want to change more than needed so I went with the extra file instead. I think some of the added require's could be removed again but we should go towards each file importing the augmentations they need anyway.

What's left now is mainly a decision on babel or not. For me the choice is clear, babel can make for simpler code and doesn't incur a runtime/size cost. As long as we're using UglifyJS (at least the current version) we can't start using any ES6 features since it doesn't understand them; e.g. it will crash on const foo = 1;. It would be great if we can set up eslint to warn if we use for example Array.includes (Not in IE11), but at least it will warn if we try to use Promise.

[Zalastax]

@lmccart could you chime in on how you want to proceed?
We have rollup working and it does not require changing the whole project at once. What it gives us is access to ES6 modules. It is possible to use with or without babel.

I'd like us to

1. Replace browserify with rollup
2. Enable ESLint rule no-commonjs but add an ignore comment to every file that needs it. This makes it easy to track progress and prevents us from going in the wrong direction.
3. Convert files one by one, using p5.js augmenter files when needed.
4. Replace augmenter files with ES6 module equivalent. Since core/main.js will be in ES6 module style by then, this will be a simple task.
5. Remove rollup-plugin-commonjs

[lmccart]

Hi all, thanks for the work and discussion on this. Yes, we are moving to ES6. The target timeline is still being finalized on this. I would like to pause on rollup+ES6 and other ES6 updating until we've officially kicked things off. You can expect an update in the next month or two with more of a timeline on this. I'm also working on adding milestones for the repo, so we have a better sense of priorities moving forward. I will post milestones intended as a draft and open an issue for discussion, before we finalize.

[outofambit]

closing this since we have a discussion going in #3425 as well.