Ckeditor5: What if there was no compilation?

[Reinmar]

This thought was provoked by @pjasiun, cause according to his experience with Webpack, it's able to easily source packages from normal npm dependencies in a simple way. import x from 'a/b/c' loads b/c from node_modules/a.

I haven't researched it now, but AFAIR this behaviour is based on some configuration which that package must expose in order to tell Webpack what's the entry point and how to actually load b/c from inside of it. Mind that Node packages do not support stuff like require( 'a/b/c' ), so this is all hack.

But it's nice. It works as you'd expect things to work... only that it works in Webpack only.

Now, there are number of popular "formats" that a library may want to support. Bare es6 modules, rollup, Browserify, cjs, amd, etc. I've been thinking how many of them we could support without the compilation that we do today.

First of all, I imagine that bundlers like Webpack, Browserify, rollup have configurable "load module from ..." hooks. Webpack has loaders for instance. So there should be no big problem with them, at least up to a point where you load a dependency which requires another dependency but loads it in a way which your bundler doesn't support. E.g. ckeditor5-utils loads lodash, but what if loadash wouldn't work without special configuration for Webpack... and for Browserify... and for rollup.

Then we have cjs. CommonJS is a problem in general, because it doesn't support deep module structures... So without treating CKEditor codebase as a single package you wouldn't be able to load modules from other packages. Not good. There will be support for ES6 modules in Node.js one day, but I don't know how it's going to work and whether it will work with exactly the dependency format which Webpack requires. Actually... what if Webpack requires different imports than rollup? Yeah...

Then we have AMD about which I don't even want to think because it required hack even despite the compilation that we do.

What else? The problem is that I don't know... but I think that what we know gives enough feeling of the matter we're dealing with. For a brief moment I thought that maybe we don't need compilation if we make some concessions, but the only situation in which I can imagine this work is if we support just one architecture – e.g. Webpack or CJS. CJS would also support Webpack I guess, but not deep modules...

Heeeelp.

[pjasiun]

Mind that Node packages do not support stuff like require( 'a/b/c' ), so this is all hack.

I'm not sure if I understood you correctly, but Node does support stuff like require( 'a/b/c' ), exactly by looking for it in node_modules:
https://nodejs.org/api/modules.html#modules_loading_from_node_modules_folders

As I see webpack naturally mimic this behavior: http://webpack.github.io/docs/configuration.html#resolve-modulesdirectories or even http://webpack.github.io/docs/configuration.html#resolve-alias

And, as I think about it, I see no good reason why any compiler should do it differently. It is natural for Node.js users.

[Reinmar]

Odd, we had a problem with requiring modules from a package. It didn't work... Hm, maybe it's related to the main option in the package.json.

Edit: nope, it's not that. @szymonkups, do you remember what problem was that?

[pjasiun]

The point is, that building CKEditor will always be an option. But we could make this building simpler thanks to having valid paths already in the source code. Then this code may work in Webpack and in node.js (as soon it will support ES6 modules) without any building.

The problem I have with Webpack is that we change internal module structure during build (by skipping src folder) what is not supported there.

[Reinmar]

The problem I have with Webpack is that we change internal module structure during build (by skipping src folder) what is not supported there.

I agree that "de-hacking" this would be good. In fact, using absolute imports, just like CJS or Webpack would "de-hack" it even more and give automatic support for those environments. I'd like to investigate this further as it wouldn't be a too big deal.

The point is, that building CKEditor will always be an option. But we could make this building simpler thanks to having valid paths already in the source code. Then this code may work in Webpack and in node.js (as soon it will support ES6 modules) without any building.

I remember that the idea that we had with using ES6 modules over AMD was that they are easy to transpile to any other format and any other environment. So the code which we create can be turned into whatever we want. And it proved to be a good idea.

Perhaps, if we stick our environment to the most common standard we'll get automatic support with some popular environments and, perhaps you're also right that then compilation will always be an option still. Though, I must say that I don't know how it could work e.g. for Node.js. It would need to break the directory structure anyway, cause all the code would still need to be copied from multiple packages into one directory (cause we cannot override the content of node_modules/). I've got some doubts about this... But if Node.js will get ES6 modules support this is not going to be any problem soon.

[Reinmar]

I've just realised today that besides the code we also have to compile themes and icons. Those things are now exported by their custom tasks too, but they'd also need to be synchronised with any changes that we make to the process. E.g. if we remove custom JS compilation for most of the workflows, then we'd still need to keep some compilation for icons and themes.

[Reinmar]

Yet another bundler: https://github.com/fuse-box/fuse-box.

[Reinmar]

OK, we're going deeper with research on using Webpack and Rollup directly. What we agreed to change so far is:

• We'll use node-like, real import paths (e.g. import CKEditorError from 'ckeditor5-utils/src/ckeditorerror.js';).
• We'll bundle SVG in by using text loaders for SVG files (e.g. import boldIcon from 'text!../themes/icons/bold.svg';).

But it turns out that this is not all... we don't know what to do with SASS compilation. By default, Webpack compiles CSS in JS too (https://webpack.github.io/docs/stylesheets.html) and you can only get the "save as a separate file" functionality by tweaking Webpack configuration (https://webpack.github.io/docs/stylesheets.html#separate-css-bundle). I'm not sure that our plugins for Webpack should do this (it's up to the developer to decide).

This means that perhaps we have two options:

1. Bundle CSS using Webpack/Rollup/xyz and just let developers know that if they want to extract CSS then [[link to xyz bundler doc]] (if xyz supports it). Also, we can predefine special bundler configurations (like the current ckeditor5-dev-bundler-rollup) which will do that by default or hide this under a simple flag.
   
   
   
   • Pros: Since every JS file which needs some CSS will use explicit CSS imports, then we can produce optimised CSS (nearly impossible with the current approach).
   
   
   • Cons: Implementing multiple themes (and an option to choose between them during bundling) will require some tweaks in the module resolution logic. In fact, since theme is a separate package, we can't write in e.g. buttonview.js: import 'ckeditor5-theme-lark/themes/components/button.sass;. We'd need to write ckeditor5-theme/components/button.sass and resolve this path to a proper file on the fly. Finally, combining CSS into JS (as a defult behaviour) never sounded right to me ;/. Plus, we'll need to research yet another thing – e.g. themes interchangeability after splitting them to a separate file.
   
   
2. Bundle CSS on our own and do whatever we want.
   
   
   
   • Pros: Sounds simpler now (we've got the code already in our compiler and we just need to extend it to support multiple themes).
   
   
   • Cons: Building our own CSS compilation server into Webpack and Rollup isn't that much fun. Also, we can't optimise CSS in this approach that well. And we can face the same issues that we had with JS compilation.
   
   

TBH, it looks like the "import all files from level JS" approach is winning recently... Every bundler supports it. I don't know how developers see that – perhaps it's all ok. Plus, we don't have that many styles anyway, so, just like with the icons, it's just a little important detail.

I'd go with importing CSS in JS now. I don't feel comfortable with this, but we can always extend our plugins to implement "the extract CSS to a separate file" logic based on these imports. In high level point of view, import statements will give us info about what SASS files are required and we can then use this info as we want.

[oleq]

TBH, I'm not up-to-date with latest bundler trends either. It all runs so fast now that we could as well create the "JS trends follower" job in CKSource and that would totally consume some guy anyway.

Finally, combining CSS into JS (as a defult behaviour) never sounded right to me ;/.

👍

I'm also worried that as every month brings a new JS bundler, we'll spend a lot of time maintaining our code to support each new toy. Because once we're into this thing, there will be no easy way out. OTOH, this is how "mainstream development" is done these days, so we cannot just ignore it and turn our backs on it.

I'm not very enthusiastic about the whole thing but I'd rather go with the first solution. What worries me a lot at this moment is theme interchangeability and extensibility. Would it be possible for 3rd party developers to use Lark mixins and variables to develop styles for their own components for the editor? If not, it's sad because we're losing some of the power that we gained by bringing SASS to our ecosystem. By not sharing that power with developers, we're wasting quite a lot of a cool feature.

Sorry to say that but I'll not be able to help you more with the decision until I see the actual integration with themes, user component styles and the such.

[Reinmar]

I'm also worried that as every month brings a new JS bundler, we'll spend a lot of time maintaining our code to support each new toy.

I don't know how many bundlers we'll support. But I guess that only the most popular ones, and for now just 2. This gives us >75% coverage anyway. With Browserify that would be >90%. Plus, percentage of developers who need to integrate bundling CKEditor with their build tools is what... 20%, 50% (the rest will use prepared builds)? IDK, but that uncovered bundlers 10-25% is highly reduced by that if you consider the whole picture.

What worries me a lot at this moment is theme interchangeability and extensibility.

Extensibility should be better, because we'll de-hack the whole process by starting using real paths (although, Node.js specific). Those paths are understandable for (I guess) nearly all JS bundlers. I'm not sure how e.g. someone using node-sass directly could integrate this... Actually, I'll check if there's a module path resolution hook there.

By not sharing that power with developers, we're wasting quite a lot of a cool feature.

First of all, the biggest group of our SASS architecture consumers are plugin developers and they will be in exactly the same situation as we are. So this will always be fine. But by de-hacking our build process we may also improve interoperability with the code "outside" all this (integration mostly).

Would it be possible for 3rd party developers to use Lark mixins and variables to develop styles for their own components for the editor?

The whole point of https://github.com/ckeditor/ckeditor5/issues/375 is to be super inclusive for 3rd party plugin devs. So yes, this was and will be possible.

[fredck]

I’m very worried about embedding CSS and images in the js file. Mainly because I don't know what are the expectations out there.

One of the problems I can point with this approach is that there would be no parallel download of resources. It would be like downloading js + images + css one after the other. This may go against performance best-practices (not confirmed though).

Another point is that we'll have a bigger js file... people will complain about this, without knowing the reasons.

In the other hand, I have the impression that the (theoretical) image strip and the CSS we generate are very related to the bundle. They contain the features that the bundle contains. Different bundles would have different images and CSS then. There would be no reuse of such resources cross-bundles. I would take the risk and go with this approach.

[Reinmar]

I have exactly the same doubts. That's why I wrote that I don't feel comfortable with that – it may just look wrong for some developers as it looked wrong to me.

But:

• We can split the code (e.g. in our official releases). I'm still researching this, but as we already know Webpack allows that and Rollup CSS plugins have such features by default.
• We can explain that clearly – it's just one resource, all assets combined and it's still only XX kb (gziped). We're actually doing pretty well here and having optimised CSS and icons (we'll use only the ones which are really needed) will be our advantage.
• And yes – image strip and CSS are very related to the bundle. It actually may make more sense to split JS into many files (some needed early, some needed later) than blindly separating CSS and images from JS. After all, once editor loads, you need all of that. But you may not need the editor (and all its assets) before someone clicks "edit" or something.

[pjasiun]

I see one more profit of having styles (SASS files) included in JS instead of having compiled CSS files: thanks to it SASS files are more reusable: external plugins can mix CKE SASS mixins or import any style it needs.

[Reinmar]

That's not a benefit of having styles compiled in JS, but having predictable and reusable paths to these original files. These are, actually, two separate things.

[pjasiun]

That's not a benefit of having styles compiled in JS, but having predictable and reusable paths to these original files. These are, actually, two separate things.

This is a benefit of using source styles (SASS files) instead of complained (CSS files) and having a compilation in bundler, possibly together with the whole project which uses CKEditor.

[Reinmar]

I've been testing Webpack modules resolution options.

In our dev env we want the ckeditor5-* modules to be always loaded from the ckeditor5/node_modules directory if they exist there. If not, the normal node module resolution logic should be used (because ckeditor5-foo may require ckeditor5-bar which may not be installed in the dev version, so it will be inside ckeditor5-foo/node_modules).

Anyway, this is one option to configure Webpack like this:

    resolve: {
        modules: [
            path.resolve( __dirname, 'node_modules' ),
            'node_modules'
        ]
    },

This will favor main package's node_modules over node_modules relative to importee (package which does the import).

This works really well and is really simple.

Unfortunately, there's a pretty common case in which this would cause a huge confusion. I've been just testing that if you remove e.g. ckeditor5-utils/src/keyboard.js the editor will still build. Why? Because ckeditor5-engine/node_modules/ckeditor5-utils/src/keyboard.js still exists. So if you renamed a file and forgot to update a path to it you're really screwed.

Therefore, I like the resolution logic of our CKEditorWebpackPlugin and ckeditorRollupPlugin which check whether an entire required package exists in ckeditor5/node_modules/ and then always try to find the module there. If keyboard.js is removed, it will not try to look for in a different place.

[Reinmar]

Moar news – I'm working on icons.

First, I tried to do import boldIcon from 'raw-loader!../theme/icons/bold.svg';. It didn't work OOTB because Webpack looks for loaders in packages' node_modules. Such configuration helped:

    resolveLoader: {
        modules: [
            path.resolve( __dirname, 'node_modules' ),
            'node_modules'
        ]
    },

It's the same code as above for JS modules, but in this case it's fine since we don't expect developing loaders together with CKEditor source.

Then I tried to work on Rollup integration (thank god we're testing both things at the same time) and it turned out that raw-loader! in an import path will not be handled there and it's not a custom to use such extensions. There are Rollup loaders but you add them to the rollup() call and configure which files they are supposed to handle. See e.g. https://github.com/TrySound/rollup-plugin-string.

I tried to use the resolveId callback in our Rollup plugin but it turns out that there's no easy way to strip the prefix and make the default resolver resolve the path. Hence, I don't think we should be using those prefixes... It'd be great if we could, but we risk problems with integrating with other bundlers.

[Reinmar]

I'm working on themes now and I tried to structure imports between SASS files like I would do with JS. Unfortunately, this leads to code duplication (@import_once will be added in Sass 3) and, most importantly, to unpredictable order how these files are combined together. And as we know, this is super !important in CSS, so the whole idea of importing SASS to UI components is not going to work.

Hence, it seems that we can only resolve this by the main editor class importing its complete SASS index file. But then how's importing CSS in JS better than building it completely separately? Well, it isn't.

So, to speed things up, I'm going to include SASS into JS because this will allow us to move to Webpack faster. But then we'll need to work on how to compile CSS separately and, even more importantly, how to allow building a chosen theme.

[Reinmar]

To clarify – it's not a problem if a file containing mixins is included multiple times. But I've noticed that some components started to depend one on each other. Perhaps if we were able to untangle this, then we could use imports in JS files.

So the chain would always look like this: JS -> SASS component -> SASS helper(s)

[Reinmar]

I managed to make styles work in Webpack and Rollup. In the second case it was highly untrivial, because the module path resolution algorithm isn't implemented (see https://github.com/differui/rollup-plugin-sass/issues/25). However, I used node-sass config options and it worked well in the case in our hands.

[Reinmar]

Webpack 2:

Rollup:

1.7MB vs 1.4MB. But I think that after minification and gzipping the percentage difference will be smaller.

[Reinmar]

More comparisons (all without minification and gzipping). CKEditor used currently on https://ckeditor5.github.io is 1720kb JS, plus 61kb CSS. The JS is transpiled to ES5 in this case. The same setup without transpiling to ES5 and with CSS included is 1470kb. So transpilation to ES5 adds 300kb.

[Reinmar]

OK... so we did it. The changes landed in https://github.com/ckeditor/ckeditor5/commit/f5a225b6063ca68e9cbf2163c0f7c53b32ce4a70 and in each package's repository.

I'll be updating docs and writing more updates soon.

[pjasiun]

Reinmar commented at 31.12.2016, 16:41 CET.

...and a Happy New Year!

[Reinmar]

Within working hours :P

[oleq]

Of course, of course ;-)

[pjasiun]

You know it was Saturday yesterday, don't you?

[Comandeer]

Guys, you just don't understand that for some of us code is just THE LIFE.

[Reinmar]

The status of the transition

Done

• We killed the compiler as the pre-bundling step.
• We developed a small plugin for Webpack which drives module resolution so Webpack is able to find CKEditor modules installed in the dev version. Currently, it's possible to build CKEditor using Webpack without this plugin if Webpack is configured properly and packages aren't installed in the dev version.
• We'll try to maintain the state when you don't have to have CKEditor 5 Webpack plugin to build it, but the translations support will, in general, require you to use this plugin. So, without the plugin, editor will use the strings which we specify in t() calls. The plugin will add support for building other languages.
• The bundle contains SVG icons and CSS for the editor and it will contain the chosen the translation
• During the research phase we worked also on the plugin for Rollup. We proved that it's doable (although, we needed to change some design decisions to make it possible) and will bring it soon.
• The plugins for Webpack and Rollup (and other bundlers in the future) will use a common utilities. We'll try to make them as simple as possible.
• During course of action we merged all existing packages into one repository https://github.com/ckeditor/ckeditor5-dev with multiple packages. We use Lerna to manage it. I think that it was a good choice for these pakages, because we're the only ones to ever work more on them and the monorepo architecture works great in this case and Lerna is a very nice tool.
• Besides plugins for bundlers, we'll also have special tasks which run these bundlers with a special configurations. These tasks will be used to create special kind of releases (like the core + split packages bundles) as well as they will simplify bundling CKEditor if you don't use Rollup or Webpack. In fact, we had such a task already – ckeditor5-dev-bundler-rollup – using a simple configuration file it was able to build a very optimised bundle with specified features.
• The testing environment was rebuilt to use Webpack and CKEditor 5 plugin for Webpack. It was the toughest part of the job because sourcing CKEditor tests from node_modules and support for CI-way of testing required many tricks.
• The API documentation config was aligned to the new setup too. I also removed the whole guides building process which we dropped some time ago anyway.
• The ckeditor5-ui-default package was merged into ckeditor5-ui.

Still to do

• Tests and cleanup. We wrote 0 tests and we'll need to better structure and document the new code.
• Bring plugin for Rollup and make ckeditor5-dev-bundler-rollup use it.
• Integration samples. While working on PoCs we created set of configurations and sample HTML pages. Without them it's not possible to validate that a bundle setup works, so I'd like to introduce a new repository (e.g. ckeditor5-samples) which would contain samples of various bundlers and various special configurations. We can later add there also samples of integrations with other frameworks (React, Angular, etc.). The reason to introduce this as a new repository is that it will contain a huge list of deps in package.json. This can also be done as a multiple package.jsons in samples/<sample-name>/ directories, but I'd like to keep ckeditor5 repository as simple as possible. But in fact, ckeditor5-samples may be a fork of this repository.
• Support for building other themes. Currently, the code imports files from ckeditor5-theme-lark. This must be changed so code requires files from ckeditor5-theme and the plugin for bundler decides which package this means. This, actually, means that you won't be able to build CKEditor without such plugin anyway (this could be workarounded by creating ckeditor5-theme package which would link to ckeditor5-theme-lark, but I'm not sure if it's worth the confusion).
• I'm seeing bigger and bigger need to extract tasks introduced by ckeditor5-dev-env to a standalone shell command (like lerna). It will allow to use it in a simpler way, write proper arg parsing (you can't do gulp exec -- the command to exec e.g.), clutter the gulpfile.js less. We also need to review the design of these commands, enable support for multi-process usage (this really speeds up the work), etc, etc. Lerna is a great example of how such a tool should work.
• While doing the above we need to move the CKEditor 5 specific tasks or configurations back to ckeditor5 repository. E.g. the new package templates should be there. Now, when I change the README of packages I also need to release a new version of ckeditor5-dev-env. The same with exec tasks – I can't add new ones easily (so I maintain https://github.com/ckeditor/ckeditor5-dev-env/issues/21). Etc, etc. We actually already started creating new tasks in ckeditor5 repo – see scripts/.
• Move tickets from closed repos to the new ones (ui-default to ui and dev-* to dev).

How to get all the goodies?

In ckeditor5:

git pull
rm -rf node_modules # just to be sure – I wasted half of a day debuggin some WTFs
npm i
gulp update

Note that the gulp test --files param has changed a bit because of the far more complicated directory structure. Check out this table: https://github.com/ckeditor/ckeditor5-dev/tree/master/packages/ckeditor5-dev-tests#rules-for-converting---files-option-to-glob-pattern and examples above. We can work on this to adjust it to our needs, but I guess that we'll just get used to it.

If you want to adjust some existing PRs to the new import paths use the 3 scripts in scripts/:

gulp exec --task sh --cmd "/workspace/ckeditor5/scripts/fix-src-imports.js"
gulp exec --task sh --cmd "/workspace/ckeditor5/scripts/fix-test-imports.js"
gulp exec --task sh --cmd "/workspace/ckeditor5/scripts/remove-js-extensions-from-imports.js"

Thanks

To @pjasiun for planting the seed to get rid of compilation and @ma2ciek for the hard work on this refactoring!