Respec: Migrate to webpack/rollup and ditch requirejs

[saschanaz]

I'm not sure what we should do for existing text! thing yet.

[sidvishnoi]

We might still need to keep requirejs for backward compatibility (in a different bundle, along with jQuery).
See: w3c/json-ld-syntax/common/common.js#L46 (and more)

[saschanaz]

Hmm. @marcoscaceres Thoughts?

[saschanaz]

Tried this a bit and found that:

1. we should first remove the deps/text things. (Webpack supports importing plain text, but we probably want a way that works both in original and bundled js.)
2. ui.addCommand shouldn't import modules via a dynamic expression as bundlers cannot statically interpret it, which breaks optimization.

[marcoscaceres]

We should be able to use one of the CSS importer things for text, I think. Or are there actual text we import?

Agree about the UI. We should be able to ditch dynamic imports.

[saschanaz]

A question: Do we want to keep our babel transpiler after webpack/rollup migration? Maybe for debugging, as sourcemaps are not perfect yet?

[sidvishnoi]

We need babel until Safari supports object spread assignment though.

[saschanaz]

webpack/rollup internally have babel (or at least a plugin) so I think it will be okay.

[saschanaz]

1981 made the webpack part but not really ditched requirejs.

[saschanaz]

Currently I have two options:

1. Migrate raw text imports to fetch() with a custom webpack plugin that converts fetch() into text importer call.
2. Implement a service worker that understands text import request and serves proper JavaScript module instead.

Option 1 needs some study because I don't know details to implement such plugins. Option 2 should be easier but 1) needs a way to detect whether a request is for text import or not 2) the worker should be injected separately to a document because the webpack-built version won't need it.

PS:

One problem remains: What should be done for profile-w3c-common.js? The batch require() is currently problematic.

[marcoscaceres]

Implement a service worker that understands text import request and serves proper JavaScript module instead.

Good idea - but that's not going to work, unfortunately :( The same origin restriction would mean that each spec would also need to locally host a copy of the service worker.

If the text calls are mostly CSS... we could consider CSS-in-JS?

[saschanaz]

The same origin restriction would mean that each spec would also need to locally host a copy of the service worker.

We can keep using webpack to bundle CSS. The suggested service worker is only for non-bundled development version.

[marcoscaceres]

Ah, ok. It still doesn't feel right that we would need two versions... from experience, the bundled version should still be debugguble in WebPack (just going to be a big fat file)... so I'm still hopeful we can do this without resorting to 2.

I need to explore exactly what stuff is coming in with !text.

[saschanaz]

If the text calls are mostly CSS... we could consider CSS-in-JS?

They are mostly CSS, but personally it sounds not best... Hmm.

[saschanaz]

BTW, we can also replace require() with native import from our tests if we remove all loader-specific syntax 👍

[marcoscaceres]

They are mostly CSS, but personally it sounds not best... Hmm.

Agree, also makes me a bit 🤢. I think just have a play and experiment and see what works. I don't have a strong opinion here: just want to make sure ReSpec stays easy to set up and hack on (and hopefully we get the bundle size down a little bit more).

BTW, we can also replace require() with native import from our tests if we remove all loader-specific syntax 👍

That would be nice.

[saschanaz]

Chromium people just published an intent to implement HTML modules, so maybe one day we can do:

<style id="style">
  .myCSSRule {}
</style>
<script type="module">
  export default style.textContent;
</script>

import css from "./mystyle.css.html";

Still not perfect (basically same as CSS-in-JS) but this way at least IDEs can still syntax highlight and linters can still work 😆

[saschanaz]

We have too many blockers. Currently requirejs is only used for debugging, how about we add a non-optimized mode so that we can get a debugger-friendly build?

[marcoscaceres]

I'd be down with that - from another project I was working on, source mapping remains a disaster.

[saschanaz]

The main issue for me is that too many lines are optimized out so that breakpoints cannot be added consistently. What were your problems there?

[marcoscaceres]

What were your problems there?

Just trying to set breakpoints and then stepping... it was completely erratic (jumps to random places, or can't actually set a breakpoint, etc).

[marcoscaceres]

... but working with the non-optimized mode totally works, even if it means debugging at package with like 30,000+ lines.

[saschanaz]

Hmm... how about we add a non-optimized build option without actually removing requirejs and see how it works?

[marcoscaceres]

Sounds good to me :)

[marcoscaceres]

it seems totally fine to just have the non-optimized version for development, and hopefully be able to use webpack's development server too.

[saschanaz]

Note: the debug mode is added by #2149

[marcoscaceres]

Did a little more experimenting... I think we should do this:

1. create 📂 profiles folder.
2. profiles are called, for example, w3c.js.
3. create a dynamic Webpack config:

const path = require("path");

module.exports = {
  mode: env.DEBUG ? "development" : "production",
  entry: `./profiles/${env.profile}.js`,
  output: {
    filename: "profile-${profile}.js",
    path: path.resolve(__dirname, "../builds"),
  },
};

Profile structure

// New top level module...
import respec from "../src/core/respec.js";

const plugins = [
  // plugins... for this profile
];

(async() => {
  respec.ui.show();
  await respec.load(plugins); // waits for DOMContentLoaded, runs base-runner runAll(), 
})();

We modify base-runner to import() plugins.

What about CSS, SVG?

For most modules, the CSS is small... let's just inline it or properly CSS-in-JS it.

[saschanaz]

Webpack can track the dependencies only in import("...path...") but not in var path; import (path); format, so probably we should do:

const plugins: [
  import("./core/foo.js"),
  import("./core/bar.js"),
  // and more
];

(async() => {
  respec.ui.show();
  const plugs = await Promise.all(plugins);
  await respec.load(plugs);
})();
`

[marcoscaceres]

Another option is:

import {run as fooRun} from "./core/foo.js"
plugins = [
 fooRun,
]

[saschanaz]

That would double the lines and we have to manually name all the runs, so personally I don't think it's preferable.

[marcoscaceres]

Agree. Just mapping out all the options.

[marcoscaceres]

Kinda sad... feels like the end of an era. Oh well! 💃

[saschanaz]

It's not completely dead just yet, see #2552 😁