Html: Add first-class support for differential script loading

[mathiasbynens]

The type=module/nomodule pattern gave developers a “clean break” to ship small, modern JavaScript bundles (with minimal polyfilling + transpilation) vs. legacy bundles (with lots of polyfills + transpilated code), which is great not just for module adoption but also for web performance. However, as more features are added to the JavaScript language, more polyfilling and transpilation becomes necessary even for these “modern” type=module script bundles.

@kristoferbaxter, @philipwalton, and I have been thinking about ways to address this problem in a future-facing way, and have explored several potential solutions. One way we could introduce a new “clean break” once a year is by adding a new attribute to <script type="module">, perhaps syntax or srcset:

<script type="module"
        srcset="2018.mjs 2018, 2019.mjs 2019"
        src="2017.mjs"></script>
<script nomodule src="legacy.js"></script>

(Note that this is just an example of what a solution could look like.) The 2018 and 2019 descriptors would then refer to feature sets that browsers recognize (in particular, they do NOT refer to ECMAScript version numbers or anything like that). For more details, read our exploration doc.

At this stage we’d like to get feedback on whether others agree this is a problem worth solving. Feedback on any particular solution (such as <script type="module" srcset> vs. <script type="module" syntax> vs. something else) is also welcome, but less important at this time.

[mathiasbynens]

cc @philipwalton @nomadtechie @watilde

[domenic]

I'm -1 on this idea for the following reasons:

• I think the user-agent string already gives the correct amount of information here. Any additional information given is a privacy leak, so this proposal must be strictly less powerful if it is not to violate privacy invariants. (For example, if a user changes their UA string, the browser would need to change what it reports for these values too, in order to not add more bits of entropy. The exploration doc seems to say this is not desired, in the "Differential JavaScript for user-agent Buckets" section, but I assume the intent was not to add more fingerprinting surface, so in fact there would be no difference.) As such it's best to stick with just one source of data.

• Agreement on "yearly feature sets" is not tractable. For example, it'd be ideal to ship BigInt or private field code to today's Chrome, but this proposal would not allow doing so, because "the majority of stable user-agents" do not contain those. (Or do they? See below.) Tests should be more granular than bundling together features in this way.

• Any definition of "majority of stable user agents" is not realistic. By some definitions, that would include exactly one user agent, the most-recently-stable Chrome version. By others, it would include Chrome and Safari, excluding Firefox. By others, it would include Chrome and Firefox, excluding Safari. (It's unclear how to count Edge given recent news.) In some geographic regions, it would include UC Browser or QQ browser. This isn't even mentioning the various Chromium-based browsers which are on less-than-latest-stable-Chrome versions. In the end, only app developers have a realistic idea of what features they want to use untranspiled, and how those features sit relative to the browsers they are targeting. They should make that determination on a per-feature/per-browser basis, not based on a committee agreement of what a year represents, or what the majority of stable user agents represent.

• Script loading is complicated and has many entry points. The exploration doc tries to thread this through <script> and <link>, but misses (in roughly descending order of importance) new Worker(), import statements, import() expressions, service workers, the varied-and-growing worklet loading entry points, importScripts(), and javascript: URLs. A unified solution would involve the server taking responsibility for the scripts based on user agent, as can already be done today, instead of speccing, implementing, and waiting for wide availability of browser-side mechanisms such as the OP, and burdening all current and future script-loading entry points with the need to support this.

• This attempts to bake in a division between the JavaScript platform and the web platform which I think we should discourage, not encourage.

As to whether this is a problem worth solving, it depends on what you mean. I agree it's a worthwhile thing to do for authors to serve browsers code based on the syntax and capabilities they support. I think that problem is already solvable with today's technology though.

[matthewp]

I like the general idea of differential loading but I don't think this solution is the right one. My main problem is surrounding how these yearly feature sets will be defined. I think it would be difficult to gain consensus on what is included.

I can also see a scenario where a Popular Website uses srcset and browsers feel pressure to lie about their support, knowing that Popular Website doesn't use feature Y (the thing they don't support) anyways.

---

I don't have a firm alternative, but I feel like some combination of import maps and top-level-await provide the primitives needed for differential loading. I could see a future feature of import maps that makes it a bit cleaner to do.

[mathiasbynens]

Some initial responses:

I think that problem is already solvable with today's technology though.

It may be “solvable” through UA sniffing and differential serving, but in practice this approach somehow hasn’t gotten much traction. We commonly see websites shipping megabytes of unnecessary JavaScript. To apply the technique you describe, currently developers have to implement and maintain:

• custom tooling configuration to output multiple separate JS bundles, and
• custom server-side UA sniffing that maps exactly to the tooling configuration

If instead, we could somehow standardize on some idea of “feature sets”, then browsers and tooling could align around that, and reduce this friction altogether. Developers could then perform a one-off change to their build configuration and reap the benefits.

This attempts to bake in a division between the JavaScript platform and the web platform which I think we should discourage, not encourage.

Which division are you seeing? There’s no reason npm and Node.js couldn’t adopt the same “feature sets” we standardize on.

Script loading is complicated and has many entry points.

Why do other entry points such as dynamic import() or javascript: URLs need to be supported? The tooling that generates the output bundles would know whether import() is supported or not based on the feature set (e.g. 2019, or whatever kind of identifier we come up with) that was used to generate it. As such, the tool could decide whether or not to transpile/polyfill import() for that particular bundle.

I think it would be difficult to gain consensus on what is included.

It would depend on the chosen process. We can make this as complicated or as simple as we want. It could be as simple as just picking a date. The date then maps to a list of latest versions of stable browsers at that point in time. That list of browsers then maps to a set of features that are fully supported (by some heuristic, e.g. 100% Test262 pass rate for ECMAScript-specific features). There’s no point in arguing about which features should be included if we can just look at browser reality and figure it out from there.

[domenic]

If instead, we could somehow standardize on some idea of “feature sets”, then browsers and tooling could align around that, and reduce this friction altogether. Developers could then perform a one-off change to their build configuration and reap the benefits.

I don't think this alignment necessitates new browser features.

Which division are you seeing?

The proposal includes language features, but not web platform features.

Why do other entry points such as dynamic import() or javascript: URLs need to be supported?

Because they are other ways of loading scripts, and if the problem statement is differential script loading, then you need to ensure those also allow differential script loading.

It would depend on the chosen process. We can make this as complicated or as simple as we want. It could be as simple as just picking a date.

As I tried to point out, it is not that simple. A concept such as "latest versions of stable browsers" is itself super-fraught.

[mathiasbynens]

Which division are you seeing?

The proposal includes language features, but not web platform features.

There’s no reason it cannot include web platform features.

[bkardell]

Given all of about 15 minutes worth of thought I am a little hesitant to share a anything like a 'real' opinion here, but my gut reaction was kind of similar to what @domenic said except that I fall way short of

I don't think this alignment necessitates new browser features.

That's not to say "it does" either, just that that I also fully accept @mathiasbynens general premise what "can" technically be done doesn't seem to have caught on and is probably more challenging than it should be - but I don't know how to fix that either.

[jkrems]

FYI: In the node modules working group, we're currently exploring extending the existing import map alternatives pattern to support this kind of environment matching: https://github.com/jkrems/proposal-pkg-exports/issues/29

[daKmoR]

that seems too ambitious of a solution... (agreeing on which feature are in which "group" seems to be way too hard... and it can fastly different on which technology you are using)
imho you will always need to have some logic like

if (this and that feature is supported) { 
  load(this); 
} else if (other and more is supported) {
  load(other); 
} else if (...) {}

imho it's about having a way of getting these checks auto-generated into your index.html by bundlers.

[mathiasbynens]

@clydin I agree Subresource Integrity should be supported somehow, eventually. I don't think lack of SRI support should block an initial version of this proposal to land (just like it didn't block import()). If we were to continue down the path of <script type=module srcset>, then https://github.com/ResponsiveImagesCG/picture-element/issues/255 is the relevant discussion.

[robpalme]

A note on naming: could we call this Differentiated script loading rather than Differential?

The latter initially made me think this involved sending script patches over the wire.

[matthewp]

@kristoferbaxter

The expectation is once a HTMLScriptElement chooses a syntax version, the resource it chose is responsible for leveraging the correct references to its dependencies (Workers, import statement, import expressions, service workers, importScripts() and javascript: URLs).

This requires the script being external correct? What about inline scripts?

<script type="module">
 // What syntax am I?

 // What syntax is this worker?
 new Worker('./worker.js');
</script>

[iamakulov]

To expand on the @daKmoR’s point (https://github.com/whatwg/html/issues/4432#issuecomment-472583278). What if we target features instead of years? Just like CSS does with @supports.

This might look like this:

<script
  src="app.bundled-transpiled.js"
  support="
    (async, dynamic-import) app.modern.js,
    (async, not dynamic-import) app.bundled.js
  "
></script>

Pros:

• Easy to use in static HTML. And easy to generate with bundlers/other tools.

• Gives enough independence to browser engines. This removes the burden of browser maintainers meeting every year and deciding what to include into each yearly snapshot.

• More reliable. There’s a high chance Chrome and Firefox may ship slightly different implementations of the 2018 descriptor, and users won’t be able to rely on it. It’s way less likely if descriptors describe specific features and not feature packs.

• Works well if a browser decides to revoke a feature (like it happened with SharedArrayBuffer). If a browser revokes a feature, it would just start returning false for the corresponding supports check. With 2018/2019/etc, browsers would have to bump the version (as described in the exploration doc).

Cons:

• Requires a lot of work in the beginning to setup keywords for existing features. To reduce the work, the committee may use the existing list of features in the Kangax compat table. Further maintenance would be easier.

• Verbose. This won’t create a real issue if the descriptor list is generated automatically (nobody would edit it, so verbosity won’t complicate anything). This might be an issue if the descriptor list is created by hand; but from my experience, in most apps, you typically just need to detect a couple key features (like async or import) and won’t care about others.

[clydin]

While I agree on the utility of this feature and that getting it in the hands of developers sooner rather than later would be useful, I don't think it is prudent to make security related concerns an afterthought for a design that changes the semantics of code loading and execution.

The integrity attribute is also one of multiple current and future attributes that would potentially need to be added to the srcset syntax. srcset would most likely need to become a DSL (CSP like?) to fully encompass the feature set of the script element for each referenced resource. At which point the script element has essentially become duplicated in a different form. And although most likely not a major concern, tooling (parsers, static analyzers, generators) would need to add support for this new DSL as well.

As an alternative, what about a markup based solution? (naming/element usage for illustrative purposes):

<script type="differential"> <!-- maybe new element <scriptset>? -->
  <script type="module" syntax="2019" nonce="xxxxxxx">
    // I'm inline ES2019
  </script>
  <script type="module" syntax="2018" src="2018.js" integrity="sha384-xxxx" crossorigin="anonymous"></script>
  <script type="module" syntax="2017" src="2017.js" referrerpolicy="no-referrer"></script>
  <script nomodule src="legacy.js"></script>
</script>

Allows full reuse of the existing script element with semantics similar to picture (the first satisfying script element is used). This also allows for inline scripts. The script element with the syntax attribute could even potentially be used standalone. I think using an attribute name of ecma or standard would also be more explicit as to its purpose (assuming the threshold was specification compliance). The supports concept with individual feature flags from the above post could also be an additive (or replacement) capability in this scenario as well.

[keithamus]

I don't think this is a problem worth solving.

On one hand I think it is easy enough to solve this for people who want to today; which I imagine to be a tiny fraction of developers; I imagine most folks will continue to use Babel as a compilation step, a huge portion of these folks will only output one target (probably whatever babel-preset-env gives them), the subset of users who do end up compiling to multiple targets are probably in single digit percentages, and probably have the engineering bandwidth to implement their own solutions in JS using feature detection with dynamic imports. I think it is reasonable enough for these folks to do something like the following:

if (feaureDetectEs2018()) {
  import('./index.2018.js')
} else if (featureDetectEs2017()) {
  import('./index.2017.js')
}

Perhaps effort would be better put into a supports style interface ala CSS @supports which can be given some kind of feature set - thereby meaning less work for a roll-your-own solution.

My second point which coincides with a few commenters here is that there really is no way of knowing what something like 2018 even means in terms of support. But I'm going to go a little further to illustrate with some concrete examples:

• Safari 12 supports all of ES2018 except for Regexp Look Behinds, should it fetch the 2018 file? Probably not - but it seems a shame to ship transpiled async iterators just because regex look behinds aren't supported.
• What about browsers with subtle edge cases, like Edge 17 which supports everything in ES6 but has one tiny bug where destructuring only the second argument whilst using default assignments causes a syntax error?

Issues like the above Edge bug lead me to my next major concern with this; what happens if bugs are discovered _after_ the browser begins shipping support for this years syntax? What recourse do I have if Edge begins optimistically fetching my es2018 only to trip up on bugs it has? If I rolled my own loader (see code example above) I could mitigate this problem by adding more feature detection, what can I do with html attributes to prevent this?

[kristoferbaxter]

@kristoferbaxter

The expectation is once a HTMLScriptElement chooses a syntax version, the resource it chose is responsible for leveraging the correct references to its dependencies (Workers, import statement, import expressions, service workers, importScripts() and javascript: URLs).

This requires the script being external correct? What about inline scripts?

<script type="module">
 // What syntax am I?

 // What syntax is this worker?
 new Worker('./worker.js');
</script>

Quite a good point. The value for supported syntax being available for script would be a possibility. I'll spend some time thinking about this further.

[littledan]

If folks are interested in more granular feature testing, in the style of @supports, I'm wondering if it might make sense to do something based on import maps.

[nicolo-ribaudo]

Issues like the above Edge bug lead me to my next major concern with this; what happens if bugs are discovered after the browser begins shipping support for this years syntax? What recourse do I have if Edge begins optimistically fetching my es2018 only to trip up on bugs it has? If I rolled my own loader (see code example above) I could mitigate this problem by adding more feature detection, what can I do with html attributes to prevent this?

I also have this concern. For this reason, I think that 2018 should only mean "this browser version has been released in 2018" and not "this browser supports es2018": an engine can never be 100% sure that they are correctly implementing every edge case, and "I support es2018" may be a false claim without the browser knowing it.

Using @babel/preset-env we can easily transpile code down to what was supported in 2018, while a browsers telling us that they think they support es2018 doesn't let us know exactly what we should transpile.

[theKashey]

If scripts would be in different __files__(file name patterns) - it would be a pain to import or create Workers. With code splitting in mind, it would be also a pain to create N "full" bundles, without any name intersection. And just very slow.

If scripts would be in different __directories__ - that would solve some problems - webpack and parcel supports publicPath out of the box, and esm just supports _relative_ imports as well.

<script type="module"
        srcset="2018/index.js 2018, 2019/index.js 2019"  
        src="2017/index.js"></script>
<script nomodule src="index.js"></script>

^ it's the same index.js for all the cases, the same names and the same structure.

Plus - it's much easier to generate these _directories_ - just create the first bundle, keeping all language features, and then transpile a whole directory to a lower syntax, which could be done much faster and safer. Here is a proof of concept.

[jridgewell]

Perhaps effort would be better put into a supports style interface ala CSS @supports which can be given some kind of feature set - thereby meaning less work for a roll-your-own solution.

I think CSS @supports is actually _too_ granular. Ie, are we going to ship every permutation of (x, y, z) to browsers to hit the optimal path for all of them?

And even if we make it less granular (@supports 'es2017'), we hit path of bugs. Safari had a broken async/await implementation in 11. Now it has a broken tagged template literal implementation in 12. But I'd imagine they're still going to advertise es2017 support, and they certainly aren't going to ship a new browser patch version to disable es2017 support.

Tying this to a specific set-in-stone supports list is the wrong way to approach this. Instead, we need a way to easily group browsers into a category, and let the community decided what is supported by that category. The category should be granular enough that we can get reasonable "clean breaks" in feature support (eg, how module implies a ton of ES6 support), but not so granular that it is valuable for fingerprinting.

That's why I think browser's build-year is an excellent compromise. Having a full year in a single category means there's not much information to fingerprint (even Safari privacy-stance is allowing the OS version in the userAgent, which roughly corresponds to a yearly release cycle). And if we find out that a browser has a broken implementation of a feature, the community can adjust what level of support (both ES and web platform features!) the build-year implies.

Plus, it'll be soooo damn easy for Babel to spit out "2018", "2019", "2020" builds using @babel/preset-env. This is like a "build it and they will come" moment. There may not be many people taking advantage of this now (through either module/nomodule break or userAgent sniffing), but if we add a feature that allows it to happen easily then we can teach it to everyone as the best way to ship less code.

[domenic]

I'd definitely support Babel or other parts of the tooling ecosystem work on producing babel-preset-env configurations based on browser release year. Then someone could invest in the server-side tooling to find the release year in the UA string and serve up the appropriate babel output. That makes sense as the sort of drop-in configuration change being proposed here, and best yet, it works with existing browser code so you can use it today in all situations.

[philipwalton]

After reading through all the posts in this issue, I wanted to clarify something (since a few people have mentioned this).

From a performance perspective, the following feature detection technique is not equivalent to what @mathiasbynens is proposing:

<script>
if (/* feature detect 2019 */) {
  import('/path/to/main.2019.mjs');
} else if (/* feature detect 2018 */) {
  import('/path/to/main.2018.mjs');
} else {
  // ...
}
</script>

The problem with the above code is it requires the browser to parse the DOM (everything up until the <script> block), parse/compile the script block itself, execute the script, and only then can it start downloading the actual module file.

By contrast, the technique described in the proposal can start fetching the correct script almost immediately because the browser's preload scanner can detect it, or it can be declaratively preloaded via <link rel="modulepreload" syntax="...">.

I know this might not be obvious to people less familiar with how browsers load pages, but I wanted to emphasize that the proposal is not just optimizing for smallest download size, it's also optimizing for fastest script load.

---

With respect to this proposal in general, here are my thoughts:

1) I do believe this is a problem worth solving, because developers (and users) would benefit from an easier way to do declarative, conditional script loading (for the performance reasons I mentioned above).

2) I share some of the reservations others have expressed around ES-year not always being a meaningful feature-set descriptor as well as the likely possibility that some browsers will get this wrong, but I've spent a good amount of time trying to come up with something better, and I haven't been able to.

3) I think following the srcset/syntax attribute module (h.t. responsive images) has a lot of potential, even if the proposed feature-set descriptors don't end up being the ones that are used.

[bkardell]

@philipwalton i understand that, but i also think that making progress and getting people to use something with a short block is considerably better than the status quo, and that experimenting together as a community goal in this space before trying to leap all the way there seems useful and like it carries little risk, no?

[jridgewell]

Then someone could invest in the server-side tooling to find the release year in the UA string and serve up the appropriate babel output. That makes sense as the sort of drop-in configuration change being proposed here, and best yet, it works with existing browser code so you can use it today in all situations.

As a temporary polyfill, yes that'll work. But using the UA is both error prone (because UA's have a lot of cruft), and the responses are effectively non cacheable. Eg, Google's server infra flat out ignores caching on any response with Vary: User-Agent. Private caching could work for some, but I doubt the AMP SREs would allow me to sacrifice edge caching to get this feature.

There could be serve-side solutions that transform HTML responses. They'd sniff the UA, and rewrite the script's URL to the appropriate release year build. This is possible, but I don't have a good sense of whether the community would adopt this. And this will eliminate Signed Exchanges from benefitting (since the response is served by a third party who can't modify the response), which is another blocker for AMP (and maybe others if SXG picks up).

So we go to client side solutions. But that'll require downloading/running a client side UA sniffer (which'll be even more error prone because size is a consideration now). And it'll increase latency because the eventual module can't be discovered during pre-parsing the HTML response. @philipwalton's comment is excellent, a declarative mechanism is crucial for this to really take off.