Webcomponents: [templates] Ensure that template instantiation actually improves the platform

[smaug----]

Before adding anything as complicated as https://github.com/w3c/webcomponents/blob/gh-pages/proposals/Template-Instantiation.md it is better to ensure it improves the platform.
I'm especially interested in the performance aspect.

If a polyfill could implement the proposal without being significantly slower, would there be need to add the API natively?

[jakearchibald]

Aside from the performance aspect, @surma is interesting it writing developer-facing documentation for this proposal, to lower the bar for feedback.

[JanMiksovsky]

I spent last week implementing a polyfill for a syntax variation of the template instantiation proposal, and also ported a number of web components from the Elix library over to that polyfill to see whether template instantiation would make creating such components easier.

Upon reflection, it does not appear that template instantiation really helps us.

The web components we create are typically more complex than can be addressed with mustache syntax. We use a mixin architecture to create our components; e.g., a complex carousel component of ours is currently built from ~20 different mixins, each contributing some aspect of programmatic or interaction behavior to the component class.

Significantly, in this mixin architecture, the component class that defines the template _doesn't know_ what properties of sub-elements within that template will need to be modified by the mixins. Instead, a component asks itself (and therefore the mixins along its prototype chain) to build up a dictionary of all the updates that should be applied to the elements in its shadow tree. Then the component applies those updates.

Such separation of concerns means we cannot use the mustache syntax proposal, because that requires the complete set of attributes that will be modified to be expressed directly in the template.

One thing that might help would be a way for a component class to indicate that a collection of property updates be applied to a given element. Perhaps this would be something along the lines of React's spread syntax:

<div {...props}/>

We'd want to avoid introducing new syntax, so maybe we'd have a special property?

<div properties="{{props}}"></div>

I had a chance to talk about these experiments with @rniwa at Apple last week. During our conversation, I proposed adding something like the properties= idea above to Ryosuke. He suggested that, given our existing mixin architecture, what we'd really benefit from is something that lets us apply a collection of property updates to a _tree_ of elements.

I've gone ahead and written up that idea as a proposal for bulk property updates. (Filed as a separate issue.) That includes a hypothetical applyPropertiesById method that would achieve what Ryosuke suggested. With something like that, we wouldn't need the template instantiation proposal.

In any event, for the time being, we don't see template instantiation helping our web components library.

[rniwa]

It's true that much (if not all) of what this API provides can be implemented in JS. For us, the main benefit of this proposal is that it paves a way to come up with a declarative syntax for custom elements. It also provides a useful mechanism for template libraries to update different parts of DOM easily. Finally, the default template processor provides a mechanism to easily create a template instance without having to import a third party library.

[mildred]

Hello, I'm new to this discussion, but I'd like to point out that there is no need for special syntax with curly braces to get useful things from templates. I'd like to point out two different template engines based on the same principles each that don't require any template syntax at all:

• https://github.com/pure/pure and https://pure-js.com/
• https://github.com/tmpvar/weld

They both work on the same principles: You define your markup with absolutely no special syntax, custom attributes or anything like that. Then, you define a JSON object on the side that goes with your markup that is telling how to instantiate the markup for a given input data structure.

What I really like from this approach is that you get a clear separation between the markup and the data structure you want to template. From the beginning, web standards is all about separation of concerns, and I believe this approach fits very nicely with that.

[dmitriid]

@rniwa

The main problem with template instantiation proposal is that it attempts to solve a symptom, not provide the cure for the cause.

And there are two causes:

• no declarative API for DOM APIs
• no data binding mechanism

As all compromises, template instantiation proposal solves both poorly.

JS. For us, the main benefit of this proposal is that it paves a way to come up with a declarative syntax for custom elements.
It also provides a useful mechanism for template libraries to update different parts of DOM easily.

This "syntax" already exists, and has been battle proven over several years on thousands of web sites. As is the ability to efficiently and easily update different parts of the DOM.

And it looks more or less like this:

h('div', 
   { 
        attrs: { className: "x" }, 
        props: { duration: y }, 
        on: { click: () => z()  }
   }, 
   [ h(...), h(...) ]
);

Yes. It's virtual DOM, popularised by React, and which exists as virtual-dom, hyperscript etc. There are differences in the APIs between these libraries, but they are quite superficial, and they more or less converged on the same set of principles:

• declarative API that is pure JS, no external dependencies, no templates, just functions
• the API is essentially f( tag, { props, attrs, events, lifecycle }, [children]) which is easy to reason about, easy to analyse, debug, optimise
• it solves about 100% of Web Component's data binding woes
• it solves the whole "we don't know how to provide a declarative API for WebComponents" because it also solves "we will provide a unified consistent API for all of HTML/DOM, not just WebComponents"
• it solves server-side rendering

Meanwhile templates... I don't know what actual issues do they solve? They introduce way more problems than it's worth:

• somehow only limited to template instantiation inside custom elements. Why not use them for the rest of the DOM?

• is a new templating language:

  • has no relation to the rest of the browser
  • needs a separate parser and execution model (not a small chunk of work in itself + an increased attack surface)
  • as all templating languages will be limited (so, limited control structures, limitations on what you can call from the host environment, poor and limited available standard library, if any)
  • only provides an imperative API to work with from JS that's no better and no easier than existing DOM APIs:

  // before
  var X = template.getElementById("#part");
  X.appendChild(document.createElement('span'));
  X.appendChild(document.createTextNode('hello'));
  
  // after
  var X = template.getRequiredTemplatePartSomehow();
  X.replace([document.createElement('span'), 'hello']);
  

  The vast majority of the web isn't even using DOM APIs with Web Components, and goes straight for .innerHtml (even lit-html does that with reckless abandon). Given that the template proposal doesn't really add any new easy-to-use APIs, it will still be easier to drop to .innerHTML or reach for any of the virtual dom libraries out there than use the templates.

In my opinion, the templating proposal has its value to show some the problems that people experience with Web Components, but can never be a solution. The solution lies in better, easier-to-use declarative DOM APIs.

[bahrus]

Hi @dmitriid

I don't follow why template instantiation could only be used inside a web component. It would be useful anytime there's repetitive markup that needs customizing in each instance. I encounter that scenario quite a bit, before and after web components.

The fundamental issue templates are trying to solve, as I understand it, is that repeatedly cloning a template for large chunks of easy to parse html is faster than making lots of appendChild or innerHTML calls, which can only be done after the expensive job of parsing the JavaScript. This is an empirical claim. Are you claiming otherwise? It would be great to show your counter-factual results.

HTML templates also feel much more "declarative" to me. Templates are inert, containing a data format (xml-ish) so they can be loaded quickly with no side effects. hyperscript may also have no side effects for simple examples (other than rendering the html of course), but you can evaluate any function you want during the processing, which could have unexpected side effects. To be fair, some aspects of the template instantiation proposal (which looks quite different from how you are describing it, with your before and after -- are we looking at the same proposal?) may also allow for functions with side effects, but I could be wrong.

For the record, I'm not opposed to introducing an h function into the api, if it fulfills some useful purpose, but I fail to see how it is easier to use than tagged template literals. Could you elaborate? The performance numbers I've seen comparing lit-html and hyperHTML, compared to react (and even preact) make me wonder what your objections are?

[dmitriid]

@bahrus

I do apologise if I sound terse or rude in the text below, as I'm writing this rather quickly in a spare moment (I wanted to acknowledge your response quickly, and not have you wait for a day or two).

I don't follow why template instantiation could only be used inside a web component. It would be useful anytime there's repetitive markup that needs customizing in each instance.

The whole discussion is mostly in the context of Custom Elements and Shadow DOM (see 2. Use Cases). The proposal itself is only limited to <template> elements.

That's why in my mind it was only limited to custom elements.

However, true, you can use them elsewhere:

rniwa = {name: "R. Niwa", email: "[email protected]"};
document.body.appendChild(contactTemplate.createInstance(rniwa));

This does still leave the question of how to more complex/nested templates where parts of a template are defined in other templates etc.

but you can evaluate any function you want during the processing, which could have unexpected side effects.

That is, side effects that are desired by the developer using it ;). There's no way to use a template that has this:

{{foreach items}}
   <li class={{class}} data-value={{value}}>{{label}}</li>
{{/foreach}}

without first calling arbitrary functions to create these items and binding them to the template. And since the proposed way of creating such things is the same old DOM API with hardly any improvements, I fail to see the improvement.

And as you correctly mentioned, templates will need to be able to call arbitrary functions.

I fail to see how it is easier to use than tagged template literals. Could you elaborate? The performance numbers I've seen comparing lit-html and hyperHTML, compared to react (and even preact) make me wonder what your objections are?

The only reason lit-html works as it does is that dozens (hundreds?) of engineers spent hundreds of hours optimising to things that are frequently used and abused in JS:

• string processing and RegExps
• .innerHtml

More or less the only thing that lit-html does is parse a string at runtime, with regexps, concatenate it into an opaque string blob, and dump it into browser via .innerHtml, and let browser deal with it. (I have a separate rant on tagged template literals elsewhere).

It's not a good thing. It's a very bad thing, it only happens to work fast enough because browsers have had decades to optimise this (and ten years ago using .innerHtml was considered extremely bad practice see e.g. this StackOverflow comment, and even now .innerHtml is not optimised enough for certain cases, _and_ there are security considerations).

Meanwhile declarative DOM/virtual DOM libraries have to recreate the entire DOM model in memory, and manually diff it against the browser DOM.

The solution to all that (and to template instantiation) is definitely not, in my opinion:

• a limited templating language that can only be used in <template> elements, and whose actual API is only marginally better than the rest of DOM APIs (and introduces a host of other problems, some of which
• concatenating strings at runtime and dumping them into the DOM via .innerHTML

A declarative API (with, hopefully, browser-native DOM-diffing) solves a lot of the problems:

• data binding (it just becomes function calls with your data/parameters)
• efficient DOM updates (you only regenerate parts of the tree that change)
• instantiation (it's just a function call, and many lifecycle methods to trigger it that are already there)

It may/will still be awkward to use, obviously. The next best thing, IMO, would be a standard/declarative way to create a DOM AST that you can pass to the browser. Virtual DOM libs, in essence, do that already. To quote Dan Abramov:

// JSX is a syntax sugar for these objects.
// <dialog>
//   <button className="blue" />
//   <button className="red" />
// </dialog>
{
  type: 'dialog',
  props: {
    children: [{
      type: 'button',
      props: { className: 'blue' }
    }, {
      type: 'button',
      props: { className: 'red' }
    }]
  }
}

Unfortunately, we cannot generate this AST for the browser and let it deal with it efficiently. We have to fall back to .innerHtml or hundreds of lines of tedious imperative code (or use libs/frameworks/wrappers).

With a natively supported declarative description of the DOM/AST you still have a low-level primitive that libs/frameworks can make even easier to use, but you can also trivially use it in vanilla JS code.

I do hope I made sense in the ramblings above :)

[bahrus]

Thanks, @dmitriid, for your civil and non-rambling response :-)

I agree templates are a key part of the web component stack, and in retrospect you were replying to a message arguing why an (apparently) small library like what template instantiation would (apparently) entail should be built in to the platform, and the argument was made that it would benefit declarative custom elements.

But more generally, I for one have taken the liberty of using templates without web components on a number of occasions, and I don't think I'm alone.

I actually agree with you that the platform would benefit from some helper functions for those scenarios where a programmatic api is needed -- one that improves upon Object.assign, one that is specifically tailored for setting properties / attributes / events on a DOM element, i.e. a similar purpose to what you are laying out.

For example, I've been toying with a function I call "decorate", which I modeled after Vue/Polymer 1, but now realize, thanks to your bringing it to my attention, is quite similar to h.

The difference is that this decorate function is applied to an existing element, coming from a template or existing DOM tree, rather than being only useful for generating the HTML data structure itself.

I do think, given the wide range of libraries doing something similar, having something built in to the platform would be useful. Just my two cents. But so would template instantiation. I don't see why one precludes the other.

If a template api/syntax has built in functions to do certain things (like for-each), i.e. officially sanctioned functions, that is quite different from saying you can use any user-defined function you want. For a concrete example of what I'm getting at, I need to change the subject slightly: Take the github developers who are managing the github web site-. They are defining web components, which can enhance the markdown vocabulary. I recently realized I can use their custom elements in my markdown! Github only trusts that because they trust their own web components. They won't allow us to use arbitrary web components, for fairly obvious reasons.

I'm not saying user-defined functions must be forbidden in template languages, only that it is a significant line one is crossing, one which could be use to separate "non-declarative" vs "declarative" in a coherent (I think) way.

I suspect the author of hyperHTML would take issue with the statement that it required dozens or hundreds of engineers to match the performance of h based libraries :-).

Thanks for the link to your critique of tagged template literals. I'll take a look.

[WebReflection]

I suspect the author of hyperHTML would take issue with the statement that it required dozens or hundreds of engineers to match the performance of h based libraries

I think @dmitriid was referring to the fact these libraries (lit, lighter, or hyperHTML) are fast only because the primitives used have been made fast by browsers engineers.

I also think there's no shame in knowing, and using, fast primitives to deliver better UX, and that's naturally the goal of any performance oriented abstraction anyway 👋

[bahrus]

@dmitriid, is that what you meant? If so, apologies for misinterpreting.

I guess I was thrown by the use of the word "to" in:

The only reason lit-html works as it does is that dozens (hundreds?) of engineers spent hundreds of hours optimising to things that are frequently used and abused in JS:

followed by his comment on what the browser engineers have done:

it only happens to work fast enough because browsers have had decades to optimise this.

I.e hundreds of lit-html engineers (I can't seem to locate his reference to hyperHTML, which line is that?) used bad practices, which happened to not matter because of the decades spent by browser engineers optimizing on those bad practices.

Apologies for my lack of reading comprehension.

I agree with you -- lit-html, lit, hyperHTML are fast and easy to use, great libraries, because they were built and optimized by a handful of dedicated and smart engineers, built on fast primitives built by great browser engineering teams. Somehow I didn't quite find that sentiment shared by @dmitriid, but what do I know? @dmitriid, we're all in agreement?

[Jamesernator]

@dmitriid Neither template instantiation or lit-html use .innerHTML to update the DOM. This seems to be some FUD around how template instantiation works but template instantiation and lit-html actually work more efficiently than a virtual DOM by holding references to part of the DOM that have template parts. In lit-html's case, it only creates the initial template using .innerHTML, to actually update the DOM it inserts nodes directly into the correct location (and similar for attributes).

For example consider this template:

<template id="exampleTemplate">
  <span title="{{foo}}">This is a sample {{bar}} with instantiation</span>
  <span>Also {{foo}}</span>
</template>

when we create an instance of the template a set of references are created to the elements that have that name:

{
  foo: [
    AttributeTemplatePart {
      el: /* ref to <span title="{{foo}}"> element */
      attrName: 'title',
    },
    NodeTemplatePart {
      previousNode: /* The previous text node before {{foo}}: "Also" */
      nextNode: /* The text node after {{foo}} */
    },
  ],
  bar: [
    NodeTemplatePart {
      previousNode: /* The previous text node before {{bar}}: "This is a sample " */
      nextNode: /* The next text node after {{bar}}: " with instantiation"
    }
  ],
}

now when we call something like instance.update({ foo: 'banana', bar: 'cabbage' }) all .update needs to do is something like this:

function update(data) {
  for (const [key, value] of Object.entries(data)) {
    for (const templatePart of this._templateParts[key]) {
      templatePart.update(value)
    }
    this._templateParts[key].update(value)
  }
}

this is way more efficient than maintaining a virtual copy of the DOM as we simply implement AttributeTemplatePart and NodeTemplatePart so that they go directly to their location in the DOM and update the value.

e.g. NodeTemplatePart could be implemented like this:

class NodeTemplatePart {
  constructor(previousNode, nextNode) {
    this._previousNode = previousNode
    this._nextNode = nextNode
  }

  // This is overly simplified and assumes there's a node both before and after
  // the {{curlies}}, a real implementation would hold a reference to the parent element
  // as well and if there's no previousNode/nextNode it'd just replace the whole contents
  update(values) {
    if (typeof values === 'string') {
      values = [new Text(values)]
    }
    while (this._previousNode.nextNode !== this._nextNode) {
      this._previousNode.nextNode.remove()
    }

    for (const value of values) {
      this._nextNode.parentNode.insertBefore(value, this._nextNode)
    }
  }
}

[WebReflection]

Just for documentation and clarification sake, everything @Jamesernator said is the exact same for both _lighterhtml_ and _hyperHTML_, based indeed on the same domdiff library, which uses innerHTML once per unique template tag, on an offline template element, and update the related DOM when needed, it never uses innerHTML again (unless explicitly required by the user, but that's another story)

[dmitriid]

I think we've steered slightly off-track, so I'll try (hopefully :) ) to get back to what I was intended to say (once again, sorry if I get sidetracked again).

There are two parts to this long-winded comment:

1. What I think (and what we should still strive for)
2. What may actually need

What I think (and what we should still strive for)

In a tl;dr kind of way my thinking comes to this:

lighterhtml and hyperHTML, based indeed on the same domdiff library, which uses innerHTML once per unique template tag

There are two question arising:

1. In 2019 why is there no better API than dumping blobs of strings into DOM via .innerHtml?
2. Why are facilities like domdiff not provided natively by the platform?

In my opinion, if we answer and find solutions to these two questions, the entire template instantiation proposal will be rendered moot. To slightly re-word the reasoning behind the proposal:

The HTML5 specification ... doesn't provide a native mechanism to instantiate with some parts of it substituted, conditionally included, or repeated based on JavaScript values

And then it does correctly say:

making it hard for web developers to combine otherwise reusable components when they use different templating libraries.

The thing is though: is the answer to that a yet another incompatible templating library and syntax? Developers will continue using their own incompatible templating libraries regardless. The reason is simple: any and all templates are limited in functionality and scope. Incompatible templating systems arise because people find that some templating system X is lacking some crucial functionality.

You can see it with Web Components themselves. When they were finalised and started shipping in browsers there was abundant joy. However, just two short years later, the mood has shifted to "web components APIs are a barebones set of low-level primitives aimed at library writers" and people go out of their way to create better more useful abstractions on top. Including pushing everything into strings.

But this was a sidetrack. Let's get back to template instantiation. Among other things mentioned (or shown as use cases):

• it's hard to create DOM via DOM APIs
• it's hard to find and update required parts of the DOM
• it's hard to do multiple updates efficiently
• it's hard to compose actual components (where a component maybe a deeply nested, updatable DOM structure)

The reason is obvious: these things are hard to do because DOM APIs are low level, imperative and verbose. However, it has been proven multiple times that these APIs can be made sufficiently high-level, declarative and succinct even in userland. And that they literally solve all of the problems above:

• create DOM? A breeze with declarative APIs
• find an update required parts? Same
• multiple updates? Ditto
• compose? These are just nested function calls/ASTs, we compose things all the time

Instead of providing these facilities, the proposal gives us:

• a new templating language (incompatible with everything else)
• that is both very limited in functionality (poorly specified/underspecified if and foreach as the only control structures) and unlimited in functionality (we can call functions in the host language)
• that is alien to the system (literally nothing in the platform looks or behaves like this)
• that requires a different parser, a different execution model, hooks into the rest of the platform to make it workable
• solves the main use case of composing components poorly, or not at all (you need to nest template elements within template elements within template elements...)
• doesn't specify how you would instantiate elements not from static sources, but from dynamic data (I'm guessing .innerHtml again?)

Even though all these problems have been solved dozens of times over by simply providing a better API.

It is my continuing belief that the only right way forward is not to make the platform increasingly weird and complex, but to improve the APIs available to developers and let them figure out what to do with them.

See part 2 on what developers already do with existing APIs and how we can help them by being a better platform.

Declarative DOM tree

As I already mentioned a few thousand times :) developers already create DOM trees, diff them, and apply only changes. In userland. Why can't platform itself provide similar APIs and capabilities is beyond my understanding at this point.

However, there are now emerging technologies that may require not just native browser APIs but also a way to provide the browser with a DOM Tree and let the browser figure out what to do. I'm talking about Phoenix LiveView. The idea is as follows:

• on the server figure out the actual data that has changed
• send it to the browser
• use morphdom to figure out minimal updates to the DOM
• update DOM

In the end it does this:

(For crazy versions, see server-rendered FlappyBird or search Twitter for LiveView).

Template instantiation and updates would have hard time offering anything of value to an approach like this. A friendly native API to tell the browser exactly what you need or a way to declaratively define and provide DOM trees though?

• They would basically kill the need for all the VDOM implementations (except tiny wrappers for people who don't like the original API)
• They would obviate the need for a complex solution like template instantiation
• They would actually provide better interoperability because whatever templating system, or a DOM library you have, if in the end of the day it outputs the same DOM structures (or uses the same APIs), it doesn't matter what it looks like. For example, Vue's templates compile to code that is for all intents and purposes indistinguishable from what React's JSX compiles to. And just because these are two different userland implementations they are (somewhat) incompatible. Why not provide a common API in the platform?
• They would let people experiment even more freely with more crazy ideas (60 fps animation entirely provided by the server is already possible, see LiveView links above) because they wouldn't be fighting the limitations of the platform, but be empowered by it.

In my opinion template instantiation and the current (and the only, cca 90s) generation of DOM APIs provide none of that and are very reluctant to move forward.