Html: "decode" attribute on <img>

[smfr]

Decoding of large images can block the main thread for hundreds of milliseconds or more, interrupting fluid animations and user interaction. Currently, there's no way for a web author to specify that they want an image to be decoded asynchronously, so there are scenarios where it is impossible to avoid UI stalls.

To solve this problem we propose an "async" attribute on image elements. This attribute is a hint to the UA that the author has requested asynchronous decoding. This implies that if the UA paints an image after the "load" event has fired, but before the image has been decoded, the UA is allowed to not paint the image (rather than block on decoding it).

To notify authors when a decoded image frame is available, we propose firing a new event, "ready", on the image element. This would allow authors who require a fully-decoded image, in content that is sensitive to UI stalls, to wait for the "ready" event before doing something that brings the image into view (such as a CSS transition).

Some images repeatedly decode frames, for example, animated GIFs. In addition, the UA can throw away the decoded frame for a still image, requiring a re-decode. In these cases, we propose that the "ready" event only fires once, the first time a frame is available for display.

ISSUES:
"async" is currently used to imply async loading, and it's possible that we'd want to use it in this sense for images too. Maybe call the new attribute "asyncDecode" or something else.

This only solves the problem for image elements, not CSS images. We could add a CSS property to allow async decoding for CSS images, and fire events on the element to which it applies.

[grorg]

This only solves the problem for image elements, not CSS images. We could add a CSS property to allow async decoding for CSS images, and fire events on the element to which it applies.

We could maybe follow CSS animations/transitions here. Make a new "imageDecoded" or "imageReady" event (I suck at names) and fire it on the target that has the new CSS property.

I wonder if you'll ever want some parts of the element to have async images, and other parts to not. e.g. borders vs background. If so, it seems like it would have to go in the new image() function. (It already has tags for rtl and ltr).

[grorg]

This only solves the problem for image elements, not CSS images. We could add a CSS property to allow async decoding for CSS images, and fire events on the element to which it applies.

I do think it is confusing that async here is a slightly different meaning from script (which is "Set this Boolean attribute to indicate that the browser should, if possible, execute the script asynchronously." )

Or does it?... Would you consider image decoding to be the equivalent of script executing?

[grorg]

Of course, the HTML5 specification has a more complete definition of script async.

[smfr]

Actually reading that, I think 'async' here is quite a good match.

[duanyao]

Isn't off main thread image decoding already possible in browsers without a spec change? E.g.:
https://bugzilla.mozilla.org/show_bug.cgi?id=716140
https://bugs.webkit.org/show_bug.cgi?id=90375

Does "async" also means lazy download of the image file?

[smfr]

Isn't off main thread image decoding already possible
Possible, yes, but the main thread may still have to block on the decoder thread if it is still decoding an image when the main thread wants to paint the image. If a UA chooses not to block, then the user gets a a blank or partial image even though the load event fired (I'm not sure what Gecko does, and Mac/iOS WebKit have used main thread decoding for the past few years).

This proposal allows an author to present images with a guarantee both that the main thread is never blocked on image decoding, and that the image is ready to display when the "ready" event fires.

Does "async" also means lazy download of the image file?

No. It's purely about decoding.

[duanyao]

Once a UA implemented off main thread decoding, there is little reason for them to keep the old behavior, so still no need to specify "async" in HTML. Currently there seems no UA which (intentionally) blocks layout/paint on image download/decoding.

As to "ready" event, I think the situation can be compicated, because UA might drop decoded images that are no longer visible (out of viewport or in background tabs) to save memory. For completeness, "unload" event may be also needed.

[smfr]

Currently there seems no UA which (intentionally) blocks layout/paint on image download/decoding.

First, we're not talking about downloading here. This proposal doesn't change anything about how image downloading works.

Second, Mac and iOS WebKit block painting on image decoding (I work on WebKit for Apple, BTW). That's the problem we're trying to address. If other UAs have solved this problem, I would be interested to hear how.

[duanyao]

I am confused because I observed that UAs may display images increamentally on a slow connection, so I think decoding is not blocking painting, at least in such situation.

So does "async" also gurantee that the image is not visible when the downloading/decoding is in progress?

[smfr]

UAs can decode and paint an image that is not fully downloaded, yes. It may appear partial or low-resolution in this state, but the decode itself can still take time (100ms or more). I think the "ready" event should fire when the first full-resolution frame is available.

[annevk]

@igrigorik @yoavweiss may have opinions.

I think the API should be <img>.ready returning a promise. No opinions on the attribute, though we might in the future also want something that influences when the image is downloaded (e.g., an indication to the user agent that it's fine to load the image later).

[domenic]

Let's set aside the bikeshedding on the name and style of the event/promise for now, and try to get some multi-implementer interest in the base feature of using a new attribute to control image decoding. I think the key question is:

Second, Mac and iOS WebKit block painting on image decoding (I work on WebKit for Apple, BTW). That's the problem we're trying to address. If other UAs have solved this problem, I would be interested to hear how.

For example, a UA could choose to simply make all images have this "async" behavior and draw placeholders. Presumably this might not be great for authors (why, exactly?), thus this proposal to make it opt-in, if I am understanding correctly. If we were to, in certain circumstances, flip the default to async, this is starting to sound intervention-ish; /cc @RByers @ojanvafai

Another aspect that this is related to is off-main-thread image decoding. Some of the recent work on ImageBitmap has enabled this to be done in JavaScript; see https://developers.google.com/web/updates/2016/03/createimagebitmap-in-chrome-50. But this requires awkward contortions because then you have to draw it onto a canvas, instead of using a simple <img>. I am unsure to what extend this async proposal overlaps---would this allow off-main-thread image decoding? Is it in some sense sugar over the ImageBitmap technique that makes it less weird? /cc @junov since he's our ImageBitmap guy, both for Chrome and for this spec.

My sense is that this area is of definite interest to Chrome, and as such they're good candidates for second-vendor interest. I've tried to pepper @-mentions of people who might know more above. Any thoughts on Mozilla or Edge?

[smfr]

For example, a UA could choose to simply make all images have this "async" behavior and draw placeholders. Presumably this might not be great for authors (why, exactly?)

Some authors need a guarantee that if an image is ready, it gets painted. An image popping in later is not acceptable in some kinds of content.

If we were to, in certain circumstances, flip the default to async

Interesting idea, but I don't think it would fly compat-wise.

I am unsure to what extend this async proposal overlaps---would this allow off-main-thread image decoding?

I'm proposing this now specifically because WebKit has been doing work on off-main-thread image decoding, for some subset of images (webkit.org/b/155322 and related). I don't think a UA would be required to do off main thread decoding to implement this (for example, they could trigger decodes in something like a requestIdleCallback), but pages would still suffer from UI stalls.

[domenic]

Some authors need a guarantee that if an image is ready, it gets painted. An image popping in later is not acceptable in some kinds of content.

I think I understand what you're saying, but can you check if the following expansion is correct?

As a web developer it seems like all of my images just pop in at random times when the browser has finished with them and is ready to show them. The difference between showing a placeholder while it's fetching and then janking while decoding and then painting, vs. showing a placeholder while fetching and decoding, and then painting, is almost unobservable to me.

The difference is in the specific case where I end up listening to the load event. For example, if I create an image out of document, then wait for the load event, I anticipate that when I insert it into the document it will be painted synchronously. (This might even affect from-script-observable things like offsetWidth.)

But as an author, it seems like any time I'm not listening for the load event and assuming the image is fully ready at that time, I'd be able to sprinkle async on the <img> elements, and there would be no observable difference. Right?

[smfr]

... anticipate that when I insert it into the document it will be painted synchronously

That's exactly it. This is mostly useful for images created in script.

any time I'm not listening for the load event and assuming the image is fully ready at that time, I'd be able to sprinkle async on the elements, and there would be no observable difference

Yes. The UA may choose to do async decoding for some or all elements. Addition of the "async" attribute to an may be treated as a hint or may do nothing.

[domenic]

OK, cool, glad I understand!

What do you think of an alternate proposal, to delay the load event until decoding is finished, in all cases?

[smfr]

What do you think of an alternate proposal, to delay the load event until decoding is finished, in all cases?

You can't do that, because loading and decoding are two separate steps. Not all images that are loaded are decoded (we never decode those that don't get painted).

Note that fetching metadata from an image, to get its size, for example, doesn't count as decoding.

[ojanvafai]

I'm pretty sure Chrome strongly supports doing something roughly like this. I'm consulting with folks to make sure I say something representative of our opinion since we've had a lot of complicated discussion about this issue. Will report back soon (please ping me if I lose track of it).

[igrigorik]

Naive question: how would "ready" event work in cases where we have hardware decoding?

[smfr]

I don't think it matters. Hardware decoding happens under a software call to the decoder, just like software decoding. It just might be faster.

[junov]

Responding to @domenic 's earlier comment that there is overlap with ImageBitmap... Yes and no.

I've experimented with getting jank-free scrolling and canvas draws under intense image loading. I used XHR -> creatImageBitmap -> canvas. It worked like a charm, but it has one non-negligible drawback: it pins all the decoded images in RAM unless the app explicitly discards them, so implementing something like Facebook's infinite scrolling on top of this is a bit involved. It would require the web app to be responsible for evicting and triggering predictive redecodes in order to avoid extreme memory bloat (and OOM crashes). Considering that JS code has no visibility into the system's memory contention, using ImageBitmap as a blanket solution for de-janking image-intensive pages is really not that great.

This proposal lets the UA continue to manage the memory occupied by decoded resources, which is a big win for many use cases.

On the other hand, the ImageBitmap approach guarantees that the image _will_ be drawn and that it will be fast. This is useful for cases that don't jive with this statement of the proposal: "if the UA paints an image after the "load" event has fired, but before the image has been decoded, the UA is allowed to not paint the image (rather than block on decoding it)"

[smfr]

If the author really want the image pixels to be available on first paint, they need to either insert the image in script after receiving the "ready" callback, or toggle some CSS so that the image becomes visible.

However, there's a problem with my proposal; there's no way for the author to indicate that the UA should start decoding an image (if that image is unparented), and I don't think UAs will want to start decoding all unparented images with "async".

[domenic]

However, there's a problem with my proposal; there's no way for the author to indicate that the UA should start decoding an image (if that image is unparented), and I don't think UAs will want to start decoding all unparented images with "async".

Since this is most useful for images created in script anyway, maybe an imperative "decode me now" API? Like

img.ensureDecoded().then(() => {
  // OK, now you can insert it into the document
});

Maybe "ensureReady" if we want to hide the implementation details a little more.

[smfr]

That could work, yes. I think that means you don't need an "async" attribute for images created in JS.

It could still be useful as a hint on content images to say that you're OK with an image temporarily painting blank even after the load event fires, trading off against possible UI stalls. No doubt people will want CSS pseudo classes to style the various states of an image.

[vmpstr]

As @ojanvafai mentioned, here are our (Chrome) thoughts after having discussed with our team:

We agree that this is an important area to look into and we’re excited to work on solutions in this space!

We see two important use cases with different characteristics.
(1) Deferring: Ability to defer decodes of images in dom order to avoid janking the page.
(2) Predecoding: Ability to know when images have been decoded so they can be inserted into dom without jank

An async attribute on images solves deferring, but not predecoding. An on ready event solves predecoding, but not deferring. It seems like these could potentially be solved separately.

We have some concerns about tricky parts of the ready event:
* Does every image get "on ready" or just ones marked async?
* Would a browser need to predecode all out of dom images so that they would receive this event? Or just ones marked async?
* Is there an "on unready" event?
* If an image is evicted and then re-decoded, is there a second ready event? The original proposal said it’s just the first time, but we’re worried this is too confusing and can lead to jank in subsequent uses.
* If an author has a series of images (e.g. carousel) out of dom, how does an author communicate which ones are important to decode first?

Here's two rough proposals to handle each of these use cases.

(1) Proposal for handling deferring images:
Give the async attribute three values: async, auto, and sync. sync behaves as today’s image elements do without any attribute specified, where once an image has been loaded it will appear immediately (and be decoded synchronously) if inserted into the document. Images marked async can get loaded/decoded best-effort without janking. auto would involve browser heuristics to decide if the image could and should be loaded async or if it needs to be sync. async is therefore just a more aggressive version of auto in practice. sync is mostly just there as a safety valve for developers in case browser heuristics get it wrong.

There’s no ready event here, as it’s not part of this use case. In particular, this helps avoid confusion around what happens when a decoded image is discarded and then later decoded again (e.g. because the image was scrolled out of view and then back in). The browser can treat it the same way regardless of whether it was the first time the image was being decoded or not.

We’re not 100% sure we’ll need the async value without more experience on trying to ship changes here. It might be that we can make all images that we want to async and just have auto and sync in the end.

(2) Proposal for predecoding images (similar to what @domenic suggested):
Provide an explicit decode function for an image with some sort of callback when the image is decoded. This would notify the author that it's ready to be appended to the page without jank. The browser would promise to keep that image decoded for at least that raf frame. It would also allow an author to call this decode function again in the future if needed, and to prioritize images in an image carousel case. Finally, we might want to make this decode function return some success/failure (in case of an overloaded cache).

Possible sample code:
var img = new Image();
img.src = ‘...’;
// The decoded image is guaranteed to be kept in memory until the frame
// after the decode promise resolves.
img.decode().then(function() {
// This is guaranteed to paint the image without flicker or decoding jank.
document.body.appendChild(img);
});

Other side questions:
* What about large images that are scaled and cropped? Do these parameters need to be specified? We found that scaling and gpu uploading can be a significant part of image preparation time. Do we just punt on this?
* How is image data that isn’t image elements handled, e.g. should we add an async keyword for background images? Divorcing async from decode makes it easier to do something consistent between image elements and CSS background images.

[smfr]

I like this analysis.

(1) Proposal for handling deferring images
... Images marked async can get loaded/decoded best-effort without janking

Presumably the effects of "async" on a content image is that it can show blank after its load event has fired, right? Seems fine if so. "auto" is weird as an attribute; I would prefer that we say that "async" is a hint and the UA is free to ignore it; if it becomes a cargo cult, I don't want to have to respect it on thousands of images in a document.

(2) Proposal for predecoding images
I like the decode() Promise form; it makes it easy to figure out the behavior with repeated calls on the same image.

What about large images that are scaled and cropped? Do these parameters need to be specified? We found that scaling and gpu uploading can be a significant part of image preparation time. Do we just punt on this?

I say punt; doing scaling/cropping is usually a paint-time operation, and if you tried to do something for these, you'd end up having to do lots of things in the painting pipeline.

• How is image data that isn’t image elements handled, e.g. should we add an async keyword for background images?

Since you can't detect load on CSS images directly, it seems less important to support "async" for them.

[vmpstr]

Presumably the effects of "async" on a content image is that it can show blank after its load event has fired, right? Seems fine if so. "auto" is weird as an attribute; I would prefer that we say that "async" is a hint and the UA is free to ignore it; if it becomes a cargo cult, I don't want to have to respect it on thousands of images in a document.

To clarify a bit, the proposal is to have a single attribute called async, which could have three possible values (async/sync/auto). I agree that these values should still be just hints to the UA and it can ignore them.

The idea behind auto value and how it's different from async is that it allows the UA to adjust its heuristics to either include or exclude more images from being async based on whatever criteria it decides. In other words, async value would mean it's always safe to show a blank image after the load event fired; auto would mean that the UA should decide whether or not it is OK do this.

It's not entirely clear that we need both auto and async, but if auto is the default it would allow the UA to make these decisions for content that didn't explicitly specify what it wants.

[domenic]

Circling back to this to try to get a resolution, since we have a couple implementers definitely interested in implementing.

It sounds like there are now two distinct but complementary proposals in flight:

• A content attribute hint, tentatively async="", for the deferring use case. This would be used declaratively most of the time.
• An imperative API, tentatively decode(), for the predecoding use case. That would be used imperatively.

Let's use thread to work through the design of the content attribute, and use https://github.com/whatwg/html/issues/2037 for the decode() API.

It seems like the biggest outstanding point for the attribute to decide on is whether this is a two- or three-state attribute. Does anyone want to help further drive the discussion on that?

[domenic]

@smfr I see that in Safari Tech Preview 40 WebKit shipped an async attribute. Do you have plans to standardize that? It does seem nice and simple.

I'm curious whether @vmpstr still thinks we need a tri-state attribute, or the two-state one you implemented.

[smfr]

We would like to standardize "async", yes. As you say, it's very simple; it's just a hint from the author to the UA that the author is OK if this image is decoded asynchronously, and therefore might not be painted for one or more frames after the load event fires. We decided not to do any kind of "ready" event.

We found it impossible to do async image decoding by default in many scenarios where we wanted to without causing unwanted flashing, so this attribute allows authors to give the UA some more leeway.

[vmpstr]

We're finding similar issues with async images: it's hard to come up with a heuristic that allows us to do this without any unwanted flashes, so I think we should standardize the async attribute.

What are your thoughts on making it tri-state? Basically, it would have three states:

• "auto", the default state, would mean that UA can attempt to do async decoding in cases where it determines that there won't be flashes. This would help with long tail of web content that has large images but isn't trying to do anything fancy that can cause undesired flashes.
• "async", which means that always go async without checking heuristics. This is the async implemented by WebKit
• "sync", which means don't do async decoding even if the heuristics seem to indicate that it would be ok.

The reason we're leaning towards this is basically so that UAs still have some leeway of using a heuristic to help the performance on sites that aren't addressing large image problems on their end. At the same time, it gives the tools to the authors that do want fine grained control.

From the implementation perspective, UAs could just adopt auto = sync, which means that for WebKit the code changes would only need to change the interpretation of the attribute.

[othermaciej]

I don't think we want the "sync" state, just "async" and "auto", which is sufficiently covered by a binary attribute. The attribute is a hint, so the UA is free to apply a heuristic when "async" is not specified, at least as I see it. You can force sync-like behavior but without causing stuttering by using Image.decode(), so I'm not sure we need an additional way to force it. What's the use case you envision for explicitly forcing "sync" instead of accepting the default "auto" state?

[vmpstr]

For me it's just a safety knob in case that we do end up doing async heuristic where the user really didn't want it. You're right however that the img.decode should cover those cases.

Ok, I think for the sake of simplicity, I'm sold on just having a binary state.

[chrishtr]

The use case for the "sync" state is it's a very simple way for developers to opt out of any heuristics, without having to use the image decode API or write any extra script.

The image decode API also only works in a simple way the first time the image is decoded and inserted
into the DOM. It is much more awkward to attempt to use it later in time during the web page's lifecycle,
which may lead to situations such as the image being scrolled offscreen, falling out of the cache, then
returning.

A "sync" attribute on the image is a clean way to handle all such situations, and also makes the sync/async
image attribute self-contained without an implicit dependency on another API.

[othermaciej]

The use case for the "sync" state is it's a very simple way for developers to opt out of any heuristics

Do we have evidence that there's a need to opt out of heuristics in a simple way? With the introduction of "async", heuristics can lean conservative and we can gradually convince content authors to use async.

The image decode API also only works in a simple way the first time the image is decoded and inserted into the DOM. It is much more awkward to attempt to use it later in time during the web page's lifecycle, which may lead to situations such as the image being scrolled offscreen, falling out of the cache, then
returning.

I don't think that's the case. You can always call decode() again and wait for the promise to resolve. Why is that awkward?

[chrishtr]

The use case for the "sync" state is it's a very simple way for developers to opt out of any heuristics

Do we have evidence that there's a need to opt out of heuristics in a simple way? With the introduction of "async", heuristics can lean conservative and we can gradually convince content authors to use async.

Simple is better than not-simple. I anticipate browsers becoming more aggressive with heuristics over time to address the image decode jank problem. A simple opt-out solution for sites which fall into corner cases of those heuristics means it will be easier to implement those opt-outs and therefore more sites
will do it. It also allows a fully declarative approach, rather than the image decode API, which requires
managing the image outside of HTML and then inserting/showing it after decode.

The image decode API also only works in a simple way the first time the image is decoded and inserted into the DOM. It is much more awkward to attempt to use it later in time during the web page's lifecycle, which may lead to situations such as the image being scrolled offscreen, falling out of the cache, then
returning.

I don't think that's the case. You can always call decode() again and wait for the promise to resolve. Why is that awkward?

When do you call decode? For the scroll use case, this requires adding a listener to see when the image
might be coming near the screen again, which requires an IntersectionObserver or custom code. It also
forces the developer to assume that the image fell out of the decode cache and call decode(), even
though it might be useless.

[othermaciej]

When do you call decode? For the scroll use case, this requires adding a listener to see when the image might be coming near the screen again, which requires an IntersectionObserver or custom code. It also forces the developer to assume that the image fell out of the decode cache and call decode(), even though it might be useless.

If the site is paging parts of the DOM in and out, it will be doing all this anyway. If the image is in the DOM the whole time and just scrolled out of view, it's up to the browser to arrange to have it decoded in a timely fashion. I don't think we should give websites a way to force scroll jank for previously decoded images. I'm strongly against adding a new web platform feature that would let web content authors force bad scrolling.

[chrishtr]

If the site is paging parts of the DOM in and out, it will be doing all this anyway. If the image is in the DOM the whole time and just scrolled out of view, it's up to the browser to arrange to have it decoded in a timely fashion. I don't think we should give websites a way to force scroll jank for previously decoded images. I'm strongly against adding a new web platform feature that would let web content authors force bad scrolling.

I definitely agree that we should not add a feature which requires scroll jank. However, "sync" does not require scroll jank, it just requires displaying all content of a tile at the same time. If the browser is not done with decode and raster for a tile, it will checkerboard that tile (behavior which is already present in the composited-scrolling paths in Chrome and Safari, and I assume all browsers) but scrolling will still proceed, possibly displaying checker-boarded tiles for a brief period.

A second point: the image decode API does not allow the developer to express that two images must display at the same time as surrounding content, because each of them has a different promise with its own lifecycle. It doesn't suffice to wait for both promises, because it is only guaranteed to be valid for a short period. OTOH, placing "sync" on both expresses exactly this, which I expect is a common use-case from developers.

[othermaciej]

Are there websites that benefit from whole tiles being blank when it's slow to decode an image while scrolling, instead of the image being blank?

For the two-image case, maybe we should enhance the decode() API to specify a list if anyone actually has this use case. I don't think we want to encourage developers to insert multiple images into the DOM with forced sync decoding specified since that risks jank. If the "both images or nothing" case is really important, we should provide a way to do it without risking either flashing or jank.

Can you name specific examples of websites that would want either of these behaviors? It's hard to argue with the expectation that it's a common use case without concrete examples.

[chrishtr]

Are there websites that benefit from whole tiles being blank when it's slow to decode an image while scrolling, instead of the image being blank?

All websites currently receive this behavior. It's hard to say which would prefer the current behavior
to a new one. I'm pretty sure many will not like the new behavior. People don't like visual distractions.

For the two-image case, maybe we should enhance the decode() API to specify a list if anyone actually has this use case. I don't think we want to encourage developers to insert multiple images into the DOM with forced sync decoding specified since that risks jank. If the "both images or nothing" case is really important, we should provide a way to do it without risking either flashing or jank.

Adding a more advanced feature to the decode() API strikes me as much more complicated than
simply supporting a "sync" attribute.

Can you name specific examples of websites that would want either of these behaviors? It's hard to argue with the expectation that it's a common use case without concrete examples.

There are lots of top sites that try really hard to avoid extra blinking or visual distraction during
load or animations. Professional, advanced sites often do things like fading in after load, to
make the UX more elegant, not to mention native apps, which often tightly control when images
display. Many people also commented on the launch of Edge that its loading seemed faster and smoother than Chrome's because it didn't pop in content during load as much. Both of these are
examples of showing the equivalent of checkerboarded tiles rather than partial content.

One might argue that my anecdotes are mostly about load, but I think they demonstrate that it's
common to want all content to display at the same time, and I think the "sync"
attribute is a clear and easy way to achieve that, without requiring interaction jank.

[smfr]

I definitely agree that we should not add a feature which requires scroll jank. However, "sync" does not require scroll jank, it just requires displaying all content of a tile at the same time

This isn't true in the situations where browser still falls back to main-thread scrolling (e.g., in Safari, overflow scroll, iframe scrolling, and some scenarios with background-attachment:fixed), so "async=sync" [weird combination!] would still cause main thread stalls.

[smfr]

I do worry about cargo-cultism here. If we provide "sync", it might be seen as a big hammer to make image loading "better", defeating browser attempts to get more stuff off the main thread.

[chrishtr]

This isn't true in the situations where browser still falls back to main-thread scrolling (e.g., in Safari, overflow scroll, iframe scrolling, and some scenarios with background-attachment:fixed),

Good point.

But: Chrome aims to have composited scrolling on all scrolls over time (by implementing raster-inducing scrolling, which would be able to mutate the display list on the compositor thread and re-raster tiles). IOW the user agent has ways to make this not blocking user interaction for scroll.

I should also point out another reason why I think "sync" is a good idea: not only does it allow developers to specify sync behavior in cases where they want it, but it also allows user agents to have a freer hand to be a bit more aggressive with deferring images in "auto" mode, because developers will have an opt-out. I think this will make it easier for user agents to reduce jank on the long tail of content which uses default settings. It sounds like Safari's early implementation experience matches that of Chrome's recent work - it's hard to come up with heuristics that don't introduce blinking when sites don't want it.

so "async=sync" [weird combination!] would still cause main thread stalls.

Yeah, the name would have to be changed in this case. :)

[othermaciej]

I should also point out another reason why I think "sync" is a good idea: not only does it allow developers to specify sync behavior in cases where they want it, but it also allows user agents to have a freer hand to be a bit more aggressive with deferring images in "auto" mode, because developers will have an opt-out.

I don't think that's right. Having an explicit opt-out doesn't really make it more ok for a heuristic to break existing content. At least that's not the strategy we take.

[chrishtr]

I don't think that's right. Having an explicit opt-out doesn't really make it more ok for a heuristic to break existing content. At least that's not the strategy we take.

Could you explain why you think so? Any heuristic will have false positives.

[vmpstr]

We've had a long discussion about this and here's why we feel that the attribute should be tri-state (all of this has been more or less mentioned throughout this thread):

One of the objectives of this attribute is to give the developers a tool to give the browser a hint that the image should be asynchronous (async value). This is a hint that makes it possible to apply some set of aggressive heuristics in order to minimize jank. I still suspect that we'll have _some_ heuristics, since it may be that small images, for example, are quicker to just synchronously decode rather that rasterizing content without the image first and then rasterizing it with the image after.

The default state would then seem to imply that we should effectively do synchronous decoding. Or have a very conservative heuristic that when applied and the image is deferred, the effect is indistinguishable from synchronous decoding.

However, we would still like to try and improve the user experience on sites that may not adopt the async attribute. The approach that would be nice is to launch the async attribute as described above and then over time improve on the "default" heuristic to be more aggressive.

With this approach, it is highly desirable to have a "sync" state which would revert back to the conservative heuristic initially used for the default state. Note that sync doesn't mean we have to do the decode synchronously. It's a hint meaning that the user prefers this image is synchronous (ie it's important to display with other content). As long as the effect is indistinguishable from sync decoding, the UA is free to do whatever it needs to.

So, on the spectrum of heuristics from conservative to aggressive, we would then have this:

• sync means conservative heuristics
• async means aggressive heuristics
• default/auto means somewhere between the two.

To emphasize again, this doesn't mean that the user can force the browser into a jank. It just means that the user _prefers_ content to not be deferred. As we would spec this, the language would rely on saying "should" more than "must" when dictating the behavior of the browser.

With this, of course, naming the attribute itself "async" is a bit confusing, but we could discuss the name of the attribute if we can agree on the tri-state value.

[othermaciej]

@vmpstr You're using "the user" a lot to refer to what I think is "the content author". Could we keep those distinct? The end user is actually a relevant party here and we should not conflate the content author's preferences with the user's preferences.

On the substance of the argument: if the three values select one of three browser-specific heuristics, which may change over time even within a single browser, then I don't think authors will be able to use it correctly. If anything, this would exacerbate @smfr 's worry that the attribute will start getting used in a cargo cult manner, so it will stop being useful signal.

@chrishtr It's true that any heuristic has false positives, and you have to tune false positives vs false negatives. But I don't think an explicit opt out makes it ok to have more false positives. So it doesn't materially change this tradeoff. I can't imagine resolving a bug that says a heuristic is too aggressive by telling the reporter that it's the site's fault front adding the opt-out. Especially since (per @vmpstr ) it's not even a hard opt-out, it's just a hint to the heuristic.

Given all this, I'm still in favor of only two states. I think the third state is likely to create more problems than it solves.

[chrishtr]

On the substance of the argument: if the three values select one of three browser-specific heuristics, which may change over time even within a single browser, then I don't think authors will be able to use it correctly. If anything, this would exacerbate @smfr 's worry that the attribute will start getting used in a cargo cult manner, so it will stop being useful signal.

"sync" means "I prefer to have it all display together, please do that". That's not a heuristic, right? Regarding @vmpstr's comment about it not being required, that language is proposed to give the user agent the right to override in situations where the UX is really bad, i.e. an intervention, which is not normal behavior.

"auto" leaves it up to the browser and is the default. This is the heuristic which would evolve over time as
we optimize for existing content.

"async" means "I prefer to display content fast, and not block on this image"; it's expected the browsers
will implement this fully async (on a background thread, with invalidation at the end), so it's not a heuristic either. (I assume something like this is what WebKit has implemented? I'm actually curious, don't really know.)

@chrishtr It's true that any heuristic has false positives, and you have to tune false positives vs false negatives. But I don't think an explicit opt out makes it ok to have more false positives. So it doesn't materially change this tradeoff. I can't imagine resolving a bug that says a heuristic is too aggressive by telling the reporter that it's the site's fault front adding the opt-out.

I don't see why we shouldn't resolve a bug sometimes as "please use sync if you really want the
image to display at the same time as other content". Sometimes developers have to take action
to keep their sites working well in new browsers.

If we could re-design images fully, I think probably it would have been a two-state of sync or async. But since "auto" is the existing web, and some browsers implement sync only currently (Chrome and Safari shipping versions) or some async-like heuristic (Edge and Firefox), we have to do something to move to a future state where jank is reduced. I think there are developer use-cases for sync and async, and also "don't know/don't care". In any case, legacy content exists and it would be a real shame to be wedged into a state where we can't do anything about it without breaking developer use-cases without simple
workarounds.

[smfr]

Sometimes developers have to take action to keep their sites working well in new browsers.

I think we try really hard to avoid this; can you cite another example where the user experience on legacy web content degrades as browsers change?

[chrishtr]

I think we try really hard to avoid this; can you cite another example where the user experience on legacy web content degrades as browsers change?

Blink changes behavior regularly to improve interop between browsers and remove footguns through
the Intent process. We do so only when the percentage of broken content appears to be very small and
the severity of the break is also small (though there are exceptions to even this for changes we deem
critical, such as removal of window.showModalDialog). We regularly change DOM APIs in ways that are known to cause JS errors on a small number of sites.

We also have been making changes explicitly targeted at reducing footguns and jank. At times these
are phrased as an "Intervention" where we believe that improving UX for users outweighs fidelity
to the current status of an API. A good recent example related to scrolling is forcing event listeners
that are not explicitly marked as non-passive to be considered passive. See here for more details:
https://www.chromestatus.com/features#passive. All shipped changes are also listed under /features
there.

In the case of image decoding, the changes we would make would only make images blink somewhat
more often on some existing sites, but otherwise the site would be perfectly usable (and faster).

[othermaciej]

Have the other Blink changes you mention required or included an explicit override for web content?

[chrishtr]

Have the other Blink changes you mention required or included an explicit override for web content?

Yes, in cases where there is an identified use case for the override, we have usually provided it.
Passive event listeners are an example, where you can force it to not be passive by setting passive: false.
The change to default-passive ended up being blocked on adding the ability to opt-out. Same goes for
the changes we made on mobile to treat pinch-zoom as not changing layout - it was blocked on adding
the viewport API due to developer pushback from valid use-cases.

Closer to the use-case being discussed in this issue, there was another change we made earlier this
year regarding when to re-raster composited content on scale change. In that case we did not add an
ability to get the entire old behavior, and got quite a bit of pushback on that from developers who did not
like the new behavior. See:

https://greensock.com/will-change

In that case we made the change and did not revert it despite the negative feedback, because it resolved
a lot of complaints about blurry composited content.

That's a good example of developers really caring about exact behavior of images. In that case
we were able to eventually discover and make a tweak to our heuristic that mitigated the issue
in some (perhaps almost all, hard to say) situations, but the complaining developers really really wanted
a way to control the behavior.

[chrishtr]

@othermaciej @smfr Given the above, would you be ok with a two-state, specified as follows?

---

Images will support an "async" attribute, which can take on one of two values:

"on": The developer prefers performance over visual atomicity.
"off": The developer prefers visual atomicity over performance.

The User Agent should make an effort to respect the developer's preference. [Side note: this is intentionally weak language, allowing for the User Agent not to jank or to apply a heuristic if deemed
important.]
If the "async" attribute is missing, the User Agent should do what it feels is best for the user.

---

With the above definitions, WebKit need not change any of the current implementation interpreting
the attribute's semantics, because WebKit's current default behavior is to prefer visual atomicity, and
nothing in the above spec requires treating absence of the attribute as different from "off".

Webkit would need to change parsing of the attribute to require "async=on" instead of just "async".
We could even consider specifying "async" to be an alias of "async=on", if that seems useful, which
would mean no changes to WebKit are required at all.

With this spec in place, Blink anticipates moving its implementation over time closer to "on" for images which don't specify "async", which should reduce jank on the web.

[chrishtr]

FYI we have filed an intent to implement for the async attribute:

https://groups.google.com/a/chromium.org/forum/#!topic/blink-dev/9i6wgXv7c7c

[othermaciej]

@chrishtr It seems like your proposal is actually tri-state, not two-state. It's just that one of the states is absence of the attribute. I think we're still not on board with tri-state. Maybe we need to discuss this in some appropriate in-person venue if we can't come to agreement in issue comments.

[chrishtr]

@othermaciej @smfr Fair enough that we should talk in person at this point. I will be at TPAC
in two weeks for the CSS WG days, hopefully I will see you there.

In the meantime, would just like to re-iterate also the motivation for a tri-state or its sort-of equivalent:
It's important in my view to move to a future where scrolling and other compositor animations are not slowed down by image decode by default. (Not sure if we agree on that point, given the comments above about not breaking existing sites.) In any case, if we want to achieve this goal in the near future, that means that unmodified existing websites must somehow start avoiding jank due to decode. The only ways to get there that I can see are (a) hard change of behavior where some flashing occurs which didn't before, or (b) find some heuristic that avoids affecting almost all sites.

My concern with (b) is that it seems hard to find such a heuristic that is not brittle or has poor coverage, and is understandable to developers. This leaves (a). But if (a) is pursued, how do developers fix their site after it starts "breaking" due to the change? Hence the tri-state (default/opt-in/opt-out).

[feross]

Just want to chime in as a web content author: I prefer @chrishtr's proposal in https://github.com/whatwg/html/issues/1920#issuecomment-334821067

Additionally, for consistency with <script async> I prefer that the async boolean attribute be aliased to async="on".

[RByers]

I agree with Chris's suggested approach, but I acknowledge that this effort
at "interventions" to try to gradually fix the defaults of the web is
contentious with chromium often taking the more aggressive stance than
other engines. I would not expect any of the other engines to necessarily
be willing to take the web compat risks for uncertain long term benefit,
but I do think we all want the same outcome in the long-run (eg. a web that
is usually fast).

What we've done in other cases (eg. the passive touch listener
intervention) is to formally specify only the 100% deterministic behavior
(but in a way that leaves open the door for a 3rd state), and then
"incubate" the heuristic-based intervention (at
https://github.com/wicg/interventions/). During incubation we iterate with
experiments and shipping in chrome without always achieving consensus on
the precise spec text (eg. unlanded spec PRs). We then learn from
experience and seek to collect the data necessary to get full standards
consensus before resolving the open "intervention" issue. Though not
without bumps, I do think this process has worked pretty well in practice
(eg. I believe WebKit has now adopted our weird passive-unspecified
behavior for touch listeners, and getting specs and I'm optimistic about
getting specs and all implementations re-aligned here).

Applying that pattern to async images, would, I think look something like:

1. agree to specify only 2 states - async=on (aka. "async") and async=off
   (aka default, no async attribute)
2. open an intervention issue for violating the spec by treating a missing
   async attribute differently from async=off
3. iterate on experiments and shipping behavior in chromium, capturing hard
   data on the tradeoffs
4. use this data to eventually achieve consensus on the right behavior for
   unspecified, update the spec/tests (if needed) and resolve the open
   intervention issue.

In this approach, only #1 needs consensus in this forum (whatwg/html) at
the moment, though we of course would welcome vigorous public debate in the
right forum for 2 (wicg/interventions) and 3 (blink intent to ship thread).

P.S. sorry for the lack of formatting and links - flying in for TPAC right
now with bad in-air WiFi. I'm happy to discuss more F2F at TPAC if people
are interested.

[smfr]

I would like to hear from Mozilla/Edge, whose decoding behavior is more async in general, to see if they are willing to implement sync decoding for async="off"

[othermaciej]

@RByers

1. agree to specify only 2 states - async=on (aka. "async") and async=off
   (aka default, no async attribute)
2. open an intervention issue for violating the spec by treating a missing
   async attribute differently from async=off

That's still three states, you're just failing to specify one of them. Also, the standard approach to HTML boolean attributes is that only presence or absence matters, and the value is ignored. Using explicit on/off values creates a wart on the platform if it turns out that we only need two states.

Finally, I don't think Apple is willing to make lack of attribute force synchronous decoding. We already have some heuristic async cases and we don't want to take them out. So async=off and lack of attribute wouldn't really mean the same thing, even if it was specified that way. I believe it's a WHATWG principle that specs should not be fiction.

To be clear, it's not the heuristic approach for unspecified that we object to, it's a mode that would force synchronous in all cases.

[chrishtr]

That's still three states, you're just failing to specify one of them. Also, the standard approach to HTML boolean attributes is that only presence or absence matters, and the value is ignored. Using explicit on/off values creates a wart on the platform if it turns out that we only need two states.

I think that ideally all images would be async by default and the developer would opt into sync. This
makes the web fast by default. However, the web is not that way today, and user agents also
lack interop in this respect.

If image decoding on the web was async by default, then actually async=off would be the only
keyword needed (not async=on)

Finally, I don't think Apple is willing to make lack of attribute force synchronous decoding. We already have some heuristic async cases and we don't want to take them out. So async=off and lack of attribute wouldn't really mean the same thing, even if it was specified that way. I believe it's a WHATWG principle that specs should not be fiction.

The proposal is not to make absence of the attribute force synchronous decoding. It's that the browser can do what it thinks is best in this case. All of the current heuristic async cases in Safari can be left as they are. Chromium would likely start making images in the default case more and more async by default over time, to address the exact performance issues you are (rightly) concerned about. Doing so would
also increase interop with how Firefox and Edge already behave.

But if we are going to do that, there needs to be a developer recourse in case the heuristics break, hence the explicit developer preference of async=off.

To be clear, it's not the heuristic approach for unspecified that we object to, it's a mode that would force synchronous in all cases.

This is necessary as a developer recourse when the heuristic/intervention fails. But as mentioned,
the spec can say "should" rather than "must", so that a user agent can still apply some async heuristics if it believes that is in the interest of the user (over the developer), or no one would notice. I envision possible cases being:

• The async=off image is not on-screen right now (user probably won't notice)
• The async=off images is really large (too much jank)
• The device has a very slow CPU (too much jank)
• The image onload event happened since the last frame, and the image has already displayed fallback content (further delaying display will not generate an additional flash)

[othermaciej]

That ends up with one reliable deterministic mode, and two different heuristic modes. Doesn't seem like a good design to me.

I don't think developer recourse is required because we shouldn't make heuristics that cause too much breakage. If there is developer recourse, it should not be recourse that forces jank. It should be to a solution that avoids both jank and flashing. Image.decode() already provides a solution that avoids both jank and flashing. If there are edge cases that it doesn't capture, we should extend it. I don't think that any purely markup-based solution can fully avoid both jank and flashing.

[othermaciej]

Also I kind of feel like we're talking in circles. All the different proposals for tri-state or supposedly two-state are exactly the same as the original tri-state proposal if you dig into it. And then we end up arguing the exact same points.

We're comparing these two distinct proposals:

1. Two State: async boolean attribute
   
   
   
   1. When async attribute is present, force async decoding
   
   
   2. When async attribute is absent, use heuristics
   
   
   3. No way to force or "should" level encourage sync
   
   
2. Tri State: async=on, async=off and missing attribute all do potentially different things
   
   
   
   1. async=on forces async
   
   
   2. When async attribute is absent, use heuristics
   
   
   3. async=off forces sync, or maybe is a different heuristic than missing attribute which tries a little harder to be sync
   
   

Let's collect the pros and cons for these two different options.

[chrishtr]

I agree that we're talking in circles. Here is a list of pros and cons.

Two State+image.decode opt-out pros:

• Trivial opt-in for developers in all scenarios
• Avoids new API to force jank
• Fewer attribute states
• Avoids breaking content

Two State+image.decode opt-out cons:

• Requires extending image.decode to more cases in the future
• Not clear if image.decode can cover all cases (*)

Tri State pros:

• Trivial opt-out and opt-in for developers in all scenarios
• Does not require extending image.decode
• Improves interop, by allowing Chrome to more safely move to async by default and therefore match most/all Edge/Firefox behavior
• Reduces jank on existing web, by allowing Chrome to more safely move to async by default

Tri State cons:

• Potential for two unspecified, per-browser heuristics instead of one.
• Introduces a new API which forces jank

---

(*) Reasons why I don't think this approach looks promising:

• Developers can't be expected to know about behavior of image decode caches, or ways in which images may or may not appear on the screen due to DOM updates, so they don't know when to decode.
• Don't see a way to make it work for CSS images, such as background-image.

[smfr]

TPAC discussion: we agreed to have tri-state, with a "decode" attribute with values "sync" and "async", to match the naming of decode() (and future "decode-sync" and "decode-async" arguments to the CSS image() function).

Reasoning for tri-state:

• for Edge/Gecko,<img async> is a no-op and there's no declarative way to get non-flashy behavior. Hard to extend thedecode() API for the case where the UA has thrown away backing store and has to re-decode, e.g. on scrolling.
• It's OK for<img sync> to be non-sync in Blink/WebKit in some cases, since where we do async now (offscreen) is mostly not user- and not author visible.

@ChumpChief, is this OK for Edge?

[RByers]

For the record, @chrishtr, @vmpstr and I were present for chromium and supported this conclusion.

@dbaron @dholbert I believe the above notes capture what you said too, let us know if we missed anything.

[chrishtr]

Partial IRC log from discussion below.

[14:05] == chrishtr [[email protected]] has joined #images
[14:05] == rbyers [[email protected]] has joined #images
[14:05] == dholbert [[email protected]] has joined #images
[14:05] <@smfr> https://github.com/whatwg/html/issues/1920
[14:07]  github: https://github.com/whatwg/html/issues/1920
[14:09] == dbaron [[email protected]] has joined #images
[14:10]  dbaron: on low performance devices can see why decode time is a bigger problem
[14:11]  dbaron: might make sense for gecko to have a way for developers to avoid flashing
[14:11]  rbyers: on some threads heard reports of bugs filed against gecko for flashing
[14:12]  chrishtr: edge team has also seen bugs about flashing
[14:12] == smfr [[email protected]] has quit [smfr]
[14:12]  smfr: is edge team interested in a way for developers to specify avoid-flashing?
[14:12] == smfr [[email protected]] has joined #images
[14:12]  Ok we'll use IRC more
[14:12]  damn conference wifi
[14:13]  Rossen is here, looking into the Edge bugs
[14:13]  Rossen: know we have signals, there have been discussions of image.decode
[14:13]  ... not aware of what our current positions is.  Trying to get Matt Rakow to say now.
[14:14]  ... by signals, I mean bugs
[14:14]  signals === bug reports
[14:14] == vmpstr [[email protected]] has joined #images
[14:14]  dholbert: I'm enticed by two-state proposal with a user-override of some sort
[14:14]  dbaron: the decode API kind of gives you the third choice
[14:15]  (just waiting on Rossen's conversation with Matt)
[14:15]  chrishtr: decode API also doesn't cover all cases (some discussion about this in issue 1920)
[14:15]  chrishtr: and harder than a way with no script
[14:15]  smfr: In GitHub there are two proposals
[14:16]  .. 2-state vs. 3-state
[14:16]  .. 2-state is easy to opt-in to async behavior
[14:16]  .. important to webkit, no new way for authors to force the worst performing behavior
[14:16]  .. we think it's pretty web compatible because we're not going to change any existing content
[14:17]  .. downside is that decode API is only recourse for other browsers that will do async by default
[14:17]  dbaron: It feels like 99% of images on the web are fine with being async
[14:17]  .. the % that will cause flash as async is very low
[14:17]  smfr: We found that is not the case
[14:17]  .. when we enabled it by default, we saw flashing on a lot of common sites
[14:18]  .. the author was dynamically changing content and expecting they could replace one image with another and not see any intermediate state
[14:18]  dbaron: Still thing those cases are a pretty small percentages
[14:18]  smfr: All cases we say where author was modifying script.  Agree most images are static images and those are OK.
[14:19]  .. in some of those cases, it's OK if they flash in - eg. replacing placeholder with image
[14:20]  rbyers: Simon isn't not wanting an 'off' option while also never changing default inconsistent?
[14:20]  smfr: Yes seems like it
[14:20]  .. worried about cargo-culting
[14:20]  smfr: Let's talk about 3-state
[14:20]  .. async=off would fix flashing in your browsers
[14:21]  .. we wouldn't want to do sync when we knew there was no point
[14:21]  .. (eg. not visible)
[14:21]  .. so we'd want this to still be a hint
[14:21]  rossen: Is the difference even detectable?
[14:21]  smfr: Not by script, but it could be user-visible
[14:22]  dbaron: What % of images have load event listener
[14:22]  .. then you know it's not detectable
[14:22]  smfr: we were going to assume async was OK when no load listener, but there were a bunch of cases that didn't work
[14:22]  .. can't remember which
[14:23]  smfr: maceij objected to default and 'off' both being heuristics
[14:23]  dholbert: For FF default will not be the worst behavior
[14:23]  chrishtr: chrome wants to move to more async by default, for perf and interop w/edge & firefox
[14:23]  rbyers: chromium wants to try to be aggressive about eventually changing defaults
[14:23] * rbyers smiles
[14:24]  smfr: are you OK with an attribute that lets developers opt-in to a more sync behavior
[14:24]  dbaron: I'd support async=off
[14:24]  .. google images
[14:27]  vmpstr: decode api works well when the image is first added to the page, but not so well when the image is scrolled into view
[14:27]  q+
[14:28]  image.decod doesn't work for CSS images
[14:28]  go ahead chris
[14:28]  image.decode
[14:33]  image search has a low res image when you click on one image
[14:33]  which is then replaced by a high res image
[14:35]  tri-state helps with interop, simple approach to no-flash use case
[14:36]  interop is not good right now
[14:37]  FYI decode API doesn't work for two images, second decode of in-DOM image
[14:39]  second decode happens if an image falls out of the decoded images cache
[14:39]  doesn't work for two images because the promise doesn't say what happens if there are two promises
[14:39]  ie, you can't make two images appear in the same frame with decode api
[14:39]  ..and css images :)
[14:40]  maybe it can be extended, but that has implementation and developer confusion costs
[14:49]  In the brief discussion in the CSSWG on Monday, we discussed CSS images like background-images
[14:49]  async could be an argument to this new paint() function instead of url() that tab mentioned
[14:49]  just a comment, I am not an expert on this aspect of CSS
[14:50]  image function, sorry
[14:50]  image() function
[14:50]  s/paint/image :)
[14:50]  you mean the image() function, right?
[14:50]  yes
[14:50]  by bad
[14:50]  my bad
[14:51]  decode=sync, decode=async sgtm as well
[14:53]  great and on the whiteboard is
[14:53]  +1 to what vmpstr staid
[14:53]  said
[14:53]  background-image: image(foo.png, decode-async)
[14:53]  background-image: image(foo.png, decode-sync)
[14:53]  sgtm

[chrishtr]

Some of the IRC attribution got cut off in the previous comment, trying again:

[14:05] == chrishtr [[email protected]] has joined #images
[14:05] == rbyers [[email protected]] has joined #images
[14:05] == dholbert [[email protected]] has joined #images
[14:05] @smfr> https://github.com/whatwg/html/issues/1920
[14:07] chrishtr> github: https://github.com/whatwg/html/issues/1920
[14:09] == dbaron [[email protected]] has joined #images
[14:10] chrishtr> dbaron: on low performance devices can see why decode time is a bigger problem
[14:11] chrishtr> dbaron: might make sense for gecko to have a way for developers to avoid flashing
[14:11] chrishtr> rbyers: on some threads heard reports of bugs filed against gecko for flashing
[14:12] chrishtr> chrishtr: edge team has also seen bugs about flashing
[14:12] == smfr [[email protected]] has quit [smfr]
[14:12] chrishtr> smfr: is edge team interested in a way for developers to specify avoid-flashing?
[14:12] == smfr [[email protected]] has joined #images
[14:12] rbyers> Ok we'll use IRC more
[14:12] rbyers> damn conference wifi
[14:13] rbyers> Rossen is here, looking into the Edge bugs
[14:13] rbyers> Rossen: know we have signals, there have been discussions of image.decode
[14:13] rbyers> ... not aware of what our current positions is.  Trying to get Matt Rakow to say now.
[14:14] rbyers> ... by signals, I mean bugs
[14:14] chrishtr> signals === bug reports
[14:14] == vmpstr [[email protected]] has joined #images
[14:14] rbyers> dholbert: I'm enticed by two-state proposal with a user-override of some sort
[14:14] rbyers> dbaron: the decode API kind of gives you the third choice
[14:15] rbyers> (just waiting on Rossen's conversation with Matt)
[14:15] chrishtr> chrishtr: decode API also doesn't cover all cases (some discussion about this in issue 1920)
[14:15] chrishtr> chrishtr: and harder than a way with no script
[14:15] rbyers> smfr: In GitHub there are two proposals
[14:16] rbyers> .. 2-state vs. 3-state
[14:16] rbyers> .. 2-state is easy to opt-in to async behavior
[14:16] rbyers> .. important to webkit, no new way for authors to force the worst performing behavior
[14:16] rbyers> .. we think it's pretty web compatible because we're not going to change any existing content
[14:17] rbyers> .. downside is that decode API is only recourse for other browsers that will do async by default
[14:17] rbyers> dbaron: It feels like 99% of images on the web are fine with being async
[14:17] rbyers> .. the % that will cause flash as async is very low
[14:17] rbyers> smfr: We found that is not the case
[14:17] rbyers> .. when we enabled it by default, we saw flashing on a lot of common sites
[14:18] rbyers> .. the author was dynamically changing content and expecting they could replace one image with another and not see any intermediate state
[14:18] rbyers> dbaron: Still thing those cases are a pretty small percentages
[14:18] rbyers> smfr: All cases we say where author was modifying script.  Agree most images are static images and those are OK.
[14:19] rbyers> .. in some of those cases, it's OK if they flash in - eg. replacing placeholder with image
[14:20] rbyers> rbyers: Simon isn't not wanting an 'off' option while also never changing default inconsistent?
[14:20] rbyers> smfr: Yes seems like it
[14:20] rbyers> .. worried about cargo-culting
[14:20] rbyers> smfr: Let's talk about 3-state
[14:20] rbyers> .. async=off would fix flashing in your browsers
[14:21] rbyers> .. we wouldn't want to do sync when we knew there was no point
[14:21] rbyers> .. (eg. not visible)
[14:21] rbyers> .. so we'd want this to still be a hint
[14:21] rbyers> rossen: Is the difference even detectable?
[14:21] rbyers> smfr: Not by script, but it could be user-visible
[14:22] rbyers> dbaron: What % of images have load event listener
[14:22] rbyers> .. then you know it's not detectable
[14:22] rbyers> smfr: we were going to assume async was OK when no load listener, but there were a bunch of cases that didn't work
[14:22] rbyers> .. can't remember which
[14:23] rbyers> smfr: maceij objected to default and 'off' both being heuristics
[14:23] rbyers> dholbert: For FF default will not be the worst behavior
[14:23] chrishtr> chrishtr: chrome wants to move to more async by default, for perf and interop w/edge & firefox
[14:23]  rbyers: chromium wants to try to be aggressive about eventually changing defaults
[14:23] * rbyers smiles
[14:24] rbyers> smfr: are you OK with an attribute that lets developers opt-in to a more sync behavior
[14:24] rbyers> dbaron: I'd support async=off
[14:24] rbyers> .. google images
[14:27] vmpstr> vmpstr: decode api works well when the image is first added to the page, but not so well when the image is scrolled into view
[14:27] chrishtr> q+
[14:28] chrishtr> image.decod doesn't work for CSS images
[14:28] rbyers> go ahead chris
[14:28] chrishtr> image.decode
[14:33] vmpstr> image search has a low res image when you click on one image
[14:33] vmpstr> which is then replaced by a high res image
[14:35] chrishtr> tri-state helps with interop, simple approach to no-flash use case
[14:36] chrishtr> interop is not good right now
[14:37] chrishtr> FYI decode API doesn't work for two images, second decode of in-DOM image
[14:39] chrishtr> second decode happens if an image falls out of the decoded images cache
[14:39] chrishtr> doesn't work for two images because the promise doesn't say what happens if there are two promises
[14:39] vmpstr> ie, you can't make two images appear in the same frame with decode api
[14:39] chrishtr> ..and css images :)
[14:40] chrishtr> maybe it can be extended, but that has implementation and developer confusion costs
[14:49] chrishtr> In the brief discussion in the CSSWG on Monday, we discussed CSS images like background-images
[14:49] chrishtr> async could be an argument to this new paint() function instead of url() that tab mentiond
[14:49] chrishtr> just a comment, I am not an expert on this aspect of CSS
[14:50] chrishtr> image function, sorry
[14:50] chrishtr> image() function
[14:50] chrishtr> s/paint/image :)
[14:50] rbyers> you mean the image() function, right?
[14:50] chrishtr> yes
[14:50] chrishtr> by bad
[14:50] chrishtr> my bad
[14:51] vmpstr> decode=sync, decode=async sgtm as well
[14:53] rbyers> great and on the whiteboard is
[14:53] chrishtr> +1 to what vmpstr staid
[14:53] chrishtr> said
[14:53] rbyers> background-image: image(foo.png, decode-async)
[14:53] rbyers> background-image: image(foo.png, decode-sync)
[14:53] chrishtr> sgtm
[14:58] chrishtr>

[domenic]

The above says tri-state, but only gives two states, "sync" and "async". What is the third state?

Perhaps a related question, what are the invalid value default and missing value default?

[smfr]

Third state is no attribute (i.e. UA default behavior).

[annevk]

It would be easier if the third state also had a value, given that we also have to define an IDL attribute.

[smfr]

Does that imply that you want decode="auto" to behave like no attribute?

[vmpstr]

I'm starting to draft a proposal, and I forgot that we already have img.decode() as a function. I think we should name the attribute something different (to disambiguate what img.decode refers to). I propose decodingmode. Thoughts?

[annevk]

@smfr yeah, or "default". We need something for the IDL attribute, which should just be an enumeration. @vmpstr we can have different names for the content and IDL attribute, even if a bit unusual.

[domenic]

I think matching IDL and content attribute names is a good idea. Bikeshed: decodeMode?

[othermaciej]

Ideally it would be something that feels meaningful and not too wordy in markup:

\ - not bad but would conflict in IDL
\ - icky to read, long
\ - icky to read, even longer
\ - weird to have mixed case in HTML
\ - maybe this one?

myImg.decoding = "async" also makes sense, though one might expect it to affect what img.decode() does, but it doesn't. All variants based on decode have this issue though.

[vmpstr]

"decoding" sounds good to me! I've updated the pull request with that and other review comments.

[chrishtr]

See also https://github.com/w3c/csswg-drafts/issues/1984 for what kind of fallback content
should be shown when an image is not displayed due to async decode.

[domenic]

@othermaciej @smfr can you confirm you are OK with the name decoding? Otherwise I think we're good to go over in #3221.

[smfr]

I think "decoding" is fine.

[othermaciej]

Agree with @smfr

[dbaron]

So I was thinking about this a little more, and considering the idea that a bunch of the use cases for decoding=sync don't actually require sync decoding that blocks the main thread -- they just require an event that fires when decoding is complete, whether that's the load event or the result of the decode() promise. I wonder if either decoding=sync should (instead of forcing sync decoding) force decoding to happen before the load event fires, or whether there should be another decoding attribute value that works that way?

[smfr]

You don't want to have to decode before firing the load event, because decoding involves allocating memory for the image backing store, and at that time you don't know if that image will ever be painted.

[othermaciej]

a bunch of the use cases for decoding=sync don't actually require sync decoding that blocks the main thread -- they just require an event that fires when decoding is complete, whether that's the load event or the result of the decode() promise.

Those use cases should use image.decode() instead of decoding=sync. Do we need a second way to do what image.decode() does?

[vmpstr]

decoding=sync don't actually require sync decoding that blocks the main thread -- they just require an event that fires when decoding is complete

I'm not sure that there is even a requirement for the event to fire when decoding is complete. The way the proposal specs this, there is no requirement for any javascript code to be able to determine the effect of different values of "decoding" attribute (other than inspecting the attribute itself of course).

I agree with @othermaciej that I think the cases where the event is desired are best suited for the img.decode() api.

[aghassemi]

@vmpstr @smfr @othermaciej One common use-case is hiding a placeholder (e.g. small, in-lined pixalated version of an image) or loading indicator as soon as image is painted.

If we use decoding=async and use load event to hide the placeholder, we may hide it too early.
If we use image.decode() before appending the image instead of attr, we lose progressive decoding.

An event to know decoding is finished combined with decoding=async seems to be needed.

@smfr, as a workaround, I am curious if we can still use image.decode() purely as a signal to know when decoding is done and combine it with the decoding=async attribute. Can this cause double work?

[smfr]

An event to know decoding is finished combined with decoding=async seems to be needed.

But the browser doesn't want to decode images that it doesn't know are going to get painted. This is extra work and possibly massive memory use.

@smfr, as a workaround, I am curious if we can still use image.decode() purely as a signal to know when decoding is done and combine it with the decoding=async attribute. Can this cause double work?

image.decode() is for forcing an (async) decode, not just for detecting when it's done. I don't think using the two things together makes much sense.

[aghassemi]

@smfr I didn't have the expectation that it will fire for every image. Maybe an event name like painted is more suitable than decoded to convey that behavior. This will still be very useful to hide partial loading overlays or background placeholders perfectly in sync with image painting.

[vmpstr]

The decision to paint (rasterize) content can happen pretty far down the pipeline and independently of the main thread. I think bubbling up an event whenever we decode an image is a lot of overhead, especially if there are only a few cases where it is useful. It's also a bit unclear if the event should be fired for subsequent decodes that may happen on the same image.

In general, the situation you describe would be best suited for (intended use of) the decode api. That is, show a low res image, call a decode() on a full res image and append/replace the content in the promise resolution. You are right that since this implies the load event, we'd lose the progressive decoding.

Just thinking out loud, but I don't readily see an elegant way to get all three of the following:

1. async behavior
2. progressive decoding
3. a signal that the final image is decoded

1+2 sounds like it could be handled by decoding=async
2+3 is implied by not specifying anything (ie decoding=auto or maybe even decoding=sync)
1+3 is the decode() api

[annevk]

Also thinking out load: I guess in the future we could add something to decode() that roughly means "take your time"? So you can invoke it early, but the browser knows you're just using the resolved promise as a signal and is therefore not in a hurry to decode.

[aghassemi]

@vmpstr curious why bubbling up an asyc flag from down the stack is expensive? I assume events like playing for video come from deep parts of the stack as well.

For now we are doing 1+2 and will assess if the transition between placeholder/image would flicker. /cc @cramforce

@annevk that's what I sort of assumed decode() already does since I was picturing it as being just the imperative version of the decoding=async attribute.

[vmpstr]

The difficulty with something like a decoded attribute is that its value may change at times that I would say are not interesting. For example, in Chromium images are sometimes decoded outside of the viewport based on some heuristics that suggests that we may soon need to rasterize content that uses these images.

However, due to user interactions we may not actually use the image that we decoded. Furthermore, the decode's memory backing may be discarded meaning the image is no longer decoded, but doing the query for this information is done only if we actually need the image to rasterize. Once we use an image to rasterize content, it may also be evicted since its memory backing is no longer necessary to present content.

If I'm understanding this correctly, to implement a decoded attribute, each event that causes a decode or eviction would have to come with corresponding communication with the main thread so that javascript can observe the change. (and this would have to happen for every image?)

So perhaps saying that this is a lot of overhead was premature. I wouldn't necessarily claim that this is either cheap or expensive; I'm simply skeptical that the value this attribute brings warrants this complexity. If there's an interest in this type of attribute, maybe we can split off the discussion into a separate issue?

[cramforce]

I think a decode event is too coupled to the implementation. What would be interesting is an event for "I'm about to or I could without extra decoding work paint pixels for this image. Either the full image or the initial progressive render."

[othermaciej]

If the requirement is to have a signal when the image is decoded enough to paint, even if not fully decoded, maybe that could be a parameter to image.decode(), which would then start decoding as usual, but trigger the promise at minimally ready to paint time instead of fully decoded time.

[othermaciej]

(I don't think you want to trigger based on the image actually being painted, since it might not even be inserted into the DOM until ready to paint, and since layout engines might be smart enough not to paint it at all if it's fully covered by an opaque placeholder.)

[chrishtr]

Hi all,

I propose to consider this issue complete for the purpose of committing the spec update in
https://github.com/whatwg/html/pull/3221. The discussion happening now is good, but probably
best in a new issue to separate concerns.

Any objections?

[annevk]

I also lean in that direction. We can have a new issue to discuss a signal of sorts for the scenarios discussed above. I suggest we merge #3221 (and close this issue) somewhere Tuesday to give folks a few more days to raise concerns with this line of action.

[sideshowbarker]

Added https://wpdev.uservoice.com/forums/257854-microsoft-edge-developer/suggestions/32541025-img-decoding-attribute