Html: Add "auto sizes" for lazy-loaded images

[zcorpan]

From https://github.com/whatwg/html/pull/3752#issuecomment-490386837

@r00dY I believe what you're asking for is essentially the data-sizes="auto" feature in lazySizes. I think this has been discussed before (maybe somewhere in https://github.com/ResponsiveImagesCG/picture-element/issues ), though I can't find it now. It was blocked on native lazy loading, but that is being introduced in this PR, so you're right that it is good timing to revisit "auto sizes".

The idea with sizes is:

• browsers need to start loading critical images as early as possible; waiting for external CSS would regress page loading time
• the browser is in a better position to make a decision on which image to load, given a list of images and their dimensions (srcset) + layout size of the image (sizes). https://ericportis.com/posts/2014/srcset-sizes/

The layout size information in sizes is redundant with layout information in CSS, but for images that are critical to initial paint it's necessary to have the information in the HTML.

But for lazy-loaded images, CSS will usually be available before the image load begins. The browser could take the actual width of the image as specified in CSS, and use that as if it was the image's sizes.

A syntax proposal is sizes="auto", or omitting the sizes attribute. Only allow this on loading=lazy images.

Issues:

• What should happen for loading=eager images? Use 100vw (like today)?
• What should happen for loading=auto images? Depend on whether the UA decides to load it eagerly or lazily?

[eeeps]

What should happen for loading=eager images? Use 100vw (like today)?

eager is how images load now, so: yeah, I’d say 100vw.

I've been planning to research the distribution of actual vs. sizes sizes for, like, years. Perhaps one day this research will be done and we can have a conversation about changing the default. But 100vw is reasonable and has years of developer education behind it.

What should happen for loading=auto images? Depend on whether the UA decides to load it eagerly or lazily?

I'd be all for re-speccing sizes so that it’s just a hint, rather than the law – and if the browser ever has directly-measurable information about the layout size of the image, it’s free to use it. Which would mean less predictability and more raciness for developers, but better outcomes for users.

@yoavweiss has told me, in informal conversation over the years, that the raciness that would result from this behavior would be Very Bad. Perhaps not platform compatible? Discuss.

---

Another issue: what to do with images whose CSS width is set to auto. Lazysizes explicitly forbids this. Fall back to 100vw?

@aFarkas probably has a lot of insight about this (and any other open questions or potential gotchas about making this a platform-native feature).

[aFarkas]

I'm extremely happy to see this discussed.

Here some thoughts:

• sizes="auto" should turn on the feature even if loading="lazy" is not set. It is up to the browser to either handle it exactly as loading="lazy" then or to introduce a new loading mechanism where the browser waits until all blocking styles are loaded and then load it eagerly. (loading="defer"???).
• Omitting sizes in combination of loading="lazy" should be handled as sizes="auto", but it should be still invalid markup.
• combining loading="eager" with sizes="auto": I have no feelings about the fallback (either handle it as loading="eager" with sizes="100vw" or defer loading and use sizes="auto"), it's clear that it's an invalid combination and it is wise to log also an error message in the dev console.

Other issues I see:

loading="defer"
I would like to see this feature also able to be implemented even if the browser does not fully support the lazy loading spec. To implement this feature you don't have to fully support all styles of loading. It's most about waiting until the layout is calculable.

Range request
I have no idea about the lazy loading spec. But as I understood you do some range request to determine the layout of images preemptively (before CSS is loaded)? If that is the case you should spec which image candidate is chosen to do this or to even defer the range request until CSS is loaded. Probably based on sizes="auto"? As long as all image candidates have the same host it shouldn't be that big of an issue.

Calculating layout width
It's not strongly about width: auto;. Any CSS technique that makes sure that the layout width is calculable is allowed. But for most developers it's more easier to understand width: auto is not allowed. So the issue here is the browser has to detect wether the width is calculable (not possible in JS) and if this is not the case define a fallback. A good fallback would be to walk up the dom tree and find the nearest ancestor who's width is calculable otherwise sizes="100vw is good to go. In any case I really would like to see a warning in the dev console if the image's width is not calculable. If it is also impossible to spec or to detect the width calculability a simple detection of width: auto should be ok I guess?

Calculating layout height in case of object-fit
As soon as the author uses object-fit the sizes="auto" calculation has also take the physical height of the image (or at least the physical aspect ratio) in to account as also the layout height of the image. lazySizes uses for detecting the physical height h descriptors which has to be used at least for one candidate in one srcset (== not all candidates have to have h descriptors) and uses a attribute data-aspectratio or width/height content attributes as fallback (even though the later is not useable for source and also has some implications on the layout (and not the physical aspect ratio) it was highly demanded and appreciated by developers). I would be happy to see h descriptors finally inside the spec (and you don't even have to specify now how the sizes attribute has to look like ;-)). But maybe you could - at least for the fallback behavior - solve this by doing the range request, although I would prefer a markup solution.

For determining the layout height all of the layout width applies the only thing here is that a fallback strategy to detect the right height (100vh or go up the dom tree) is not practical. The general fallback would be to simply use the auto calculated sizes width and do not take the aspect ratios (physical and layout) into account.

At the end this is not a pure and only height based calculation but an aspect ratio based calculation which simply calculates a corrected width size.

Post edits:

sizes="100px, auto"

I don't think that explicitly set sizes="100px" attributes should be seen as a hint and been overwritten by sizes="auto" if possible. We should give the developer the benefit of a doubt that he knows better why he is doing it. Maybe he wants to animate the image or he uses this image on another part of the page that will come soon and opted for one higher candidate instead of two candidates.

Respecting transform

It's also interesting wether you want to take CSS transform into account. While lazySizes detects visibility using getBoundingClientRect lazysizes uses offsetWidth/clientWidth to get the layout width.

[OliverJAsh]

for lazy-loaded images, CSS will usually be available before the image load begins

How do we know this will be guaranteed?

One fear I have about this is that developers will just be lazy and lazy load all of their images just so they can avoid having to define sizes, which of course would have a negative impact on performance. 🤔

[zcorpan]

We get to specify how this is to behave. If we want lazy images to not start loading until all pending stylesheets have loaded, then we require that (and write tests to verify).

Web developers can lazyload all images today if they want to (and use lazySizes's "auto sizes" feature). Has this been a problem?

[r00dY]

Wow, also very happy this is discussed and it's really great that you guys responded to my doubts about current state of standard. Thanks @zcorpan for creating a separate thread for this issue!

Idly wondering if loading=eager should use sizes only and blindly default to 100vw.

My experience shows that sizes parameter is usually not well maintained. Even if it's correct at first, then it's easy to forget to update it. Devs update CSS so that layout looks good but frequently forget about sizes. And I don't blame them, it's easy to miss one resolution (like tablet) and sizes is obviously a duplication of layout logic.

And question is: how often browser does know image width and height for loading=eager images? Doesn't that happen often when DOM is updated dynamically (not during page load?) like in many single page apps / PWAs? Or can't it happen in any other case because of various optimisations taking for example bad network conditions into account? Or can it use some heuristics to properly estimate image width quickly (without making full layout)?

What if sizes is not well maintained and browser downloaded 400px image and when after layout it becomes obvious that 1200px from srcset would be much better choice (and network is good)?

Instead of sizes=auto we could keep sizes as it was and add autosizes boolean attribute. Or incorporate it into sizes like @aFarkas suggested (sth like sizes="[sizes], auto").

autosizes=false would mean that browsers should blindly follow sizes - it's the option for guys who "know what they're doing" (as @aFarkas suggested we should give devs benefit of the doubt) and might want some untypical behaviour and still keep full control.

autosizes=true means that if <img> width is calculated, browser will always download image automatically if it's layout is available (no matter which loading mode, eager, auto or lazy). Arguably, it could also mean that browser can "correct" downloaded image (for better quality version) if sizes turned out to be wrong (leaving this to discuss).

And also, loading=auto is not a problem with this approach. Browsers can even make a decision to load image eagerly/lazy based on autosizes + sizes pair. If autosizes=true they might want to postpone image download a bit until layout is ready. But if sizes is there, they would be more inclined into treating image as good candidate for eager download.

To sum up, let's be honest, from software architecture/maintenance point of view, sizes is an awful practice, it's brutal duplication of layout logic which is already there in CSS / JS. It's hard to maintain and so easy to forget about. The only reason it exists is optimisation. I'd love to see the world where by default sizes is used as little as possible, no matter which loading mode we use.

@OliverJAsh

One fear I have about this is that developers will just be lazy and lazy load all of their images just so they can avoid having to define sizes, which of course would have a negative impact on performance. 🤔

Honestly, I think the opposite could happen. Setting all images to loading=lazy would definitely make page load UX worse, because some above-the-fold images would show up later, that's true. But on the other hand, all the images with sizes set wrong or not set at all would get fixed "automatically" and this would bring a lot of positive effect on performance. Taking into account the fact that sizes is extremely hard to maintain, I think the net effect could be actually pretty good.

[aFarkas]

Have no mind about a new attribute like autosizes/autosize only positive thing I see here is that you can easily feature detect it.

[zcorpan]

Honestly, I think the opposite could happen. Setting all images to loading=lazy would definitely make page load UX worse, because some above-the-fold images would show up later, that's true. But on the other hand, all the images with sizes set wrong or not set at all would get fixed "automatically" and this would bring a lot of positive effect on performance. Taking into account the fact that sizes is extremely hard to maintain, I think the net effect could be actually pretty good.

Browsers today eagerly load images, and even speculatively load images (from the speculative parser), in order to improve performance. When sizes was specified, it was because browsers were not willing to regress on this optimization when responsive images were used.

Are things different today? I could take a guess, but I would prefer to look at data. Are there any studies for this already?

An experiment could be to make a build of Chromium that lazy-loads all images, and compare performance metrics (preferably a metric that reflects when above the fold images are loaded) with a normal Chromium build.

[eeeps]

@zcorpan with the “Enable Lazy Loading” flag enabled in Chrome, it lazyloads all images by default.

@dougsillars did some perf testing of this mode nine months ago, and found Speed Index scores to be ~500ms slower with lazyloading, vs without it, for one test page: https://dougsillars.com/2018/09/21/chromes-experimental-image-lazy-loading-flag/

In some informal personal testing, I've found that preloading an oversized image easily wins, across a broad range of connection speeds/latencies, over waiting for layout, and then loading an appropriately sized image. Loading CSS & doing layout takes a comparative eternity, vs preloading.

[erkstruwe]

@eeeps

@zcorpan with the “Enable Lazy Loading” flag enabled in Chrome, it lazyloads all images by default.

Images without the loading attribute are currently only lazy-loaded on Chrome for Android with Data Saver mode enabled. For all other platforms and settings, images are loaded eagerly (https://css-tricks.com/a-deep-dive-into-native-lazy-loading-for-images-and-frames/#article-header-id-3). There are plans to extend this behavior to other platforms, though (last paragraph of https://docs.google.com/document/d/1e8ZbVyUwgIkQMvJma3kKUDg8UUkLRRdANStqKuOIvHg/), if performance tests show the expected benefits.

[eeeps]

@erkstruwe Oops! You're correct, & sorry for the misinformation.

[nhoizey]

Taking into account the fact that sizes is extremely hard to maintain, I think the net effect could be actually pretty good.

I agree sizes is sometimes painful, but maybe not "extremely hard"… 😅

If losing it means decreased performance, I would vote against it.

Being difficult on the DX side is not an excuse if the UX benefits from it. There are extensions to check validity of sizes values. Even if I don't use them, I remember there are some ways to keep CSS and HTML sizes synced. And of course, people can be professional and deal with it.

[zcorpan]

@annevk wrote in #3752

I also found #4654 which largely seems wontfix? I think we want developers to specify width/height for intrinsics.

I think there's a misunderstanding here. Yes, we want developers to specify width and height for intrinsic ratio. However, the CSS set a width that is different from what is given in the width attribute (this is normal for fluid images or when using different layouts for different viewport widths). So specifying width/height is good but doesn't remove the desire for auto sizes for lazy images.