Html: <img>: Have an autorotate="" attribute that honours orientation metadata in the image (e.g. EXIF)

[zcorpan]

https://www.w3.org/Bugs/Public/show_bug.cgi?id=25508#c26

It seems like there is not enough interest in trying to rotate all images by default. It's also not clear to me that it's a good idea to drop the CSS property, since one might want to rotate e.g. 'content: url(image)' images etc.

I suggest we do this:

• Keep the CSS property.
• Add autorotate to <img>
• Add img[autorotate] { image-orientation: from-image; } to Rendering.
• Clarify that image-orientation affects naturalWidth, drawImage(), etc.
• Require that https://html.spec.whatwg.org/multipage/browsers.html#read-media has an autorotate attribute set on the <img> (for top-level images and images in <iframe>s).

or, we could change the initial value of image-orientation from none to from-image (cc @tabatkins ), so all images are rotated "correctly" by default. Use counter data from 2016 found affected images to be ~0.00% of all images.

cc @dbaron

[eeeps]

Anecdotally, over the years, I've encountered multiple novice users publishing to platforms like email newsletters who are very confused as to why their images rotate as soon as they enter a web view.

e.g. https://stackoverflow.com/questions/35011966/why-my-image-is-rotate

From the (great!) answer there, it sounds like most default OS viewers respect Orientation. How many point-zero-zeroes do we need on that percentage-of-affected-images to change the initial value to from-image?

[zcorpan]

I filed an issue for the CSSWG since it affects the css-images spec.
https://github.com/w3c/csswg-drafts/issues/3799

[emilio]

Clarify that image-orientation affects naturalWidth, drawImage(), etc.

I don't think that's great. That means that naturalWidth and such need to update layout, which is a common perf footgun that in this case is easy to avoid. Would it be better to have a naturalOrientation on the image or such?

[zcorpan]

Why would it need to update layout?

[zcorpan]

Oh I see. It needs the latest styles in case image-orientation has changed.

[emilio]

Right.

[zcorpan]

Ok. So to be a bit more consistent, there could be orientation that takes CSS into account, and naturalOrientation that reflects just the EXIF Orientation.

[emilio]

I think it's not too hard to just use getComputedStyle() instead of exposing a new attribute, but...

[annevk]

Can we perhaps split the various issues here? I'm counting a lot of them and it's not entirely clear to me where each is at:

1. By default, do images ignore their intrinsic orientation or not? (Status quo is yes, but OP suggests changing that, right?)
2. Should we expose the intrinsic orientation to script?
3. Should we expose the used orientation to script?
4. Should the CSS-declared orientation impact anything beyond the rendering of the img element?
5. Should the intrinsic orientation impact anything beyond the rendering of the img element?
6. Should canvas APIs accept orientation as an input in some manner?
7. Should the img element get an attribute to override the intrinsic/CSS/used orientation or some such?

[noell]

Yeah ^^^^ good questions

https://bugs.chromium.org/p/chromium/issues/detail?id=158753#c56

[zcorpan]

@noell your recommendation in that comment includes

Change the initial value to be "from-image"

I agree. That is issue https://github.com/w3c/csswg-drafts/issues/3799

Because we're defaulting to rotation

• there's no need for an autorotate attribute on HTML <img>.

Also agree.

[noell]

Thank you, Simon. Still our proposals in crbug.com/158753#c56 is up-for-debate for browser vendors.

Second, hard data. 28-day aggregation April 17, 2019. Let 1 Trillion be defined as 1,000,000,000,000.

Blink.DecodedImage.Orientation, Google Chrome Stable Channel.

Top Left | X (elided) Trillion | 99.99%
Top Right | 2,392,540 | 00.00%
Bottom Right | 14,562,472 | 00.00%
Bottom Left | 6,927 | 00.00%
Left Top | 18,885 | 00.00%
Right Top | 178,935,821 | 00.00%
Right Bottom | 1,103 | 00.00%
Left Bottom | 33,551,146 | 00.00%

[noell]

^^^ that data is for JPEG btw, being the only image format in Chrome (at this time) that has a _Blink.DecodedImage.Orientation_ metric.

[zcorpan]

https://github.com/w3c/csswg-drafts/issues/3799 has been fixed.

So that's (1) in Anne's comment. Leaving this issue open for now, but we should probably split this up as @annevk suggested.

[smfr]

CSS WG resolved to have all images respect EXIF orientation by default, which would make an "autorotate" attribute unnecessary.

I think https://html.spec.whatwg.org/multipage/embedded-content.html#the-img-element needs some edits to clarify that 'width', 'height', 'naturalWidth' and 'naturalHeight' are the dimensions after applying any EXIF orientation.

https://html.spec.whatwg.org/multipage/canvas.html#drawing-images may also need some words about images drawn to canvas respecting EXIF by default.

[oriadam]

i agree,
supporting image-orientation:none seems to me the same as image-orientation:random which doesn't make any sense...

i'm worried that exposing any information about image orientation to css/js would result in ugly wrapper codes that try to fix orientations themselves in effort to just display the image correctly.
when we can just silently always display the image correctly.

html/css/js expect image objects, not files.
width should always represent the width of the image, not the width represented by the original file bytes.
imho image orientation should be honored by JPEG to image interpreter.

for example lets say SVG or Canvas team forgot to honor the orientation.
if any information about image orientation is exposed we will very soon find terrible codes similar to:

    let img = new Image(image_file);
    let naturalWidth = img.naturalWidth;
    if (/img.style['image-orientation']=='origin' && (img.naturalOrientation == 8 || img.naturalOrientation == 6)) {
        naturalWidth = img.naturalHeight;
        elem.style.transform.rotate = 
     ....

[noell]

@smfr Does Safari nightly report 'naturalWidth' and 'naturalHeight' that are the dimensions after applying any EXIF orientation? Chrome Canary does not, and has no test, and we have an open bug about dealing with this 'naturalWidth' and 'naturalHeight' issue.

[smfr]

Yes, in WebKit height and width are computed after rotation (where rotation is affected by the image-orientation style).

[emilio]

Hmm, so naturalWidth needs to update style? That sounds unfortunate.

[smfr]

Good point. WebKit does not, and should (it references a renderer). This feel out of some work done for other reasons (https://trac.webkit.org/changeset/97448/webkit) and may not be intentional.

[annevk]

It might make sense for naturalWidth to respect EXIF (5 in https://github.com/whatwg/html/issues/4495#issuecomment-481653964), but respecting style (4) seems weird. We don't look at style for the width itself either after all.

[smfr]

Good point.

[heycam]

Following up on all the questions Anne had last year:

Can we perhaps split the various issues here? I'm counting a lot of them and it's not entirely clear to me where each is at:

1. By default, do images ignore their intrinsic orientation or not? (Status quo is yes, but OP suggests changing that, right?)

https://drafts.csswg.org/css-images-3/#the-image-orientation has changed to have from-image be the initial value, and this change will ship in Chrome and Firefox soon.

1. Should we expose the intrinsic orientation to script?
2. Should we expose the used orientation to script?

The used orientation will only be different if image-orientation is none, so I don't think we need to expose a used orientation separately from the intrinsic orientation, as the author can look at the image-orientation value.

As for whether it's useful to expose the intrinsic orientation -- maybe, though I'd like to know what kinds of uses people would have for it.

1. Should the CSS-declared orientation impact anything beyond the rendering of the img element?

Images produced by the content property, at least. (That's what css-images says.)

1. Should the intrinsic orientation impact anything beyond the rendering of the img element?

Beyond images produced by the content property, Chrome is also planning to honor the intrinsic orientation for canvas-drawn images, per https://bugs.chromium.org/p/chromium/issues/detail?id=158753#c111. Is there a separate issue on the spec open for that? @schenney-chromium?

1. Should canvas APIs accept orientation as an input in some manner?

At most, this should be a flag to say whether the intrinsic orientation should be ignored.

Either way, it should be defined whether the value of image-orientation on img element that is painted directly using drawImage() should be honored.

1. Should the img element get an attribute to override the intrinsic/CSS/used orientation or some such?

image-orientation: none would be the mechanism for this. I'm not sure there is much of a need for specific orientation values.

---

I think I agree with Anne that it makes sense for naturalWidth to respect the EXIF orientation and not take image-orientation: none into account.

[heycam]

For what it's worth, when using drawImage() with an img that has orientation data in Chrome Canary, it honors the value of the image-orientation property on the canvas element itself, and not the referenced img.

[annevk]

@heycam that CSS Images document suggests that image-orientation itself might go away once there's convergence on using the intrinsic orientation (i.e., image-orientation: none might not be an option going forward).

cc @whatwg/css

[schenney-chromium]

Here are my thoughts after getting this implemented in Chrome (and as far as I know chromium embedders will be matching Chrome).

As stated, Chrome 81 defaults to image-orientation: from-image and also supports the "none" value. It turns out the latter is important for some developers at this time, though that is mostly due to issues with canvas in the initial Chrome implementation.

Regarding intrinsicSize, naturalSize etc, we will not consider style so will always report oriented or un-oriented size. I believe we report the oriented dimensions at this time but I need to write tests to verify the behavior.

We apply image orientation everywhere the image is used. This is motivated by consistency of the default behavior across use-cases. A patch just landed to honor it by default in canvas use cases. We honor the image-orientation style property from the canvas element because that seems to provide the most consistent behavior regardless of how the image finds its way into the canvas.

I don't think we need attributes at this time but we'll see if there's any push back from developers.

Finally, we are aware that the entire image-orientation property may disappear in favor of always using the EXIF orientation. We don't support adding orientation parameters to canvas methods or anything else because it seems likely that oriented images will be the overwhelming preference. For now we're providing "none" primarily to give developers an opt out and as a backstop for working around bugs. We'll add some metrics to figure out how much it is used to inform future deprecation decisions.

[heycam]

We apply image orientation everywhere the image is used. This is motivated by consistency of the default behavior across use-cases. A patch just landed to honor it by default in canvas use cases. We honor the image-orientation style property from the canvas element because that seems to provide the most consistent behavior regardless of how the image finds its way into the canvas.

The css-images spec says that image-orientation must not be honored for CSS images (e.g. background-image), though I see now that there was a resolution to apply it to CSS images ~6 months ago, which hasn't made it back into the spec yet. :( https://github.com/w3c/csswg-drafts/issues/4165#issuecomment-519304078

[schenney-chromium]

Right. I should have emphasized that we were going with interop with Safari when it comes to content images, as we understood that to be where the spec was. Our implementation was mostly guided by discussion on these issues: https://github.com/w3c/csswg-drafts/issues/4164 and then https://github.com/web-platform-tests/wpt/issues/18549

Note the latter references this as the spec change: https://github.com/w3c/csswg-drafts/issues/3799

[heycam]

Can we get agreement on naturalWidth honoring the orientation data, and ignoring what image-orientation says? We may or may not want to add something to expose the orientation metadata, but I think that can be separate.

[tabatkins]

+1 from me, the whole point of natural* is to expose the original values from the image.

[zcorpan]

Yes, agree. Ignoring CSS there also avoids having to force recalculate style.

[annevk]

Thanks to @heycam's #5358 how naturalWidth and naturalHeight work (and image orientation in general) is now clarified. I suggest we close this issue and if there's anything remaining, such as naturalOrientation (we also still need naturalAspectRatio), that has sufficient interest a new issue is opened to track that. Sounds good?

[emilio]

I missed the comment above, but yeah it sounds good.

Why is naturalAspectRatio useful compared to naturalWidth / naturalHeight?

[annevk]

@emilio SVG can lack an intrinsic width/height, but still have an aspect ratio.

[mzur]

If I followed this issue correctly you concluded that an autorotate attribute is not necessary as the images should always be rotated according to EXIF and the developers can use image-orientation: none if they want to disable this behavior. However, image-orientation is only applied if the element is in the DOM (as noted in the Chromium bug). For image and canvas elements that are not in the DOM, currently no method exists to disable auto-rotation according to EXIF. I suggest that this is still a valid use case for an attribute like autorotate (or similar). What do you think?

[annevk]

Probably, though I suspect you want to start by enumerating some concrete scenarios of what you'd like to do and then file a new issue with those. See https://whatwg.org/faq#adding-new-features. Hope that helps.

[oriadam]

If I followed this issue correctly you concluded that an autorotate attribute is not necessary as the images should always be rotated according to EXIF and the developers can use image-orientation: none if they want to disable this behavior. However, image-orientation is only applied if the element is in the DOM (as noted in the Chromium bug). For image and canvas elements that are not in the DOM, currently no method exists to disable auto-rotation according to EXIF. I suggest that this is still a valid use case for an attribute like autorotate (or similar). What do you think?

can't think of any reasonable use case for disabling the correct image orientation in canvas or anywhere else. if there is a use case for this, exposing the image meta data should suffice, and the canvas can rotate the image as it pleases.

[mzur]

can't think of any reasonable use case for disabling the correct image orientation in canvas or anywhere else

My use case is backwards compatibility. Changing the default way many images are displayed is a huge change for web applications that work with images. In my case, I'm maintaining an image annotation application where you can mark things on the images. Now suddenly some images are rotated and the annotations are no longer at the right place. As the behavior is different across different browsers, users see the images differently and produce inconsistent annotations. To provide some advanced image manipulation features, the images are loaded and drawn onto canvas elements outside the DOM. Hence, I can't disable the auto-rotation using CSS. IMHO the results of image manipulation with image and canvas elements shouldn't depend on CSS anyway.

if there is a use case for this, exposing the image meta data should suffice, and the canvas can rotate the image as it pleases.

Is there a proposal for exposing the image meta data somewhere? I'm happy to support any solution that lets me disable image auto-rotation. My suggestion would be something like an orientation attribute for the img element. Values could be auto (default, EXIF orientation), portrait, landscape or data (I couldn't think of a better name). The latter value makes the image display the way the data is stored in the file. A canvas element should not affect the configured orientation of the image. I can open a new feature request for this if you think that makes sense.

Edit: There is also a proposal to change drawImage or getContext('2d', ...) to enable/disable auto-rotation for canvas elements. However, img elements will still be auto-rotated (unless disabled via CSS) and one wouldn't know which dimensions to use for the canvas element if the image should be drawn completely, as the width and height may be swapped. With an orientation attribute for img as suggested above, this wouldn't be a problem and drawImage or getContext wouldn't need to be changed.

[inukshuk]

I would like to emphatically echo @mzur's concerns. Is there currently any way to tell whether a given Image instance was rotated or flipped because of the Exif orientation field (or, even better, the actual orientation value that was used)?

I am, too, working on an Chromium-based application that previously parsed and applied the Exif orientation separately; there are also annotations, stored based on the raw bitmap coordinates so an orientation attribute as suggested above, with a data-like value would be ideal. Without this, at least a way to tell what orientation was applied, in order to reverse-map coordinates would be great. I was hoping naturalWidth and naturalHeight would report the original, un-oriented values, but they are identical to width and height even when the Image was rotated because of the orientation metadata.