Ckeditor5: Image orientation/rotation not being displayed correctly...

[Methodician]

Is this a bug report or feature request? (choose one)

🐞 Bug report

💻 Version of CKEditor

CKEditor 5 Build Inline (custom build) all versions 10..+ (editor-inline @11.0.1, [email protected], etc...)

📋 Steps to reproduce

Not clear, but the following images aren't displaying right:
https://firebasestorage.googleapis.com/v0/b/cosourcerytest.appspot.com/o/articleBodyImages%2FFglLuO3VJj1pQXgaHBBL%2F20181031_183926.jpg?alt=media&token=6f2f861e-d32a-4290-8042-b11956783ff8

https://firebasestorage.googleapis.com/v0/b/cosourcerytest.appspot.com/o/articleBodyImages%2FFglLuO3VJj1pQXgaHBBL%2Fsoaking_roots_before_replanting.jpg?alt=media&token=10506ed8-9aef-4bbe-a86e-68af1bf8d7af

You can see an example here:
https://cosourcerytest.firebaseapp.com/article/FglLuO3VJj1pQXgaHBBL

✅ Expected result

I would expect the images to display with the right orientation...

❎ Actual result

Images display on their side.

📃 Other details that might be useful

This is not isolated to CKEditor. It's a common problem with displaying images online. However, I would expect CKEditor to have a solution for it though.

[Methodician]

I may be able to catch the file being uploaded and do some stuff client-side or server-side, but is there some way we can just have things displayed with the right EXIF orientation?

https://stackoverflow.com/questions/20600800/js-client-side-exif-orientation-rotate-and-mirror-jpeg-images

[Reinmar]

This is not isolated to CKEditor. It's a common problem with displaying images online. However, I would expect CKEditor to have a solution for it though.

The solution is to use the right backend engine for uploading images. Sadly, I just checked and both Easy Image and CKFinder services don't autorotate the image.

cc @czerwonkabartosz @zaak @wwalc

I may be able to catch the file being uploaded and do some stuff client-side or server-side, but is there some way we can just have things displayed with the right EXIF orientation?

I don't think that a client-side solution is a solution here. It would not be a viable solution in practice when you, later on, send that content to your clients and make their browser autorotate it. It would be highly unoptimal so you'd want to autorotate those images on your server anyway, before sending them to clients.

[Reinmar]

I'm leaving this as a bug+discussion for now until we decide what Easy Image and/or CKFinder could do about it.

[zaak]

Related CKFinder issue https://github.com/ckfinder/ckfinder/issues/326.

[czerwonkabartosz]

I've found a few libs for this very quickly, so it could be possible on server-side. We will check libs as research for this. Unfortunately, I don't know ETA.

[Reinmar]

I guess that in case of Easy Image autorotating would not be considered a big deal because, as a service, it's mainly built for web content. But I wonder if CKFinder could introduce such an option. If we see at the "file manager" part of its purpose, then changing image data would not be good. But when integrating with a rich text editor, that would be desirable. Perhaps this should be a flag then, only used when outputting URLs to a rich text editor?

[Livijn]

Is there any updates regarding this? I am using Easy Image with a custom upload adapter but this problem might force me to switch editor as users cannot upload their images correctly.

The exif data is persisted, so I could rotate it on the backend, but the image will still be displayed in wrong orientation until the image is uploaded.

Can I somehow hook into the image select flow?

[Reinmar]

Is there any updates regarding this? I am using Easy Image with a custom upload adapter but this problem might force me to switch editor as users cannot upload their images correctly.

If you're using a custom upload adapter then you're not using Easy Image. Easy Image is a service + a CKEditor 5 plugin which enables it. If you're using a custom adapter, then you use only the image upload feature of CKEditor 5.

The update regarding Easy Image is that the autorotate feature will land today. Once the image is uploaded, its orientation will be used so all images will properly display in the browsers.

The exif data _is_ persisted, so I could rotate it on the backend, but the image will still be displayed in wrong orientation until the image is uploaded.

Rotating images while they are being uploaded (so on the client) isn't something that we planned and it doesn't seem to be a trivial feature. Unless there's some trick I'm not aware of, we'd need to include an image parsing library in CKEditor itself and use it to extract exif information and rotate the image ourselves. Sounds like a lot of work to fix an issue which occurs for a couple of seconds max (while the image is being uploaded). Plus, I wonder how would a performance of such a solution look.

cc @Comandeer perhaps you've heard about some simple solution?

[czerwonkabartosz]

The update regarding Easy Image is that the autorotate feature will land today.

Autorotate feature in Easy Image service just landed in our cloud.

[Livijn]

My bad for mixing up the terms.

If you're using a custom adapter, then you use only the image upload feature of CKEditor 5.

Correct, I am not using Easy Image, but rather the @ckeditor/ckeditor5-upload package with a custom adapter.

Rotating images while they are being uploaded (so on the client) isn't something that we planned and it doesn't seem to be a trivial feature.

Instead of rotating the image while its being uploaded, you could do it before uploading it, therefor sparing your backend service the extra work.

Unless there's some trick I'm not aware of, we'd need to include an image parsing library in CKEditor itself and use it to extract exif information and rotate the image ourselves.

Yes, I believe this is correct. exif-js seems to be stable source of getting the exif data. And this stackoverflow answer seems to be a popular advice on rotating the images using canvas.

Sounds like a lot of work to fix an issue which occurs for a couple of seconds max (while the image is being uploaded). Plus, I wonder how would a performance of such a solution look.

Uploading a ~10MB image and then rotating it on the server actually can take a while. Also, since the image url needs to be returned in the request there is no room for queueing the conversions and then sending the rotated image URL through sockets. This means a lot of extra load on the app servers – which shouldn't handle these kind of tasks (bottleneck inc 🙅‍♂️🍾). Also, a minor issue is that the upload progress bar is somewhat incorrect since the actual progress of handling the request is disregarded.

[Reinmar]

Thanks for your feedback.

I do agree that rotating the image before showing it to the user would create a better UX. That's for sure.

Delegating this to the client will indeed offload your server too. However, if we're talking about 10MB images, the question is whether it won't make the client far less responsive. Currently, the APIs that allow us to read the file from the clipboard are synchronous, meaning that there's a visible pause when the image is dropped. This is not a great UX. If we'd add autorotation on the client, we may make the situation even worse. We'd need to have a PoC of such a solution to tell which situation is less painful for the user.

[Nothing-Works]

@Reinmar hi if I understand this issue correctly, then I think this is going to be a problem for the people who implemented the inlining images using base64 URLs feature by themselves (myself is one of them), using a custom upload adapter. I guess in this scenario the client-side solution makes sense, as I do not want to bother the server for doing it. I want everything happens on the client-side and then when the user finishes editing, I send the whole gigantic string to the server and persist it into the database. Please let me know what's your thought, thanks.

[Reinmar]

Good point. In this case, doing this on the client sounds like the only option.

[Methodician]

Just a little update:

After posting this I did a torrent of research and concluded that processing EXIF orientation is definitely something to be done client side. That's essentially how EXIF data is intended to be consumed (and there are plans to add a "orientation: from-exif" option to css)

Due in part to the barriers erected by CKEditor it's not exactly trivial on an app-by-app basis. However, FWIW it may be relatively trivial to add this into CKEditor so everyone doesn't have to hack together a work-around.

I'll paste in some code that worked in an Angular project but this is far from an explanation and I would write this differently if I went back over it today. I also used "exif-js" to get the metadata but that's not a necessary dependency if you have time and skill. Hopefully it will offer some inspiration to others... There's a lot of extra junk in here so I may come back and clean it up for a more generic example if @Reinmar and friends confirm that this isn't in the road map for the open sourced version.

```
onCKEditorChanged({ event, editor }: ChangeEvent) {
// We're not using CKEditor as a normal FormControl because its scripts would mark the form as "dirty" even when the data was coming from DB.
// This approach allows us to manually mark it as dirty only when the changes are local.
const contents = editor.getData();
// If onCKEditorReady hasn't run, this will still run with no images to process.
if (this.ckEditorReady) {
// setTimeout with 0 delay still pushes this down the stack so we get the updated body.
// Otherwise when deleting an image, we'll still process the deleted image.
setTimeout(() => {
this.processCKEditorImages();
}, 0);
}
this.articleEditForm.markAsDirty();
this.articleEditForm.patchValue({ body: contents });
}

onCKEditorReady($event) {
this.ckEditorReady = true;
this.processCKEditorImages();
}

// CKEditor image processing (would like to move some of this outside the component)
processCKEditorImages() {
const figures = document.getElementsByClassName('image');
for (let i = 0; i < figures.length; i++) {
const fig = figures[i];
const img = fig.firstChild as HTMLImageElement;
if (img.complete) {
// Processes images when for one reason or another they are already loaded but may not be rotated.
this.rotateImage(img);
} else {
img.onload = ev$ => {
// Processes images when form first loaded
this.rotateImage(img);
};
}
}
}

async rotateImage(img) {
if (img.src.includes('data:image')) {
return;
}
const storage = firebase.storage();
const imgPath = storage.refFromURL(img.src).fullPath;
const imgCode = imgPath.split('/')[imgPath.split('/').length - 1];

let rotation: orientationDegrees = 0;
// check if it's been rotated, if so, don't do any extra stuff
if (img.style.transform && img.style.transform.includes('rotate')) {
  return;
  // check if it's in the map, if so, set rotation via its orientation
} else if (this.articleEditForm.value.bodyImages[imgCode]) {
  rotation = this.exifOrientationToDegrees(
    this.articleEditForm.value.bodyImages[imgCode].orientation,
  );
  // else add it to the map with it's correct orientation
} else {
  let orientation = await this.getExifOrientation(img);
  orientation = orientation ? orientation : 0;
  rotation = this.exifOrientationToDegrees(orientation);

  const imageObject: BodyImageMeta = {
    path: imgPath,
    orientation: orientation,
  };

  this.articleEditForm.value.bodyImages[imgCode] = imageObject;
}

img.setAttribute(
  'style',
  `transform:rotate(${rotation}deg); margin: 80px 0 `,
);

}

getExifOrientation(img): Promise {
const promise = new Promise((resolve, reject) => {
try {
exif.getData(img as any, function() {
const orientation = exif.getTag(this, 'Orientation');
return resolve(orientation);
});
} catch (error) {
// console.log('Can\'t get EXIF', error);
return reject(error);
}
});
return promise;
}

exifOrientationToDegrees(orientation): orientationDegrees {
switch (orientation) {
case 1:
case 2:
return 0;
case 3:
case 4:
return 180;
case 5:
case 6:
return 90;
case 7:
case 8:
return 270;
default:
return 0;
}
}

```

[Methodician]

I'm revisiting this now that exif-js is pretty much deprecated (doesn't work with strict mode, not updated for years, etc...) exif-js was also pretty poor on performance. However, to do it right/optimally I do need to intercept the file being uploaded, client-side. I have a custom plugin for CKEditor5-build-inline but where is the instance of "FileReader"? (maybe I should open another issue?)