Web-stories-wp: Tenor: Render as video rather than GIF

[choumx]

Feature Description

To reduce network load and improve latency, embedded media from Tenor should be rendered as .mp4 or .webm in <amp-video> rather than .gif in <amp-img>.

Alternatives Considered

N/A

Additional Context

This should be the last thing that's holding back the launch of Tenor integration (#4599).

---

_Do not alter or remove anything below. The following sections will be managed by moderators only._

Acceptance Criteria

Implementation Brief

[diegovar]

• The Media3P API Returns both mp4 and webm formats, you probably want to just use the mp4s for now. Example query is: https://media3p.googleapis.com/v1/media?filter=provider%3Atenor+type%3Agif&page_size=20&key=KEY
• The easiest approach is possibly to have the canvas (and design panel) treat the GIF as an image, and then at the time of rendering the element, render it as a video with autoplay, mute and no controls.

[swissspidy]

The easiest approach is possibly to have the canvas (and design panel) treat the GIF as an image, and then at the time of rendering the element, render it as a video with autoplay, mute and no controls.

As I mentioned previously, I think the cleanest way to achieve this would be by making gif a first-class citizen here.

To give an example, we currently have a couple of instances like this in the code base:

if (resource.type == 'gif') {
  resource.type = 'image';
}

Adding more checks like this is going to make things overly complex and prone to errors (should we ever forget to add such a check anywhere).

By establishing the gif resource type we can save ourselves this trouble.

We already have assets/src/edit-story/elements/media, assets/src/edit-story/elements/image, and assets/src/edit-story/elements/video.

So we could just introduce assets/src/edit-story/elements/gif that takes the best of both worlds as suggested - treat as image, render as video. That also means sharing some components like VideoOutput for instance. These shared components could be moved to a shared location if needed.

Things to consider:

• Since the Media3P API returns both mp4 and webm formats, we might want to support multiple source elements in VideoOutput.
• GIFs are also quite heavy to display in the media library, and it would be wise looking into whether we can use the mp4 files there too.
• <amp-video> ideally have a poster image (not sure right now if it's a requirement). The API doesn't seem to provide that. @diegovar Could this be added? Seems like Tenor provides a preview image

[cvolzke4]

Note that stickers require transparency support, which means .gif or .webm formats should be used instead of .mp4 (or both webm and mp4/gif in the AMP output to support all browser types). Stickers aren't used today, but it'll be easy to add them once the Tenor backend supports filtering by sticker.

Adding a poster image from the backend is straightforward, we just have to choose which 'preview' image from tenor and return it in the 'image_urls' field of media3p:
go/media3p-tenor-image-urls

The frontend can then filter by non-gif images to find the poster image (it's a png).

[cvolzke4]

Let us know if you need an overview of the backend service, or send us a CL if we can help with code review.

[embarks]

@cvolzke4 Thank you for the info on transparency. Is there a way to return the png preview image from the backend now? Currently the front end receives image urls which include the gif size variants and video urls which have the mp4/webms and size variants of those.

The API request just has the provider and type like:
https://media3p.googleapis.com/v1/media?filter=provider:tenor type:gif&page_size=20&key=KEY

[swissspidy]

It sounded like preview image support is something we‘ll have to add from our end

[embarks]

@swissspidy cool :) When you say "from our end" I'm assuming there are changes to the media3p api which would not be done from the web-stories context. I see the preview image would be available on the response object according to tenor's API (which I believe is what you were referring to here:

ideally have a poster image (not sure right now if it's a requirement). The API doesn't seem to provide that. diegovar Could this be added? Seems like Tenor provides a preview image

Tenor gif api

[diegovar]

If poster images are not a requirement for AMP, perhaps we can just not add them? Not sure it adds a lot of value given the mp4 and webm files for these gifs are pretty tiny.

[swissspidy]

I just checked and the video poster is mandatory in stories (reference). I can take a pass at writing a CL for that this week as suggested by Clinton, as I assume you won't have bandwidth for that.

While poster support will be worked on, the rest in this ticket shouldn't be blocked by that.

[choumx]

A real poster image would be great but a stopgap could be a Tenor or generic image logo.

[swissspidy]

Interestingly the AMP validator does not complain about poster="" so... let's just use that instead of a Tenor logo/generic image?

[csossi]

Verified in QA