Gutenberg: Video Block: Subtitles

[Soean]

Description

To add, edit and delete subtitles in the video block, we need an UI.

Current Classic Editor:

A video tag can have multiple tracks elements.
A track element has 4 Attributes.

• srclang (Valid BCP 47 language tag)
• label (Title for player UI)
• kind (Captions, Chapters, Descriptions, Metadata or Subtitles)
• src (URL of the text file)

Related: #7501

[karmatosed]

Can I ask how you get to this from the existing editor without a plugin? It would be great for steps so people can replicate and explore.

[Soean]

1. Add Video via 'Add Media'

2. Click on the edit button on top of the video
   

3. Add a Subtitle file

4. You see the Subtitle details
   

Here you can download example .mp4 and .srt files: https://github.com/mediaelement/mediaelement-files

[karmatosed]

This is a really rough mock but I am thinking something similar? That or using a pattern like playlist does could work?

[melchoyce]

An idea for adding subtitles:

The flag emoji would change based on the language detected in the subtitle file, since seems to be functionality we already have?

[Soean]

Hey @melchoyce
thanks for your design. I have a few comments on this.

• Flags are not ideal for this, because they represent states not languages, see http://www.flagsarenotlanguages.com/blog/why-flags-do-not-represent-language/
• There is no language detection, the default is english in the classic editor
• The user should be able to set these attributes for the track file:
  srclang (Valid BCP 47 language tag)
  label (Title for player UI)
  kind (Captions, Chapters, Descriptions, Metadata or Subtitles)
• I added the Accessibility label because it's new UI. Maybe we have to change the button labels? See https://github.com/WordPress/gutenberg/issues/11991#issuecomment-443238953

[Ipstenu]

Is there any movement on this at all?

As it stands I cannot add subtitles at all, let alone per language, to videos, which is not a good experience for my users :(

[paaljoachim]

What about something like this?

I just added a settings icon. Click the icon and it opens something to add the subtiles.

(I am working on keeping options outside of the sidebar.)

[afercia]

As commented on a similar issue, not sure the sidebar is the best place for tracks. Anything content related should be within the block.

And yes, flags are not languages 🙂Also, prefer plain text when possible instead of icons.

[anevins12]

@afercia In the most ideal scenario, where would you expect the tracks to appear? I think that's a really tough question to answer but should help drive this ticket forward

[afercia]

@anevins12 tough question. I don't even understand why clicking the "Edit" button (the pencil) switches back the editor to the "placeholder". Instead, in the Classic Editor it opens the details modal, where there's plenty of space for all the settings:

In Gutenberg the available space is way more limited. The sidebar is only 300 pixels wide. Also, we know it's very hard to use for persons with accessibility needs.

I'd like to see more settings in the "placeholder":

According to the Gutenberg design principles, the primary actions / settings should be in the block UI. I'd say alternate sources and tracks are directly related to content and as such are "primary".

[joedolson]

It would be great to be able to upload and assign subtitle/caption tracks directly in the placeholder. In addition to making an easier interface, it would also greatly increase people's awareness of the feature and improve the likelihood it gets used.

[melchoyce]

I'll try an updated version with the fields in the placeholder!

Any thoughts on whether the sidebar would feel like an appropriate place for changing or updating subtitles after the fact? My inclination still leans towards the sidebar as a good place to house subtitles, but I want to get y'alls a11y opinions first.

[joedolson]

You mean fields for switching out already-added subtitles for a different set of subtitles?

Sidebar might be OK as a place to change subtitles - while I think adding subtitles in the first place is a first-tier activity, I can see changing them as a secondary activity. It should be a considerably less frequent task.

That said, I do agree with @afercia that content changes should generally take place within the block itself - and captions & subtitles are definitely content. It seems like there should be ample space in the placeholder container or some other in-block interface to manage these.

[melchoyce]

We can probably take inspiration from the image change flow in #14142 for changing or adding new subtitles.

[afercia]

Noting that this issue was reported in the WPCampus accessibility reports, see #15281, and as such should be moved to the related project in the "To do".

[youknowriad]

This was discussed today in the #core-editor triage session (link requires registration):

https://wordpress.slack.com/archives/C02QB2JS7/p1557753241107800

Based on the reading and the discussions, it seems that this issues still needs design proposals and thoughts as it's not really clear what should be implemented as a v1. I'd encourage everything to also think pragmatic here and come up with a version that can be iterated on. Since this feature is not available entirely, having something even if not perfect design-wise would be appreciated.

[afercia]

Related: https://core.trac.wordpress.org/ticket/47559

It appears the srclang and label attributes in the media views are hardcoded to English 😞

[enriquesanchez]

I'm not sure if the option to add subtitles should appear in the block placeholder, or at least not until a video has been uploaded/selected.

The way I see it, you first need to upload/select a video in order to then add subtitles. Please correct me if I'm wrong, but once you've uploaded/selected a video, the placeholder is replaced by the video itself.

I'm wondering if, once a video has been uploaded/selected and the block is focused, we can instead show an option that takes you to the attachment details modal:

Once on the details modal, the user can then edit/add title, caption, description, subtitles, etc:

This should also cover the case where more than one subtitle track needs to be added. We would enough space in the UI to accomodate all of these settings and fields.

[afercia]

In the Classic Editor the UI to add subtitles isn't in the "select" media modal. Instead, it's in the Edit Video details modal:

That means users have to add a video to the post content first, then edit the details of the video they just added. I guess this is one of the reasons why this UI isn't very discoverable.

For Gutenberg, I can think of a few options:

• Enhance the "select" media modal, adding a second step for additional settings including the UI for subtitles. This could work much like the existing Gallery or Playlist modals, where the second step allows to edit the details _before_ inserting media in the post
• The above would require a decent amount of work in the media views
• Or: once a video has been inserted in a block, give access to the existing "Video Details" modal: the subtitles UI is there
• Or: move away from the media modal and provide a new UI within the Block to edit the video details

Existing issues:

once you've uploaded/selected a video, the placeholder is replaced by the video itself.

Correct. Also, when clicking the "Edit video" button within the block toolbar:

• the button disappears: this is a problem because technically it's a focus loss
• there's no way to switch back from the placeholder to the preview other than:
  
  
  
  • click on "Insert from URL" (which is totally counterintuitive)
  
  
  • accept the current URL value and press the "Apply" button
  
  

Personally, amongst the various options I'd recommend one that helps in making the subtitles UI _discoverable_. Having it only in the "Edit" mode after the video has been already added isn't that great for discoverability. Instead, given the importance subtitles have for accessibility and internationalization, the subtitles UI should have the same prominence as the Caption one.

[melchoyce]

Improvement on the "Edit" menu item:

Agree we need to improve discoverability, and maybe explore what it could look like to try a multi-step video uploading process.

[enriquesanchez]

Instead, given the importance subtitles have for accessibility and internationalization, the subtitles UI should have the same prominence as the Caption one.

@afercia do you think that the first mockup on my comment above (or an iteration of a similar idea) could help us with this?

I also really like @melchoyce's explorations on the Edit menu item above 👍

[Ipstenu]

If we go with Mel's Edit Menu, that should also include the ability to edit the poster image.

That said, we can edit/add poster images on the block edit sidebar. Would that not also be a proper place to have a 'customize subtitles and language' option, or even a button to click to GET to the old screen?

[afercia]

do you think that the first mockup on my comment above (or an iteration of a similar idea) could help us with this?

Well this is more up to the design and Gutenberg teams. I have the impression one of the goals is to move away from the media modals and build new UI within Gutenberg. Personally, I don't have any problem in keeping the media modals. They're also a focus for the accessibility team in the 5.3 release cycle so they will see some accessibility improvements. I'd tend to think that if they're going to be replaced, the new UI should be better and provide clear advantages compared to what WP has now.

If we go with Mel's Edit Menu, that should also include the ability to edit the poster image.

Yep, good point!

Re: the sidebar: it's one of the most inaccessible parts of the Gutenberg UI, both in an a11y and literal sense. Jumping to the sidebar _while keeping the block selected_ is very hard for keyboard and assistive technology users. For this reason, the sidebar should contain only secondary settings, while anything related to content production should live within the block (which is also one of the Gutenberg design principles).

[audrasjb]

Hi there,

I'm not convinced we should handle that in the Editor, since there is other places where subtitles can be added: widgets first, but also in plugin’s custom media modals.

Then, I worked on a patch for the Core media modal. I need some design feedback on the shared screenshot before generating the full patch.

For reference: https://core.trac.wordpress.org/ticket/47559#comment:9 (and screenshot provided just below that comment)

Cheers,
Jb

[mapk]

If we worked in the Media flow component here, would that help?

The "Open Media Library" option should probably still open the Media Library, but if we added an option here to "Open Video Details" that took the user to that particular modal, how's that sound?

[Soean]

Yes, we can try this approach. I am waiting for the merge of this PR and then I'll continue working on the ticket.

[mapk]

@Soean, it's been merged! 🎉 Can you resume work here?

[Soean]

Yes I will resume in the next days.

[youknowriad]

This still needs more design work I think. We can start prototyping things but the design is not complete yet.

How do you fill the "lang"? How do you type the "label"? How do you choose the "kind"?
I'm also assuming that for the "src", we should have a file input to upload the file?

[afercia]

How do you fill the "lang"? How do you type the "label"? How do you choose the "kind"?

Currently, after a track file has been selected, the media views add a value like this to the track input field:

<track srclang="en" label="English" kind="subtitles" src="http://local.wordpress.test/wp-content/uploads/2019/05/test.vtt">

where srclang, label, kind are pre-filled with sensible defaults. I tend to think these are advanced settings for users who know what they're doing. If they need to change these values, they can do it by manually editing. I wouldn't be opposed to just replicate the current behavior as a first step. Then iterate and consider if a new UI is worth it.

[joedolson]

There are two primary use cases for adding a track, and they'd have different default values: adding a translation and adding captions. For a translation, the srclang & label would need to vary widely.

I think that a UI to handle these values is important to support standard use cases, and something we shouldn't leave as only available to advanced users.

[afercia]

@joedolson I get your point. On the other hand, this feature is still missing and Gutenberg lacks parity with the Classic Editor since more than one year. Could we consider to get a first working implementation and then iterate on the UI?

[mapk]

I'm wondering if, once a video has been uploaded/selected and the block is focused, we can instead show an option that takes you to the attachment details modal:

Could we consider to get a first working implementation and then iterate on the UI?

I agree with these points.
I don't think the "Add subtitles" button should be inline with the caption text because it can interfere with the captions. Now that we have the MediaReplaceFlow component, I don't think it makes sense to add to that popover either. We currently show a "Select Poster Image" button in the sidebar video settings which opens the Media Modal, and I think we can just use this pattern as well for subtitles which would also open the Media attachment details modal... at least for now.

Something like this:

I know it's not ideal, but it's following an existing pattern and getting it into Gutenberg. Does the interface exist currently in the Media Modal? I couldn't find it, so if we need to create it in the Media Modal, how do people feel about @enriquesanchez mockups here: https://github.com/WordPress/gutenberg/issues/7673#issuecomment-515607606?

[joedolson]

As long as we clearly understand that doing this only moves towards parity with the classic editor, and does not actually provide new accessibility support.

[afercia]

Does the interface exist currently in the Media Moda

• activate Classic Editor
• add a video to a post
• once inserted into the post, select the video (click on it) then click the "Edit" pencil icon
• in the modal that opens, click "Add Subtitles" in the left menu
• if I remember correctly, "Add Subtitles" appears only if you have a .vtt file uploaded in your media library so here's one attached"
  test.txt
• rename the file to test.vtt (GitHub doesn't allow files with a .vtt extension)
• now after you click "Add Subtitles" you should see your track file
• select it and "Add Subtitles" in the bottom right corner of the modal
• now you're back to the "Video Details" modal view
• scroll the modal, see the track details at the bottom
• click "Update"
• the modal closes
• save/update the post
• see the post on the front end
• play the video and activate the caption by pressing the CC button in the player UI (select English)
• see the captions working

A _very straightforward_ flow worth improving 🙂

[jorgefilipecosta]

I'm moving this ticket how of WordPress 5.4 must-have because it seems it will not be ready. That said this issue seems very important and it is something that should be prioritized right after beta 1.
If I'm missing any update to this issue and there is a chance it may be ready please let me know.

[mtias]

I'm adding back "needs design" because it's not clear to me what's the outcome here to implement. Video captions / subtitles are an important feature to surface as primary, so I'd rather we explore how to show it clearly in the toolbar if possible.

[enriquesanchez]

I've created a first pass at a proposal and prototype. I've considered the following as requirements to meet:

1. A user should be able to add and remove subtitle tracks to a video
2. A video can have multiple tracks elements.
3. Each track element has 4 Attributes:
   
   
   
   • srclang: (Valid BCP 47 language tag)
   
   
   • label: (Title for player UI)
   
   
   • kind: (Captions, Chapters, Descriptions, Metadata or Subtitles)
   
   
   • src: (URL of the text file)
   
   
4. All of this, should be exposed from the block’s toolbar.

---

The following prototype covers the case of adding multiple tracks. Each track has the option to be edited or deleted.

The flow shown above would bring us to parity with what's currently available in the classic editor.

In order to make the editing of the track's attributes friendlier, we'd need to hide the track's markup and instead expose each attribute as a separate input field, as suggested above.

I'll continue exploring ideas around this.

For now, I'd appreciate feedback on the prototype I shared:

• Does it bring us to parity with the classic editor? Is there anything missing?
• How should we handle the case where there are a lot of tracks?
• Both in the classic editor and here, the prompt is to "Add subtitles", but we then call these "Tracks". Also, these tracks can have a different kind attribute: Captions, Chapters, Descriptions, Metadata, or Subtitles. Should we use a different term? Would "Tracks" be generally understood?

[paaljoachim]

Hey Enrique!

Does it bring us to parity with the classic editor? Is there anything missing?

An overview of todays method:

1. Add video.
   

2. The video is added to classic editor layout. Clicking edit brings up the Video Details screen. One then clicks the left sidebar - Add subtitles text link, and the title becomes Add Subtitles.

1. Subtitle is added to the video. Which is seen in the Video Details screen.

Based on the above Classic Editor. Now transferred to Gutenberg. After having added a video one then might want to add a subtitle.

It is really helpful to have a subtitle icon in the toolbar. Making it easy to upload/media library and edit inline the existing subtitle(s). Inline editing keeps it directly in place without having to go to another screen to edit the subtitle.

I did not see anything missing in relation to subtitles from your prototype.

How should we handle the case where there are a lot of tracks?

I am not sure what you mean with this (just send me a message and I can rephrase/edit this comment), but the way you have set it up with the subtitles icon with upload/media library works well for one or multiple tracks/subtitles.

Both in the classic editor and here, the prompt is to "Add subtitles", but we then call these "Tracks". Also, these tracks can have a different kind attribute: Captions, Chapters, Descriptions, Metadata, or Subtitles. Should we use a different term? Would "Tracks" be generally understood?

As I experience it tracks in relation to video can be confusing while subtitles is a common term.
For instance. Tracks on a CD. I would not say tracks in relation to a video. I did a search and it gives an idea how it is used: https://support.firecore.com/hc/en-us/articles/215090907-Subtitles-and-Audio-Tracks
Audio tracks and video subtitles. I do not think we would say audio subtitles and video tracks. As that is mixing the terms.

But as I read your comment again. It seems that video track is something above in the hierarchy of video subtitles. Tracks is used inside the inline editing of a subtitle and also used above the edit area. Having the word tracks so prominent just above the subtitles to me seems like a mistake. Instead of the word TRACKS it should say Subtitles, as right below the subtitles are linked. Inside the subtitle edit area it could for instance say attribute instead of tracks.

[joedolson]

"Tracks" is the most correct option here. The track element in HTML5 is used for subtitles and captions, but is also the element that provides support for chapters and other metadata. If this only said 'subtitles', it would strongly imply that it only supported that kind of track.

From a terminology standpoint, "tracks" should be considered more similar to tracks in audio mixing (e.g., parallel simultaneous tracks), rather than tracks on a CD (discrete sequential tracks.)

However, the UI certainly has room for improvement; it might be better to expose each type of track as an independent upload option matching the type of data. E.g., 'Subtitles', 'Captions', 'Chapters', etc.

However, that's not really necessary to reaching parity, and might be better as a later interface improvement.

[pablohoneyhoney]

Perhaps the bubble refers to chat or conversation, and we can have something more direct like a screen with two lines alluding to text. Like:

[enriquesanchez]

Thanks for the feedback @paaljoachim, @joedolson and @pablohoneyhoney! 🙏

I spent a little more time working on the UI and looking at ways to make the editing of track attributes a little friendlier. Here's a new prototype:

---

There's a new step after uploading a track file that lets the user modify the label, srclang and kind attributes:

This new UI also appears when the user chooses to 'Edit' a track.

There's a "(?)" in the "Source language" field label that I'm thinking can link to the W3C reference on language tags and how to use them.

The 'Kind' dropdown should let the user pick between Captions, Chapters, Descriptions, Metadata, or Subtitles.

---

• How does the prototype feel with this new step to edit track attributes?
• Are we improving the accessibility support compared to the Classic editor?

[StefanXhunga]

Hi All, thank you for your assistance!

[afercia]

Aside: Unsubscribing from this issue as, for what concerns accessibility, it's already in the good hands of @joedolson and @enriquesanchez.

[paaljoachim]

@enriquesanchez I really like the additional editing screen you made. Which makes it a lot easier to edit a track.
Thanks!

[enriquesanchez]

Here's an updated prototype. Same flow just some minor refinements to the UI.

---

I think this is in a good spot, it brings us to parity with the classic editor and makes editing of track attributes easier. Unless anyone thinks there are blockers, we should move forward with a PR and get it tested.

[joedolson]

This is a good starting point, and is an improvement on the equivalent process in the classic editor. I'd like the 'Edit' button to look more button like, but that's just design, and shouldn't stop this from moving forward.

[mtias]

Are "tracks" a clear enough term for users to use as the entry point? (Tooltip and heading within the dropdown menu.) I like the explanatory text, I think we should be using more of these contextually.

[enriquesanchez]

Are "tracks" a clear enough term for users to use as the entry point?

@mtias I'm not sure, as others mentioned above, while technically accurate the term feels more audio-related to me.

On the other hand, "subtitles" does not fully represent all the different types of tracks you can upload (captions, description, chapters, etc.).

We could call it "Captions & subtitles"? I've seen these two as the most common terms used for these text alternatives.

[enriquesanchez]

Looking at https://developer.mozilla.org/en-US/docs/Web/HTML/Element/track, maybe "Text tracks" is a better umbrella term for all of these types?

[enriquesanchez]

@Soean The design feels like in a good shape to start working on a PR. Would you have availability to work on this?

[joedolson]

I think that 'Text Tracks' seems like a good choice. It conveys more clearly the type of content that's in common to all of the supported fields without implying that it only supports a subset of those choices.

If it's called "Captions & subtitles", that would suggest that it doesn't support chapters or descriptions.

As far as I know, the player does not currently support descriptions, however.

[joedolson]

Would love to have a status update on this; this is a major accessibility gap in the feature set of Gutenberg, and if there's any chance it can get into a near-future release, I'd like to know that!

[jorgefilipecosta]

If no one is already working on an implementation of this feature, I can work on it :)

[enriquesanchez]

Thank you @jorgefilipecosta!

Please let me know if you have any question about the design and flow.

[jorgefilipecosta]

An implementation was proposed in https://github.com/WordPress/gutenberg/pull/25861.

[enriquesanchez]

Would love to have a status update on this; this is a major accessibility gap in the feature set of Gutenberg, and if there's any chance it can get into a near-future release, I'd like to know that!

@joedolson The feature has just been merged 🎉

[joedolson]

Just saw that earlier today; happy to see it!