Hls.js: Support mp3 audio (mp4a.40.34) in mp4 video container

[sompylasar]

For now, I see a single option -- implement MP3 parsing and re-coding into AAC.

References:

• https://github.com/dailymotion/hls.js/issues/234#issuecomment-181895148
• https://github.com/dailymotion/hls.js/issues/61#issuecomment-159108556
• https://github.com/dailymotion/hls.js/issues/44#issuecomment-181895640
• for MP3 in MP4, refer to https://github.com/gpac/mp4box.js/issues/78#issuecomment-181879290
• https://github.com/tchakabam/multimedia.js
• https://github.com/audiocogs/mp3.js (GPLv2 :-1:)

[mangui]

MP3 in MP4, audio config box : https://github.com/tchakabam/multimedia.js/blob/master/src/mp4-mux.js#L311

MP3 parsing in JS : https://github.com/tchakabam/multimedia.js/blob/master/src/mp3-parser.js

[nochev]

Hello guys,

Based on issues and shared links I tried to add mp3 audio support in HLS.js plugin.
Here you can see my changes to the plugin: https://github.com/nochev/hls.js/commit/64b54c254d12ca8cc3945f4f24c543bd8b02545c and demo.

There are several issues and will be glad someone to share experience with that.

The good news is that mp3 works pretty well in Firefox - mp3 is feeding in mp4 container in Quicktime way (MPEG-1/2 Part 3 - QuickTime File Format audio/mp4; codecs="mp3").

Chrome doesn't support audio/mp4 container with mp3 codec, but it can play with audio/mpeg which is RAW mp3 container. Here comes the first issue: in Chrome when trying to seek on timeline where chunks are not loaded yet, it hangs. Maybe the problem here is that mp3 frames do not have timestamps and browser expects stream to be consistent. After seek video.buffered property shows only one buffered range, but in Firefox they are two - for current and seek.

Another issue is with Microsoft Edge browser. I can't play mp3 sound there at all. It supports ISO/IEC 11172-3 (audio/mp4; codecs="mp4a.69"), ISO/IEC 13818-3 (audio/mp4; codecs="mp4a.6B") and ISO/IEC 14496-3 (audio/mp4; codecs="mp4a.40.34") as result from MediaSource.isTypeSupported. I tried three of codecs and always get media error MEDIA_ERR_SRC_NOT_SUPPORTED after feeding data to the SourceBuffer. The interesting thing here is that I wrote data to file with codec mp4a.6B and file is playable as src attribute in audio tag, but gives same error when trying to play it through MSE. I found similar issue https://developer.microsoft.com/en-us/microsoft-edge/platform/issues/8972303/ and could be some kind of bug in the Edge browser or I'm missing something?

Any comments and tips are much appreciated.

Mladen

[sompylasar]

@nochev Great work so far. Can't help with the issues though, sorry.

[mangui]

nice work, congrats @nochev ! pinging @dalecurtis for Chrome mp3 support through MSE.
regarding MS Edge, if anybody has any contact @ Microsoft, it would be great to put them into the loop.

[nochev]

I posted the issue in Edge forum, so hope the guys from MS will give us some tips. Here is the test page.

[wolenetz]

I work on Chrome MSE (and am MSE spec co-editor).
Raw mp3 container is supported on Chrome, and per the bytestream spec registry entry for it (https://www.w3.org/TR/mse-byte-stream-format-registry/), user agents auto-generate timestamps and update SourceBuffer.timestampOffset during coded frame processing as part of appendBuffer() operation. To place an mp3 stream explicitly where you want in the MSE timeline, try the following:

• (in case of a previous mp3 appendBuffer leaving the parser in the middle of "PARSING_MEDIA_SEGMENT" state, best practice is to reset the parser): SourceBuffer.abort() (and await 'updateend' if 'updating' was true at time of issuing abort() call), then:
• SourceBuffer.timestampOffset = desired point in the timeline where the next frames should be appended
• SourceBuffer.appendBuffer(mp3 byte stream). Upon 'updateend', observe that timestampOffset has been adjusted automatically (per the spec).

Chrome has a few known issues around this:

• https://crbug.com/607372 : Though Chrome internally correctly treats mp3 as 'sequence' appendMode, it doesn't expose this implicitly overridden mode via the SourceBuffer.mode attribute (yet). Neither does it disallow changing the mode for mp3 bytestreams.
• https://crbug.com/592889: canPlayType() incorrectly reports "probably" for many containers with "mp3" codec.
• https://crbug.com/587303: we should unify our media mime type parsing code between isTypeSupported and canPlayType, and consider including mp4a.6B and mp4a.69 as allowable audio codecs for {audio,video}/mp4 in MSE.

[wolenetz]

I've also filed https://crbug.com/661749 to track adding mp3-in-mp4 support to Chrome MSE.

[nochev]

Many thanks @wolenetz for detailed information about mp3 MSE behavior in Chrome. It would be great if Chrome enables support for audio/mp4 with codecs mp3 or even mp4a.6B and mp4a.69.

With suggested tips I managed to fix issue with seek in Chrome, but also I noticed that a similar issue occurs with multibitrate live streaming when switching between resolutions (current level). I tried to make a general solution for live/vod streams by comparing timestampOffset of audio buffer and PTS from parsed fragment - everything works well, except that for live streams sometimes it takes a little bit more time for resolution switch. You can see my changes in the following commit: https://github.com/nochev/hls.js/commit/2737c71b8bf7377f01be69cce14c5594de6e8e2d - any feedback and tips about it are welcome. I prepared live demo, vod demo and vod demo 2 where my last changes are applied.

Unfortunately still no development and information about the problem with MS Edge.

[mangui]

great job @nochev !
I guess black disabling MP3 support on MS Edge until it is sorted out would do the trick.

[nochev]

Thanks @mangui ! Yes, I left MS Edge to play video without feeding audio buffer, until the issue is sorted out. I removed unnecessary code for now, as you can see in the commit https://github.com/dailymotion/hls.js/pull/808/commits/ccf26355e978b8907af55f2ebfa3059dd162921f. I think the feature is currently at good stage and created a pull request https://github.com/dailymotion/hls.js/pull/808

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[tchakabam]

Hey @nochev, Do you know if the issue on MP3 codec support via MediaSource in latest Edge version (16, 17) is still present?

At the time of the last discussion I remember we were still at a very early version of Edge browser :)

The microsoft issue link you gave was marked as duplicate (see https://developer.microsoft.com/en-us/microsoft-edge/platform/issues/8972303/)

It seems that this issue has been fixed now by now :) https://developer.microsoft.com/en-us/microsoft-edge/platform/issues/13951005/

If we can verify that (I will try to check this asap) we could remove the note about Edge & MP3 from the readme :)

[nochev]

Hello @tchakabam,
As far I remember we disabled audio MP3 stream processing for Edge browser because of errors in SourceBuffer. I will check that and will do tests with latest version of Edge and will let you know.

[xqq]

I can confirm that MP3 contained in audio/mpeg has been working well on latest Edge (17.17134).
Test by flv.js with mp3 audio track.

[nochev]

I can confirm that MPEG audio works in latest Edge (17.17134). Unfortunately there is some issue with audioBuffer.timestampOffset, values are not correct and this forces changing audio timestamp offset constantly. For example in Chrome if timestampOffset is 18s in Edfe is 0.40523 ... Probably this should be reported in Microsoft too.

[thenewguy]

I am struggling with mp3 in mp4. What is the correct codec string to get Safari playback? Is cross browser support possible with 1 codec string?