mpv drops frames at VO for specific AV1 WebM

[CounterPillow]

mpv version and platform

mpv 0.28.0-638-g08a6827b3d Copyright © 2000-2018 mpv/MPlayer/mplayer2 projects
 built on Fri Jul 20 19:07:42 CEST 2018
ffmpeg library versions:
   libavutil       56.18.102
   libavcodec      58.21.105
   libavformat     58.17.101
   libswscale      5.2.100
   libavfilter     7.26.100
   libswresample   3.2.100
ffmpeg version: N-91498-g9888a19db4

Arch Linux x86_64

Reproduction steps

1. Play back mpv --no-config https://fratti.ch/tmp/wanderers.webm
2. Watch the VO frame drops
3. Try a different VO, notice how it drops there too

Expected behavior

Don't drop frames, like vlc, Chromium and ffplay do.

Actual behavior

Severe VO framedrops

Log file

https://0x0.st/sVHf.log

Sample files

• https://fratti.ch/tmp/wanderers.webm
• mp4 remux exhibiting the same issue: https://0x0.st/sVHO.mp4

[mc4man]

So why does the main website where your sample is show as being seized by the nsa?

[Hrxn]

With which Chromium versions is this supposed to work?

[CounterPillow]

@Hrxn ones which have a devendored ffmpeg that supports AV1, like the one Arch Linux ships.

[aufkrawall]

I only get one frame drop in the very beginning (nothing unusual) and video plays fine otherwise for me on Windows and Linux.

[aufkrawall]

Odd thing: I'm getting lots of delayed frames with lachs0r's build, but not with shinchiro's.

[mia-0]

@aufkrawall: Try the latest one.

@CounterPillow: Well. Consider the following:

1. Both VLC and Chromium use vendored libaom in Arch
2. ffplay might not implement frame dropping/synchronization (I don’t know)
3. Arch’s libaom package fails to enable platform-specific optimization (I blame aom’s dumb build system but openSUSE’s multimedia maintainers pointed this out to me right after I had submitted a package with the same mistake)

I was able to reproduce the judder on my system with my own Windows builds until I changed the build options. For the record, see this spec file:

https://build.opensuse.org/package/view_file/multimedia:libs/libaom/libaom.spec?expand=1

[aufkrawall]

Yep, fine now.

[Hrxn]

@lachs0r

Small nit, don't want to open an issue for this:

PS F:\Tools> mpv --V
mpv 0.29.0 Copyright © 2000-2018 mpv/MPlayer/mplayer2 projects
 built on UNKNOWN
ffmpeg library versions:
   libavutil       56.18.102
   libavcodec      58.22.100
   libavformat     58.17.101
   libswscale      5.2.100
   libavfilter     7.26.100
   libswresample   3.2.100
ffmpeg version: 4.1+809.g269daf5985

PS F:\Tools> mpv
mpv 0.29.0 Copyright © 2000-2018 mpv/MPlayer/mplayer2 projects
 built on UNKNOWN
ffmpeg library versions:
    [...]
    [...]

built on UNKNOWN

Not sure, but probably not intentional? Older builds definitely had their date in that place..

[CounterPillow]

@Hrxn that's to make the builds reproducible, AFAIU.

[Hrxn]

Good to know, thanks.

Then please disregard.. or maybe scrap the built on UNKNOWN line entirely?

[CounterPillow]

@lachs0r I've built aom with optimisations now and that didn't change a thing about mpv's framedrops.

[qyot27]

Since this didn't go through last night when I tried to post it:

--disable-build-date is what results in 'built on UNKNOWN'. I think it was one of the Debian maintainers that wanted to be able to disable it for determinism reasons or something. Point releases may or may not default to using that option, hence that output. Git builds show the date.

Some years back there was a bug report that wanted it to be more comprehensive and if the --disable-build-date option was used, for the 'built on' line to be omitted completely. I didn't particularly care to open a pull request just for that, but I did whip up a patch to do it.

EDIT: I went ahead and opened a pull request.

[mc4man]

Seems rather finicky about dropped frames. The linked vid was pretty short so did some local encodes of a bit longer. Seems minor changes to encoding options can create more or less dropped frames in mpv, don't have any other player to compare.
As ex. these 2 vids - the 30 sec one I get no dropped, the 45 sec. I get 11. If I encode diffently the dropped can increase for both.
Both done thru ffmpeg, multithreaded encoding, about 30% of orig. bitrate/file size, 2 pass. 2nd pass options for 30 sec, -

ffmpeg -hide_banner -nostdin -y -i "file:/home/doug/Videos/Special Edition Trailer-avatar-specialeditiontrailer.mov" -map 0:v:0 -map 0:a:0 -threads 8 -c:v libaom-av1 -cpu-used 4 -strict experimental -tile-columns 3 -tile-rows 3 -b:v 3814k -auto-alt-ref 1 -lag-in-frames 25 -g 128 -pix_fmt yuv420p -c:a libopus -b:a 128k -f webm -pass 2 -passlogfile /tmp/boram-198764pEX5Nbjj2ik -metadata title=Avatar "file:/home/doug/Desktop/Special Edition Trailer-avatar-specialeditiontrailer.webm" Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'file:/home/doug/Videos/Special Edition Trailer-avatar-specialeditiontrailer.mov'

https://0x0.st/sxjk.tar
(- both files in a .tar folder, if uploaded & then downloaded as .webm files then both showed increased frame drops when played, from the downloaded/extracted .tar they behave as the local ones, ..no idea why that

[Kagami]

I have the same issue with AV1 1080p of this video from AV1 Beta Launch Playlist.

$ mpv --version            
mpv 0.29.0-30-g6eb59fea2f-dirty Copyright © 2000-2018 mpv/MPlayer/mplayer2 projects
 built on Sun Sep 16 23:58:12 MSK 2018
ffmpeg library versions:
   libavutil       56.19.101
   libavcodec      58.30.100
   libavformat     58.18.101
   libswscale      5.2.100
   libavfilter     7.32.100
   libswresample   3.2.100
ffmpeg version: N-91961-g5109c38162
$ aomdec --help | grep av1
    av1    - AOMedia Project AV1 Decoder 1.0.0-590-g6fa400604

$ youtube-dl -f 399 -o dua.mp4 k2qgadSvNyU
$ time ffmpeg -v warning -i dua.mp4 -frames:v 250 -f null -
269% cpu 4.216 total

As you can see, it takes only 4.2 seconds to decode 10 seconds of video (25 fps) on my machine but when I'm trying to play it with mpv I get a lot of VO frame drops.

[aufkrawall]

This issue is still present with latest ffmpeg + mpv builds (both Windows & Linux).
Open
mpv https://www.youtube.com/watch?v=2nXYbGmF3_Q
with
ytdl-format="((bestvideo[vcodec=av01.0.05M.08]/)+bestaudio)/best"
and notice stutter and frames reported as mistimed.

[veikk0]

I have a theory that AV1 decoding is limited to a single thread since FFmpeg doesn't support multithreaded encoding for AV1 either. So there's a precedent there.

To test it, download a 1080p AV1 video from Youtube:

youtube-dl -f 399 https://www.youtube.com/watch?v=mIAfxj7nd9k -o av1test.1080p.mp4

Then play it with mpv and record the CPU use of the process with time:

ThinkPad-T420:~/Videos/av1_yt$ /usr/bin/time -f "\ntime\t%E,\nCPU\t%P,\nRAM\t%Mk" mpv av1test.1080p.mp4 
Playing: av1test.1080p.mp4
[ffmpeg/demuxer] mov,mp4,m4a,3gp,3g2,mj2: Unknown AV1 Codec Configuration Box version 129
 (+) Video --vid=1 (*) (av1 1920x1080 23.976fps)
VO: [gpu] 1920x1080 yuv420p
V: 00:00:22 / 00:02:38 (14%) Dropped: 149 Cache: 136s+24MB
[input] No key binding found for key 'Ctrl+Alt+SPACE'.
V: 00:02:38 / 00:02:38 (99%) Dropped: 905 Cache:  0s


Exiting... (End of file)

time    3:20.06,
CPU 89%,
RAM 313636k

Playback was stuttery with a large number of dropped frames and took 3 minutes 20 seconds to complete for a 2 minute 38 second video. And as you can see, CPU usage was only 89% of the 400% maximum on a 2-core 4-thread CPU. While observing with htop I saw only 2 mpv processes, one with CPU usage below 100% and another that fluctuated from about 2% to 6.6%. Normally you'd see many more processes, when decoding 1080p VP9 for example I see at least 6 processes.

Now that the latest version of Firefox (v. 63) also supports AV1 decoding we can test that too. The media.av1.enabled flag needs to be set to true in about:config for it to work. From playing back the video in a new tab and observing the CPU usage from htop, it seems Firefox is using about 4 threads/processes for playback and their combined CPU usage is 200%+. Playback is smooth with no apparent frame drops, though there seems to be a bug where the rendering of the video playback can stop in a specific frame but according to htop's CPU usage the video is still being decoded in the background.

If you can't replicate the test result, then you probably aren't using a dual-core laptop CPU from 7 years ago (mine is an i3-2520M) and can just brute-force through the threading issue. In that case you can try one of Elecard's Holi festival AV1 test files. I'd imagine even the beefiest CPU will run into issues when trying to play the 2160p version on one thread. I can't even play the 720p 3.9 mbps file without dropped frames on mpv, while Firefox is again able to play it just fine.

[mc4man]

At least here mpv & ffplay are ok with av1 files encoded with libaom0, particularly if done thru ffmpeg, higher rez's can have some dropped frames (I'm ok to 1080p. )
The elecard files 1080p or less play back ok.

However the youtube ones don't, neither in mpv or ffplay. They do in vlc but vlc does not use use ffmpeg libs for decoding av1. What's the difference between local, elecard & youtube I've no idea..

As far as mt encoding in ffmpeg, that is quite possible, ie. with -tile-columns & -tile-rows , ffmpeg needs to be patched for -tile-rows.
Additionally here for any locally encoded av1 or the elecard samples mpv & ffplay decoding is multithreaded. Again not so for the youtube samples..

Maybe by next year there will be a better decoder & mpv/ffmpeg will perform better. (dav1d

[richardpl]

Why this is still open, mpv does not have own decoders any more...

[Kagami]

Because ffmpeg decodes video faster than realtime but there are frame drops in mpv?

[CounterPillow]

@richardpl maybe you should actually try to reproduce this before making a stupid comment; other applications using the same decoder work fine, and mpv drops frames at vo.

[aufkrawall]

Issue disappears when using libdav1d instead of libaom.

[richardpl]

Off course.... I told you so.

[CounterPillow]

Closing this, since it's probably a combination of the following:

• libaom is slow
• framedrops appear as VO instead of decoder because mpv in recent versions doesn't drop frames at the decoder anymore (my main source of confusion attributing this to mpv)
• Chromium/ffplay probably don't properly implement frame dropping or something

With dav1d I still get stutters but they're less severe, so I guess it is just the decoder after all.

[aufkrawall]

Do you get any anomalies reported? If I'm not mistaken, libdav1d is entirely free of stutter for me. Would have to test this more thoroughly though.

[CounterPillow]

I still get delayed frames, but that's likely because this is a 2 core 4 thread laptop.

[sergeevabc]

mpv 0.32.0-515-g685bd6a5f5
https://x0.at/HKo.mp4 AV1, 720p, 5.5 MiB
gpu-hq, deband=no
getting frame drops… aggrrrhhh

[aufkrawall]

Plays fine on Windows here. Open a new ticket and don't ignore the issue template, thanks.

[sergeevabc]

For anyone looking for a quick solution, vd-queue-enable=yes helped in my case.
Further investigation revealed a few threads with the same, and even tuned, solution.
However that amateur tuning seems to a bit of overkill, official advice would be appreciated.

[ghost]

Is suspect using a higher value for --vd-lavc-threads=N has the same or better effect.