Mpv: Multi input playback using lavfi-complex not working as expected

[kydouglas]

using the latest release I am looking to create a 12 video mosiac from multiple inputs since i cannot tile and autofit multiple players via script using mac. I have observed suggestions using something like this but it is not working just for 2 let alone 12. any suggestions on how I can make this function. also a lavfi-complex example for 8,12, or 16 tiles would also be very appreciated if it is possible.

mpv --external-file= --lavfi-complex='[vid1] [vid2] hstack [vo]'

output error:
[ffmpeg] Parsed_hstack_0: Input 1 height 252 does not match input 0 height 720.
[ffmpeg] Parsed_hstack_0: Failed to configure output pad on Parsed_hstack_0
failed to configure the filter graph

[richardpl]

Have you ever read documentation of FFmpeg filters?

You can not horizontally stack videos of different height, If you really need to stack videos of different sizes then resize them to same video size, before doing hstack.

[ghost]

Yes, reading error messages and docs helps. This is basically a libavfilter shortcoming, nothing we can do about. (Other than providing functions to automatically create a filter strings.)

Also, 12 videos are probably going to be slow to decode, stack, and render...

[ghost]

Oh, and the solution is of course resizing or padding the videos before stacking. Which will also be slow.

[kydouglas]

I searched for that error message in ffmpeg docs and some other forums but came up empty. I see im not the only one that had an issue or questioned if it should be a feature request, I wanted to open a thread to get more feedback to be sure i'm not missing a simpler way to approach this but it looks like I just get RTFM. Thanks for suggestion on resizing and padding and any other options others use to tile 12 low quality streams on high end hardware using an open solution id be interested as so far mpv is my goto but loading multiple players seems less than ideal.

[kydouglas]

Below is working example to output 2 x 2 with same videos that I used in last example. Still working on this for mpv but wanted to post what worked for me using ffmpeg

ffmpeg \
-i url_0 \
-i url_1 \
-i url_2 \
-i url_3 \
-filter_complex \
        "nullsrc=size=640x480 [base];\
        [0:v] setpts=PTS-STARTPTS, scale=320x240 [upperleft];\
    [1:v] setpts=PTS-STARTPTS, scale=320x240 [upperright];\
    [2:v] setpts=PTS-STARTPTS, scale=320x240 [lowerleft];\
    [3:v] setpts=PTS-STARTPTS, scale=320x240 [lowerright];\
    [base][upperleft] overlay=shortest=1 [tmp1];\
    [tmp1][upperright] overlay=shortest=1:x=320 [tmp2];\
    [tmp2][lowerleft] overlay=shortest=1:y=240 [tmp3];\
    [tmp3][lowerright] overlay=shortest=1:x=320:y=240"\
-c:v libx264 output.mkv

[richardpl]

overlay is sloow

[kydouglas]

hstack/vstack does seem improved but trying to stack more than 2 params isnt working suggestions on how to 3x3 or 4x4 or docs that would explain this function for more than a 2x2 mosaic?

cmd = """
ffmpeg \
-i '{0}' \
-i '{1}' \
-i '{2}' \
-i '{3}' \
-filter_complex \
"[0:v] setpts=PTS-STARTPTS, scale=320x240[vid1];\
 [1:v] setpts=PTS-STARTPTS, scale=320x240[vid2];\
 [2:v] setpts=PTS-STARTPTS, scale=320x240[vid3];\
 [3:v] setpts=PTS-STARTPTS, scale=320x240[vid4];\
 [vid1][vid2]hstack[top];\
 [vid3][vid4]hstack[bottom];\
 [top][bottom]vstack"\
 -c:v libx264 videos/output.mp4 -y
""".format(url[0], url[1], url[2], url[3])

[wiiaboo]

But really, have you tried actually reading the documentation? You can stack more than two videos.

[kydouglas]

you guys should just label this trolling not question. the docs around this function are poor and when i use this with --lavfi-complex= for mpv i just get the option to scroll through broken videos. If anyone has positive feedback that would be more ideal

9.74 hstack
Stack input videos horizontally.

All streams must be of same pixel format and of same height.

Note that this filter is faster than using overlay and pad filter to create same output.

The filter accept the following option:

inputs
Set number of input streams. Default is 2.

shortest
If set to 1, force the output to terminate when the shortest input terminates. Default value is 0.

9.169 vstack
Stack input videos vertically.

All streams must be of same pixel format and of same width.

Note that this filter is faster than using overlay and pad filter to create same output.

The filter accept the following option:

inputs
Set number of input streams. Default is 2.

shortest
If set to 1, force the output to terminate when the shortest input terminates. Default value is 0.

[wiiaboo]

Documentation looks pretty clear to me. Default inputs is 2, so if you want to have more you have to pass the number to inputs=n. If you think the documentation is not clear, you're welcome to send patches to clear it up to ffmpeg-devel mailing list.

Or you can just assume this is trolling and go ask for help somewhere else. Whatever floats your boat.

[kydouglas]

Just in time to get it working thats the missing piece. I was not sure where in syntax inputs was required and still see an issue using this with mpv as it just displays a single video on input. Ill keep trying this and post if i get it cleared up.

# FFMPEG COMMAND TESTING
cmd = """
ffmpeg \
-i '{0}' \
-i '{1}' \
-i '{2}' \
-i '{3}' \
-i '{4}' \
-i '{5}' \
-i '{6}' \
-i '{7}' \
-i '{8}' \
-filter_complex \
"[0:v] setpts=PTS-STARTPTS, scale=512x288[vid1];\
 [1:v] setpts=PTS-STARTPTS, scale=512x288[vid2];\
 [2:v] setpts=PTS-STARTPTS, scale=512x288[vid3];\
 [3:v] setpts=PTS-STARTPTS, scale=512x288[vid4];\
 [4:v] setpts=PTS-STARTPTS, scale=512x288[vid5];\
 [5:v] setpts=PTS-STARTPTS, scale=512x288[vid6];\
 [6:v] setpts=PTS-STARTPTS, scale=512x288[vid7];\
 [7:v] setpts=PTS-STARTPTS, scale=512x288[vid8];\
 [8:v] setpts=PTS-STARTPTS, scale=512x288[vid9];\
 [vid1][vid2][vid3]hstack=inputs=3[top];\
 [vid4][vid5][vid6]hstack=inputs=3[middle];\
 [vid7][vid8][vid9]hstack=inputs=3[bottom];\
 [top][middle][bottom]vstack=inputs=3"\
 videos/output.mp4 -y
""".format(url[0], url[1], url[2], url[3],url[4], url[5], url[6], url[7], url[8])

[kydouglas]

Heres what im doing in case its obvious to someone else

#MPV
cmd = """
mpv -v \
'{0}' \
--external-file {{'{1}','{2}','{3}','{4}','{5}','{6}','{7}','{8}'}} \
--lavfi-complex=\
"[0:v] setpts=PTS-STARTPTS, scale=512x288[vid1];\
 [1:v] setpts=PTS-STARTPTS, scale=512x288[vid2];\
 [2:v] setpts=PTS-STARTPTS, scale=512x288[vid3];\
 [3:v] setpts=PTS-STARTPTS, scale=512x288[vid4];\
 [4:v] setpts=PTS-STARTPTS, scale=512x288[vid5];\
 [5:v] setpts=PTS-STARTPTS, scale=512x288[vid6];\
 [6:v] setpts=PTS-STARTPTS, scale=512x288[vid7];\
 [7:v] setpts=PTS-STARTPTS, scale=512x288[vid8];\
 [8:v] setpts=PTS-STARTPTS, scale=512x288[vid9];\
 [vid1][vid2][vid3]hstack=inputs=3[top];\
 [vid4][vid5][vid6]hstack=inputs=3[middle];\
 [vid7][vid8][vid9]hstack=inputs=3[bottom];\
 [top][middle][bottom]vstack=inputs=3"
""".format(url[0], url[1], url[2],url[3], url[4], url[5],url[6], url[2], url[3])

[cplayer] Running hook: ytdl_hook/on_preloaded
[cplayer] Run command: hook-ack, flags=0, args=[on_preloaded]
[ffmpeg] Parsed_scale_1: w:512 h:288 flags:'bilinear' interl:0
[ffmpeg] Parsed_scale_3: w:512 h:288 flags:'bilinear' interl:0
[ffmpeg] Parsed_scale_5: w:512 h:288 flags:'bilinear' interl:0
[ffmpeg] Parsed_scale_7: w:512 h:288 flags:'bilinear' interl:0
[ffmpeg] Parsed_scale_9: w:512 h:288 flags:'bilinear' interl:0
[ffmpeg] Parsed_scale_11: w:512 h:288 flags:'bilinear' interl:0
[ffmpeg] Parsed_scale_13: w:512 h:288 flags:'bilinear' interl:0
[ffmpeg] Parsed_scale_15: w:512 h:288 flags:'bilinear' interl:0
[ffmpeg] Parsed_scale_17: w:512 h:288 flags:'bilinear' interl:0
[cplayer] filter pad without name label
[cache] Terminating cache...
[cache] Cache exiting...
[cache] Terminating cache...
[cache] Cache exiting...
[cplayer] finished playback, loading failed (reason 4)
[cplayer]
[cplayer]
[cplayer] Exiting... (Errors when loading file)
[osx] Exiting...
[ytdl_hook] Exiting...
[stats] Exiting...
[osc] Exiting...

[wiiaboo]

https://mpv.io/manual/master/#options-lavfi-complex

mpv's --lavfi-complex label of the tracks is different from ffmpeg's.

[0:v] -> [vid1], [1:v] -> [vid2], etc.

[kydouglas]

I think that is a valid syntax still since error is still present but no longer see scaling messages after removing 0:v lines, just prior to final cplayer messages. Appreciate the suggestion though.

[global] config path: 'sub' -/-> '/usr/local/bin/sub'
[global] config path: 'sub' -/-> '/usr/local/etc/mpv/sub'
[cplayer] Running hook: ytdl_hook/on_preloaded
[cplayer] Run command: hook-ack, flags=0, args=[on_preloaded]
[cplayer] filter pad without name label
[cache] Terminating cache...
[cache] Cache exiting...
[cache] Terminating cache...
[cache] Cache exiting...
[cplayer] finished playback, loading failed (reason 4)
[cplayer]
[cplayer]
[cplayer] Exiting... (Errors when loading file)
[osx] Exiting...
[stats] Exiting...
[ytdl_hook] Exiting...
[osc] Exiting...

[ghost]

I think that is a valid syntax

What makes you think that? Seriously...

[kydouglas]

because I have it working like that as long as the sources are not live hls stream. everything works using ffmpeg so ill stop taking the abuse here and use that. you guys should drop the egos a bit.

[ghost]

FFmpeg uses exactly the same code (libavfilter), but has different stream specifiers. So I don't know why you're so surprised.

I don't know why you think you're being abused either If that's the thanks we get for helping you for free and dealing with your attitude, please leave.

[chanhbc]

Thanks you #kydouglas

Below is working example to output 2 x 2 with same videos that I used in last example. Still working on this for mpv but wanted to post what worked for me using ffmpeg

ffmpeg \
-i url_0 \
-i url_1 \
-i url_2 \
-i url_3 \
-filter_complex \
        "nullsrc=size=640x480 [base];\
        [0:v] setpts=PTS-STARTPTS, scale=320x240 [upperleft];\
    [1:v] setpts=PTS-STARTPTS, scale=320x240 [upperright];\
    [2:v] setpts=PTS-STARTPTS, scale=320x240 [lowerleft];\
    [3:v] setpts=PTS-STARTPTS, scale=320x240 [lowerright];\
    [base][upperleft] overlay=shortest=1 [tmp1];\
    [tmp1][upperright] overlay=shortest=1:x=320 [tmp2];\
    [tmp2][lowerleft] overlay=shortest=1:y=240 [tmp3];\
    [tmp3][lowerright] overlay=shortest=1:x=320:y=240"\
-c:v libx264 output.mkv

[richardpl]

You are using overlay, so you will get: slower result and worse quality (by default it scales everything to yuv420p) result.

[chanhbc]

Yes, i know. i want make grid video custom

[richardpl]

It can be custom with hstack too....

[chanhbc]

If I use the above solution the output video has the default time under the shorter video duration. I want to customize the timing of the output video or the duration of the longer video. What is the solution? sorry for my bad english

[varenc]

When I use setpts=PTS-STARTPTS on a complex filter with multiple video inputs combined with hstack, the audio always gets out of sync after syncing. It's fine if there's no audio or if I start playing the file at "00:00:00", but as soon as I seek things mess up.

Instead what I've been doing is just setting fps=23.976 for each video stream. That's the original fps on all of the input files, so this has the desired effect of syncing up each video's frames so that the combined video has the same frame rate and each component video ticks a frame a the same time. But using fps here seems non-desirable since it results in some frame dups and drops, and can lead to very slight off by one frame synchronization issues.

Any ideas?

[richardpl]

you mean it get out of sync after seeking? thats normal and expected. You would need to use asetpts filter too. but i think that is out of scope of this project bug report page. so post filtergraph usage questions on ffmpeg user mailing list.

[ghost]

Why are you using setpts? Of course that will cause problems.

[varenc]

Why are you using setpts? Of course that will cause problems.

i was using setpts because that's what kydouglas included in his working solution posted here and I didn't see anyone else reporting problems with that. This issues comes up when searching for other people doing multi input playback on mpv.

When I look for ffmpeg solutions to syncing frames in multiple overlayed videos, setpts is also the only cited solution I could find. So I'd say this problem is somewhat mpv specific since for most any ffmpeg use case you just encode the video in one go and setpts works fine for that. I did try using asetpts with mpv, but ran into other issues and thought I'd try throwing a question into the void as well.

I've sort of resolved this by pre-processing the input videos with ffmpeg and the setpts=PTS-STARTPTS filter. Since the input files have identical constant frame rates, after doing that the videos will now have same time codes for every frame and I can vstack them in mpv without using fps but still have the frames synced. But the pre-processing is annoying. Though perhaps this will be helpful to anyone else that finds this on Google

[varenc]

oh wow, I might have found a better solution.

I basically just want to align the timecode for each frame in two videos that already have nearly identical frame time codes. So I just round each frame's pts to the nearest 1/fps interval. (24 in my case) and forunatenly they sync up.

--lavfi-complex="[vid1]setpts=floor(PTS*24)/24,scale=1200:900[v1];
[vid2]setpts=floor(PTS*24)/24,scale=1200:900[v2];
[v1][v2]  hstack [vo]"

Since I'm not using the STARTPTS variable in my setpts expression, seeking works fine. I believe this only works since each video's frame time codes are already nearly frame for frame identical with just a little bit of occasional jitter, so when I do the rounding the matching pair of frames align to the same PTS. But I tried this with some more varied files and it didn't work.

[ghost]

Your problem is probably that the timestamps of the different streams are slightly different (by milliseconds), and libavfilter's syncing method will skip some frames because of that?
In that case, it would be much more reliable to delay streams other than the master stream by a few milliseconds.