Ffmpeg-python: Brainstorming on how to implement the split filter (or other variable multiple-output filters)

[Depau]

I was gonna need split but noticed it's not implemented yet.
I was wondering how we could do that. Maybe some new filter_multi_o(parent, name, number_of_outputs, *a, **kw) function that will return a list/tuple of number_of_outputs nodes, with a wrapper named split.

What do you think?

[kkroening]

I think it should happen automatically. If you refer to a stream in multiple places it should automatically generate a split under the covers.

(Actually, it's probably a bug that ffmpeg-python doesn't do this already)

What do you think - is there any reason you wouldn't want a split to appear if you're using a stream in multiple places?

[kkroening]

Example:

flipped = (ffmpeg
    .input('input.mp4')
    .hflip()
)
out = (ffmpeg
    .concat(
        flipped,
        flipped.vflip()
    )
    .output('output.mp4')
)

_(green: input, yellow: filter, blue: output)_

Since flipped is used in two places, the graph compiler should put a split filter in there for the output of the hflip filter to be used in both places.

In general, any filter that is used in more than one place (i.e. has more than one outgoing arrow) should automatically have a split created in the ffmpeg command-line filter args, and the graph compiler should keep track of sending it to the right place. The real magic of using a library like this is that it can be inferred from the graph structure, and users don't have to manually keep track of splitting+numbering/naming streams.

[kkroening]

For other multi-output filters, it could be something like

tmp = in.somefilter()
out1 = tmp[0]
out2 = tmp[1]

We could allow the split filter to be explicitly specified if someone really wants to, and then you could use the above syntax, but I think it should be optional and that the library should be able to infer that it's needed based on the graph structure (to make things less tedious for the user).

[Depau]

I agree with the fact that ffmpeg-python should also do it automatically. It should also be available for whoever want's to do something we didn't think of beforehand.

Anyway I tried to .output some stream, then add some more filters to it and output the resulting streams. ffmpeg would complain because I'm reusing it.

I like the getitem syntax (like any Python syntax-porn feature 😂), though it may cause a bit of confusion. I think a split function/method would be more readable, considering that other built-in python objects have such method (str, bytes, unicode).

[Depau]

P.S. what do you use to render graphs like this?

[kkroening]

Anyway I tried to .output some stream, then add some more filters to it and output the resulting streams. ffmpeg would complain because I'm reusing it.

We should probably make sure ffmpeg-python immediately catches this, if it doesn't already.

P.S. what do you use to render graphs like this?

yEd Graph Editor

I use the jar version

[Depau]

I'm not sure how to automatically detect potential stream-reuses, but I can work on a split filter (maybe something "internal", if you don't want to expose it in the "public" API) on Tuesday, so we can use it when we understand how to detect when it's needed.

[kkroening]

I think for now the automatic part is actually easier since it's mostly in one function: _get_filter_arg

I should put some more documentation for this, but the _get_filter_arg function generates the ffmpeg -vf ... argument by going through all of the filter (non-input/output/etc) nodes. By the time it runs, we already have a representation of the entire graph, and we can ask questions such as "how many inputs and outputs does a particular node have."

To tell if a filter node is used in multiple places, the question becomes "does the filter node have more than one outgoing edge," and if so it needs a split (for example, hflip in the example above has two arrows coming out of it so it needs split).

The child_map dict in get_args provides a mapping of all the downstream edges for each node. A filter that's reused will have len(child_map[node]) > 1. We'll need to pass child_map to _get_filter_arg and then put splits in there.. something like this:

def _get_filter_arg(filter_nodes, stream_name_map, child_map):
    stream_num = 0
    filter_specs = []
    for node in filter_nodes:
        filter_specs += [_get_filter_spec(stream_num, node, stream_name_map)]
        stream_num += 1

        children = child_map[node]
        if len(children) > 1:
            # needs `split`!
            filter_specs += [_get_split_spec(stream_num, children, stream_name_map)]
            stream_num += len(children)  # one stream-id for each child

    return ';'.join(filter_specs)

Keeping track of stream IDs is done in stream_name_map. As we go through the graph and generate filter specs, we fill in stream_name_map saying "this node has such and such stream ID," so the stream_name_map is a mapping from Node reference to output stream-id string.

The tricky thing here is that we need to keep track of stream IDs and we currently assume every node has exactly one output stream ID, and that's not true when we have splits. Let's say we have the graph in the earlier comment above:

[0]hflip[v0];
[v0]split[v1][v2];
[v1]vflip[v3];
[v2][v3]concat=n=2[v4]

It assumes nodes have only a single output stream, and that's going to have to change. Basically it has to become a (node, child_node) -> stream_id mapping. Some refactoring will have to be done, but it should be completely self contained in _run.py.

This is probably the quickest way to get it up and running since it doesn't require any API changes, but longer term we should figure out a way to support both, as well as potentially support other multi-output filters.

I think what should happen in the long run is that at the beginning of the graph compilation (get_args) we pre-process the graph and unify it so that splits are injected into the graph just like any other graph node, and if the user happened to put them in the graph already then it just skips over them. It would also be able to detect incorrectly structured graphs. But I'd say this is a little bit further down on the roadmap.

I should be able to work on this again on Tuesday or tomorrow night and sync up then.

[kkroening]

Btw, re graph rendering.. I was thinking of adding something like this to the project:

from __future__ import unicode_literals
from ffmpeg._run import _topo_sort
import graphviz

from ffmpeg.nodes import (
    InputNode,
    OutputNode,
    FilterNode,
    operator,
)


@operator()
def view(parent):
    sorted_nodes, child_map = _topo_sort(parent)
    graph = graphviz.Digraph()
    graph.attr(rankdir='LR')

    for node in sorted_nodes:
        if isinstance(node, InputNode):
            color = '#99cc00'
        elif isinstance(node, OutputNode):
            color = '#99ccff'
        elif isinstance(node, FilterNode):
            color = '#ffcc00'
        else:
            color = None
        graph.node(str(node._hash), node._name, shape='box', style='filled', fillcolor=color)
        child_nodes = child_map.get(node, [])
        for child_node in child_nodes:
            kwargs = {}
            graph.edge(str(node._hash), str(child_node._hash), **kwargs)

    graph.view('graph.png')


some_node.view()

Output:

It requires graphviz though, so it would have to be conditionally enabled based on whether graphviz is available. Or just have it as a separate package altogether ffmpeg-python-graphviz? (haven't thought about it for more than a couple of minutes)

Could be useful for debugging, if nothing else.

(Do pip install ipython graphviz and then paste the above code if you wanna try it ;replace some_node with your ffmpeg filter graph)

[Depau]

I just read this very quickly and I didn't get very well a couple points. Maybe I'll check back tomorrow or on Tuesday.

[Depau]

• Graph rendering
  I tried it and it works well (I rendered the complex filter graph example in the README):
  
  Only thing is it doesn't show input/output filenames.

• Split
  Actually I have a deadline for a project I'm working on, and it's kind of close. We need to deploy soon, so for the time being I won't help you on this particular issue and try to avoid this kind of situation. We're making a web-based video editor, the browser "fakes" all the effects, sends a request to our web server which in turns requests an editing job to a dedicated server that runs the program I'm working on. This program uses this module (BTW I'll have to check the Apache license terms to protect your intellectual property as this is likely not going to be open-source :/ ) to process the job, generating multiple quality outputs + thumbnails.
  The configuration that triggers this issue happens when I try to output thumbnails too, because I apply an fps filter to output one every # seconds.
  For the moment I'll just try to output thumbnails separately, then as soon as we get the basic editing features working I'll have time to help you on this and other issues.

This is a sample graph that needs split:

P.S.: I changed your code to this to display filenames:

@operator()
def view(parent):
    sorted_nodes, child_map = _topo_sort(parent)
    graph = graphviz.Digraph()
    graph.attr(rankdir='LR')

    for node in sorted_nodes:
        name = node._name
        if '_kwargs' in dir(node) and 'filename' in node._kwargs:
            name = os.path.basename(node._kwargs['filename'])
        if isinstance(node, InputNode):
            color = '#99cc00'
        elif isinstance(node, OutputNode):
            color = '#99ccff'
        elif isinstance(node, FilterNode):
            color = '#ffcc00'
        else:
            color = None
        graph.node(str(node._hash), name, shape='box', style='filled', fillcolor=color)
        child_nodes = child_map.get(node, [])
        for child_node in child_nodes:
            kwargs = {}
            graph.edge(str(node._hash), str(child_node._hash), **kwargs)

    graph.view('graph.png')

[kkroening]

We're making a web-based video editor, the browser "fakes" all the effects, sends a request to our web server which in turns requests an editing job to a dedicated server that runs the program I'm working on.

That's way cool. Exactly the kind of thing I was hoping someone would eventually use this for. It's awesome to see someone actually doing it.

For the moment I'll just try to output thumbnails separately, then as soon as we get the basic editing features working I'll have time to help you on this and other issues.

Yeah, for now duplicating the entire upstream portion of the graph and running separate ffmpeg commands should work. I'll work on this today though and hopefully have something good out soon. I'll keep you posted.

I changed your code to this to display filenames:

Great idea. I'll incorporate that when I make a PR for this.

[Depau]

That's way cool. Exactly the kind of thing I was hoping someone would eventually use this for. It's awesome to see someone actually doing it.

I might eventually remake an open-source implementation of it at some point, it'd be a good thing to have for free.

(everything else)

Awesome :)

[kkroening]

It is done: https://github.com/kkroening/ffmpeg-python/pull/20

[kkroening]

Also #21 for graph visualization.

[Depau]

(I just wanted to let you know that I'm not dead, we had some issues with other components of our project and had to pause the video-processing-related part. I'll get back to it in a couple weeks.)

[kkroening]

Haha, okay I'll call off the ambulance. Just kidding :)

Thanks for your contributions. I'm going to work on the audio stream stuff in the next couple of days (#26). Feel free to let me know any other issues or share thoughts on the approach for audio support.