Server: FFMPEG Producer Auto Conversion

[ronag]

To what degree do we want CasparCG to perform auto conversion? This is a balance between "play anything" and "play with small and/or cheap hardware".

Trust interlaced flag?

Sometimes the interlace flag of files is incorrect (common with files from e.g. bloomberg). We can automatically detect and correct this at a performance cost.

De-interlace

Should we automatically de-interlace to progressive?

Framerate conversion of interlaced video?

This one is tricky since interlaced video does not lend itself well to framerate conversion. We basically need to deinterlace or deinterleave, convert framerate and then re-interlace. A quite costly operation which is not always possible to avoid. Performance implication depends on whether we want to trust the reported framerate (which is not always correct) or not...

I guess we could hide these options under some global configuration option...?

Ping @5opr4ni @dotarmin @TomKaltz

[premultiply]

I think CasparCG should handle only progressive full framerate RGBA video inside to make compositing easy and clean. So all producers should convert to this format if necessary. And all consumers need to convert again if their destination format does requiere it.

The metadata of the sources (files, streams or devices) must be trusted.
Shure, if frames are not flagged as interlaced the will be handled as progressive. The file or stream is broken if interlaced frames are flagged as progressive. But it is the same problem if other metadata like field dominance, orientation, aspect ratio, framerate, colorspace or colorspace is wrong. Something will be wrong...
And yes there may be some sophisticated filters which try to detect some of these errors by analysing the pictures but they may even do lots of errors and also use lots of ressources. Please do not use them by default. Keep it simple and clean. If anyone needs this stuff he can manually add filters by sending individual AMCP commands.

But if you feel better with it it may be an idea to provide a configuration option to adjust the last two parameters of the bwdif interlacer. It may be better to force the adaptive _bwdif_ deinterlacer to deinterlace all frames regardless of the flag as to use a seperate _idet_ filter.

[TomKaltz]

I think we should do the "right thing" as far as de-interlacing and
frame-rate conversion based on the best guess from the metadata but also be
able to disable certain high cost conversions via config params if we the
user doesn't have the hardware firepower.

Thomas R. Kaltz III

[phone] cell

On Sat, Jan 13, 2018 at 3:45 PM, premultiply [e-mail]
wrote:

I think CasparCG should handle only progressive full framerate RGBA video
inside to make compositing easy and clean. So all producers should convert
to this format if necessary. And all consumers need to convert again if
their destination format does requiere it.

The metadata of the sources (files, streams or devices) must be trusted.
Shure, if frames are not flagged as interlaced the will be handled as
progressive. The file or stream is broken if interlaced frames are flagged
as progressive. But it is the same problem if other metadata like field
dominance, orientation, aspect ratio, framerate, colorspace or colorspace
is wrong. Something will be wrong...
And yes there may be some sophisticated filters which try to detect some
of these errors by analysing the pictures but they may even do lots of
errors and also use lots of ressources. Please do not use them by default.
Keep it simple and clean. If anyone needs this stuff he can manually add
filters by sending individual AMCP commands.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/CasparCG/Server/issues/672#issuecomment-357465981,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AC0_IcvzScvx4594wUR_lWpdeEjd-Op9ks5tKRXmgaJpZM4RdWxt
.

[jesperstarkar]

What I wish for, perhaps not in 2.1, but in the future, is that de-interlacing will be bypassed if not needed.

If you have interlaced input, interlaced output and NO mixer effects or other things going on that need full frames to calculate, it would be cool to see sources (FFmpeg and SDI inputs) going straight through without the de-interlace + re-interlace roundtrip.

This is just wishful thinking from my side, I have no knowledge of the codebase, processing pipeline or i.e. Decklink APIs to back this concept with.

[Julusian]

I think CasparCG should handle only progressive full framerate RGBA video inside to make compositing easy and clean.

At the moment I believe the compositor is interlacing each layer as it gets rendered to to a texture, before then flattening the layer down to the final channel image. From what I've seen, the compositing doesn't really care either way, as it is done after the transforms at which point only layer blending is left to do and so is unaffected.

What I wish for, perhaps not in 2.1, but in the future, is that de-interlacing will be bypassed if not needed.

I too would like this.

I think it is reasonable to trust the metadata is correct, and to have a parameter to the LOAD/PLAY command to force enable deinterlacing.
I suppose we should also consider playback speed as part of framerate conversion, as that will be doing a very similar thing. I think it would be fine to deinterlace and reinterlace as needed when it comes to converting the framerate. If it is too costly to the user then they can always convert the file to the correct format themselves before loading it into caspar.
And again I think we should trust the metadata for the framerate as that sounds like a very rare edgecase. I do remember seeing a couple of reports of the framerate claiming to be 0 when playing a stream, so perhaps when it looks very wrong it could be calculated instead. Also, what would happen during normal playback if the framerate metadata is wrong?

Could we precalculate these fields before playback begins? Would it help to make it happen as part of the LOAD, and if the video isnt pre-loaded then we just assume the data is accurate? It is already recommended to LOAD before PLAY, and this way we should be able to reduce the impact of actual playback.
Alternatively, it sounds like something we could do as part of populating the CINF cache in the background, or while generating thumbnails. It would need to be heavily rate limited, but if we can do the checks with the first few seconds of the file then that might be a better solution and allow for more optimal playback.

[premultiply]

For any transform (even simple scaling to other size) it is requiered to deinterlace and upsample (25i to 50p for example) any interlaced input first. Framerateconversion is just another transform of many others. This is also the way any professional equipment is working. Even very expensive commercial playout videoservers or standalone up-down-cross-converters do it this way.
And in CasparCG there are many transforms on the mixer stage which are applied to the the layer contents. And I am pretty shure it is already implemented like this. Please remember that CasparCG is more then a simple videoserver...

Deinterlacing or better say upsampling ("double" fields to frames) and re-interlacing (removing the doubled half of every frame) without any other transformation itself is nearly lossless. So it does not make sense to have separate and complicated processing paths for interlaced and progressive material.
The bwdif adaptive deinterlacer/upsampler is very high quality. It beats many professional implementations which can be found in NLE software, processing devices or displays. bwdif was implemented by TV professionals.

Even for playback of DV-video (reversed field order) it is a good solution to deinterlace it so full framerate first. The other way you have to shift the image by one line, delay the audio tracks by 1 frame or start/end with black field. And in CasparCG you would like to mix it with other files or input with different resolutions, progressive frames, other framerates... And at the end you want to sent it to different sdi cards, screen consumers, stream encoding and file recorders in parallel which all have different video properties.

So I hope you see that it is not a good idea to have an interlaced processing path. CasparCG has to many features. :-)

[ronag]

Here is the automatic postfix graph for video if anyone has any comments.

There is a fast path for interlaced output when input and output framerate guess matches. This won't work 100% time but should be ok most of the time.

const auto frame_rate = q2r(decoder_->framerate);

// TODO Do we want a fast path?
if (frame_rate == format_desc.framerate && format_desc.field_count > 1) {
    // NOTE CasparCG mixer does not support interlaced scaling.
    // TODO This will currently scale progressive input as well. Modify vf_scale?
    filter_spec += (boost::format(",scale=%d:%d:interl=-1")
        % format_desc.width % format_desc.height
    ).str();
} else {
    filter_spec += (boost::format(",idet,bwdif=mode=%s:parity=auto:deint=interlaced")
        % (frame_rate == format_desc.framerate ? "send_frame" : "send_field")
    ).str();

    filter_spec += (boost::format(",fps=fps=%d/%d")
        % (format_desc.framerate.numerator() * format_desc.field_count) % format_desc.framerate.denominator()
    ).str();

    if (first_pts_ != AV_NOPTS_VALUE) {
        filter_spec += (boost::format(":start_time=%f")
            % av_q2d(AVRational{ first_pts_, AV_TIME_BASE })
        ).str();
    }

    if (format_desc.field_count == 2) {
        // TODO CasparCG mixer could do this.
        filter_spec += (boost::format(",scale=%d:%d,interlace=scan=")
            % format_desc.width % format_desc.height
            % (format_desc.field_mode == core::field_mode::upper ? "tff" : "bff")
        ).str();
    }
}

TODO

• [ ] Auto letter box 4:3?
• [ ] Move some stuff to the GPU Mixer (e.g. the scale + interlace step)
• [ ] Use framerate filter instead of fps filter? Currently, framerate filter does not support alpha.
• [ ] Evaluate whether idet is worth it or not.

[ronag]

What's the consensus regarding framerate vs fps filters?

👍 for framerate
👎 for fps

[ronag]

What's the consensus regarding idet filter?

👍 for idet
👎 for no idet

[premultiply]

Imho better to do
_bwdif=mode=%s:parity=auto:deint=all_
(which is the default) and skip the _idet_ filter.

[ronag]

@premultiply that will cause progressive frames to be deinterlaced as well... which we don't want.

But we can skip the idet filter if we want.

[ronag]

What´s the consensus regarding the interlaced output fast path?

👍 for fast path
👎 for no fast path

[ronag]

Going with @premultiply we would end up with something like.

filter_spec += (boost::format(",bwdif=mode=send_field:parity=auto:deint=interlaced")
).str();

filter_spec += (boost::format(",fps=fps=%d/%d")
    % (format_desc.framerate.numerator() * format_desc.field_count) % format_desc.framerate.denominator()
).str();

if (first_pts_ != AV_NOPTS_VALUE) {
    filter_spec += (boost::format(":start_time=%f")
        % av_q2d(AVRational{ first_pts_, AV_TIME_BASE })
    ).str();
}

if (format_desc.field_count == 2) {
    filter_spec += (boost::format(",scale=%d:%d,interlace=scan=")
        % format_desc.width % format_desc.height
        % (format_desc.field_mode == core::field_mode::upper ? "tff" : "bff")
    ).str();
}

[premultiply]

@premultiply that will cause progressive frames to be deinterlaced as well... which we don't want

@ronag I think that will be no difference. At the end the deinterlacer will not do any visual "deinterlacing" on progressive frames after the motion estimation as there is no "interlaced motion" even it is told by all. But I will check tomorrow and talk to the bwdif developer. Autoskipping (less CPU load) of progressive frames may be requested by interlaced.

[ronag]

@premultiply interesting observation... I need to think about it a little... it has performance implications but if we are going for correctness over performance that might be the correct direction.

[Julusian]

@ronag What is the performance cost of doing this? In particular, knowing the impact relative to 2.1beta2 or 2.0.7 would be useful. Because, if performance is similar (within a few % cpu), then I have no problems with doing it all, but if it is a significant increase for many video formats then it would be worth discussing further.

Something else to remember is that we can always come back to this in a later version to optimize it or add config options.

[walterav1984]

@ronag

This is a balance between "play anything" and "play with small and/or cheap hardware".

I really like the intention of this, but might a performance gain/loss and quality gain/loss "matrix" be a better solution and choose your processing path from there? Because more (over)processing steps might not always improve quality (which is subjective) since it can be a tradeof for example less scaling/compositing artifacts vs loss of detail because of de/re-interlacing?

@premultiply

Please remember that CasparCG is more then a simple videoserver...

This is also very true, there is the option of correct interlaced native YUV 422 playback with mlt-framework (which is very limited in flexability with compositing/effects) since casparcg always does YUV>RGB>YUV if I'm not mistaking. Maybe define two kind of operations/focus for casparcg server like prioritize broadcast interlaced which assumes correct crafted/flagged assets/footage with less correction steps vs flexible throw in any footage format fps/resolution/field/frame for progressive RGB output with the condition of loss in detail but smoother playback?

@jesperstarkar

If you have interlaced input, interlaced output and NO mixer effects or other things going on that need full frames to calculate, it would be cool to see sources (FFmpeg and SDI inputs) going straight through without the de-interlace + re-interlace roundtrip.

I'd like the idea of no de/re-interlacing when there are no further compositing operations, but non-square pixel aspect ratios like HDV 1440x1080 and some XDCAM just need horizontal scaling and I would like that also without going through de/re-interlacing.

However most important for interlacing imo, when will interlaced chroma upsampling be introduced in this discussion this was a real show stopper for me to stop looking at casparcg as professional until I knew a workaround. Will this interlaced chrome upsampling also be introduced in this auto path @ronag ?
https://github.com/CasparCG/Server/issues/328
https://github.com/CasparCG/Server/issues/41

I think we'll agree that for pixel/fps/color perfect operation it means that all input format properties will be the same as output, but we also know this is not the reality most live in.
There will be SD footage to be played back in HD setups and the other way around, also progressive in interlaced and the other way around... but how to focus and implement and discuss with respect to each usage is very challenging. Thanks for the opportunity for asking input.

[ronag]

However most important for interlacing imo, when will interlaced chroma upsampling be introduced in this discussion this was a real show stopper for me to stop looking at casparcg as professional until I knew a workaround. Will this interlaced chrome upsampling also be introduced in this auto path

This will be fixed.