Handbrake: Add support for GNU/Linux VAAPI (AMD, Intel, ..) HW-Encoding

[sgothel]

Feature request

Similar to issue #259 (supporting NVidia's NVenc), hardware accelerated encoding for AMD and Intel should be supported via GNU/Linux VAAPI as supported by FFMPEG.
See https://trac.ffmpeg.org/wiki/HWAccelIntro

Similar to issue #259, this shall be based on issue #974, i.e. based on FFMPEG.

[sr55]

Intel QuickSync should, in theory, work through mfx via ffmpeg/libav so I would avoid a second code path for that.

[sgothel]

@jstebbins @sr55 currently working on this item .. on top of ffmpeg + nvenv-encoder branch
Edit: Might support VAAPI based h264, hevc and vp8/vp9 encoding.
Not so sure about the latter vp8/vp9 though, but some support seems to be available in ffmpeg sources.

[rathann]

@sr55 Wouldn't it make more sense to have a single code path for both Intel and AMD (and NVIDIA via vdpau-to-vaapi wrapper library)? Intel fully supports VAAPI as far as I know. Also, VAAPI should work on non-x86 architectures while libmfx is x86-only.

[sr55]

It wouldn't be a single code path though. You need separate implementations for Windows and Mac.

Also, general purpose API's tend to have problems. VT on Mac cripples the hardware encoders feature set, DXVA on windows is horrifically unreliable and in areas, fundamentally broken for example.

So, I'd then ask the questions:

1. Does VA-API support the full feature set of these encoders, including the filters and do we actually have up-to-date driver support (the intel site doesn't indicate support for Kaby/Coffee lake features for example)
2. Is it actually reliable?
3. Does it support zero-copy?

Since we need libmfx and the nvidia/vce sdks's via ffmpeg anyway for Windows, everything should just use those same paths so we have consistent behaviour on all platforms. That's easier to support.

We already have a wrapper around using libav encoders, so it's not a giant leap to extend that and use ffmpeg/libav's built-in support that already exists.

QuickSync is Intel Only, so it doesn't matter that it's x86 only library. Also, we don't formally support or have any intention of supporting any other hardware platforms. (Supporting what we already do is a stretch for a project with limited resources)

[sgothel]

added initial pull #1111
@sr55 your questions are partly answered and repeated/extended in the commit and pull request
It is a work in progress (WIP) and we need to solve the question I asked first :)
I believe it could become a nice added feature when it properly supports AMD and Intel.
On Debian 9.3 'stretch' I was able to get some results on AMD (erratic) and Intel (working but big file size, 'normal' as I have read)

Edit: I may add that primary motivation for VAAPI support is due to enabling AMD hw encoding using the open-source driver.
I agree with your assessment that generic API's need good maturity to compete with specific solutions, like QuickSync (Intel).
Therefor allowing VAAPI in HB (when it's working) in parallel to specific solutions might be desired. Only when VAAPI reaches parity with the individual encoding API one might consider its retirement.

[DocMAX]

When will it be done?

[sr55]

@DocMAX There is no ETA and we haven't decided whether to accept this yet, or stick to vendor specific implementations so it'll be some time away.

[mirh]

It depends on #1078 to the very least, so wait for that for starters.

[sr55]

Closing in favour of sticking to native support.
Nvenc and quicksync both have support on Linux. AMF looks like it might add support too so we'll pick that up when available.

[survivor303]

No support from AMD yet.. so perhaps this can be opened again?

[userofryzen]

for me vaapi is mature enough to reopen this.

[userofryzen]

and I have not seen any amf documentation.

[userofryzen]

oh, yes, i have found amf package in arch, but.. why this and not vaapi ones? so more standard I think, since MPV to firefox apps are using it for decoding and obs is using it for encoding one

[mathstuf]

I spent some time today looking into supporting VAAPI. I got it most of the way plumbing-wise, but it seems it runs into some assumptions HandBrake is making about its process:

• VAAPI needs an example frame uploaded before it can work, but HandBrake is creating the encoder pipeline before starting the decoder, so no frames are available yet
• plumbing the device to use needs some configuration/job storage (I currently just open the default device where needed)

My branch is here: https://github.com/mathstuf/HandBrake/tree/vaapi-support

Looking to continue discussion here since the PR requests an issue and this seems to be the issue that fits best (rather than opening yet another VAAPI-related issue).

[sr55]

We already have QuickSync, VCE and NVEnc working on Linux. We will not be accepting patches for alternate implementations for these as that's simply adds a support burden.

[mathstuf]

I don't have an nvidia card, so that isn't viable. It's an Intel chip so it doesn't have VCE (that sounds AMD-only). I don't see QuickSync options. I have a 10th gen Intel processor, so what am I missing? Is HandBrake just not showing me any hardware-accelerated options because I don't have them or that HandBrake is not detecting QuickSync support?

[sr55]

If you don't have QuickSync support, either you the build your using isn't compiled with it enabled, or there is an issue with the driver installation / setup.

Best option is to use our Flatpak builds from the website or flathub and the associated quicksync plugin. That rules out a fair amount of environmental issues and pretty much leaves:

1. Hardware disabled in bios or not supported.
2. Driver issues.

[mathstuf]

The Fedora (RPMFusion) build is linking to libmfx, so it is at least supposed to be available there. I'll try the Flatpak.

If that doesn't work, is there an FAQ/Wiki page for getting it set up properly? I see https://handbrake.fr/docs/en/latest/technical/video-qsv.html but I'd expect "driver issues" on a stock Fedora+RPMFusion (or even Ubuntu I imagine) setup to have docs for installing any packages that may be necessary there. BIOS changes are hard to document, but a mention that one should check the BIOS menu would be appreciated.

[userofryzen]

We already have QuickSync, VCE and NVEnc working on Linux. We will not be accepting patches for alternate implementations for these as that's simply adds a support burden.

Vce is not supported by newer cards of amd in Linux. It's vcn so... It doesn appear very much helpful

[mirh]

Vce is not supported by newer cards of amd in Linux. It's vcn so... It doesn appear very much helpful

You are just debating semantics, I'm pretty sure the AMF api is still the same.

If any, the thing is that it sounds dumb to support every single proprietary interface one by one, rather than the actual free standard.

[mathstuf]

Note to self, #1111 seems to have fixes for the code bridging the issues I had run into.

[userofryzen]

I spent some time today looking into supporting VAAPI. I got it most of the way plumbing-wise, but it seems it runs into some assumptions HandBrake is making about its process:

* VAAPI needs an example frame uploaded before it can work, but HandBrake is creating the encoder pipeline before starting the decoder, so no frames are available yet

* plumbing the device to use needs some configuration/job storage (I currently just open the default device where needed)

My branch is here: https://github.com/mathstuf/HandBrake/tree/vaapi-support

Looking to continue discussion here since the PR requests an issue and this seems to be the issue that fits best (rather than opening yet another VAAPI-related issue).

Will you mantain the fork? I think I am going to recommend this fork before the original ones since vaapi support is more easy to have and open source onto de systems. For arch. Manjaro ..

[mathstuf]

If someone wants to submit a PR to adapt #1111's changes to make the hardware frame context available to the encoder part (or maybe its code is just better and it should be the starting point instead? I don't know, I haven't investigated it myself yet.). I have a number of encode jobs that I'd really like to shuttle off to dedicated hardware rather than bake my CPU, but I likely won't have time to integrate the branches myself until the weekend (and iffy even if then).

I missed it because I thought it wasn't implemented because it wasn't done, not rejected, so I didn't think to search closed PRs. Supporting the umpteen proprietary APIs on each platform seems like more work than VAAPI/macOS framework/MediaFramework to me and it covers hardware "everybody" has, but I'm just a contributor here, so not much weight unfortunately.

[jstebbins]

The Fedora (RPMFusion) build is linking to libmfx, so it is at least supposed to be available there. I'll try the Flatpak.

Distro builds of HandBrake are notorious for being built wrong because they have rules that directly conflict with how we require HandBrake be built. So they modify things to suite their rules and pretty much always break something.

This is why I created the flatpaks. It's an environment we can control and the flatpak system was specifically designed for applications like ours. So we get no pushback from the flathub maintainers.

[mathstuf]

To be clear, I'm fine with rebasing on master as needed, but it will likely be on a "please rebase" manual poke method (open an issue).

Distro builds of HandBrake are notorious for being built wrong because they have rules that directly conflict with how we require HandBrake be built.

I'm aware of the bundling problems around the project and distro :) . I'll hopefully get a chance to try the flatpak after work some time this week.

[jstebbins]

@mathstuf you are correct that maintaining a single VAAPI interface would be easier. But we already have implementations of the other interfaces that are superior to what VAAPI provides. For example, with QSV we can do "zero-copy" encoding using the QSV decoder and encoder (windows only currently) without bouncing buffers through CPU memory.

Since we do not want to drop these better implementations, we would end up with multiple encoder options that do the "same thing" differently which would be confusing.

That said, I personally would like to see a hardware encoder implementation that works with the open source drivers. I never install closed source drivers because they are often buggy and installing custom drivers on Fedora using signed binaries is a royal PITA. NVEnc and AMF both require closed source drivers. So I wouldn't mind having VAAPI as an option that isn't enabled by default in the official builds.

I would also suggest that individual hardware encoder options available through VAAPI be selectable at build time. That way, if some new hardware encoder comes along that we don't have support for already, we can enable it (and only it) through the VAAPI interface.

[mathstuf]

For example, with QSV we can do "zero-copy" encoding using the QSV decoder and encoder (windows only currently) without bouncing buffers through CPU memory.

VAAPI should support this as long as it is used for decoding as well. I don't know what kind of filtering is possible though. FFmpeg's command line hints that it can do some, but that's the extent of what I know about it.

So I wouldn't mind having VAAPI as an option that isn't enabled by default in the official builds.

That's fine with me. Fedora (and other distros) can enable it and its packages won't get so many "why doesn't hardware encoding work?" questions since it's on a stack that can actually be debugged on Fedora's side :) .

I would also suggest that individual hardware encoder options available through VAAPI be selectable at build time. That way, if some new hardware encoder comes along that we don't have support for already, we can enable it (and only it) through the VAAPI interface.

This is OK with me too. Though H.264 and H.265 should probably be unconditional.

[jstebbins]

This is OK with me too. Though H.264 and H.265 should probably be unconditional.

I made this comment explicitly with these in mind since they are already covered by QSV, NVenc, and AMF which will always be enabled in an official build. If we wish to add say hardware AV1 support through VAAPI, we don't want that to automatically bring in other encoders.

[jstebbins]

VAAPI should support this as long as it is used for decoding as well

Maybe I'm mistaken, IIRC someone said ffmpeg does not support zero-copy in any hardware encoder yet. And this would go through ffmpeg's vaapi implementation no?

[mathstuf]

I made this comment explicitly with these in mind since they are already covered by QSV, NVenc, and AMF which will always be enabled in an official build

Ah. I made it with VAAPI being the conditional part, but if available, always make H.264 and H.265 available since not everyone has hardware/drivers for the other ones handy. Sure, HandBrake supports those, but without me digging into why QSV isn't being detected on my machine, I'd be stuck without any hardware accel if VAAPI is never made available for them in a stock build.

Maybe I'm mistaken, IIRC someone said ffmpeg does not support zero-copy in any hardware encoder yet. And this would go through ffmpeg's vaapi implementation no?

There's at least some support per this Reddit comment. Though surrounding comments seem to suggest they might be patches in Plex's FFmpeg fork (though tracking them down for GPL compliance if they use libx264 or libx265 might be something). And that's just for nvidia offloading. No idea about VAAPI zero-copy support.

[mathstuf]

OK, so I took some time to fetch and try the Flatpak: no QSV/QuickSync options are offered in it either, so without debugging that driver stack (no known BIOS fiddling and nothing showing up in package searches for "QSV" or "QuickSync") and without VAAPI, a 10th gen Intel processor is without hardware encoding in HandBrake AFAICT.

[sr55]

What Driver are you running here?

[mathstuf]

Mesa's Intel DRI driver.

% glxinfo | grep Mesa
client glx vendor string: Mesa Project and SGI
    Device: Mesa Intel(R) UHD Graphics (CML GT2) (0x9bca)
OpenGL renderer string: Mesa Intel(R) UHD Graphics (CML GT2)
OpenGL core profile version string: 4.6 (Core Profile) Mesa 20.0.7
OpenGL version string: 4.6 (Compatibility Profile) Mesa 20.0.7
OpenGL ES profile version string: OpenGL ES 3.2 Mesa 20.0.7

Is there another driver I should be looking for?

[mathstuf]

I do have intel-mediasdk-20.1.0-1.fc32.x86_64 installed already if that's the driver you're referring to.

[jstebbins]

Did you also install the MediaSDK flatpak plugin? It's available on both our download page and on flathub. Flatpak is a self-contained environment. It doesn't matter what you have installed on your host. You need the our MediaSDK plugin.

[mathstuf]

I installed the plugin. I tried to do one of the jobs I have after migrating the queue and changing the encoder and this error appeared in the log:

qsv_hevc_make_header: MFXVideoENCODE_EncodeFrameAsync failed (-5)

That error code seems to be MFX_ERR_NOT_ENOUGH_BUFFER.

[jstebbins]

Can you post the full log please.

[mathstuf]

[17:12:35] gtkgui: HandBrake 1.3.2 (2020050600) - Linux x86_64 - https://handbrake.fr
[17:12:35] hb_display_init: attempting VA driver 'iHD'
libva info: VA-API version 1.5.0
libva info: va_getDriverName() returns 0
libva info: User requested driver 'iHD'
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/iHD_drv_video.so
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/intel-vaapi-driver/iHD_drv_video.so
libva info: Found init function __vaDriverInit_1_5
libva info: va_openDriver() returns 0
Cannot load libnvidia-encode.so.1
[17:12:35] hb_init: starting libhb thread
[17:12:35] hb_init: starting libhb thread
[17:12:35] hb_init: starting libhb thread
[17:12:55] gtkgui: Modified Preset: /Matroska/H.265 MKV 2160p60
[17:12:55] Starting work at: Thu Jun 18 17:12:55 2020

<snip job json dump>

[17:12:55] CPU: Intel(R) Core(TM) i7-10710U CPU @ 1.10GHz
[17:12:55]  - logical processor count: 12
[17:12:55] Intel Quick Sync Video support: yes
[17:12:55]  - Intel Media SDK hardware: API 1.32 (minimum: 1.3)
[17:12:55]  - H.264 encoder: yes
[17:12:55]     - preferred implementation: hardware (any) via ANY
[17:12:55]     - capabilities (hardware):  breftype icq+la+i+downs vsinfo opt1 opt2+mbbrc+extbrc+trellis+ib_adapt+nmpslice
[17:12:55]  - H.265 encoder: yes (8bit: yes, 10bit: yes)
[17:12:55]     - preferred implementation: hardware (any) via ANY
[17:12:55]     - capabilities (hardware):  bpyramid icq vsinfo opt1

<snip scan dump>

<start snip of job configuration>

[17:14:59]  * video track
[17:14:59]    + decoder: mpeg2video
[17:14:59]      + bitrate 200 kbps
[17:14:59]    + filters
[17:14:59]      + Comb Detect (mode=3:spatial-metric=2:motion-thresh=1:spatial-thresh=1:filter-mode=2:block-thresh=40:block-width=16:block-height=16)
[17:14:59]      + Decomb (mode=39)
[17:14:59]      + Framerate Shaper (mode=2:rate=27000000/450000)
[17:14:59]        + frame rate: 23.976 fps -> peak rate limited to 60.000 fps
[17:14:59]      + Crop and Scale (width=720:height=362:crop-top=56:crop-bottom=62:crop-left=0:crop-right=0)
[17:14:59]        + source: 720 * 480, crop (56/62/0/0): 720 * 362, scale: 720 * 362
[17:14:59]    + Output geometry
[17:14:59]      + storage dimensions: 720 x 362
[17:14:59]      + pixel aspect ratio: 32 : 27
[17:14:59]      + display dimensions: 853 x 362
[17:14:59]    + encoder: H.265 (Intel Media SDK)
[17:14:59]      + preset:  balanced
[17:14:59]      + profile: auto
[17:14:59]      + level:   auto
[17:14:59]      + quality: 16.00 (ICQ)
[17:14:59]      + color profile: 6-1-6

<rest of job configuration>

[17:14:59] sync: expecting 155380 video frames
[17:14:59] hb_display_init: using VA driver 'iHD'
libva info: VA-API version 1.5.0
libva info: va_getDriverName() returns 0
libva info: User requested driver 'iHD'
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/iHD_drv_video.so
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/intel-vaapi-driver/iHD_drv_video.so
libva info: Found init function __vaDriverInit_1_5
libva info: va_openDriver() returns 0
[17:14:59] qsv_hevc_make_header: MFXVideoENCODE_EncodeFrameAsync failed (-5)
[17:14:59] encqsvInit: qsv_hevc_make_header failed
[17:14:59] Failure to initialise thread 'Quick Sync Video encoder (Intel Media SDK)'

[jstebbins]

Thanks, What CPU do you have?

[mathstuf]

Intel(R) Core(TM) i7-10710U CPU

[jstebbins]

Strange. Can't reproduce that here. I'm running the 1.3.3 release and MediaSDK plugin both installed from flathub.

My CPU specifics for reference

07:20:19] CPU: Intel(R) Core(TM) i9-9880H CPU @ 2.30GHz
[07:20:19]  - Intel microarchitecture Kaby Lake
[07:20:19]  - logical processor count: 16
[07:20:19] Intel Quick Sync Video support: yes
[07:20:19]  - Intel Media SDK hardware: API 1.32 (minimum: 1.3)
[07:20:19]  - H.264 encoder: yes
[07:20:19]     - preferred implementation: hardware (any) via ANY
[07:20:19]     - capabilities (hardware):  breftype icq+la+i+downs vsinfo opt1 opt2+mbbrc+extbrc+trellis+ib_adapt+nmpslice
[07:20:19]  - H.265 encoder: yes (8bit: yes, 10bit: yes)
[07:20:19]     - preferred implementation: hardware (any) via ANY
[07:20:19]     - capabilities (hardware):  bpyramid icq vsinfo opt1

@maximd33 or @agalin89 any ideas?

[maximd33]

@mathstuf can you please try nightly build ?
we had similar issue fixed , https://github.com/HandBrake/HandBrake/pull/2862

[jstebbins]

The nightly build is here:
https://nightly.handbrake.fr/fr.handbrake.ghb-20200614151529-ed54f307f-master-x86_64.flatpak

But this fix should also be in the 1.3.3 release since it was backported here:
https://github.com/HandBrake/HandBrake/commit/7c11f02eb89d73db82786ffc451234c189896fa8

[sgothel]

@jstebbins I have updated my vaapi work in hw-encoder-vaapi
(as well as general hw-encoder-base and hw-encoder-nvenc - all names of my new branches).

Please feel free to comment, pick, merge ..

The usual TODO: I still have some hardcoded features in vaapi and nvenc,
which I like to have added to the extra-options field.
To enable this for the vaapi/nvenc codecs and have it stored in presets,
I need a little help from you.

Detailed description in the vaapi commit message,
working quite well on a AMD NAVI10 using my mesa-vaapi patch.

1) hw-encoder-base https://github.com/sgothel/HandBrake/tree/hw-encoder-base
1.1) hw-encoder-nvenc https://github.com/sgothel/HandBrake/tree/hw-encoder-nvenc
1.2) hw-encoder-vaapi https://github.com/sgothel/HandBrake/tree/hw-encoder-vaapi
2) hw-encoder-joint https://github.com/sgothel/HandBrake/tree/hw-encoder-joint

Notable, using vaapi with Handbrake (utilizing the hardcoded soft-frame -> NV12 -> VAAPI)
is faster than ffmpeg commandline: ~150 fps -> ~200 fps

I compared quality differences with x264 ~5Mbit/s and while I couldn't see
any visually really, PSNR and SSIM difference to orig were below 1%.

As usual, great job of all of you working on Handbrake!
I regularly use it, hence my sporadic contributions.
Will help with integration, if intended by your team.

[sgothel]

hw-encoder-vaapi commit in question
https://github.com/sgothel/HandBrake/commit/c0fd9eeccbaa48936b02f3bda5d92cd9d3fb079b

[sgothel]

The mesa-vaapi patch included is also available here https://github.com/sgothel/mesa/commit/5e10db6f3f22a7d687ad7eeded3bbf0866978628

I might need to communicate this one with the mesa and ffmpeg team.

[sgothel]

Last one: W/o this mesa-vaapi patch, Matroska can't be stored - mp4 doesn't care.
Hence is usable w/o in a limited format fashion.