Nixpkgs: ffmpeg Cannot load libcuda.so.1

[tbenst]

Describe the bug
When trying to encode with ffmpeg -i my_vid.mp4 -c:v h264_nvenc nvenc_test.mp4, I get the following error:

[h264_nvenc @ 0x5e28c0] Cannot load libcuda.so.1
Error initializing output stream 0:0 -- Error while opening encoder for output stream #0:0 - maybe incorrect parameters such as bit_rate, rate, width or height

I believe this is a continuation of https://github.com/NixOS/nixpkgs/issues/67780 & https://github.com/NixOS/nixpkgs/pull/67790

cc @ivan

[flokli]

https://github.com/NixOS/nixpkgs/pull/67790 should have fixed cuda for ffmpeg, shouldn't it?

Can you check with strace where if/where it tries to load libcuda.so.1 from?

[tbenst]

huh now it's working (on NixOS). Maybe it didn't work on ubuntu. Closing for now

[tbenst]

@flokli nvm I reproduced it using strace -o ffmpeg.txt ffmpeg -f lavfi -i nullsrc -c:v h264_nvenc -gpu list -f null -. It worked before because it was run in an impure nix-shell and it found libcuda from ~/.conda.

openat(AT_FDCWD, "/nix/store/m7s79nz19n92s0xb3y2h89rs992nmxgy-systemd-243.3-lib/lib/libcuda.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/nlqa5imjflsdigdwdpc37f0xjzl6iy8c-fontconfig-2.12.6-lib/lib/libcuda.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/3dkxqw4r625xqli01xms8k5v2z8k0jri-ffmpeg-full-4.2.1/lib/libcuda.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/vsvs3c8im9x5ypm9pywxx2i7n56fvspc-libvpx-1.7.0/lib/libcuda.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/fh06q7mzlvjjmvxmw3fy21knwfgirv61-libwebp-1.0.3/lib/libcuda.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/qn1mi06vw2li6d2lpmjyncwipi7vbflj-dav1d-0.5.2/lib/libcuda.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/ylpi31dwsgm9c9rndp0j5b4na749d97z-zlib-1.2.11/lib/libcuda.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/7ia9l74y521b1gbzqmw7jyshi0kzmnjn-openjpeg-2.3.1/lib/libcuda.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/nxrvxj72piv71w9mvqwx2zg0kb84f8q8-libopus-1.3.1/lib/libcuda.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/3r2a36z89bblni4pmcja2ds9bxdla0zv-speex-1.2.0/lib/libcuda.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/mwl3w3lv8y6ghysnsq762xwxn5cnrsfg-libvorbis-1.3.6/lib/libcuda.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/8jqx1k60gzcpa1d3nx1z5i8fwp374bag-x264-20190517-2245-lib/lib/libcuda.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/111nfy1k696sb0czjn1grvp33xbr0j98-x265-3.2/lib/libcuda.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/qlxras2nqmnm62z7078246b0v74fyx4d-libva-2.5.0/lib/libcuda.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/5vkls3ragpxwicb6d5jyhhg8d7d6xd32-intel-media-sdk-19.3.1/lib/libcuda.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/fa8dnypsji2k6nzbmki70mz81rxv7vfn-celt-0.11.3/lib/libcuda.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/ivf7kv7hp657xnl10ja2mv3a5b33yyd9-gsm-1.0.18/lib/libcuda.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/fhm00v7lbikqj216r2z5sl5ac0910x6n-lame-3.100-lib/lib/libcuda.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/p13vbx1nk26f9j77sfsrzrmzv83kdrdh-libtheora-1.1.1/lib/libcuda.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/6bz6zqhsbbgcjdxps792a3754ql4hhq4-xz-5.2.4/lib/libcuda.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/071s0bq4376g7g9qs3r06ryqjbalb7xb-opencore-amr-0.1.5/lib/libcuda.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/mn459kyziiyhmisg64q2xls9s87kiwad-vo-amrwbenc-0.1.3/lib/libcuda.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/6kxnxyx8g2f6gyy1s63al5vmb67ryklj-wavpack-5.1.0/lib/libcuda.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/0570iisrg8d587980a4fmba3x8gpj75d-xavs-55/lib/libcuda.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/i1r1n3s3ns4f4wwxc8w1r69fnxxngjd5-xvidcore-1.3.5/lib/libcuda.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/xhpwab5kavygbr1fswawmdyqvmn3wa4i-glibc-2.27/lib/libcuda.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/xhpwab5kavygbr1fswawmdyqvmn3wa4i-glibc-2.27/lib/libcuda.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)

full output

[flokli]

Looks like ffmpeg doesn't look into /run/opengl-driver(-32)/lib - see pkgs/applications/misc/blender/default.nix and pkgs/applications/video/mpv/default.nix, you might want to do something similar here.

[tbenst]

@flokli what about the scenario where someone is not using nvidia for a display driver, but only for compute—do you know if this approach would still work? I wonder if better to just include CUDA in ffmpeg is better when NVENC is enabled

[flokli]

I assume you can still set

hardware.opengl.enable = true;
hardware.opengl.extraPackages = [ ... ];

on a headless system, which will fill /run/opengl-driver(-32)/lib, and once you apply the same fixes as done in blender or mpv, ffpmeg will work.

But please test, and file a PR if it works :-)

[tbenst]

@flokli that should be fine for headless NixOS but won’t work on headless Ubuntu using Nix. As a general rule, I think a derivation should function regardless of system configuration if possible, so I’m leaning towards a fix that brings in CUDA

[flokli]

@tbenst It's not that simple, as it depends on the graphic / accelerator card being used, and we don't want to build a multitude of packages in all kinds of variations, which is why the "impure" /run/opengl-driver(-32)/lib is used to load whatever is enabled at runtime.

This obviously doesn't work on non-NixOS, as the location doesn't exist there.

One option for non-NixOS might be to use https://github.com/guibou/nixGL. However, in general it's complicated, as the dynamic library used on the non-NixOS distro might be incompatible with the things built by NixOS.

[flokli]

So the TLDR for GPU-specific stuff built with Nix on non-NixOS is either using nixGL, or setting LD_LIBRARY_PATH to make it find libcuda.so.1 from Ubuntu.

There's not really another way to fix this, sorry.

[AleXoundOS]

So, what's the solution on NixOS?

[flokli]

On NixOS, you should see /run/opengl-driver[-32] as soon as hardware.opengl.enable is set to true, and binaries/libraries trying to load mesa, vdpau, cuda, … libraries should have their RUNPATH set look in these locations, either by themselves, or in some dispatcher libraries (like libglvnd).

This clearly could benefit from some more docs (both how it works in principle, and what components make use of it) - @ambrop72, @abbradar, @guibou, would you be willing to PR some docs on that?

Looking at what appears in /run/opengl-driver[-32], and searching in nixpkgs for /run/opengl-driver should give at least some hints for the time being.

[ambrop72]

@flokli I can write something here https://nixos.org/nixpkgs/manual/#sec-opengl (and in fact fix it because it is out of date saying we rely on LD_LIBRARY_PATH), does that seem like the right place?

[flokli]

@flokli I can write something here https://nixos.org/nixpkgs/manual/#sec-opengl (and in fact fix it because it is out of date saying we rely on LD_LIBRARY_PATH), does that seem like the right place?

That'd be :heart:! Feel free to highlight be on the PR for review.