Wgpu-rs: wgpu doesn't find Vulkan device on NixOS with nvidia drivers

[Philipp-M]

I'm currently trying to get the wgpu examples running on NixOS using NixGL.
But it seems that it doesn't find any adapter (tried multiple examples).

Trying the hello-triangle example with on master:
RUST_BACKTRACE=full nixVulkanNvidia cargo run --example hello-triangle:

thread 'main' panicked at 'called `Option::unwrap()` on a `None` value', examples/hello-triangle/main.rs:11:19
stack backtrace:
   0:     0x5623cab1aea4 - backtrace::backtrace::libunwind::trace::he144fab28a4aed2d
                               at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.46/src/backtrace/libunwind.rs:86
   1:     0x5623cab1aea4 - backtrace::backtrace::trace_unsynchronized::h54d2de49d4561d5b
                               at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.46/src/backtrace/mod.rs:66
   2:     0x5623cab1aea4 - std::sys_common::backtrace::_print_fmt::hc03d55f811cceef4
                               at src/libstd/sys_common/backtrace.rs:78
   3:     0x5623cab1aea4 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::hbc44d25334fa89cd
                               at src/libstd/sys_common/backtrace.rs:59
   4:     0x5623cab42a2c - core::fmt::write::hdf236390fbd68d3d
                               at src/libcore/fmt/mod.rs:1069
   5:     0x5623cab18103 - std::io::Write::write_fmt::h4ee562ef1f300991
                               at src/libstd/io/mod.rs:1532
   6:     0x5623cab1d6b5 - std::sys_common::backtrace::_print::hc4186a5ac838159c
                               at src/libstd/sys_common/backtrace.rs:62
   7:     0x5623cab1d6b5 - std::sys_common::backtrace::print::h61fb789361bb8109
                               at src/libstd/sys_common/backtrace.rs:49
   8:     0x5623cab1d6b5 - std::panicking::default_hook::{{closure}}::hcd876594d255c932
                               at src/libstd/panicking.rs:198
   9:     0x5623cab1d3f2 - std::panicking::default_hook::hfc205bc5ce834a89
                               at src/libstd/panicking.rs:218
  10:     0x5623cab1dd12 - std::panicking::rust_panic_with_hook::h98fcd55ca23bf6e3
                               at src/libstd/panicking.rs:477
  11:     0x5623cab1d8fb - rust_begin_unwind
                               at src/libstd/panicking.rs:385
  12:     0x5623cab40821 - core::panicking::panic_fmt::hd101a87121fa411f
                               at src/libcore/panicking.rs:89
  13:     0x5623cab4076d - core::panicking::panic::h02171c407fa1462f
                               at src/libcore/panicking.rs:52
  14:     0x5623ca1f8c7f - core::option::Option<T>::unwrap::h303854b1bf306fcb
                               at /home/philm/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libcore/macros/mod.rs:10
  15:     0x5623ca228d2f - hello_triangle::run::{{closure}}::ha26ffd5f6c0ec31c
                               at examples/hello-triangle/main.rs:11
  16:     0x5623ca1e5627 - <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll::h9b8e522bf846e618
                               at /home/philm/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libcore/future/mod.rs:61
  17:     0x5623ca21fd06 - futures_executor::local_pool::block_on::{{closure}}::h42e41fe7063002ce
                               at /home/philm/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-executor-0.3.1/src/local_pool.rs:289
  18:     0x5623ca21fbd9 - futures_executor::local_pool::run_executor::{{closure}}::hb0a8ec324126e03e
                               at /home/philm/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-executor-0.3.1/src/local_pool.rs:70
  19:     0x5623ca1e7b3d - std::thread::local::LocalKey<T>::try_with::hd50c80ef3044031d
                               at /home/philm/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd/thread/local.rs:263
  20:     0x5623ca1e7a43 - std::thread::local::LocalKey<T>::with::hefab92d77b994bd1
                               at /home/philm/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd/thread/local.rs:239
  21:     0x5623ca21fb34 - futures_executor::local_pool::run_executor::h0e41aa84d64e50fd
                               at /home/philm/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-executor-0.3.1/src/local_pool.rs:66
  22:     0x5623ca21fc75 - futures_executor::local_pool::block_on::h138106198f2ff586
                               at /home/philm/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-executor-0.3.1/src/local_pool.rs:289
  23:     0x5623ca233616 - hello_triangle::main::h196b4435f00d939d
                               at examples/hello-triangle/main.rs:150
  24:     0x5623ca1af52b - std::rt::lang_start::{{closure}}::h7636faf286a4f882
                               at /home/philm/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd/rt.rs:67
  25:     0x5623cab1e158 - std::rt::lang_start_internal::{{closure}}::h86be2e33c1b2fc02
                               at src/libstd/rt.rs:52
  26:     0x5623cab1e158 - std::panicking::try::do_call::h9468ed432eddf5f1
                               at src/libstd/panicking.rs:297
  27:     0x5623cab1e158 - std::panicking::try::hda2b1d9056481eed
                               at src/libstd/panicking.rs:274
  28:     0x5623cab1e158 - std::panic::catch_unwind::h8cbb401c5be5ca4d
                               at src/libstd/panic.rs:394
  29:     0x5623cab1e158 - std::rt::lang_start_internal::h2403699d8e6ba133
                               at src/libstd/rt.rs:51
  30:     0x5623ca1af507 - std::rt::lang_start::h72f500ddc61713de
                               at /home/philm/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd/rt.rs:67
  31:     0x5623ca2336ca - main
  32:     0x7fecd1d24d8b - __libc_start_main
  33:     0x5623ca1af1aa - _start
                               at ../sysdeps/x86_64/start.S:120
  34:                0x0 - <unknown>

My first thought was that Vulkan might not be working, or is not setup correctly.
I have now almost everything Vulkan related installed on my system: vulkan-tools, vulkan-headers, vulkan-loader, vulkan-validation-layers.

But I just tried https://github.com/SaschaWillems/Vulkan
and there everything is working as intended.

My Notebook which still has Arch Linux with Intel graphics (on a docking station also with a dedicated Nvidia GPU) runs all the examples flawlessly.
Any thoughts what might cause this issue?

Here is a vulkaninfo pastebin

[kvark]

Thank you for filing!
Could you check ash and gfx examples as well?

[P-E-Meunier]

Same here, with an old-ish Intel integrated GPU and NixOS. I can also confirm that Vulkano works.

[P-E-Meunier]

Fixed! The issue is that the rpath of the produced binaries doesn't have the paths to the dynamically-linked libraries. This causes binaries to depend on environment variables such as LD_LIBRARY_PATH on Linux, which is not great (because it is less reproducible), but usually works (sometimes by telling users to change their .bashrc).

However, NixOS users tend to avoid such hacks, and I would argue that they are right, since this allows them to deploy compute shaders to remote machines, and get reproducible computation for free without requiring endless tuning of the remote system.

Fortunately, there is a simple fix, with three different options:

1. The easiest way would be to add a build script (such as the one below) to the various backends of GFX (in particular Vulkan on Linux, since the other platform already have other problems in the way of reproducibility).

2. Another way would be to add the same build script to this crate, wgpu.

3. Meanwhile, users of this crate can add the following line to their build scripts:

fn main() {
    if cfg!(target_os = "linux") {
        println!("cargo:rustc-link-lib=X11");
        println!("cargo:rustc-link-lib=Xcursor");
        println!("cargo:rustc-link-lib=Xrandr");
        println!("cargo:rustc-link-lib=Xi");
        println!("cargo:rustc-link-lib=vulkan");
    }
}

[kvark]

@P-E-Meunier
I'm not completely following. gfx-backend-vulkan is built to load the libvulkan.so dynamically. We don't know where exactly it's supposed to be on the target system, since it depends on the distribution and such. What would rustc-link-lib=vulkan fix here, exactly? Make our library aware of where Vulkan is on the developers machine?

[P-E-Meunier]

That is correct. There is something called the "rpath" in ELF binaries on Linux, which is where libraries are looked up first. If they aren't found, it's up to the binary to specify the path, which is usually done with LD_LIBRARY_PATH. Compiling with these flags allows the dynamic library to work. One way to see it is to run cargo run with an empty LD_LIBRARY_PATH.

[kvark]

@P-E-Meunier ok, thank you for explaining!
Now, correct me if I see this wrong. We have 2 kind of use cases here:

1. wgpu and gfx libraries are built within an application that should be runnable on different systems, e.g. for distribution. This is our common case. We don't know ahead of time what systems it will run on, so it doesn't make sense to encode the path to dynamic libraries that the development (or CI) machine had at the time of building.
2. you are building wgpu and gfx in order "to deploy compute shaders to remote machines, and get reproducible computation for free without requiring endless tuning of the remote system." In this case, you can put that build script into your build.rs.

Did I capture this right?

[P-E-Meunier]

You're almost right: for use case 1, the rpath will be set at compile time. When distributing on Linux, you're either distributing:

1a. In the form of source code, in which case it doesn't matter, since the end user will recompile. Setting the rpath has the benefit that wgpu and gfx would work on absolutely all Linux distributions, since the ELF format is not specific to any of them. In the particular case of NixOS, we do know in advance where the exact version of libvulkan will be (that is actually the main promise of NixOS), and NixOS automatically guarantees that it will always be there, by reading the ELF when installing your binary package.

1b. Using a package format such as deb or rpm. In this case, the package has dependencies, and there are two options: either set the rpath to the path where the Vulkan dependency placed vulkan.so.1, typically /usr/lib/vulkan.so.1 on Ubuntu/Debian/RedHat/… or ask users to set their LD_LIBRARY_PATH to the appropriate value, in case they prefer use their own variant of libvulkan, rather than the one installed on the system. I would argue that this is a strange use of a package manager, but why not?

For use case 1 and 2 the end user can totally set the rpath by themselves by writing a build script, but IMHO this breaks the everything-works-out-of-the-box experience I'm used to with Cargo.

[kvark]

@P-E-Meunier I'm still concerned about this line of thought. So we can skip talking about (1a) case and focus on (1b).

either set the rpath to the path where the Vulkan dependency placed vulkan.so.1, typically /usr/lib/vulkan.so.1 on Ubuntu/Debian/RedHat/… or ask users to set their LD_LIBRARY_PATH to the appropriate value

I don't think you listed all the options. The other option here is - do nothing. Instead, just assume libvulkan is a standard library on a system and will be dynamically looked up as all the other standard libraries, i.e (from here):

The list of directories in the file /etc/ld.so.conf. This file can include other files, but it is basically a list of directories - one per line.
Default system libraries - usually /lib and /usr/lib

The most sane approach is to let the target system figure out how it installs Vulkan and just rely on the standard discovery mechanism. Does nixOS not have Vulkan loader registering anywhere within the standard paths? That seems weird.

If we are talking about distro packages, it's also a bit iffy, since they are built for specific distro, and in this build they can expect a specific path. It would make more sense to talk about, say, Itch.io binaries that you download, or even Steam game binaries. Do they have rpath to Vulkan/X11/etc, in the executables? I'd be surprised if they rely on that.

[P-E-Meunier]

I don't think you listed all the options. The other option here is - do nothing.

"Standard" in your argument there means "on the LD_LIBRARY_PATH", so you're not actually doing nothing, you are in fact asking users to set it to the desired location, if it's not already set by their package manager.

(as a side note, I've spent several hours in the last few weeks holding people's hands via zoom to get them to compile Vulkano and GFX on various flavours of Linux, OSX and Windows, and I don't know anymore what "standard" means ;-).

Instead, just assume libvulkan is a standard library on a system and will be dynamically looked up as all the other standard libraries, i.e (from here):

Well, the rpath in the ELF format is really even more standard than all possible system organisations one could imagine, and setting it is really part of "letting the target system figure out".

[P-E-Meunier]

Steam game binaries. Do they have rpath to Vulkan/X11/etc, in the executables? I'd be surprised if they rely on that.

They probably don't, which is why they first check on their rpath, and then resort to checking the LD_LIBRARY_PATH. I now see what you're saying: you don't want to add stuff to the rpath that is not going to be used anyway. What about a mixture of both: add a build script, and check if LD_LIBRARY_PATH has the Vulkan libraries. If so, don't do anything. If not, print the flags. When building in debug mode, always print the flags.

This would allow first-time users of the crate to get the optimal experience, while also making it maximally flexible for experienced users and people already thinking about distribution of their binaries.

[kvark]

"Standard" in your argument there means "on the LD_LIBRARY_PATH"

This is confusing to me. LD_LIBRARY_PATH is supposed to be empty unless the user hacks something. It's not a standard location of dynamic libraries. In the link I've read, see "Runtime Search Path" section. By "standard" I'm talking about points (4) and (5), while "LD_LIBRARY_PATH" is on point (2). Does nixOS not have any standard discovery for shared libraries?

Well, the rpath in the ELF format is really even more standard than all possible system organisations one could imagine, and setting it is really part of "letting the target system figure out".

That part I don't understand. How is this "letting the target system figure out" if you are providing a concrete path on the target system (and we have no idea what the target system/distro is when we are building it).

they first check on their rpath, and then resort to checking the LD_LIBRARY_PATH

This sounds extremely weird to me. Rpath may locate the shared library on the same ditro that an app was built on, but it has no value if we are running on a different distro/platform. And LD_LIBRARY_PATH is basically user interventoin. So what you are saying is that there is no way to build a Linux application (that uses standard libraries like X11) that would run well on other systems (without user intervention). That paints a rather grim picture... Moreover, it's not clear to me why points (4) and (5) of the link I provided wouldn't make this happen automatically. The Vulkan SDK would install itself in a standard location, like /usr/lib/libvulkan.so, and/or add itself to the /etc/ld.so.conf and be done with it.

add a build script, and check if LD_LIBRARY_PATH has the Vulkan libraries. If so, don't do anything. If not, print the flags.

Who is going to be setting LD_LIBRARY_PATH for this? Also, this is meant to be a run-time environment variable, not compile time.

[P-E-Meunier]

How is this "letting the target system figure out" if you are providing a concrete path on the target system

Because the system "figures it out" by looking up paths in order, starting with the rpath.

Does nixOS not have any standard discovery for shared libraries?

No, unless you hack something, and that's a (really) good thing.

there is no way to build a Linux application

There is a way: this is precisely what package managers are for.

Moreover, it's not clear to me why points (4) and (5) of the link I provided wouldn't make this happen automatically.

They would! For example point (7) of that link lists where the binaries look for libraries. There is no /etc/ld.so.conf on NixOS, and no "default system libraries", because NixOS tries to make builds reproducible. So, we're left with the rpath, the runpath, and LD_LIBRARY_PATH.

Who is going to be setting LD_LIBRARY_PATH for this?

All I'm saying is, right now the examples do not work "out-of-the-box" for everybody, as evidenced by this issue (which I didn't open myself, btw). There could be a simple way to make this work out-of-the-box, at least in debug mode, so that people can try it out more easily. Maybe LD_LIBRARY_PATH isn't the way, and maybe you can check it in a different way -- or maybe you can just add the path to the rpath in debug mode, and you don't have to compromise your principles for release mode.

It's ok to say "I don't want to support exotic platforms", users of exotic systems are kind of expecting that sort of stuff anyway.

[kvark]

There is a way: this is precisely what package managers are for.

Not sure I understand this answer. If you build a "deb" package on one distro and install it on another, how is the package manager solving the problem at hand (about standards library discovery)?

For example point (7) of that link lists where the binaries look for libraries. There is no /etc/ld.so.conf on NixOS, and no "default system libraries", because NixOS tries to make builds reproducible. So, we're left with the rpath, the runpath, and LD_LIBRARY_PATH.

Ok, interesting, so it does come down to basically NixOS not having a notion of a standard shared library. How is this solved in other nix packages? I imagine anything that uses X11, for example, would need to know where to look for it. How does your build system know where to look for X11 and Vulkan libs?

All I'm saying is, right now the examples do not work "out-of-the-box" for everybody

One of the possible "solutions" is to make it so the examples have this build script. This is totally doable on gfx-rs, where the examples are a separate crate, but in wgpu-rs they are a part of the same crate. I'm really wary of adding any logic in build.rs - it's a hack around the Rust build system, and it's much easier to reason about the build if it's not scripted.

It's ok to say "I don't want to support exotic platforms", users of exotic systems are kind of expecting that sort of stuff anyway.

That's not what I'm saying (just yet), I'm merely trying to access the situation and evaluate our possible solutions.

One idea that could work is having a little library crate somewhere that has no code but only a build.rs with the changes you suggested. We'd make it a dependency of gfx/wgpu examples, and that seems harmless. It could live in gfx-rs repo, say in src/auxil/gfx-nixos-deps folder.

[P-E-Meunier]

If you build a "deb" package on one distro and install it on another, how is the package manager solving the problem at hand (about standards library discovery)?

Deb packages (1) specify the exact paths for all their files, and (2) typically have dependencies, also specifying the exact paths for all their files. This solves the problem, since in order to install your game, you must first install a package named "vulkan-1.42.deb", which hopefully is already installed.

Ok, interesting, so it does come down to basically NixOS not having a notion of a standard shared library. How is this solved in other nix packages? I imagine anything that uses X11, for example, would need to know where to look for it. How does your build system know where to look for X11 and Vulkan libs?

This is actually a desirable feature: NixOS is meant to be "functional" in the sense that installing a package doesn't touch anything on the disk outside a closed sandbox meant exclusively for the package. In other package managers like Debian's, there is a hack called "triggers", which are, well, "triggered" when you install a package, if another known package is also installed. Triggers are mostly used to change configuration files shared by multiple packages, such as /etc/ld.so.conf.

But this is not a great design, since these files tend to "age", and your entire system degrades slowly over time, unless you're very careful and know exactly which packages you install.

Back to dynamic linking, there are three cases:

• when you compile a package on NixOS, the package manager automatically looks at the resulting binary and hardcodes the exact paths of the exact version with which the package was built (that path is itself computed from a hash of the library, so this method is 100% reproducible).
• when you write a "wrapper" package around a debian package (or another binary), you have to edit the ELFs to include the paths yourself.
• LD_LIBRARY_PATH.

[P-E-Meunier]

One idea that could work is having a little library crate somewhere that has no code but only a build.rs with the changes you suggested. We'd make it a dependency of gfx/wgpu examples, and that seems harmless. It could live in gfx-rs repo, say in src/auxil/gfx-nixos-deps folder.

This can even be disabled via a feature, so it's quite harmless. That library will probably be never need to be touched. You seem to have strong opinions against build scripts (at least in the main repository), may I ask why?

[kvark]

There seem to be a huge gap between how we see things, and communicating is hard. Thank you for your patience!
For some background: I highly appreciate the architectural decisions of NixOS, would like to try it out sometime.

Deb packages (1) specify the exact paths for all their files, and (2) typically have dependencies, also specifying the exact paths for all their files. This solves the problem, since in order to install your game, you must first install a package named "vulkan-1.42.deb", which hopefully is already installed.

This means that, basically, if your distro supports "deb" packages, it has to have a fixed layout of libraries and files. So this isn't really "solving" the portability between distros, as much as just making all little deb-supporting distros a single meta-distro.

when you compile a package on NixOS, the package manager automatically looks at the resulting binary and hardcodes the exact paths of the exact version with which the package was built

You wrote 3 paragraphs (of good info!) without actually answering my question, or maybe I'm just totally missing it. The question was:

How does your build system know where to look for X11 and Vulkan libs?

Or more specifically, when you do the build.rs changes as you suggested on NixOS, how does it know where to look up the X11 and Vulkan libraries?

You seem to have strong opinions against build scripts (at least in the main repository), may I ask why?

You are suggesting to hack around the Rust build system to insert the link instructions. If you check the documentation, it has a set of recommendations on how this should be done. Basically, whenever you want to link to standard native libraries, you'd best do:

1. insert links = in the package manifest
2. have it all contained within a -sys kind of package

Point (2) raises a big red flag to me with regards to adding this to gfx-rs or wgpu, since these libraries aren't system library wrappers.

So I filed https://github.com/MaikKlein/ash/issues/312 upstream, since we use Ash to talk to Vulkan today. If you agree, let's close this one and continue there?

[P-E-Meunier]

Or more specifically, when you do the build.rs changes as you suggested on NixOS, how does it know where to look up the X11 and Vulkan libraries?

I did misunderstand your question, I thought you were asking about how NixOS found paths for games downloaded in binary form.

The way my change works is by letting Rust add the path to Vulkan to the rpath.

[P-E-Meunier]

We can indeed close.