Nixpkgs: Restoring OpenCL support in Mesa

[athas]

Nixpkgs packages Mesa (maintained by @vcunat), which has a reasonable operational OpenCL implementation that works at least on AMD GPUs (and whose OpenGL implementation is probably better than AMDs proprietary one). Unfortunately, OpenCL support was explicitly removed from the package in d21b6702a3fae4c0c5003b08741f1a1d4d6c815a by @edolstra, apparently to remove dependency bloat (I guess the reason is that OpenCL needs an entire C compiler frontend at runtime).

I'm a Nix novice, but I've poked around a bit to see how to fix this. First, libclc must be updated to work at all (already made a pull request). Then, reverting (in spirit) d21b6702a3fae4c0c5003b08741f1a1d4d6c815a makes the derivation at least compile the OpenCL bits again, most importantly libMesaOpenCL.so (a whopping 53MiB file!). The mesa package already contains a separate output "drivers" that depends on LLVM, so perhaps the OpenCL bits could be moved there? Alternatively, a new "opencl" output could be made.

There is one more problem. OpenCL specifies a directory /etc/OpenCL/vendors, which should contain text files containing names of ICDs (for example libMesaOpenCL.so). In Nix, this is instead /run/opengl-driver/etc/OpenCL/vendors. I don't understand how to make stuff available there.

[abbradar]

We could try making a separate opencl output, this sounds good. To add something to /run/opengl-driver add a package to hardware.opengl.extraPackages.

[matthewbauer]

Does opencl even need clang to work? This seems surprising if it did. On the other hand, lots of drivers need llvm to work.

[athas]

I don't know if it needs the CLI program clang, but it certainly needs a C frontend, which I assume is built with libclang.

[athas]

It looks like just reverting d21b670 and patching up a few omissions makes something that at least builds libMesaOpenCL.so (in mesa-drivers) and even /run/opengl-driver/etc/OpenCL/vendors/mesa.icd seems to show up correctly. I guess mesa-drivers is already in hardware.opengl.extraPackages. Before I create an opencl output, I'd like to understand whether it's still worthwhile. mesa-drivers already looks pretty large, so is the overhead of carrying libclang all that significant?

Also, I can't yet claim that it actually works. The Mesa OpenCL ICD is picked up just fine, but my device (Vega 64 GPU) is not found. Haven't investigated closely yet. It should be supported, as it works fine in the Mesa that Fedora ships.

In case anyone is curious, these are the changes I made:

diff --git a/pkgs/development/libraries/mesa/default.nix b/pkgs/development/libraries/mesa/default.nix
index 5bc1dfaf878..4826722c956 100644
--- a/pkgs/development/libraries/mesa/default.nix
+++ b/pkgs/development/libraries/mesa/default.nix
@@ -10,6 +10,7 @@
 , vulkanDrivers ? null
 , eglPlatforms ? [ "x11" ] ++ lib.optionals stdenv.isLinux [ "wayland" "drm" ]
 , OpenGL, Xplugin
+, libclc
 }:

 /** Packaging design:
@@ -69,6 +70,7 @@ in
 let
   version = "18.3.4";
   branch  = head (splitString "." version);
+  clang = if llvmPackages ? clang-unwrapped then llvmPackages.clang-unwrapped else llvmPackages.clang;
 in

 let self = stdenv.mkDerivation {
@@ -131,7 +133,9 @@ let self = stdenv.mkDerivation {
     "--enable-llvm-shared-libs"
     (enableFeature stdenv.isLinux "omx-bellagio")
     (enableFeature stdenv.isLinux "va")
-    "--disable-opencl"
+   "--enable-opencl"
+   "--enable-opencl-icd"
+   "--with-clang-libdir=${clang}/lib"
   ];

   nativeBuildInputs = [
@@ -147,7 +151,7 @@ let self = stdenv.mkDerivation {
   buildInputs = with xorg; [
     expat llvmPackages.llvm libglvnd xorgproto
     libX11 libXext libxcb libXt libXfixes libxshmfence libXrandr
-    libffi libvdpau libelf libXvMC
+    libffi libvdpau libelf libXvMC libclc clang
     libpthreadstubs openssl /*or another sha1 provider*/
   ] ++ lib.optionals (elem "wayland" eglPlatforms) [ wayland wayland-protocols ]
     ++ lib.optionals stdenv.isLinux [ valgrind-light libomxil-bellagio libva-minimal ];
@@ -173,7 +177,9 @@ let self = stdenv.mkDerivation {
       $out/lib/vdpau         \
       $out/lib/bellagio      \
       $out/lib/libxatracker* \
-      $out/lib/libvulkan_*
+      $out/lib/libvulkan_*   \
+      $out/lib/gallium-pipe  \
+      $out/lib/libMesaOpenCL*

     # Move other drivers to a separate output
     mv $out/lib/dri/* $drivers/lib/dri # */

[matthewbauer]

Looks good! d21b6702a3fae4c0c5003b08741f1a1d4d6c815a actually predates multiple outputs, so I suspect it is not a problem anymore?

[abbradar]

Just in case let's measure closure size difference; worst case it'll be simple enough to move it to a separate output.

On June 10, 2019 11:59:05 PM GMT+03:00, Matthew Bauer [e-mail] wrote:

Looks good! d21b6702a3fae4c0c5003b08741f1a1d4d6c815a actually predates
multiple outputs, so I suspect it is not a problem anymore?

--
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/NixOS/nixpkgs/issues/62933#issuecomment-500592304

--
Nikolay.

[vcunat]

It does not predate multi-output mesa derivation (I suppose you meant libclang and I understood it wrong); you can have a look at the whole file around the diff. Mesa drivers will be present on any graphical NixOS installation that uses free drivers (basically all except those using binary nvidia), so I can understand this default would better not be too bloated.

This new lib + libclang output might be roughly 100 MB of new stuff in NixOS closures (even though C-like developers will often have some libclang already; *2 with 32-bit included), so from this quick estimate I'd expect it should better be a separate output.

[athas]

What I'm currently stuck on is libMesaOpenCL.so, which as built contains a reference to /nix/store/i9xsrfn3isrpzdw1xn2rxdfiwg3n230d-mesa-noglu-18.3.4/lib/gallium-pipe.so, but this file has to be located in -drivers (or -opencl) to avoid circular references. The common solution that I find in other packages is to perform a text substitution in the corresponding .la file, but libMesaOpenCL.so is not linked like a normal library, but instead dlopen()ed by the ICD loader (the actual libOpenCL.so, the ocl-icd package in Nix).

I'll see if Mesa has some compilation options I can tweak, but how is this kind of stuff otherwise handled normally?

[vcunat]

IIRC we now don't really need to separate .out and .drivers, as package closures should only refer to the libglvnd stubs. We indirect some paths through ${libglvnd.driverLink} as well, and dlopen() cases would be likely suspects for that. (I haven't really thought about these ATM.)

[athas]

So you're suggesting that we remove .drivers altogether?

[vcunat]

I expect it's a possibility, in case it helps dealing with the cycles.

[athas]

It's fine with me, as it'll certainly simplify my task here (rebuilding now to see whether that's actually the case). I'm too much of a novice to be able to evaluate the impact, however. I assume the split was made for a reason, and as I've been using NixOS for all of two days, I don't really know if those reasons have changed since then.

[abbradar]

There is a potential problem with merging separate outputs: I grepped around a little and noticed that xorgserver depends on mesa:

➜  nixpkgs git:(stable) ✗ ldd result/lib/xorg/modules/libglamoregl.so
    libgbm.so.1 => /nix/store/10n2qp973b7vb6fd09mwmvcx47axawxn-mesa-noglu-18.3.4/lib/libgbm.so.1 (0x00007fe47cb56000)

@athas wouldn't replacing path in .icd file do? It should specify path to libMesaOpenCL.so.

[athas]

The problem isn't the reference to libMesaOpenCL.so - that works fine. The problem is that libMesaOpenCL.so then contains a hard-coded string referencing mesa/lib/gallium-pipe, but this file is in our case in mesa-drivers/lib/gallium-pipe. I'm poking around the Mesa build system to see whether this can be changed.

[athas]

I'm fairly confident the offending line is this one: https://github.com/mesa3d/mesa/blob/18.3/src/gallium/auxiliary/pipe-loader/pipe_loader_drm.c#L156

Trying to figure out whether that can be changed.

[abbradar]

@athas Hacky way: NIX_CFLAGS_COMPILE = [ "-UPIPE_SEARCH_DIR" "-DPIPE_SEARCH_DIR=\"${placeholder "drivers"}/lib/gallium-pipe\"" ];

[athas]

@athas Hacky way: NIX_CFLAGS_COMPILE = [ "-UPIPE_SEARCH_DIR" "-DPIPE_SEARCH_DIR=\"${placeholder "drivers"}/lib/gallium-pipe\"" ];

Thanks, this works, and everything seems to resolve now. Unfortunately. trying to runtime-compile any OpenCL kernel fails with a dubious error:

Intrinsic has incorrect return type!
i8 addrspace(2)* ()* @llvm.amdgcn.dispatch.ptr

I'll see if I can figure out what's going on.

[athas]

It's probably because I have been testing my changes on top of NixOS 19.03, which has an old LLVM, and libclc requires at least LLVM 7. I'll rebuild everything on top of unstable.

[athas]

I got things working with the OpenCL parts in the .drivers output. How do I check the growth in closure size?

[vcunat]

For example with nix path-info -S ./result-drivers (or arbitrary path).

[athas]

With OpenCL support:

$ nix path-info -S /nix/store/am103lnxa5h15z7xj0ziy5m26rnwnfr1-mesa-18.3.4-drivers/
/nix/store/am103lnxa5h15z7xj0ziy5m26rnwnfr1-mesa-18.3.4-drivers  1177379384

Without (I think; the noglu/txc stuff is a bit confusing to me):

$ nix path-info -S /nix/store/073km24qkqr808q7zvz785smzp6ck592-mesa-noglu-18.3.4-drivers
/nix/store/073km24qkqr808q7zvz785smzp6ck592-mesa-noglu-18.3.4-drivers     210949288

So that's a five-factor increase! Not surprising given that it probably includes all of LLVM.

[athas]

I managed to move the OpenCL parts into a separate .opencl output, which means the .drivers output size is essentially unchanged compared to without OpenCL. I can make a pull request with my changes to discuss further, but they still depend on first or concurrently bumping libclc (pull request here).

[stale[bot]]

Hello, I'm a bot and I thank you in the name of the community for opening this issue.

To help our human contributors focus on the most-relevant reports, I check up on old issues to see if they're still relevant. This issue has had no activity for 180 days, and so I marked it as stale, but you can rest assured it will never be closed by a non-human.

The community would appreciate your effort in checking if the issue is still valid. If it isn't, please close it.

If the issue persists, and you'd like to remove the stale label, you simply need to leave a comment. Your comment can be as simple as "still important to me". If you'd like it to get more attention, you can ask for help by searching for maintainers and people that previously touched related code and @ mention them in a comment. You can use Git blame or GitHub's web interface on the relevant files to find them.

Lastly, you can always ask for help at our Discourse Forum or at #nixos' IRC channel.

[athas]

Still important to me, but not high priority. I'll look at it again eventually.