Nixpkgs: NixOS Blender OpenCL GPU acceleration?

[lukego]

Project description

Nixpkgs includes Blender but I haven't found any instructions for enabling OpenCL GPU acceleration for use with AMD Radeon hardware. Is this supported in any way by the nixpkgs Blender package and/or NixOS?

[MetaDark]

I just tried enabling it through Preferences ‣ System ‣ Cycles Render Devices, but I get:

No compatible GPUs found for path tracing
Cycles will render on CPU

I'm using a Radeon RX 590.

It seems like OpenCL drivers aren't enabled by default, but can be enabled through hardware.opengl.extraPackages.

[MetaDark]

Hmm, I couldn't find any drivers packaged yet for AMD, but it looks like they can be built from https://github.com/RadeonOpenCompute/ROCm

[danieldk]

No experience with this at all, but just wanted to note that @acowley has packaged the ROCm stack, including OpenCL drivers for NixOS:

https://github.com/nixos-rocm/nixos-rocm

[lukego]

@acowley has packaged the ROCm stack

Thanks for pointing that out. This packaging works for me. Blender system settings then offers blender "Navi 10" under "Cycles Render Devices -> OpenCL."

Can also be that I've already been experiencing GPU-accelerated Blender via the Eevee renderer using OpenGL and that OpenCL is only needed for the Cycles rendering backend. I should do some research on this! Meanwhile thanks again I'm glad to have all drivers working (albeit with seemingly many segfault crashes in the first minutes playing with OpenCL/Cycles and the overlay...)

[acowley]

There is also a ROCm 3.5 branch on my fork that gets pushed to cachix every night. The binary cache (cachix use nixos-rocm) should make it relatively quick to find out if 3.5 improves things for you.

[lukego]

Thanks @acowley. Closing this issue since everything seems to be moving forward in NixOS/OpenCL-land.

[danieldk]

@acowley Are you planning to submit the base ROCm derivations to nixpkgs in the future? I don't know much about the specifics, but as a happy user of your package set (we have one machine with Radeon VIIs for machine learning), I think that this would benefit many AMD GPU-wielding NixOS users.

[acowley]

@danieldk With ROCm 3.5 this is much more bearable since we now need 1 rather than 3 builds of LLVM+clang. Still it is currently a 6+ hour build for CI (cachix makes this only a couple minutes for most daily rebuilds). Do you think the folks tending to the main nixpkgs hydra would be okay with that?

[danieldk]

@danieldk With ROCm 3.5 this is much more bearable since we now need 1 rather than 3 builds of LLVM+clang. Still it is currently a 6+ hour build for CI (cachix makes this only a couple minutes for most daily rebuilds). Do you think the folks tending to the main nixpkgs hydra would be okay with that?

For everything (including PyTorch, Tensorflow, etc.) or just the ROCm stack? CI here is GitHub Actions, right?

I think it would already help if the core ROCm libraries are in nixpkgs for use cases like this issue. Since ROCm is the only fully open source stack that has the potential to compete with CUDA. IMO it would be good to support that.

The largest impact would probably be during staging merges. @FRidh @vcunat would adding (part of) ROCm to nixpkgs be possible from a Hydra perspective?

[acowley]

@danieldk For just OpenCL and HIP, the GitHub Actions build time is only around 2 hours. I'm not sure what you'd want to include as core libraries, but the 6 hour number does not yet get up to PyTorch or Tensorflow, so perhaps 6 hours is the right figure (things like MIOpen, rocBLAS, and rocFFT).

I like the idea of easing discoverability and making installation easier, so I would be happy if someone wanted to maintain derivations in nixpkgs that followed a nixos-rocm fork. I would not want to maintain nixos-rocm itself in nixpkgs, as ROCm changes so rapidly, it takes quite a long time for PRs to be merged into nixpkgs, and we just got this CI sorted out with cachix that is amazingly helpful 😀. So I think it makes sense to maintain infrastructure for the overlay independent from nixpkgs, but I agree that it would be a feather in NixOS's cap to make ROCm easily installable.

[veprbl]

I think you don't need to worry about the Hydra. Worst case ROCm expressions will be not built on it, but at least they will be in sync with the rest of nixpkgs.

it takes quite a long time for PRs to be merged into nixpkgs

I'm sorry to hear that. It's often the case that some changes just don't have enough manpower with relevant interests or hardware to do the peer review properly. However, it seems like there are some interested people in this issue.

[acowley]

@veprbl We currently do a daily CI build against nixos-unstable and push those artifacts to cachix, so we're good about staying in sync. I'm a big fan of nixpkgs, this is just a case where the maintenance burden is already probably too much for me, and I do not want to sign myself up for anything that will make it harder. ROCm updates rapidly, let's say once a month, which involves new LLVM and new clang builds, along with all of their own pieces. How long it takes to update the nix packaging varies a lot -- it's not instantaneous as something usually breaks -- let's say two weeks. If the ROCm packages are not built on hydra, then we'll want to steer people to the cachix setup in any case since the build is so heavy.

I'm totally okay with someone mirroring things in nixpkgs as long as it doesn't preclude use of the overlay, but perhaps this is a case where we could talk about an external overlay in the nixpkgs manual? In some ways, I think NixOS is currently the best ROCm platform out there, but I do appreciate that my fork of an obscure overlay is not as visible as being in nixpkgs itself.

[danieldk]

I'm totally okay with someone mirroring things in nixpkgs as long as it doesn't preclude use of the overlay,

I am willing to invest the time to prepare a base set of packages (probably everything necessary for OpenCL) and maintain it. I'll try to stay as closely to your packages so that it would be easy override things with your overlay.

Let me make a draft PR. Is it ok to ping you once it's done?

but perhaps this is a case where we could talk about an external overlay in the nixpkgs manual?

:+1: nixos-rocm is a great addition to the ecosystem. I think some of my colleagues were jealous how easy it is to use ROCm on NixOS. It would be great if it was featured more prominently.

[lukego]

@acowley I'm trying to test 3.5.x to see if that improves stability but I'm hitting this problem with nixpkgs master:

while evaluating definitions from `/home/luke/git/nixpkgs/nixos/modules/hardware/opengl.nix':
while evaluating 'dischargeProperties' at /home/luke/git/nixpkgs/lib/modules.nix:467:25, called from /home/luke/git/nixpkgs/lib/modules.nix:396:137:
while evaluating the attribute 'value' at /home/luke/git/nixpkgs/lib/types.nix:269:40:
while evaluating the attribute 'passAsFile' of the derivation 'opengl-drivers' at /home/luke/git/nixpkgs/pkgs/build-support/trivial-builders.nix:7:7:
while evaluating the attribute 'src' of the derivation 'rocm-opencl-icd' at /home/luke/git/nixos-rocm/pkgs/development/libraries/rocm-opencl-icd.nix:5:3:
while evaluating the attribute 'text' of the derivation 'amdocl64.icd' at /home/luke/git/nixpkgs/pkgs/build-support/trivial-builders.nix:7:7:
while evaluating the attribute 'buildInputs' of the derivation 'rocm-opencl-runtime-3.5.0' at /home/luke/git/nixos-rocm/pkgs/development/libraries/rocm-opencl-runtime.nix:25:3:
while evaluating the attribute 'buildInputs' of the derivation 'rocclr-3.5.0' at /home/luke/git/nixpkgs/pkgs/stdenv/generic/make-derivation.nix:192:11:
while evaluating the attribute 'buildInputs' of the derivation 'comgr-3.5.0' at /home/luke/git/nixpkgs/pkgs/stdenv/generic/make-derivation.nix:192:11:
while evaluating the attribute 'depsTargetTargetPropagated' of the derivation 'clang-unwrapped-wrapper-3.5.0' at /home/luke/git/nixpkgs/pkgs/stdenv/generic/make-derivation.nix:192:11:
while evaluating 'getOutput' at /home/luke/git/nixpkgs/lib/attrsets.nix:464:23, called from undefined position:
while evaluating anonymous function at /home/luke/git/nixpkgs/pkgs/stdenv/generic/make-derivation.nix:160:17, called from undefined position:
while evaluating anonymous function at /home/luke/git/nixpkgs/pkgs/top-level/aliases.nix:27:22, called from undefined position:
while evaluating 'removeDistribute' at /home/luke/git/nixpkgs/pkgs/top-level/aliases.nix:15:22, called from /home/luke/git/nixpkgs/pkgs/top-level/aliases.nix:27:29:
while evaluating 'isDerivation' at /home/luke/git/nixpkgs/lib/attrsets.nix:305:18, called from /home/luke/git/nixpkgs/pkgs/top-level/aliases.nix:16:8:
while evaluating 'removeRecurseForDerivations' at /home/luke/git/nixpkgs/pkgs/top-level/aliases.nix:8:33, called from /home/luke/git/nixpkgs/pkgs/top-level/aliases.nix:28:31:
while evaluating 'checkInPkgs' at /home/luke/git/nixpkgs/pkgs/top-level/aliases.nix:22:20, called from /home/luke/git/nixpkgs/pkgs/top-level/aliases.nix:29:32:
libstdcxx hook has been removed because cc-wrapper is now directly aware of the c++ standard library intended to be used.

Any tips for toubleshooting? Should I report this somewhere else?

[acowley]

@lukego Is that from my fork? I haven't merged the 3.5.x branch to master on the main repo yet since some higher level libraries are broken, and development happens on separate forks because I don't have permissions to properly setup CI on the main nixos-rocm repo.