Nixpkgs: [20.09][21.03] PRIME render offload doesn't work on latest builds

[Alderaeney]

Issue description

PRIME render offload stopped working on the latest update both on 20.09 channel and unstable channel, I've found that changing the bootloader and removing nvidia-drm.modeset=1 sometimes makes it work and sometimes don't, but maybe it's something else I changed.

xrandr --listproviders

Providers: number : 1
Provider 0: id: 0x46 cap: 0xf, Source Output, Sink Output, Source Offload, Sink Offload crtcs: 3 outputs: 4 associated providers: 0 name:modesetting

nvidia-offload glxinfo

name of display: :1
X Error of failed request:  BadValue (integer parameter out of range for operation)
  Major opcode of failed request:  152 (GLX)
  Minor opcode of failed request:  24 (X_GLXCreateNewContext)
  Value in failed request:  0x0
  Serial number of failed request:  39
  Current serial number in output stream:  40

nvidia-smi

No devices were found

Steps to reproduce

1.- Install 20.09 or 21.03
2.- Run nixos-rebuild boot --upgrade
3.- Reboot

Technical details

configuration.nix

{ config, pkgs, ... }:

let
  nvidia-offload = pkgs.writeShellScriptBin "nvidia-offload" ''
    export __NV_PRIME_RENDER_OFFLOAD=1
    export __NV_PRIME_RENDER_OFFLOAD_PROVIDER=NVIDIA-G0
    export __GLX_VENDOR_LIBRARY_NAME=nvidia
    export __VK_LAYER_NV_optimus=NVIDIA_only
    exec -a "$0" "$@"
  '';
in
{
. . .
  # Xserver configuration
  services.xserver = {
    enable = true;

    # Xserver keyboard configuration
    layout = "es";
    xkbOptions = "eurosign:e";

    # Use libinput for trackpad support
    libinput.enable = true;

    # Wacom tablet support
    wacom.enable = true;

    # Use nvidia drivers
    videoDrivers = [ "nvidia" ];

    # Gnome3 desktop configuration
    displayManager = {
      gdm = {
        enable = true;
        wayland = false;
      };
    };
    desktopManager.gnome3.enable = true;
  };

  hardware.nvidia = {
    powerManagement.enable = true;
    prime = {
      offload.enable = true;
      intelBusId = "PCI:0:2:0";
      nvidiaBusId = "PCI:1:0:0";
    };
  };

Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.

• system: "x86_64-linux"
• host os: Linux 5.8.10, NixOS, 21.03pre244416.daaa0e33505 (Okapi)
• multi-user?: yes
• sandbox: yes
• version: nix-env (Nix) 2.3.7
• channels(root): "nixos-21.03pre244416.daaa0e33505"
• nixpkgs: /nix/var/nix/profiles/per-user/root/channels/nixos

[eadwu]

journalctl output?

GPU?

I'm not really sure how to go about making it more resilient. I personally have a 1050 TI and have never had any problems (though I never go on stable since I'm following master so that might be where the source of issues are).

It looks like you're missing NVIDIA-G0 from your providers, might be a issue regarding glamor then from an initial look?

[Alderaeney]

Sorry, I have a 1050 4gb, I've been using this same configuration.nix since some months ago and never failed to work, so I'm assuming this is an issue with a recent update, here are the journalctl and journalctl -xe outputs:
https://gist.github.com/Alderaeney/e52282c609f8d1d0cc5f88e38bc07336
https://gist.github.com/Alderaeney/d31759eb6588672306a22988733a2d88

[kevincox]

I see the following in the log:

sep 27 21:19:51 link-gl63-8rc /nix/store/2ngdc7ln6zsg5bajxqi4fjiwv8fa0cf7-gdm-3.34.1/libexec/gdm-x-session[1864]: (II) NVIDIA GLX Module  450.66  Wed Aug 12 19:41:37 UTC 2020
sep 27 21:19:51 link-gl63-8rc /nix/store/2ngdc7ln6zsg5bajxqi4fjiwv8fa0cf7-gdm-3.34.1/libexec/gdm-x-session[1864]: (II) NVIDIA: The X server supports PRIME Render Offload.
sep 27 21:19:51 link-gl63-8rc /nix/store/2ngdc7ln6zsg5bajxqi4fjiwv8fa0cf7-gdm-3.34.1/libexec/gdm-x-session[1864]: (EE) NVIDIA(GPU-0): Failed to initialize the NVIDIA GPU at PCI:1:0:0.  Please
sep 27 21:19:51 link-gl63-8rc /nix/store/2ngdc7ln6zsg5bajxqi4fjiwv8fa0cf7-gdm-3.34.1/libexec/gdm-x-session[1864]: (EE) NVIDIA(GPU-0):     check your system's kernel log for additional error
sep 27 21:19:51 link-gl63-8rc /nix/store/2ngdc7ln6zsg5bajxqi4fjiwv8fa0cf7-gdm-3.34.1/libexec/gdm-x-session[1864]: (EE) NVIDIA(GPU-0):     messages and refer to Chapter 8: Common Problems in the
sep 27 21:19:51 link-gl63-8rc /nix/store/2ngdc7ln6zsg5bajxqi4fjiwv8fa0cf7-gdm-3.34.1/libexec/gdm-x-session[1864]: (EE) NVIDIA(GPU-0):     README for additional information.
sep 27 21:19:51 link-gl63-8rc /nix/store/2ngdc7ln6zsg5bajxqi4fjiwv8fa0cf7-gdm-3.34.1/libexec/gdm-x-session[1864]: (EE) NVIDIA(GPU-0): Failed to initialize the NVIDIA graphics device!
sep 27 21:19:51 link-gl63-8rc /nix/store/2ngdc7ln6zsg5bajxqi4fjiwv8fa0cf7-gdm-3.34.1/libexec/gdm-x-session[1864]: (EE) NVIDIA(G0): Failing initialization of X screen

I also see that if I enable modesetting. But have the same issue if I don't. It seems like the GPU is available and online but the GLX is not set up. For example I can see nvidia-smi.

% sudo nvidia-smi
Wed Oct  7 20:04:40 2020       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 450.66       Driver Version: 450.66       CUDA Version: 11.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  GeForce GTX 960M    On   | 00000000:01:00.0 Off |                  N/A |
| N/A   57C    P8    N/A /  N/A |      3MiB /  2004MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      1039      G   ...-xorg-server-1.20.8/bin/X        3MiB |
+-----------------------------------------------------------------------------+

But if I pass the __GLX_VENDOR_LIBRARY_NAME=nvidia I get:

% __GLX_VENDOR_LIBRARY_NAME=nvidia glxinfo
name of display: :0
X Error of failed request:  BadValue (integer parameter out of range for operation)
  Major opcode of failed request:  156 (NV-GLX)
  Minor opcode of failed request:  6 ()
  Value in failed request:  0x0
  Serial number of failed request:  86
  Current serial number in output stream:  86

[bryanasdev000]

@kevincox just got it working at https://github.com/NixOS/nixpkgs/issues/90152#issuecomment-711425977.

[lightdiscord]

I had the same issue with gdm, switching to lightdm fixed it. (I'm currently searching how to fix it for gdm).

[kevincox]

Weird, it works fine for me with gdm.

[Alderaeney]

I've tested the solution on #90152 and doesn't work for me, the issue I'm having is that gdm fails to initialize the GPU and consequently it can't configure prime render offload, I'm pretty sure this had to be an update to something (quite probably gdm) because the days before it worked fine with the same configuration.

This is basically the root of the issue:

sep 27 21:19:51 link-gl63-8rc /nix/store/2ngdc7ln6zsg5bajxqi4fjiwv8fa0cf7-gdm-3.34.1/libexec/gdm-x-session[1864]: (II) NVIDIA GLX Module  450.66  Wed Aug 12 19:41:37 UTC 2020
sep 27 21:19:51 link-gl63-8rc /nix/store/2ngdc7ln6zsg5bajxqi4fjiwv8fa0cf7-gdm-3.34.1/libexec/gdm-x-session[1864]: (II) NVIDIA: The X server supports PRIME Render Offload.
sep 27 21:19:51 link-gl63-8rc /nix/store/2ngdc7ln6zsg5bajxqi4fjiwv8fa0cf7-gdm-3.34.1/libexec/gdm-x-session[1864]: (EE) NVIDIA(GPU-0): Failed to initialize the NVIDIA GPU at PCI:1:0:0.  Please
sep 27 21:19:51 link-gl63-8rc /nix/store/2ngdc7ln6zsg5bajxqi4fjiwv8fa0cf7-gdm-3.34.1/libexec/gdm-x-session[1864]: (EE) NVIDIA(GPU-0):     check your system's kernel log for additional error
sep 27 21:19:51 link-gl63-8rc /nix/store/2ngdc7ln6zsg5bajxqi4fjiwv8fa0cf7-gdm-3.34.1/libexec/gdm-x-session[1864]: (EE) NVIDIA(GPU-0):     messages and refer to Chapter 8: Common Problems in the
sep 27 21:19:51 link-gl63-8rc /nix/store/2ngdc7ln6zsg5bajxqi4fjiwv8fa0cf7-gdm-3.34.1/libexec/gdm-x-session[1864]: (EE) NVIDIA(GPU-0):     README for additional information.
sep 27 21:19:51 link-gl63-8rc /nix/store/2ngdc7ln6zsg5bajxqi4fjiwv8fa0cf7-gdm-3.34.1/libexec/gdm-x-session[1864]: (EE) NVIDIA(GPU-0): Failed to initialize the NVIDIA graphics device!
sep 27 21:19:51 link-gl63-8rc /nix/store/2ngdc7ln6zsg5bajxqi4fjiwv8fa0cf7-gdm-3.34.1/libexec/gdm-x-session[1864]: (EE) NVIDIA(G0): Failing initialization of X screen

[bryanasdev000]

@Alderaeney this may be relevant:

https://forums.developer.nvidia.com/t/nvidia-driver-not-yet-supported-for-linux-kernel-5-9/157263

I am without my notebook for now, but as soon as I can I will try to test PRIME on unstable.