Nixpkgs: loginctl attach fails with "Could not attach device: Read-only file system" message

[akamaus]

Describe the bug
A modern way to configure multiseat systems seems to is to use systemd's loginctl.
https://wiki.debian.org/Multi_Seat_Debian_HOWTO

But if I try creating a new seat by attaching a GPU to it, I get an error.

To Reproduce
Steps to reproduce the behavior:
Issue following command (I guess any peripherial device present in your /sys will go) :

% sudo  loginctl attach seat-right  /sys/devices/pci0000:00/0000:00:02.0/0000:4b:00.0/drm/card0                                   
Could not attach device: Read-only file system

Expected behavior
A new seat named seat-right should be created, it should be visible with
loginctl list-seats

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context
I did some quick investivation, traces lead to https://github.com/systemd/systemd/blob/9494b39dac37894628cf5ef405e6bbc18f69bba9/src/login/logind-dbus.c#L1374

Looks like systemd tries to create a custom udev rule on static etc.

Notify maintainers
@flokli @andir

Metadata
Its basically a nixos-20.03 from channels plus a few additional commits adding some packages on top of it.

nix-shell -p nix-info --run "nix-info -m"                                                                                              ~/src/systemd2 
 - system: `"x86_64-linux"`
 - host os: `Linux 5.4.35, NixOS, 20.03.git.c9d6f28b852 (Markhor)`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.3`
 - channels(root): `""`
 - channels(test): `""`
 - channels(dvyal): `""`
 - nixpkgs: `/home/dvyal/nixpkgs`

Maintainer information:

# a list of nixos modules affected by the problem
module: systemd

[flokli]

Nice find! Of course, /etc/udev/rules.d is read-only on NixOS.

Even on non-NixOS, these rules will stay around on the disk on a reboot. Is that intended behaviour?

If it's not, I wonder if it would make sense to have udev load rules from /run/udev/rules.d (or another, ephemeral location) as well.

Care to open a discussion upstream?

[akamaus]

I think it's ok for these rules to be persistent. Basically you arrange your input devices and gpus to different seats once and then work for long time in this setting.

Care to open a discussion upstream?

You mean persistent vs ephemeral? I guess It's a question of usability and I have no experience with multiseat.

BTW, in the meanwhile I tried a dirty hack of remounting /nix/store rw. To my surprise, problem persists.

[aanderse]

Warning: my multiseat knowledge is almost 10 years old and comes from a time before loginctl and seatctl were considered usable for the most part... everything was managed through xorg.conf and kdm/gdm configuration files.

@akamaus you're right in that most people probably want persistent settings. On NixOS ideally we would want to declaratively configure multiseat through configuration.nix instead of imperatively through loginctl, but it would be nice if logintl still worked (though the results were ephemeral).

[akamaus]

@aanderse, agree, combination of declarative and imperative approach is preferable, like we have with users.

[akamaus]

@aanderse I tried implementing multiseat old way, without loginctl, using separate xorg.conf files. Looks like it's no longer possible, If I just start several X servers on different gpus only one is active at the moment (I can switch using ctrl-alt-Fx), moreover, keyboards and mouses all work with currently active server, although I added InputDevice sections with specific Device options.

I tried setting -seat option for X, this way servers render pictures simultaneously, but one with nondefault -seat can't access input devices.

[ 78290.037] (**) Option "CorePointer"
[ 78290.037] (**) mouse-right: always reports core events
[ 78290.037] (**) evdev: mouse-right: Device: "/dev/input/mouse1"
[ 78290.037] (EE) evdev: mouse-right: Unable to query fd: Inappropriate ioctl for device
[ 78290.056] (EE) PreInit returned 2 for "mouse-right"
[ 78290.056] (II) UnloadModule: "evdev"
[ 78290.056] (II) Using input driver 'evdev' for 'keyboard-right'
[ 78290.056] (EE) systemd-logind: failed to take device /dev/input/by-id/usb-SIGMACHIP_USB_Keyboard-event-kbd: Operation not permitted
[ 78290.056] (**) Option "CoreKeyboard"
[ 78290.056] (**) keyboard-right: always reports core events
[ 78290.056] (**) evdev: keyboard-right: Device: "/dev/input/by-id/usb-SIGMACHIP_USB_Keyboard-event-kbd"
[ 78290.072] (--) evdev: keyboard-right: Vendor 0x1c4f Product 0x26
[ 78290.072] (--) evdev: keyboard-right: Found keys
[ 78290.072] (II) evdev: keyboard-right: Configuring as keyboard
[ 78290.072] (II) XINPUT: Adding extended input device "keyboard-right" (type: KEYBOARD, id 6)

Here configs I used
https://gist.github.com/akamaus/b406b7e137c0dcbb2a85a007824258b5

[flokli]

@akamaus to get to this functionality, you probably need loginctl cooperation to tag various devices to various seats (scroll up 4 lines from your link to https://github.com/systemd/systemd/blob/9494b39dac37894628cf5ef405e6bbc18f69bba9/src/login/logind-dbus.c#L1374).

/run/udev/rules.d is also a valid location for ephemeral udev rules. I'd guess upstream made a deliberate choice on having these persisted across rebuilds.

We have all NixOS config-generated udev rules in /etc/udev/rules.d and don't really have a separate place for "rules created imperatively that should survive reboots". We might want to have, but that's another topic too broad for the scope of this issue.

So my proposal here: What about patching logind-dbus.c to put rules in /run/udev/rules.d and warn users to persist these rules in their NixOS configuration, by printing a message when running loginctl attach?

[akamaus]

@flokli your suggestion works indeed. I applied the following patch to systemd

--- a/pkgs/os-specific/linux/systemd/default.nix
+++ b/pkgs/os-specific/linux/systemd/default.nix
@@ -169,6 +169,8 @@ in stdenv.mkDerivation {

     substituteInPlace src/journal/catalog.c \
       --replace /usr/lib/systemd/catalog/ $out/lib/systemd/catalog/
+
+    substituteInPlace src/login/logind-dbus.c --replace /etc/udev/rules.d /run/udev/rules.d
   '';

   # These defines are overridden by CFLAGS and would trigger annoying

Unfortunately, it triggered MASS rebuild, so I just issued nix-build -A systemd and manually replaced systemd-logind binary. Ugly, but loginctl attach now works, rules appear in /var/run/udev/rules.d, as expected.

Settings seem to be active without a reboot, I was able to attach gpu, mouse and keyboard to new seat and X server launched with X :1 --seat new_seat detected them all.

What else have to be done to merge it?

[flokli]

@akamaus you could have set a custom systemd via systemd.package to try this out, so it wouldn't trigger a mass rebuild, but your approach obviously also worked, even though it's a bit more hacky ;-)

When it comes to "how to merge it":

The best way to fix this long-term would be to upstream a fix. I'm not sure if /etc/udev/rules.d would be writable in NixOS any time soon, nor should we just silently add a patch changing the path to some location that's wiped after a reboot.

IIRC, systemd already has some graceful fallbacks if / is mounted read-only. What about filing a PR to systemd, which falls back to /run/udev/rules.d (and emits a warning changes will be lost after a reboot) if it fails to create the rule inside /etc?

[stale[bot]]

I marked this as stale due to inactivity. → More info

[aanderse]

Still relevant. @Mic92 I could be wrong but were you mentioning something somewhat related to this in #nixos-systemd the other day? Do you have a PR open with upstream?