Nixpkgs: [18.09] X-Server fails to start since latest update

[matthiasbeyer]

Issue description

I cannot start my X server since the latest stable channel update.

The channel was forwarded: 235487585ed to 4c0be408155.
All relevant updates _I_ see are kernel updates (I'm running 4.14 LTS kernel). But maybe I'm missing something.

The journal tells me:

[...]
Mär 04 07:32:46 terry display-manager[1281]: (II) No input driver specified, ignoring this device.
Mär 04 07:32:46 terry display-manager[1281]: (II) This device may have been added with another device file.
Mär 04 07:32:46 terry display-manager[1281]: (II) config/udev: Adding input device (unnamed) (/dev/tty8)
Mär 04 07:32:46 terry display-manager[1281]: (II) No input driver specified, ignoring this device.
Mär 04 07:32:46 terry display-manager[1281]: (II) This device may have been added with another device file.
Mär 04 07:32:46 terry display-manager[1281]: (II) config/udev: Adding input device (unnamed) (/dev/tty9)
Mär 04 07:32:46 terry display-manager[1281]: (II) No input driver specified, ignoring this device.
Mär 04 07:32:46 terry display-manager[1281]: (II) This device may have been added with another device file.
[...]

Steps to reproduce

Not sure.

(Edit: Running a ThinkPad X220 here, maybe someone can reproduce with this information alone)

Technical details

• system: "x86_64-linux"
• host os: Linux 4.14.104, NixOS, 18.09.2292.4c0be408155 (Jellyfish)
• multi-user?: yes
• sandbox: yes
• version: nix-env (Nix) 2.1.3
• channels(root): "nixos-18.09.2292.4c0be408155"
• nixpkgs: /nix/var/nix/profiles/per-user/root/channels/nixos

More information can be provided, of course.

Pinging

Pinging stable channel maintainers and kernel guys here.

@NeQuissimus updated the 4.14 kernel in this cycle.
Other kernel people: @dezgeg @FRidh (just taking randomly two instead of pinging all of you).

[matthiasbeyer]

If this becomes important: The kernel was updated from 4.14.102 to 4.14.104. The kernel log can be shown with this:

# in the kernel repository
$ git remote add stable git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git
$ git fetch stable linux-4.14.y
$ git log --oneline 5461ace2e2f7..30921fc1e5fc

Lets just hope the bug is in nixpkgs/nixos.

---

My system configuration changed slighly between my last generation and my problematic one, but I only removed xfce4 stuff because I switched back to i3. So nothing too crazy. And the journal does not show anything xfce-specific as far as I can tell.

[NeQuissimus]

Any chance your services.xserver.videoDriver is incorrect?
I have not seen any issues with the updated kernels yet.

[matthiasbeyer]

I did not change (nor touch) this setting on this device:

$  git log --grep videoDrivers --format="%ai"
2016-05-24 11:42:52 +0200
2016-05-24 11:41:38 +0200
2016-05-22 18:46:19 +0200

[vcunat]

(II) lines from X log tend to have low importance. Are there no (EE) or at least (WW) lines? I have logs (also) in /var/log/X.0.log{,.old}.

[matthiasbeyer]

$ journalctl -b -1 | grep -E "(EE|WW)"
Mär 04 07:31:18 terry kernel: DMI: LENOVO 4290A11/4290A11, BIOS 8DET66WW (1.36 ) 10/22/2012
Mär 04 07:31:19 terry kernel: ata1.00: ACPI cmd f5/00:00:00:00:00:a0 (SECURITY FREEZE LOCK) filtered out
Mär 04 07:31:19 terry kernel: ata1.00: ACPI cmd f5/00:00:00:00:00:a0 (SECURITY FREEZE LOCK) filtered out
Mär 04 07:31:19 terry kernel: thinkpad_acpi: ThinkPad BIOS 8DET66WW (1.36 ), EC unknown
Mär 04 07:31:21 terry smartd[785]: Device: /dev/sda [SAT], Samsung SSD 850 PRO 256GB, S/N:S251NX0HC00131F, WWN:5-002538-8402a77a3, FW:EXM02B6Q, 256 GB
Mär 04 07:31:21 terry display-manager[844]:         (WW) warning, (EE) error, (NI) not implemented, (??) unknown.
Mär 04 07:31:21 terry display-manager[844]: (WW) Open ACPI failed (/var/run/acpid.socket) (No such file or directory)
Mär 04 07:31:21 terry display-manager[844]: (WW) Falling back to old probe method for cirrus
Mär 04 07:31:21 terry display-manager[844]: (WW) Falling back to old probe method for modesetting
Mär 04 07:31:21 terry display-manager[844]: > Internal error:   Could not resolve keysym XF86WWAN
Mär 04 07:31:22 terry display-manager[844]: > Internal error:   Could not resolve keysym XF86WWAN
Mär 04 07:31:26 terry display-manager[974]:         (WW) warning, (EE) error, (NI) not implemented, (??) unknown.
Mär 04 07:31:26 terry display-manager[974]: (WW) Open ACPI failed (/var/run/acpid.socket) (No such file or directory)
Mär 04 07:31:26 terry display-manager[974]: (WW) Falling back to old probe method for cirrus
Mär 04 07:31:26 terry display-manager[974]: (WW) Falling back to old probe method for modesetting
Mär 04 07:31:26 terry display-manager[974]: > Internal error:   Could not resolve keysym XF86WWAN
Mär 04 07:31:26 terry display-manager[974]: > Internal error:   Could not resolve keysym XF86WWAN
Mär 04 07:31:32 terry display-manager[1071]:         (WW) warning, (EE) error, (NI) not implemented, (??) unknown.
Mär 04 07:31:32 terry display-manager[1071]: (WW) Open ACPI failed (/var/run/acpid.socket) (No such file or directory)
Mär 04 07:31:32 terry display-manager[1071]: (WW) Falling back to old probe method for cirrus
Mär 04 07:31:32 terry display-manager[1071]: (WW) Falling back to old probe method for modesetting
Mär 04 07:31:32 terry display-manager[1071]: > Internal error:   Could not resolve keysym XF86WWAN
Mär 04 07:31:32 terry display-manager[1071]: > Internal error:   Could not resolve keysym XF86WWAN
Mär 04 07:31:40 terry kernel: Hardware name: LENOVO 4290A11/4290A11, BIOS 8DET66WW (1.36 ) 10/22/2012
Mär 04 07:31:40 terry kernel: Hardware name: LENOVO 4290A11/4290A11, BIOS 8DET66WW (1.36 ) 10/22/2012
Mär 04 07:31:40 terry kernel: Hardware name: LENOVO 4290A11/4290A11, BIOS 8DET66WW (1.36 ) 10/22/2012
Mär 04 07:32:26 terry display-manager[1208]:         (WW) warning, (EE) error, (NI) not implemented, (??) unknown.
Mär 04 07:32:26 terry display-manager[1208]: (WW) Open ACPI failed (/var/run/acpid.socket) (No such file or directory)
Mär 04 07:32:26 terry display-manager[1208]: (WW) Falling back to old probe method for cirrus
Mär 04 07:32:26 terry display-manager[1208]: (WW) Falling back to old probe method for modesetting
Mär 04 07:32:26 terry display-manager[1208]: > Internal error:   Could not resolve keysym XF86WWAN
Mär 04 07:32:26 terry display-manager[1208]: > Internal error:   Could not resolve keysym XF86WWAN
Mär 04 07:32:46 terry display-manager[1281]:         (WW) warning, (EE) error, (NI) not implemented, (??) unknown.
Mär 04 07:32:46 terry display-manager[1281]: (WW) Open ACPI failed (/var/run/acpid.socket) (No such file or directory)
Mär 04 07:32:46 terry display-manager[1281]: (WW) Falling back to old probe method for cirrus
Mär 04 07:32:46 terry display-manager[1281]: (WW) Falling back to old probe method for modesetting
Mär 04 07:32:46 terry display-manager[1281]: > Internal error:   Could not resolve keysym XF86WWAN
Mär 04 07:32:46 terry display-manager[1281]: > Internal error:   Could not resolve keysym XF86WWAN

$ ls /var/log | grep X | wc -l
0

[vcunat]

Hmm, I can't see anything fishy in there either :-/

[matthiasbeyer]

It sounds more and more like a kernel bug, at least to me (hint: I have no experience with kernel bugs, so maybe this is stupid).

How to proceed?

[vcunat]

Looking at diff between the two commits, kernel seems the most likely. Without some concrete info like logs, it seems hard to report anywhere, so I would most likely try to find the particular commit causing this. Bisection should work in the worst case. One may be able to tip-off the candidate by looking at commit log, but I probably can't do that confidently, and log2 grows so slowly...

[vanzef]

I have just opened somehow similar issue: https://github.com/NixOS/nixpkgs/issues/57014
(though my X server is starting but freezes right afterwards)

Mine problem was in "removing obsolete xorg.conf" during switching to a new generation:

removing obsolete symlink ‘/etc/X11/xkb’...
removing obsolete symlink ‘/etc/X11/xorg.conf’...

Can it be that you have the same problem?

Hotfix is to copy the old xorg.conf to the new generation.

[NeQuissimus]

There should be another kernel bump incoming.

Does the same happen with 4.19?

[matthiasbeyer]

Can it be that you have the same problem?

I don't know ... I guess journalctl does not give me the nixos-rebuild logs, right?

There should be another kernel bump incoming.

I'll wait for that.

Does the same happen with 4.19?

I'll test after the 4.14 bump if it doesn't work.

bisect

I guess bisecting is not a option for me right now as I'm on a low-power device (Thinkpad X220) and rebuilding the kernel takes ... long. Too long on that device. And my workstation is not yet connected to the internet as I moved two weeks ago and do not yet have a provider.

Without some concrete info like logs, it seems hard to report anywhere

What kind of logs would we need?

[NeQuissimus]

According to http://howoldis.herokuapp.com/ you should be getting 4.14.105 already

[matthiasbeyer]

(The following might be completely false. Take a big grain of salt NOW):

So what I see in the diff (git diff --name-only) is that there are changes in drivers/gpu. All other stuff (like security or arch) seem to be not relevant here from my understanding.

If we take that as a starting point, we're down to 2 (two) patches that changed the kernel in drivers/gpu between 4.14.102 and 4.14.104:

6964c1d5062d drm/i915/fbdev: Actually configure untiled displays
170c71065fec drm/meson: add missing of_node_put

But the commit messages and the code do not help (at least me).

[vanzef]

I don't know ... I guess journalctl does not give me the nixos-rebuild logs, right?

I'm not sure, but you can try copying xorg.conf from the working generation (or rebuild generation) to verify if it is the problem.

[matthiasbeyer]

Unfortunately, I get the same error with kernel 4.14.105

[matthiasbeyer]

I will now test with kernel commit 6964c1d5062de1fb26864f59b41a5ad00f641db9 reverted. Maybe it's a lucky shot.

(More info: I will build starting from the channel generation that worked for me, with kernel 4.14.104 but the one patch reverted.)

[matthiasbeyer]

Fuckin hell. I compiled for two hours... And when I tried to boot it tells me that it cannot find the fuckin /dev/sda

Yeah thanks, I'm going to the club now

[matthiasbeyer]

So after a night of thinking, I wonder why my custom kernel does not boot. Because all that is (should be) different is the kernel, build with one damn patch reverted... why doesn't it find the root filesystem? Of course you guys cannot tell me why, but it feels somewhat strange, I think you understand...

[matthiasbeyer]

Okay, now the fun beginns:

I tried the following:

1. Updating the channel (to 37694c8cc0)
2. git checkout the configuration from the last update that worked
3. build the system
4. reboot
5. works

Damn.

Now here's my config diff from "works" to "doesn't work" (excluding some private stuff):

$ git diff HEAD..master  --name-only
nixos/lucy.nix
nixos/pkgs/desktopPackages.nix
nixos/programs/utils.nix
nixos/services/x-noautostart.nix
nixos/services/x.nix
nixos/terry.nix
vdirsyncer/config

$ git diff HEAD..master -- nixos/
[ excluded because different machine: /nixos/lucy.nix ]
diff --git a/nixos/pkgs/desktopPackages.nix b/nixos/pkgs/desktopPackages.nix
index c3af2c8..39e02b8 100644
--- a/nixos/pkgs/desktopPackages.nix
+++ b/nixos/pkgs/desktopPackages.nix
@@ -15,9 +15,6 @@ let
     ((utils pkgs).optimizeForThisHost (muttWithoutThings pkgs)
     );

-  xfce4term = pkgs:
-    (utils pkgs).optimizeForThisHost pkgs.xfce4-12.xfce4terminal;
-
 in
 pkgs: [
   pkgs.acpi
@@ -59,35 +56,6 @@ pkgs: [
   pkgs.youtube-dl
   pkgs.zathura

-  # Packages for desktop env
-  #pkgs.xfce4-12.xinitrc
-  pkgs.arc-icon-theme
-  pkgs.arc-theme
-  pkgs.adapta-gtk-theme
-  pkgs.papirus-icon-theme
-  pkgs.xfce.xfce4_xkb_plugin
-  pkgs.xfce4-12.gtk-xfce-engine
-  pkgs.xfce4-12.xfce4-sensors-plugin
-  pkgs.xfce4-12.xfce4_appfinder
-  pkgs.xfce4-12.xfce4_genmon_plugin
-  pkgs.xfce4-12.xfce4_battery_plugin
-  pkgs.xfce4-12.xfce4_datetime_plugin
-  pkgs.xfce4-12.xfce4_mpc_plugin
-  pkgs.xfce4-12.xfce4_mailwatch_plugin
-  pkgs.xfce4-12.xfce4_timer_plugin
-  pkgs.xfce4-12.xfce4_power_manager
-  pkgs.xfce4-12.xfce4icontheme
-  pkgs.xfce4-12.xfce4mixer
-  pkgs.xfce4-12.xfce4notifyd
-  pkgs.xfce4-12.xfce4panel
-  pkgs.xfce4-12.xfce4session
-  pkgs.xfce4-12.xfce4settings
-  (xfce4term pkgs)
-  pkgs.xfce4-12.xfce4volumed
-  pkgs.xfce4-12.xfconf
-  pkgs.xfce4-12.xfdesktop
-  pkgs.xfce4-12.xfwm4
-
   pkgs.plymouth

   # Finance packages
diff --git a/nixos/programs/utils.nix b/nixos/programs/utils.nix
deleted file mode 100644
index e76fb7a..0000000
--- a/nixos/programs/utils.nix
+++ /dev/null
@@ -1,16 +0,0 @@
-{
-  makei3config = { alacrittyConfigFilePath, i3Config }:
-    (builtins.toFile "i3.conf" ''
-      ${i3Config}
-
-      # start a terminal
-      bindsym Mod4+Return exec xterm -bg grey10 -fg grey90
-
-      # start a terminal in floating mode
-      bindsym Mod4+Shift+Return exec xterm -bg grey10 -fg grey90 -geometry 80x24 -class "floating_term" -name "floating_term"
-      for_window [class="floating_term"] floating toggle
-
-      exec xterm -bg grey10 -fg grey90
-    '');
-
-}
[ excluded because not in configuration: /nixos/services/x-noautostart.nix ]
diff --git a/nixos/services/x.nix b/nixos/services/x.nix
index 1b5cd32..b335e14 100644
--- a/nixos/services/x.nix
+++ b/nixos/services/x.nix
@@ -1,7 +1,4 @@
-{ pkgs
-, i3Config
-, ...
-}:
+{ pkgs, configDir, ...  }:

 {
   autorun    = true;
@@ -21,6 +18,6 @@

   windowManager.i3 = {
     enable = true;
-    configFile = i3Config;
+    configFile = "${configDir}/i3/config";
   };
 }
diff --git a/nixos/terry.nix b/nixos/terry.nix
index 888d83e..3102c6b 100644
--- a/nixos/terry.nix
+++ b/nixos/terry.nix
@@ -23,16 +23,6 @@ let
     sidebar
   ];

-  # Funny shit here: We generate a custom alacritty config per hostName
-  #
-  # That config is then used in the i3Config for the bindings for starting
-  # terminals
-  #
-  # And that config is then used in the x-server service configuration for i3
-  alacrittyConfig = import ./programs/alacritty.nix { textsize = "7.0"; };
-
-  defaulti3Config = (builtins.readFile ../i3/config);
-
   utils = import "${configDir}/programs/utils.nix";
 in

@@ -71,12 +61,9 @@ in
   boot.initrd.luks.cryptoModules = [ "aes" "sha256" "sha1" "cbc" ];

   services.xserver = (import "${configDir}/services/x.nix" {
-      pkgs = pkgs;
-      i3Config = (utils.makei3config {
-        i3Config = defaulti3Config;
-        alacrittyConfigFilePath = alacrittyConfig;
-      });
-    });
+    pkgs = pkgs;
+    configDir = configDir;
+  });

   networking = {

@@ -98,7 +85,6 @@ in
     CONFIG_DIR  = "/home/m/config";
     HOST_COLOR  = ''\e[1;30m'';
     EDITOR      = "vim";
-    TERMINAL    = "xfce4-terminal";
   };

   # Program configuration

So from what I see (and maybe I'm missing something, but I looked really hard several times) I do _not_ see anything that could cause the _xserver_ to crash on startup. I removed some xfce packages from my configuration, removed some utility functions, removed alacritty-config-building and removed an environment variable. Nothing that could cause an xserver to crash, right?

---

Either way, this seems _not_ to be a kernel bug (thank god!), so please @NeQuissimus remove the kernel label.

[matthiasbeyer]

I don't know why, but this fixes my problem (tested with nixos-rebuild build-vm && ./result/bin/run-terry-vm):

diff --git a/nixos/services/x.nix b/nixos/services/x.nix
index b335e14..8af015c 100644
--- a/nixos/services/x.nix
+++ b/nixos/services/x.nix
@@ -1,4 +1,4 @@
-{ pkgs, configDir, ...  }:
+{ pkgs, ...  }:

 {
   autorun    = true;
@@ -16,8 +16,5 @@
     };
   };

-  windowManager.i3 = {
-    enable = true;
-    configFile = "${configDir}/i3/config";
-  };
+  windowManager.i3.enable = true;
 }

[vcunat]

With references like this I'm often unsure whether I get reference to the source path or to a copy inside the nix store :-) but perhaps that wouldn't make a difference in bootability in _your_ case?

[matthiasbeyer]

Yeah, maybe that was the issue. configDir is set to /home/m/config, but if that is joined onto a nix path it might end up being /nix/store/somehash-foo/home/m/config, ... Maybe putting the config into the store in this step would be the right thing to do here.