Nixpkgs: `nixos-rebuild switch --upgrade` over ssh may put system into an unusable state

[nagisa]

Issue description

Upgrading with nixos-rebuild switch --upgrade may put system into an unusable state by disabling services and terminating the SSH session, thus killing the rebuild process(?)

$ sudo nixos-rebuild switch --upgrade
unpacking channels...
building Nix...
building the system configuration...
these derivations will be built:
  /nix/store/ngm996cnslb1yz15g09hcd8bzhsln138-system-path.drv
  /nix/store/0b3agdfcv1ax3qn9a2bi4ri4c5a0wrpm-dbus-1.drv
  /nix/store/bcy7asyvxa5xza1bdy5ph8qx0viadfmr-unit-dbus.service.drv
  /nix/store/8v6mdb50nbr73x3k18v39cg2wyiahbn0-user-units.drv
  /nix/store/i3q3ycxzpnqyy811c4l67p75pbl5spvp-unit-polkit.service.drv
  /nix/store/rp6j33aazxv86pm5j6r4apc9pqknpdib-unit-systemd-fsck-.service.drv
  /nix/store/hpgr2fllnbmmjhpb74ip1wlb8zrj63y2-system-units.drv
  /nix/store/yc462a9jb3qwsc9yfygdwcyk35krbssd-etc.drv
  /nix/store/8hl7rjwzd27zasf87nxlj6jkwfcpkmpv-nixos-system-shirobox-18.09pre137207.92f0d31b949.drv
these paths will be fetched (194.31 MiB download, 1694.29 MiB unpacked):
  /nix/store/3cv3s9lv79qjkqyafbclbk03gz5z813w-ghc-8.2.2
  /nix/store/9vgm81xw5z9psmizacwbi83c88xabam8-clang-wrapper-5.0.1
  /nix/store/nz60fx1vyva1a2rsw0jp0qsrjq91xhbw-clang-5.0.1
copying path '/nix/store/3cv3s9lv79qjkqyafbclbk03gz5z813w-ghc-8.2.2' from 'https://cache.nixos.org'...
copying path '/nix/store/nz60fx1vyva1a2rsw0jp0qsrjq91xhbw-clang-5.0.1' from 'https://cache.nixos.org'...
copying path '/nix/store/9vgm81xw5z9psmizacwbi83c88xabam8-clang-wrapper-5.0.1' from 'https://cache.nixos.org'...
building '/nix/store/ngm996cnslb1yz15g09hcd8bzhsln138-system-path.drv'...
Use of uninitialized value $propagated in split at /nix/store/9g4wc31j7a2xp22xpgwr0qssfxahxdzl-builder.pl line 146.
created 6358 symlinks in user environment
building '/nix/store/0b3agdfcv1ax3qn9a2bi4ri4c5a0wrpm-dbus-1.drv'...
building '/nix/store/i3q3ycxzpnqyy811c4l67p75pbl5spvp-unit-polkit.service.drv'...
building '/nix/store/rp6j33aazxv86pm5j6r4apc9pqknpdib-unit-systemd-fsck-.service.drv'...
building '/nix/store/bcy7asyvxa5xza1bdy5ph8qx0viadfmr-unit-dbus.service.drv'...
building '/nix/store/hpgr2fllnbmmjhpb74ip1wlb8zrj63y2-system-units.drv'...
building '/nix/store/8v6mdb50nbr73x3k18v39cg2wyiahbn0-user-units.drv'...
building '/nix/store/yc462a9jb3qwsc9yfygdwcyk35krbssd-etc.drv'...
building '/nix/store/8hl7rjwzd27zasf87nxlj6jkwfcpkmpv-nixos-system-shirobox-18.09pre137207.92f0d31b949.drv'...
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
    LANGUAGE = (unset),
    LC_ALL = (unset),
    LANG = "en_GB.UTF-8"
    are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").
stopping the following units: audit.service, cpufreq.service, kmod-static-nodes.service, network-link-enp24s0.service, network-local-commands.service, network-setup.service, nix-daemon.service, nix-daemon.socket, nscd.service, rngd.service, rtkit-daemon.service, samba-nmbd.service, samba-smbd.service, samba-winbindd.service, system.slice, systemd-modules-load.service, systemd-sysctl.service, systemd-timesyncd.service, systemd-tmpfiles-clean.timer, systemd-tmpfiles-setup-dev.service, systemd-udev-settle.service, systemd-udev-trigger.service, systemd-udevd-control.socket, systemd-udevd-kernel.socket, systemd-udevd.service, transmission.service, zfs-import-sb.service, zfs-scrub.timer, zfs-snapshot-daily.timer, zfs-snapshot-frequent.timer, zfs-snapshot-hourly.timer, zfs-snapshot-monthly.timer, zfs-snapshot-weekly.timer
Connection to 192.168.1.5 closed by remote host.
Connection to 192.168.1.5 closed.

Steps to reproduce

No idea. I ran nixos-rebuild switch --upgrade over a ssh session.

Technical details

Please run nix-shell -p nix-info --run "nix-info -m" and paste the results.

Can’t access my machine now.

[nixy]

Since it tried to restart networking this upgrade probably won't be able to be done using nixos-rebuild switch over an SSH connection.

The nixos-rebuild command can also operate in boot and test modes which should help you work around this. If you do nixos-rebuild boot --upgrade then it won't switch into the new configuration till you reboot (where it would have to start up networking again anyways) and nixos-rebuild test --upgrade would immediately switch you into it without making it the boot default.

You may need to try rolling back and using the boot option to get back to a working state.

[nyanloutre]

A good habit may be to run commands that may break (or temporarily down) the network in a tmux session, so if the SSH link break it will continue to run on the server

[nagisa]

I interface with this machine chiefly over ssh and since this is a local network, I don’t have a pretext to be worried about network breaking, which is why I don’t tend to use something like nohup or tmux. This is the first time over, I think, a year of nixos use that nixos-rebuild switch decides to restart networking in such a crude manner – it never happened before even after I changed network configuration.

However, no network configuration changes have been made so I had no reason to believe nixos-rebuild would restart networking (it seems it actually restarted every service there is) this time either.

It would be ideal to me if there was a way to block nixos-rebuild switch if it would involve killing networking or the ssh daemon – otherwise this will definitely hit me again when I expect this the least.

---

I’ll keep in mind to only ever use --upgrade in conjunction with boot.

[ElvishJerricco]

It might be good if nixos-rebuild did the activation in a systemd-run. This way all forms of TTY terminating are non-damaging. I'd still want the activation log to go to the TTY though; not sure if that's possible.

[ElvishJerricco]

@nagisa In the meantime, you can probably just do alias nixos-rebuild="systemd-run -t --wait /run/current-system/sw/bin/nixos-rebuild". I believe that should be resilient to this kind of failure.

[deliciouslytyped]

https://github.com/NixOS/nixpkgs/issues/34902

[stale[bot]]

Thank you for your contributions.

This has been automatically marked as stale because it has had no activity for 180 days.

If this is still important to you, we ask that you leave a comment below. Your comment can be as simple as "still important to me". This lets people see that at least one person still cares about this. Someone will have to do this at most twice a year if there is no other activity.

Here are suggestions that might help resolve this more quickly:

1. Search for maintainers and people that previously touched the related code and @ mention them in a comment.
2. Ask on the NixOS Discourse.
3. Ask on the #nixos channel on irc.freenode.net.