Nixpkgs: Update from 16.03 to 16.09, problems with /var/setuid-wrappers

[bsoudan]

Issue description

I just ran an update from 16.03 to 16.09, and at the end of the update, I noticed this:

umount: /var/setuid-wrappers: target is busy
        (In some cases useful info about processes that
         use the device is found by lsof(8) or fuser(1).)
rm: cannot remove '/var/setuid-wrappers': Device or resource busy

Then, in another window, tried a sudo operation but received:

sudo: /run/current-system/sw/bin/sudo must be owned by uid 0 and have the setuid bit set

I poked around a bit, and /var/setuid-wrappers looked a bit wacky, it looked like there were too many levels, as if a symlink operation went bad:

[root@bsoudan-linux:/home/bsoudan]# cd /var/setuid-wrappers/

[root@bsoudan-linux:/var/setuid-wrappers]# ls -al
total 4
drwxr-xr-x 2 root root   60 Oct 25 11:22 .
drwxr-xr-x 8 root root 4096 Oct 10 10:25 ..
lrwxrwxrwx 1 root root   51 Oct 25 11:22 setuid-wrappers.AvZKKe7ojo -> /run/setuid-wrapper-dirs/setuid-wrappers.AvZKKe7ojo

[root@bsoudan-linux:/var/setuid-wrappers]# cd setuid-wrappers.AvZKKe7ojo/

[root@bsoudan-linux:/var/setuid-wrappers/setuid-wrappers.AvZKKe7ojo]# ls -al
total 260
drwxr-xr-x 2 root root         560 Oct 25 11:22 .
drwxr-xr-x 3 root root          60 Oct 25 11:22 ..
-r-s--x--x 1 root root       12856 Oct 25 11:22 chfn
-rw-r--r-- 1 root root          65 Oct 25 11:22 chfn.real
-r-sr-x--- 1 root messagebus 12856 Oct 25 11:22 dbus-daemon-launch-helper
-rw-r--r-- 1 root root          90 Oct 25 11:22 dbus-daemon-launch-helper.real
-r-s--x--x 1 root root       12856 Oct 25 11:22 fusermount
-rw-r--r-- 1 root root          69 Oct 25 11:22 fusermount.real
-r-s--x--x 1 root root       12856 Oct 25 11:22 newgidmap
-rw-r--r-- 1 root root          70 Oct 25 11:22 newgidmap.real
-r-s--x--x 1 root root       12856 Oct 25 11:22 newuidmap
-rw-r--r-- 1 root root          70 Oct 25 11:22 newuidmap.real
-r-s--x--x 1 root root       12856 Oct 25 11:22 ping
-r-s--x--x 1 root root       12856 Oct 25 11:22 ping6
-rw-r--r-- 1 root root          70 Oct 25 11:22 ping6.real
-rw-r--r-- 1 root root          69 Oct 25 11:22 ping.real
-r-s--x--x 1 root root       12856 Oct 25 11:22 pkexec
-rw-r--r-- 1 root root          71 Oct 25 11:22 pkexec.real
-r-s--x--x 1 root root       12856 Oct 25 11:22 polkit-agent-helper-1
-rw-r--r-- 1 root root          91 Oct 25 11:22 polkit-agent-helper-1.real
-r-s--x--x 1 root root       12856 Oct 25 11:22 su
-r-s--x--x 1 root root       12856 Oct 25 11:22 sudo
-r-s--x--x 1 root root       12856 Oct 25 11:22 sudoedit
-rw-r--r-- 1 root root          66 Oct 25 11:22 sudoedit.real
-rw-r--r-- 1 root root          66 Oct 25 11:22 sudo.real
-rw-r--r-- 1 root root          66 Oct 25 11:22 su.real
-r-s--x--x 1 root nogroup    12856 Oct 25 11:22 unix_chkpwd
-rw-r--r-- 1 root root          81 Oct 25 11:22 unix_chkpwd.real

I tried running another 'nixos-rebuild switch' operation, and that appeared to fix everything. No errors this time:

[root@bsoudan-linux:/run/setuid-wrapper-dirs/setuid-wrappers.AvZKKe7ojo]# nixos-rebuild switch
building Nix...
building the system configuration...
activating the configuration...
setting up /etc...

[bsoudan@bsoudan-linux:~]$ cd /var/setuid-wrappers

[bsoudan@bsoudan-linux:/var/setuid-wrappers]$ ls -al
total 260
drwxr-xr-x 2 root root         560 Oct 25 11:26 .
drwxr-xr-x 4 root root          80 Oct 25 11:26 ..
-r-s--x--x 1 root root       12856 Oct 25 11:26 chfn
-rw-r--r-- 1 root root          65 Oct 25 11:26 chfn.real
-r-sr-x--- 1 root messagebus 12856 Oct 25 11:26 dbus-daemon-launch-helper
-rw-r--r-- 1 root root          90 Oct 25 11:26 dbus-daemon-launch-helper.real
-r-s--x--x 1 root root       12856 Oct 25 11:26 fusermount
-rw-r--r-- 1 root root          69 Oct 25 11:26 fusermount.real
-r-s--x--x 1 root root       12856 Oct 25 11:26 newgidmap
-rw-r--r-- 1 root root          70 Oct 25 11:26 newgidmap.real
-r-s--x--x 1 root root       12856 Oct 25 11:26 newuidmap
-rw-r--r-- 1 root root          70 Oct 25 11:26 newuidmap.real
-r-s--x--x 1 root root       12856 Oct 25 11:26 ping
-r-s--x--x 1 root root       12856 Oct 25 11:26 ping6
-rw-r--r-- 1 root root          70 Oct 25 11:26 ping6.real
-rw-r--r-- 1 root root          69 Oct 25 11:26 ping.real
-r-s--x--x 1 root root       12856 Oct 25 11:26 pkexec
-rw-r--r-- 1 root root          71 Oct 25 11:26 pkexec.real
-r-s--x--x 1 root root       12856 Oct 25 11:26 polkit-agent-helper-1
-rw-r--r-- 1 root root          91 Oct 25 11:26 polkit-agent-helper-1.real
-r-s--x--x 1 root root       12856 Oct 25 11:26 su
-r-s--x--x 1 root root       12856 Oct 25 11:26 sudo
-r-s--x--x 1 root root       12856 Oct 25 11:26 sudoedit
-rw-r--r-- 1 root root          66 Oct 25 11:26 sudoedit.real
-rw-r--r-- 1 root root          66 Oct 25 11:26 sudo.real
-rw-r--r-- 1 root root          66 Oct 25 11:26 su.real
-r-s--x--x 1 root nogroup    12856 Oct 25 11:26 unix_chkpwd
-rw-r--r-- 1 root root          81 Oct 25 11:26 unix_chkpwd.real

I'm worried though if I would have rebooted my system would have been foobar'd, even if I booted into an old system.

• System:

[bsoudan@bsoudan-linux:~]$ /run/booted-system/sw/bin/nixos-version 
16.03.1242.c5cbda2 (Emu)

[bsoudan@bsoudan-linux:~]$ /run/current-system/sw/bin/nixos-version 
16.09.824.e4fb65a (Flounder)

• Nix version:

[bsoudan@bsoudan-linux:~]$ /run/current-system/sw/bin/nix-env --version
nix-env (Nix) 1.11.4

[bsoudan@bsoudan-linux:~]$ /run/booted-system/sw/bin/nix-env --version
nix-env (Nix) 1.11.2

• Nixpkgs version: "16.09.824.e4fb65a

[bsoudan]

Related to #18186 and #18124 probably?

[FRidh]

cc @domenkozar

[domenkozar]

Did you upgrade to 16.09 with sudo?

[bsoudan]

Probably, but I couldn't say for sure, the upgrade is definitely long out of my bash history. :(

[res0nat0r]

I appear to be getting hit by this.

$ sudo su -
sudo: /run/current-system/sw/bin/sudo must be owned by uid 0 and have the setuid bit set

$ ls -l /run/current-system/sw/bin/sudo
lrwxrwxrwx 1 root root 66 Jan  1  1970 /run/current-system/sw/bin/sudo -> /nix/store/sgw7jnji7fclpfw7kn049ilrnsrn06x5-sudo-1.8.19p2/bin/sudo

$ ls -l /run/current-system/sw/bin/sudo -L
-r-xr-xr-x 1 root root 149064 Jan  1  1970 /run/current-system/sw/bin/sudo

$  /run/booted-system/sw/bin/nixos-version
16.09.1512.6b28bd0 (Flounder)

$ /run/current-system/sw/bin/nixos-version
17.03pre101839.53a2baa (Gorilla)

I did run nixos-rebuild switch via sudo. I'm not sure how to salvage this now that I'm not able to get root access as I don't have a manual root passwd currently set.

[domenkozar]

Do you still have logs for the upgrade?

[res0nat0r]

@domenkozar I haven't rebooted the host, is there a specific logfile where this is stored that I could attach for you?

[domenkozar]

It's just in the standard output. My hypothesis is, since you use sudo it's not able to run the migration for the wrappers.

[res0nat0r]

@domenkozar Ah, sorry unfortunately then I do not. I was adding some NFS mounts to my configuration.nix and restarting the network services killed my ssh session so I don't have the upgrade log anymore.

If I look into booting into single user mode and chmod u+s to the sudo binary would that be enough to fix this issue or do you think there are bigger issues still lingering?

[res0nat0r]

Actually now I see that the sudo binary does in fact have the correct permissions. I logged out of the host and logged in again and am now able to sudo su - to root.

$ ls -lL `which sudo`
-r-s--x--x 1 root root 17728 Mar  1 19:11 /run/wrappers/bin/sudo

Could it be that my existing shell which ran the sudo nixos-rebuild switch just wasn't properly pointing to the new sudo binary due to some shell env swap breakage and that the upgrade in fact worked as intended?

[domenkozar]

Ah yes, it probably points to the old /run/setuid-wrappers/sudo.

[qknight]

i've upgraded from 16.09 to master and i couldn't use su anymore:

➜  ~ su
Password: 
su: Authentication failure
➜  ~ which su   
/run/current-system/sw/bin/su

but running a different su tool:

➜  ~ /run/wrappers/bin/su 
Password: 
root@lenovo-t530>

using the other binary it is working again as a quick solution!

thanks @domenkozar

[Mic92]

@qknight you logout and login again to get /run/wrappers/bin/ in your PATH.

[teh]

Minimal test case for a fresh 16.09 system:

sudo nano /root/.nix-channels # update to 17.03
sudo nix-channel --update
sudo nixos-rebuild switch

[tom@test:~]$ sudo
sudo: /run/current-system/sw/bin/sudo must be owned by uid 0 and have the setuid bit set

@ixmatus cc'ing you in case you have a simple fix in mind, otherwise I'll give it a shot.

[qknight]

@teh: simply 'source /etc/profile' and your commands should be working.

are you using zsh?

[teh]

@qknight Using bash. To be clear: I added a test case so we can fix. I also tested running as root wich works as intended. @domenkozar's theory:

It's just in the standard output. My hypothesis is, since you use sudo it's not able to run the migration for the wrappers.

seems to be correct - the combination of sudo and switch breaks

[ixmatus]

The paths change due to the switch to using /run/wrappers/bin so you need
to log back in to pick up the PATH changes. Does that fix the issue?

How do we communicate this in a upgrade?

[domenkozar]

It would be easiest just to not switch paths. It's more pain than gain

On Wed, Mar 8, 2017, 12:07 Parnell Springmeyer [e-mail]
wrote:

The paths change due to the switch to using /run/wrappers/bin so you need
to log back in to pick up the PATH changes. Does that fix the issue?

How do we communicate this in a upgrade?

On Wed, Mar 8, 2017, 4:29 AM teh [e-mail] wrote:

@qknight https://github.com/qknight Using bash. To be clear: I added a
test case so we can fix. I also tested running as root wich works as
intended. @domenkozar https://github.com/domenkozar's theory:

It's just in the standard output. My hypothesis is, since you use sudo
it's not able to run the migration for the wrappers.

seems to be correct - the combination of sudo and switch breaks

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/NixOS/nixpkgs/issues/19862#issuecomment-285005040,
or mute the thread
<
https://github.com/notifications/unsubscribe-auth/AAB3-gLP1UoKgk2rTDQ3p7rENFJ-LmMAks5rjoMAgaJpZM4KgJEG

.

>

Parnell Springmeyer
0xDCCF89258EAD874A
http://pgp.mit.edu/pks/lookup?op=get&search=0xDCCF89258EAD874A

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/NixOS/nixpkgs/issues/19862#issuecomment-285013167,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAHtg_fEYW8pTD3W7jMK2JOJeXk4wRluks5rjovogaJpZM4KgJEG
.

[ixmatus]

I originally had kept it that way but was asked to remove the indirection
if the var path.

[ixmatus]

Well that's not entirely true, actually, my original changes even with the
var path would have broken this but it was in an attempt to unify all of
this name things accurately.

[ixmatus]

What if we have a symlink of the old setuid path to this new path??

[ixmatus]

Also, I'd like to provide the fix for this since I should have thought
about the migration path more thoroughly. I can do so in about two hours.

[teh]

I'm actually surprised that the combination of sudo and switch doesn't break more often. I guess it's because the PATH doesn't change a lot.

But generally sudo spawns a new shell and on exit it loses the environment changes. I'm not 100% clear that a symlink is the best solution as it's going to hang around after when its no longer needed.

[teh]

The more I think about this the I'm leaning towards adding an entry to the release notes for now. re-sourcing /etc/profile or logging out and in again doesn't seem like an undue burden after an os upgrade with sudo. Still suboptimal though!

In general it'd be useful to have a mechanism that re-sources the environment after a switch, no matter how that switch was executed. direnv solves that by running a no-output function when rendering the prompt, but it seems heavy to opt-in every user to a mechanism like that.

For this specific case we also have a chicken-and-egg problem, because we can't deliver a fix that will arrive in users' open shells before they do the os upgrade.

[ixmatus]

I'm available to discuss and work on this. Personally, I don't want to disrupt people's workflows too much but I also fall on the side of just getting this over with since I think it needs to happen.

A symlink pointing to the new path isn't too dirty and can be removed after the 17.03 release for the 17.09 release, I think that's fairly reasonable.

[domenkozar]

I'm skeptical symlink will work as wrappers assert the path they're being ran from (at least that was the case before the refactoring).

[ixmatus]

@domenkozar shoot, you're right.

[ixmatus]

In any case, I think rm'ing the /var/setuid-wrappers was a bad move on my part not thinking about running from sudo itself; I will put up a PR to eliminate that step which should fix some of the issues people have experienced as reported in this issue.

[ixmatus]

@domenkozar @teh we can move discussion of how to handle the migration over to #23641. I'm at-minimum removing the code that deletes the old wrapper dirs as that will significantly reduce migration pain I think.

I'm testing on my EC2 machine from scratch with some of the actions people reported here.

[qknight]

@domenkozar
i had been using zsh with oh-my-zsh and this hardcoded the PATH into the shell script. removing that variable made it work!

thanks for all the help!

[globin]

The 17.03 stuff is handled in #23641