Crouton: entering and exiting chroot on external device seemingly creates many rogue mounts eventually preventing chroot entry

[rmartin16]

I've searched and looked around for similarly documented issues but alas.

Please paste the output of the following command here: sudo edit-chroot -all
chronos@localhost /media/removable/USB Drive/Russell/crouton $ sudo sh ./bin/edit-chroot -c ./chroots/ -all
name: xenial_steam
encrypted: no
Entering /media/removable/USB Drive/Russell/crouton/chroots/xenial_steam...
crouton: version 1-20180611074759~master:3697b784
release: xenial
architecture: amd64
xmethod: xorg
targets: keyboard,chrome,core,extension,xfce
host: version 10575.58.0 (Official Build) stable-channel caroline
kernel: Linux localhost 3.18.0-17554-g9194949d4df2 #1 SMP PREEMPT Fri Jun 22 17:21:44 PDT 2018 x86_64 x86_64 x86_64 GNU/Linux
freon: yes
Unmounting /media/removable/USB Drive/Russell/crouton/chroots/xenial_steam...
name: xenial_test
encrypted: no
Entering /media/removable/USB Drive/Russell/crouton/chroots/xenial_test...
crouton: version 1-20180611074759~master:3697b784
release: xenial
architecture: amd64
targets: core
host: version 10575.58.0 (Official Build) stable-channel caroline
kernel: Linux localhost 3.18.0-17554-g9194949d4df2 #1 SMP PREEMPT Fri Jun 22 17:21:44 PDT 2018 x86_64 x86_64 x86_64 GNU/Linux
freon: yes
Unmounting /media/removable/USB Drive/Russell/crouton/chroots/xenial_test...
name: xenial_vm
encrypted: no
Entering /media/removable/USB Drive/Russell/crouton/chroots/xenial_vm...
crouton: version 1-20180611074759~master:3697b784
release: xenial
architecture: amd64
xmethod: xorg
targets: chrome,extension,keyboard,xfce
host: version 10575.58.0 (Official Build) stable-channel caroline
kernel: Linux localhost 3.18.0-17554-g9194949d4df2 #1 SMP PREEMPT Fri Jun 22 17:21:44 PDT 2018 x86_64 x86_64 x86_64 GNU/Linux
freon: yes
Unmounting /media/removable/USB Drive/Russell/crouton/chroots/xenial_vm...

Please describe your issue:

This issue only seems to occur for chroots on my external devices; chroots on internal storage do not create the issue.

Upon exiting the chroot, the numbers of mounts can be significantly more than prior to entering the chroot. The problem seems to quickly compounding with subsequent entries to the chroot.

Mounts prior to entering the externally hosted choot (51 entries):
mounts_before_chroot_entry.txt

Mounts while in the chroot for the first time (164 entries):
mounts_while_in_chroot_first_time.txt

Mounts after leaving chroot for the first time (75 entries):
mounts_after_exiting_chroot_first_time.txt

Mounts while in the chroot for the second time (389 entries):
mounts_while_in_chroot_second_time.txt

Mounts after leaving chroot for the second time (187 entries):
mounts_after_exiting_chroot_second_time.txt

Mounts while in chroot for the third time (2429 entries):
mounts_while_in_chroot_third_time.txt

Mounts after leaving chroot for the third time (1435 entries):
mounts_after_exiting_chroot_third_time.txt

Mounts while in chroot for the fourth time (~80000 entries):
This is when everything goes to hell.

When I try to enter the chroot the fourth time, here's the output I see:
chronos@localhost /media/removable/USB Drive/Russell/crouton $ sudo sh bin/enter-chroot -c ./chroots/ -n xenial_test
Entering /media/removable/USB Drive/Russell/crouton/chroots/xenial_test...
mount: mount /sys on /var/run/crouton/media/removable/USB Drive/Russell/crouton/chroots/xenial_test/sys failed: No space left on device
Unmounting /media/removable/USB Drive/Russell/crouton/chroots/xenial_test...

It gets stuck here trying to unmount everything and umount sits around 100% CPU. Usually, I'll see the mounts drop to around 60000 entries but nothing really seems to change after that.

If known, describe the steps to reproduce the issue:

Repeat steps above.

Questions

I don't understand the nature of the seemingly rogue mounts left around after exiting the chroot but they appear recursive. Maybe the unmount process is unintentionally leaving some mounts around....and then for some reason the mount process goes crazy with them...maybe something in my setup...

Is there something I can do to prevent this situation? My xenial_test chroot was a fresh install I just created.

TIA.

[dnschneid]

Could you please run md5sum /media/removable/*/Russell/crouton/bin/* and post the output?

[rmartin16]

Sure thing.

chronos@localhost / $ md5sum /media/removable//Russell/crouton/bin/
65d56e24823ce6d4541ea0d4b3943612 /media/removable/USB Drive/Russell/crouton/bin/crash_reporter_wrapper
54a4e1d7b40fdc23970ce41a659d25f0 /media/removable/USB Drive/Russell/crouton/bin/delete-chroot
54a4e1d7b40fdc23970ce41a659d25f0 /media/removable/USB Drive/Russell/crouton/bin/edit-chroot
73d125af61adbf14b33a19acfe07b806 /media/removable/USB Drive/Russell/crouton/bin/enter-chroot
d15e1b38590e55bc557bc40fec988526 /media/removable/USB Drive/Russell/crouton/bin/mount-chroot
7b6a7c5c975b2d30f0cfa6bb0fda08ba /media/removable/USB Drive/Russell/crouton/bin/startxfce4
fadaed964561c12c52e2a9985def8e67 /media/removable/USB Drive/Russell/crouton/bin/unmount-chroot

And not sure if this was your intent based on wildcard for the removable directory but below is the output for my SD Card as well if that's useful.

chronos@localhost / $ md5sum /media/removable//crouton/bin/
65d56e24823ce6d4541ea0d4b3943612 /media/removable/SD Card/crouton/bin/crash_reporter_wrapper
54a4e1d7b40fdc23970ce41a659d25f0 /media/removable/SD Card/crouton/bin/delete-chroot
54a4e1d7b40fdc23970ce41a659d25f0 /media/removable/SD Card/crouton/bin/edit-chroot
73d125af61adbf14b33a19acfe07b806 /media/removable/SD Card/crouton/bin/enter-chroot
d15e1b38590e55bc557bc40fec988526 /media/removable/SD Card/crouton/bin/mount-chroot
1448638efcec446b513b58f47f5d004d /media/removable/SD Card/crouton/bin/startcli
c55f2d637233376f7ce37dd6df248eda /media/removable/SD Card/crouton/bin/startunity
e8ebcaa901698d904d3f42011345bdca /media/removable/SD Card/crouton/bin/startxiwi
fadaed964561c12c52e2a9985def8e67 /media/removable/SD Card/crouton/bin/unmount-chroot

[dnschneid]

Wildcard was just to avoid having to deal with spaces when pasting commands in issue trackers :)

Cool, looks like there haven't been any issues updating. It seems the logic around shared mounts might be wrong when using unencrypted chroots on external drives. Give this a shot:

sudo sed -i '/mount -i -o/a mount --make-private "$CHROOT"' /media/removable/*/Russell/crouton/bin/mount-chroot

This should add a line to the mount-chroot script that makes the shadow mount private, hopefully avoiding propagating the submounts back to the system's /media. Run that line, reboot, and see if everything behaves better...

[rmartin16]

ahh, gotcha.

Ran the command and that did it; the mounts returned to normal after exiting the chroot. Thanks a lot!

Should this be left open to potentially generalize this solution?

[dnschneid]

Yeah, I'll push the fix once we test it a bit more.

[dnschneid]

OK, pushed.