Toolbox: Cannot enter toolbox: conmon.pid: permission denied

[cjao]

Since the latest system update for Fedora Silverblue 30, I am unable to enter newly created toolbox.

Versions:
toolbox 0.0.10-1
podman 1.3.1-1.git7210727

toolbox -v create

toolbox: resolved absolute path for /usr/bin/toolbox to /usr/bin/toolbox
toolbox: TOOLBOX_PATH is /usr/bin/toolbox
toolbox: checking if 'podman system migrate' exists
toolbox: migration not needed: 1.3.1 is unchanged
toolbox: Fedora generational core is f30
toolbox: base image is fedora-toolbox:30
toolbox: container is fedora-toolbox-30
toolbox: checking value /var/run/.heim_org.h5l.kcm-socket (Stream) of property Listen in sssd-kcm.socket
toolbox: parsing value /var/run/.heim_org.h5l.kcm-socket (Stream) of property Listen in sssd-kcm.socket
toolbox: checking if 'podman create' supports --dns=none and --no-hosts
toolbox: 'podman create' supports --dns=none and --no-hosts
toolbox: looking for image localhost/fedora-toolbox:30
toolbox: looking for image registry.fedoraproject.org/f30/fedora-toolbox:30
toolbox: base image fedora-toolbox:30 resolved to registry.fedoraproject.org/f30/fedora-toolbox:30
toolbox: checking if container fedora-toolbox-30 already exists
toolbox: trying to create container fedora-toolbox-30
Created container: fedora-toolbox-30
Enter with: toolbox enter

toolbox -v enter

toolbox: resolved absolute path for /usr/bin/toolbox to /usr/bin/toolbox
toolbox: TOOLBOX_PATH is /usr/bin/toolbox
toolbox: checking if 'podman system migrate' exists
toolbox: migration not needed: 1.3.1 is unchanged
toolbox: Fedora generational core is f30
toolbox: base image is fedora-toolbox:30
toolbox: container is fedora-toolbox-30
toolbox: checking if container fedora-toolbox-30 exists
toolbox: trying to start container fedora-toolbox-30
toolbox: looking for /etc/profile.d/toolbox.sh in container fedora-toolbox-30
ERRO[0000] open /var/home/casey/.local/share/containers/storage/overlay-containers/38cc7f3195305881c42edb26b7e84071d5d108497ed6250870acc7c7811a0b46/userdata/conmon.pid: permission denied 
toolbox: copying /etc/profile.d/toolbox.sh to container fedora-toolbox-30
ERRO[0000] open /var/home/casey/.local/share/containers/storage/overlay-containers/38cc7f3195305881c42edb26b7e84071d5d108497ed6250870acc7c7811a0b46/userdata/conmon.pid: permission denied 
toolbox: unable to copy /etc/profile.d/toolbox.sh to container fedora-toolbox-30

and ls -l overlay-containers/38cc7f3195305881c42edb26b7e84071d5d108497ed6250870acc7c7811a0b46/ yields

total 4
drwx------. 3 100000 100000 4096 May 30 14:58 userdata

The contents of /etc/subuid is
casey:100000:65536

[darkmuggle]

I can confirm this behavior as well. If I whack .local/share/containers and run toolbox create it makes podman useless:

Starting from a clean-slate:

ben@porpentina:~ $ podman ps
CONTAINER ID  IMAGE  COMMAND  CREATED  STATUS  PORTS  NAMES

ben@porpentina:~ $ podman images
REPOSITORY   TAG   IMAGE ID   CREATED   SIZE

ben@porpentina:~ $ podman pull fedora
Trying to pull docker.io/library/fedora...Getting image source signatures
Copying blob 01eb078129a0 done
Copying config d09302f77c done
Writing manifest to image destination
Storing signatures
d09302f77cfcc3e867829d80ff47f9e7738ffef69730d54ec44341a9fb1d359b

ben@porpentina:~ $ podman images
REPOSITORY                 TAG      IMAGE ID       CREATED        SIZE
docker.io/library/fedora   latest   d09302f77cfc   2 months ago   283 MB

# Create a new toolbox
ben@porpentina:~ $ toolbox create 
Image required to create toolbox container.
Download registry.fedoraproject.org/f30/fedora-toolbox:30 (500MB)? [y/N]: y
Created container: fedora-toolbox-30
Enter with: toolbox enter

# Try to get into the container
ben@porpentina:~ $ toolbox enter
toolbox: unable to copy /etc/profile.d/toolbox.sh to container fedora-toolbox-30

# Now I can't list my images
ben@porpentina:~ $ podman images
ERRO[0000] open /home/ben/.local/share/containers/storage/overlay-containers/887f887c73f30ce9166713b528a6c7ea969f3d853a8202fd77caff590281b034/userdata/conmon.pid: permission denied 

[fbruetting]

Also is the case with v0.9 for me.

[heyakyra]

Looking at the source, this is the line which breaks. https://github.com/debarshiray/toolbox/blob/master/toolbox#L278

and the result of running it by hand:

podman cp /etc/profile.d/toolbox.sh container-name:/etc/profile.d
ERRO[0000] open /var/home/me/.local/share/containers/storage/overlay-containers/91a9744f21bb0d59f3f84ac3d434d895b162f9a992b03eb09f2087aec36fc486/userdata/conmon.pid: permission denied 

or, if I add sudo:

Error: invalid arguments /etc/profile.d/toolbox.sh, container-name:/etc/profile.d you must use just one container

[cjao]

Fortunately I was able to roll back the system update. Under toolbox 0.0.9 and podman 1.2.0, the corresponding directory .local/share/containers/storage/overlay-containers/[HASH]/userdata is owned by my real UID, not the mapped UID:

ls -l .local/share/containers/storage/overlay-containers/fd8bd9e2e54158f7c2e27d9c92cd227b764646efa87f267cba9943cdfc03bca8/
total 4
drwx--x--x. 3 casey casey 4096 May 31 06:50 userdata`

[juhp]

I reported at containers/libpod#3219

(toolbox-0.0.10 and podman-1.2 are working okay for me)

[kurobeats]

Tipped off by @cjao's comment, I was able to work around the issue by doing the following:

sudo chown -R USERNAME:USERNAME .local/share/containers/storage/overlay-containers/[HASH]/

[juhp]

On a fresh Fedora 30 Workstation install with latest updates and using latest podman snapshot from Copr:

$ sudo dnf -y copr enable baude/Upstream_CRIO_Family
$ sudo dnf -y install podman-1.3.2 toolbox
$ sudo rm -rf ~/.local/share/containers/
$ toolbox -y -v enter
toolbox: resolved absolute path for /usr/bin/toolbox to /usr/bin/toolbox
toolbox: TOOLBOX_PATH is /usr/bin/toolbox
toolbox: checking if 'podman system migrate' exists
toolbox: migration not needed: 1.3.2-dev is unchanged
toolbox: Fedora generational core is f30
toolbox: base image is fedora-toolbox:30
toolbox: container is fedora-toolbox-30
toolbox: checking if container fedora-toolbox-30 exists
toolbox: container fedora-toolbox-30 not found
toolbox: found 0 containers
toolbox: Fedora generational core is f30
toolbox: base image is fedora-toolbox:30
toolbox: container is fedora-toolbox-30
toolbox: checking value /var/run/.heim_org.h5l.kcm-socket (Stream) of property Listen in sssd-kcm.socket
toolbox: parsing value /var/run/.heim_org.h5l.kcm-socket (Stream) of property Listen in sssd-kcm.socket
toolbox: checking if 'podman create' supports --dns=none and --no-hosts
toolbox: 'podman create' supports --dns=none and --no-hosts
toolbox: looking for image localhost/fedora-toolbox:30
toolbox: looking for image registry.fedoraproject.org/f30/fedora-toolbox:30
toolbox: pulling image registry.fedoraproject.org/f30/fedora-toolbox:30
Trying to pull registry.fedoraproject.org/f30/fedora-toolbox:30...Getting image source signatures
Copying blob eb96ef222807 done
Copying blob 4c4fc0337a75 done
Copying config 42cdab313e done
Writing manifest to image destination
Storing signatures
toolbox: base image fedora-toolbox:30 resolved to registry.fedoraproject.org/f30/fedora-toolbox:30
toolbox: checking if container fedora-toolbox-30 already exists
toolbox: trying to create container fedora-toolbox-30
toolbox: trying to start container fedora-toolbox-30
toolbox: looking for /etc/profile.d/toolbox.sh in container fedora-toolbox-30
Error: unable to find user petersen: no matching entries in passwd file
toolbox: copying /etc/profile.d/toolbox.sh to container fedora-toolbox-30
Error: cannot copy into running rootless container with pause set - pass --pause=false to force copying
toolbox: unable to copy /etc/profile.d/toolbox.sh to container fedora-toolbox-30
$ toolbox enter
⬢[petersen@toolbox ~]$ 

(Does not work with podman-1.3.1. ;-)

[debarshiray]

All the conmon.pid: permission denied problems are caused by https://github.com/containers/libpod/issues/2935

Toolbox 0.0.10 works around it by calling podman system migrate. You can try calling it yourself when all your containers are stopped. eg., right after booting the host OS.

[debarshiray]

Other than that, all I can say is that the podman-1.3.1 release had plenty of regressions. I'd suggest reverting back to the podman-1.2.0 build or as @juhp says use podman-1.3.2 from the Copr.

[debarshiray]

I am closing this because the original problem and at least one of the related breakages are known. Please open a new issue if reverting Podman and migrating your containers don't solve this problem.

Thanks for getting in touch! :)