Toolbox: Error: invalid entry point PID of container

[kuba3351]

Hello

I am using an Fedora Silverblur Rawhide 35 and after toolbox update to version 0.0.99.2 I cannot start any container created before or after the upgrade, because of error: invalid entry point PID of container

I am attaching a logs with verbose:

[jakub@NBJSIERZEGA sbpwsdlvalidator]$ toolbox enter -vvv
DEBU Running as real user ID 1000                 
DEBU Resolved absolute path to the executable as /usr/bin/toolbox 
DEBU Running on a cgroups v2 host                 
DEBU Checking if /etc/subgid and /etc/subuid have entries for user jakub 
DEBU Validating sub-ID file /etc/subuid           
DEBU Validating sub-ID file /etc/subgid           
DEBU TOOLBOX_PATH is /usr/bin/toolbox             
DEBU Migrating to newer Podman                    
DEBU Toolbox config directory is /var/home/jakub/.config/toolbox 
INFO[0000] podman filtering at log level debug          
DEBU[0000] Called version.PersistentPreRunE(podman --log-level debug version --format json) 
DEBU[0000] overlay storage already configured with a mount-program 
DEBU[0000] Merged system config "/usr/share/containers/containers.conf" 
DEBU[0000] overlay storage already configured with a mount-program 
DEBU[0000] Using conmon: "/usr/bin/conmon"              
DEBU[0000] Initializing boltdb state at /var/home/jakub/.local/share/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /var/home/jakub/.local/share/containers/storage 
DEBU[0000] Using run root /run/user/1000/containers     
DEBU[0000] Using static dir /var/home/jakub/.local/share/containers/storage/libpod 
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp      
DEBU[0000] Using volume path /var/home/jakub/.local/share/containers/storage/volumes 
DEBU[0000] overlay storage already configured with a mount-program 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] overlay: mount_program=/usr/bin/fuse-overlayfs 
DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=false 
DEBU[0000] Initializing event backend journald          
DEBU[0000] configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument 
DEBU[0000] configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument 
DEBU[0000] Using OCI runtime "/usr/bin/crun"            
INFO[0000] Found CNI network podman (type=bridge) at /var/home/jakub/.config/cni/net.d/87-podman.conflist 
DEBU[0000] Default CNI network name podman is unchangeable 
INFO[0000] Setting parallel job count to 25             
DEBU[0000] Called version.PersistentPostRunE(podman --log-level debug version --format json) 
DEBU Current Podman version is 3.3.0-dev          
DEBU Creating runtime directory /run/user/1000/toolbox 
DEBU Old Podman version is 3.3.0-dev              
DEBU Migration not needed: Podman version 3.3.0-dev is unchanged 
DEBU Resolving container and image names          
DEBU Container: ''                                
DEBU Distribution: ''                             
DEBU Image: ''                                    
DEBU Release: ''                                  
DEBU Resolved container and image names           
DEBU Container: 'fedora-toolbox-35'               
DEBU Image: 'fedora-toolbox:35'                   
DEBU Release: '35'                                
DEBU Checking if container fedora-toolbox-35 exists 
INFO[0000] podman filtering at log level debug          
DEBU[0000] Called exists.PersistentPreRunE(podman --log-level debug container exists fedora-toolbox-35) 
DEBU[0000] overlay storage already configured with a mount-program 
DEBU[0000] Merged system config "/usr/share/containers/containers.conf" 
DEBU[0000] overlay storage already configured with a mount-program 
DEBU[0000] Using conmon: "/usr/bin/conmon"              
DEBU[0000] Initializing boltdb state at /var/home/jakub/.local/share/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /var/home/jakub/.local/share/containers/storage 
DEBU[0000] Using run root /run/user/1000/containers     
DEBU[0000] Using static dir /var/home/jakub/.local/share/containers/storage/libpod 
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp      
DEBU[0000] Using volume path /var/home/jakub/.local/share/containers/storage/volumes 
DEBU[0000] overlay storage already configured with a mount-program 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] overlay: mount_program=/usr/bin/fuse-overlayfs 
DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=false 
DEBU[0000] Initializing event backend journald          
DEBU[0000] configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument 
DEBU[0000] configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument 
DEBU[0000] Using OCI runtime "/usr/bin/crun"            
INFO[0000] Found CNI network podman (type=bridge) at /var/home/jakub/.config/cni/net.d/87-podman.conflist 
DEBU[0000] Default CNI network name podman is unchangeable 
INFO[0000] Setting parallel job count to 25             
DEBU[0000] Called exists.PersistentPostRunE(podman --log-level debug container exists fedora-toolbox-35) 
DEBU Inspecting mounts of container fedora-toolbox-35 
INFO[0000] podman filtering at log level debug          
DEBU[0000] Called inspect.PersistentPreRunE(podman --log-level debug inspect --format json --type container fedora-toolbox-35) 
DEBU[0000] overlay storage already configured with a mount-program 
DEBU[0000] Merged system config "/usr/share/containers/containers.conf" 
DEBU[0000] overlay storage already configured with a mount-program 
DEBU[0000] Using conmon: "/usr/bin/conmon"              
DEBU[0000] Initializing boltdb state at /var/home/jakub/.local/share/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /var/home/jakub/.local/share/containers/storage 
DEBU[0000] Using run root /run/user/1000/containers     
DEBU[0000] Using static dir /var/home/jakub/.local/share/containers/storage/libpod 
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp      
DEBU[0000] Using volume path /var/home/jakub/.local/share/containers/storage/volumes 
DEBU[0000] overlay storage already configured with a mount-program 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] overlay: mount_program=/usr/bin/fuse-overlayfs 
DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=false 
DEBU[0000] Initializing event backend journald          
DEBU[0000] configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument 
DEBU[0000] configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument 
DEBU[0000] Using OCI runtime "/usr/bin/crun"            
INFO[0000] Found CNI network podman (type=bridge) at /var/home/jakub/.config/cni/net.d/87-podman.conflist 
DEBU[0000] Default CNI network name podman is unchangeable 
INFO[0000] Setting parallel job count to 25             
DEBU[0000] Called inspect.PersistentPostRunE(podman --log-level debug inspect --format json --type container fedora-toolbox-35) 
DEBU Starting container fedora-toolbox-35         
DEBU Inspecting entry point of container fedora-toolbox-35 
INFO[0000] podman filtering at log level debug          
DEBU[0000] Called inspect.PersistentPreRunE(podman --log-level debug inspect --format json --type container fedora-toolbox-35) 
DEBU[0000] overlay storage already configured with a mount-program 
DEBU[0000] Merged system config "/usr/share/containers/containers.conf" 
DEBU[0000] overlay storage already configured with a mount-program 
DEBU[0000] Using conmon: "/usr/bin/conmon"              
DEBU[0000] Initializing boltdb state at /var/home/jakub/.local/share/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /var/home/jakub/.local/share/containers/storage 
DEBU[0000] Using run root /run/user/1000/containers     
DEBU[0000] Using static dir /var/home/jakub/.local/share/containers/storage/libpod 
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp      
DEBU[0000] Using volume path /var/home/jakub/.local/share/containers/storage/volumes 
DEBU[0000] overlay storage already configured with a mount-program 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] overlay: mount_program=/usr/bin/fuse-overlayfs 
DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=false 
DEBU[0000] Initializing event backend journald          
DEBU[0000] configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument 
DEBU[0000] configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument 
DEBU[0000] Using OCI runtime "/usr/bin/crun"            
INFO[0000] Found CNI network podman (type=bridge) at /var/home/jakub/.config/cni/net.d/87-podman.conflist 
DEBU[0000] Default CNI network name podman is unchangeable 
INFO[0000] Setting parallel job count to 25             
DEBU[0000] Called inspect.PersistentPostRunE(podman --log-level debug inspect --format json --type container fedora-toolbox-35) 
DEBU Entry point PID is a float64                 
DEBU Entry point of container fedora-toolbox-35 is toolbox (PID=0) 
Error: invalid entry point PID of container fedora-toolbox-35

Please help for resolving the issue.

[HarryMichal]

Hi @kuba3351, thank you for opening this ticket. Could you, please, provide the output of podman start --attach fedora-toolbox-35?

[kuba3351]

Yes, I have got another error:

[jakub@NBJSIERZEGA ~]$ podman start --attach fedora-toolbox-35
toolbox: /lib64/libc.so.6: version `GLIBC_2.34' not found (required by toolbox)

[kuba3351]

Ok, I figured out how to partially workaround this issue.

I found that toolbox binary on my host machine was upgraded and built with never GLIBC, and it is mounted into the container with older GLIBC. So, I run the following commands:

podman run -it --name temporary registry.fedoraproject.org/fedora-toolbox:35 /bin/bash
sudo dnf update
exit
podman commit temporary registry.fedoraproject.org/fedora-toolbox:35

After that I can create working toolboxes based on fedora 35 but older toolboxes still not work. Any idea what to deal with it?

[kuba3351]

Created PR with proposal solution for this issue (https://github.com/containers/toolbox/pull/822)

[debarshiray]

This is basically a duplicate of https://github.com/containers/toolbox/issues/529

We need to update src/libc-wrappers/libc-wrappers.c for the new GNU C Library release.

[travier]

Do you have a doc somewhere explaining why we need to have toolbox as the entrypoint of the container? Thanks!

[travier]

It might make sense to create a separate, fully statically linked binary to use inside the toolbox for setup operations. This would thus be completely independent of the version of the glibc available inside the container. Some container images may not even have glibc installed at all.

[HarryMichal]

Do you have a doc somewhere explaining why we need to have toolbox as the entrypoint of the container? Thanks!

There's a paragraph in toolbox-init-container(1) that touches the topic:

OCI containers are inherently immutable. Configuration options passed through podman create are baked into the definition of the OCI container, and can't be changed later. This means that changes and improvements made in newer versions of Toolbox can't be applied to pre-existing toolbox containers created by older versions of Toolbox. This is avoided by using the entry point to configure the container at runtime.

[HarryMichal]

It might make sense to create a separate, fully statically linked binary to use inside the toolbox for setup operations. This would thus be completely independent of the version of the glibc available inside the container. Some container images may not even have glibc installed at all.

Agreed. Let's discuss it more after we release v0.1.

[travier]

Filed as https://github.com/containers/toolbox/issues/832 for tracking.