podman exec fails

Created on 18 Apr 2019 · 15Comments · Source: containers/podman

BUG REPORT
/kind bug

Can't exec command in running container

After starting a container I can't use exec to execute a command int the container.

Steps to reproduce the issue:

podman run --name ubuntu -it registry.hub.docker.com/library/ubuntu bash
Open new terminal window/tab
podman exec -it ubuntu bash

Describe the results you received:

mpa@mpa-MacBookPro:~$ podman exec -it ubuntu bash
exec failed: container_linux.go:345: starting container process caused "process_linux.go:91: executing setns process caused \"exit status 22\""
Error: exit status 1

Describe the results you expected:
A bash promt inside the container

Additional information you deem important (e.g. issue happens only occasionally):
Happens always

Output of podman version:

Version:            1.3.0-dev
RemoteAPI Version:  1
Go Version:         go1.10.4
OS/Arch:            linux/amd64

Output of podman info --debug:

mpa@mpa-MacBookPro:~$ podman info --debug
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0xf8 pc=0x111dda3]

goroutine 1 [running]:
github.com/containers/libpod/libpod.(*Runtime).storeInfo(0xc4208e4b60, 0x2417fb0, 0x0, 0x0)
    /build/podman-OeFyYA/podman-1.3.0/src/github.com/containers/libpod/libpod/info.go:119 +0x53
github.com/containers/libpod/libpod.(*Runtime).Info(0xc4208e4b60, 0x16aeea8, 0xc4208e4b60, 0x0, 0x0, 0x1661be6)
    /build/podman-OeFyYA/podman-1.3.0/src/github.com/containers/libpod/libpod/runtime.go:1075 +0x110
main.infoCmd(0x23f5ac0, 0x0, 0x0)
    /build/podman-OeFyYA/podman-1.3.0/src/github.com/containers/libpod/cmd/podman/info.go:59 +0xc7
main.glob..func49(0x23934c0, 0xc420374d40, 0x0, 0x1, 0x0, 0x0)
    /build/podman-OeFyYA/podman-1.3.0/src/github.com/containers/libpod/cmd/podman/info.go:32 +0x87
github.com/containers/libpod/vendor/github.com/spf13/cobra.(*Command).execute(0x23934c0, 0xc4200cc080, 0x1, 0x1, 0x23934c0, 0xc4200cc080)
    /build/podman-OeFyYA/podman-1.3.0/src/github.com/containers/libpod/vendor/github.com/spf13/cobra/command.go:762 +0x468
github.com/containers/libpod/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0x2394c80, 0x6, 0x24199e0, 0xc4201d7100)
    /build/podman-OeFyYA/podman-1.3.0/src/github.com/containers/libpod/vendor/github.com/spf13/cobra/command.go:852 +0x30a
github.com/containers/libpod/vendor/github.com/spf13/cobra.(*Command).Execute(0x2394c80, 0xc420704370, 0xc4200bc058)
    /build/podman-OeFyYA/podman-1.3.0/src/github.com/containers/libpod/vendor/github.com/spf13/cobra/command.go:800 +0x2b
main.main()
    /build/podman-OeFyYA/podman-1.3.0/src/github.com/containers/libpod/cmd/podman/main.go:150 +0x3e

Additional environment details (AWS, VirtualBox, physical, etc.):
Ubuntu 16.04 installen on a MacBook Pro 15 without any virutalization.

kinbug rootless

Source

panzerdev

All 15 comments

Where did you get the software for this? Do you have an up 2 date version of runc.
@lsm5 PTAL

rhatdan on 18 Apr 2019

I followed the Ubuntu instructions here
https://github.com/containers/libpod/blob/master/install.md
and had the latest docker installed

mpa@mpa-MacBookPro:/tmp$ runc -v
runc version 1.0.0-rc6+dev
commit: 2b18fe1d885ee5083ef9f0838fee39b62d653e30
spec: 1.0.1-dev

panzerdev on 18 Apr 2019

Info seems to be segfaulting on trying to dereference r.store. Given the way the PPA works, this is probably master, which means it has the single rootless userns patch, which I think means we always configure the rootless userns.

mheon on 18 Apr 2019

@panzerdev Can you try those commands as root, and see if they work?

mheon on 18 Apr 2019

If I run podman as root for the run and exec command it works a expected.

Have I overlooked the instruction to run it as root or so?

panzerdev on 18 Apr 2019

No, this sounds like a bug, but one specific to our rootless code - we need to do more work to get Podman running without root (setting up a user namespace so we can gain limited versions of certain capabilities), and some piece of that seems to be breaking.

mheon on 18 Apr 2019

👍1

@panzerdev is this working for you now? @giuseppe ptal if not

baude on 29 May 2019

@baude I am running

Version:            1.3.2-dev
RemoteAPI Version:  1
Go Version:         go1.10.4
OS/Arch:            linux/amd64

and it is sadly still the same issue. Do I need to change something about my installation?

panzerdev on 30 May 2019

Does podman info work now, and if so, can you post it?

If it doesn't, that still tells us something

mheon on 31 May 2019

if you have no problems to run the container, could you run a process inside of it. And from the host do a cat /proc/$(PID_OF_THE_PROCESS_INSIDE_THE_CONTAINER)/uid_map?

You could for example do:

$ podman run --rm -d registry.hub.docker.com/library/ubuntu sleep 123.4
$ cat /proc/$(pgrep -f 123.4)/uid_map

giuseppe on 31 May 2019

@giuseppe

mpa@mpa-MacBookPro:~$ cat /proc/$(pgrep -f 123.4)/uid_map
         0       1000          1
         1     100000      65536

That's the result

@mheon

mpa@mpa-MacBookPro:~$ podman ps
CONTAINER ID  IMAGE                                          COMMAND      CREATED             STATUS                 PORTS  NAMES
4faddae85d25  registry.hub.docker.com/library/ubuntu:latest  sleep 123.4  About a minute ago  Up About a minute ago         determined_wozniak
mpa@mpa-MacBookPro:~$ podman exec -it determined_wozniak bash
exec failed: container_linux.go:346: starting container process caused "process_linux.go:101: executing setns process caused \"exit status 1\""
Error: exit status 1
mpa@mpa-MacBookPro:~$ podman info
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x100 pc=0x1131cb3]

goroutine 1 [running]:
github.com/containers/libpod/libpod.(*Runtime).storeInfo(0xc4203ec1e0, 0x2461f78, 0x0, 0x0)
    /build/podman-qFWEMp/podman-1.3.2/src/github.com/containers/libpod/libpod/info.go:119 +0x53
github.com/containers/libpod/libpod.(*Runtime).Info(0xc4203ec1e0, 0x16d3998, 0xc4203ec1e0, 0xc420911100, 0x0, 0x0)
    /build/podman-qFWEMp/podman-1.3.2/src/github.com/containers/libpod/libpod/runtime.go:1114 +0x10d
main.infoCmd(0x243f7c0, 0x0, 0x0)
    /build/podman-qFWEMp/podman-1.3.2/src/github.com/containers/libpod/cmd/podman/info.go:60 +0xe8
main.glob..func64(0x23dcc60, 0x2461f78, 0x0, 0x0, 0x0, 0x0)
    /build/podman-qFWEMp/podman-1.3.2/src/github.com/containers/libpod/cmd/podman/info.go:33 +0x87
github.com/containers/libpod/vendor/github.com/spf13/cobra.(*Command).execute(0x23dcc60, 0xc42000c070, 0x0, 0x0, 0x23dcc60, 0xc42000c070)
    /build/podman-qFWEMp/podman-1.3.2/src/github.com/containers/libpod/vendor/github.com/spf13/cobra/command.go:762 +0x468
github.com/containers/libpod/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0x23de420, 0x6, 0x24639a0, 0xc42019cf00)
    /build/podman-qFWEMp/podman-1.3.2/src/github.com/containers/libpod/vendor/github.com/spf13/cobra/command.go:852 +0x30a
github.com/containers/libpod/vendor/github.com/spf13/cobra.(*Command).Execute(0x23de420, 0xc420712160, 0xc4200aa058)
    /build/podman-qFWEMp/podman-1.3.2/src/github.com/containers/libpod/vendor/github.com/spf13/cobra/command.go:800 +0x2b
main.main()
    /build/podman-qFWEMp/podman-1.3.2/src/github.com/containers/libpod/cmd/podman/main.go:142 +0x3e
mpa@mpa-MacBookPro:~$

That's the result of podman info after I tried call exec.

Before:

mpa@mpa-MacBookPro:~$ podman info
host:
  BuildahVersion: 1.8.2
  Conmon:
    package: 'conmon: /usr/libexec/crio/conmon'
    path: /usr/libexec/crio/conmon
    version: 'conmon version , commit: '
  Distribution:
    distribution: ubuntu
    version: "16.04"
  MemFree: 5928730624
  MemTotal: 16696877056
  OCIRuntime:
    package: 'cri-o-runc: /usr/lib/cri-o-runc/sbin/runc'
    path: /usr/lib/cri-o-runc/sbin/runc
    version: 'runc version spec: 1.0.1-dev'
  SwapFree: 17024049152
  SwapTotal: 17049841664
  arch: amd64
  cpus: 8
  hostname: mpa-MacBookPro
  kernel: 4.4.0-148-generic
  os: linux
  rootless: true
  uptime: 26h 47m 40.53s (Approximately 1.08 days)
registries:
  blocked: null
  insecure: null
  search: null
store:
  ConfigFile: /home/mpa/.config/containers/storage.conf
  ContainerStore:
    number: 1
  GraphDriverName: vfs
  GraphOptions: null
  GraphRoot: /home/mpa/.local/share/containers/storage
  GraphStatus: {}
  ImageStore:
    number: 1
  RunRoot: /tmp/1000
  VolumePath: /home/mpa/.local/share/containers/storage/volumes

panzerdev on 4 Jun 2019

could you please attach the output of podman --log-level=debug info --debug?

giuseppe on 6 Jun 2019

Of course @giuseppe

INFO[0000] running as rootless                          
DEBU[0000] Initializing boltdb state at /home/mpa/.local/share/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver vfs                       
DEBU[0000] Using graph root /home/mpa/.local/share/containers/storage 
DEBU[0000] Using run root /tmp/1000                     
DEBU[0000] Using static dir /home/mpa/.local/share/containers/storage/libpod 
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp      
DEBU[0000] Using volume path /home/mpa/.local/share/containers/storage/volumes 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] Not configuring container store              
DEBU[0000] Initializing event backend file              
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x100 pc=0x1131cb3]

goroutine 1 [running]:
github.com/containers/libpod/libpod.(*Runtime).storeInfo(0xc4203d61e0, 0x2461f78, 0x0, 0x0)
    /build/podman-qFWEMp/podman-1.3.2/src/github.com/containers/libpod/libpod/info.go:119 +0x53
github.com/containers/libpod/libpod.(*Runtime).Info(0xc4203d61e0, 0x16d3998, 0xc4203d61e0, 0xc420587200, 0x0, 0x0)
    /build/podman-qFWEMp/podman-1.3.2/src/github.com/containers/libpod/libpod/runtime.go:1114 +0x10d
main.infoCmd(0x243f7c0, 0x0, 0x0)
    /build/podman-qFWEMp/podman-1.3.2/src/github.com/containers/libpod/cmd/podman/info.go:60 +0xe8
main.glob..func64(0x23dcc60, 0xc4202da010, 0x0, 0x1, 0x0, 0x0)
    /build/podman-qFWEMp/podman-1.3.2/src/github.com/containers/libpod/cmd/podman/info.go:33 +0x87
github.com/containers/libpod/vendor/github.com/spf13/cobra.(*Command).execute(0x23dcc60, 0xc42003a130, 0x1, 0x1, 0x23dcc60, 0xc42003a130)
    /build/podman-qFWEMp/podman-1.3.2/src/github.com/containers/libpod/vendor/github.com/spf13/cobra/command.go:762 +0x468
github.com/containers/libpod/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0x23de420, 0x6, 0x24639a0, 0xc42019af00)
    /build/podman-qFWEMp/podman-1.3.2/src/github.com/containers/libpod/vendor/github.com/spf13/cobra/command.go:852 +0x30a
github.com/containers/libpod/vendor/github.com/spf13/cobra.(*Command).Execute(0x23de420, 0xc420702160, 0xc4200ac058)
    /build/podman-qFWEMp/podman-1.3.2/src/github.com/containers/libpod/vendor/github.com/spf13/cobra/command.go:800 +0x2b
main.main()
    /build/podman-qFWEMp/podman-1.3.2/src/github.com/containers/libpod/cmd/podman/main.go:142 +0x3e

panzerdev on 6 Jun 2019

thanks.

Something strange is happening here, I see "Not configuring container store"

it seems it is trying to use info without re-execing in the user namespace. That should never happen with info

giuseppe on 6 Jun 2019

I see the issue, I've opened a PR here: https://github.com/containers/libpod/pull/3271

giuseppe on 6 Jun 2019

👍1

Was this page helpful?

0 / 5 - 0 ratings