Podman: `podman info` fails without sudo

Created on 28 May 2019  路  17Comments  路  Source: containers/podman

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

adel@adel-pc:~  
> podman info
cannot clone: Invalid argument
panic: calling Unlock on unlocked lock

goroutine 1 [running]:
github.com/containers/libpod/vendor/github.com/containers/storage.(*lockfile).Unlock(0xc0000cdcc0)
        /build/podman/src/src/github.com/containers/libpod/vendor/github.com/containers/storage/lockfile_unix.go:144 +0x1ed
github.com/containers/libpod/libpod.makeRuntime.func2(0xc0006f4abe, 0x55b5908f2c00, 0xc0000cdcc0)
        /build/podman/src/src/github.com/containers/libpod/libpod/runtime.go:884 +0x47
github.com/containers/libpod/libpod.makeRuntime(0x55b5908e1ce0, 0xc0000c2048, 0xc0005b60f0, 0x55b5908b05a0, 0xc000814100)
        /build/podman/src/src/github.com/containers/libpod/libpod/runtime.go:897 +0x1a2f
github.com/containers/libpod/libpod.newRuntimeFromConfig(0x55b5908e1ce0, 0xc0000c2048, 0x0, 0x0, 0xc0000c47e0, 0x1, 0x1, 0xc0005b60f0, 0x0, 0x0)
        /build/podman/src/src/github.com/containers/libpod/libpod/runtime.go:564 +0x6a1
github.com/containers/libpod/libpod.NewRuntime(...)
        /build/podman/src/src/github.com/containers/libpod/libpod/runtime.go:355
github.com/containers/libpod/cmd/podman/libpodruntime.getRuntime(0x55b5908e1ce0, 0xc0000c2048, 0xc0006f5bb0, 0x0, 0x3e8, 0x0, 0x1e)
        /build/podman/src/src/github.com/containers/libpod/cmd/podman/libpodruntime/runtime.go:149 +0xbf9
github.com/containers/libpod/cmd/podman/libpodruntime.GetRuntime(...)
        /build/podman/src/src/github.com/containers/libpod/cmd/podman/libpodruntime/runtime.go:26
main.setupRootless(0x55b59179fa00, 0x55b591823320, 0x0, 0x0, 0x0, 0x0)
        /build/podman/src/src/github.com/containers/libpod/cmd/podman/main_local.go:116 +0x144
main.before(0x55b59179fa00, 0x55b591823320, 0x0, 0x0, 0x1, 0xc0000c4610)
        /build/podman/src/src/github.com/containers/libpod/cmd/podman/main.go:106 +0x6e
main.glob..func68(0x55b59179fa00, 0x55b591823320, 0x0, 0x0, 0x0, 0x0)
        /build/podman/src/src/github.com/containers/libpod/cmd/podman/main.go:75 +0x4b
github.com/containers/libpod/vendor/github.com/spf13/cobra.(*Command).execute(0x55b59179fa00, 0xc0000ba030, 0x0, 0x0, 0x55b59179fa00, 0xc0000ba030)
        /build/podman/src/src/github.com/containers/libpod/vendor/github.com/spf13/cobra/command.go:741 +0x572
github.com/containers/libpod/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0x55b5917a11c0, 0xc0000b0cc0, 0x7ffed30ea503, 0x6)
        /build/podman/src/src/github.com/containers/libpod/vendor/github.com/spf13/cobra/command.go:852 +0x2ee
github.com/containers/libpod/vendor/github.com/spf13/cobra.(*Command).Execute(...)
        /build/podman/src/src/github.com/containers/libpod/vendor/github.com/spf13/cobra/command.go:800
main.main()
        /build/podman/src/src/github.com/containers/libpod/cmd/podman/main.go:142 +0x8a

Output of podman version:

Version:            1.3.0
RemoteAPI Version:  1
Go Version:         go1.12.4
Built:              Fri May 10 11:46:12 2019
OS/Arch:            linux/amd64

Output of podman info --debug:

cannot clone: Invalid argument
panic: calling Unlock on unlocked lock

goroutine 1 [running]:
github.com/containers/libpod/vendor/github.com/containers/storage.(*lockfile).Unlock(0xc0000cfcc0)
        /build/podman/src/src/github.com/containers/libpod/vendor/github.com/containers/storage/lockfile_unix.go:144 +0x1ed
github.com/containers/libpod/libpod.makeRuntime.func2(0xc00070eabe, 0x555644731c00, 0xc0000cfcc0)
        /build/podman/src/src/github.com/containers/libpod/libpod/runtime.go:884 +0x47
github.com/containers/libpod/libpod.makeRuntime(0x555644720ce0, 0xc0000c4048, 0xc0005e80f0, 0x5556446ef5a0, 0xc000820080)
        /build/podman/src/src/github.com/containers/libpod/libpod/runtime.go:897 +0x1a2f
github.com/containers/libpod/libpod.newRuntimeFromConfig(0x555644720ce0, 0xc0000c4048, 0x0, 0x0, 0xc000010750, 0x1, 0x1, 0xc0005e80f0, 0x0, 0x0)
        /build/podman/src/src/github.com/containers/libpod/libpod/runtime.go:564 +0x6a1
github.com/containers/libpod/libpod.NewRuntime(...)
        /build/podman/src/src/github.com/containers/libpod/libpod/runtime.go:355
github.com/containers/libpod/cmd/podman/libpodruntime.getRuntime(0x555644720ce0, 0xc0000c4048, 0xc00070fbb0, 0x0, 0x3e8, 0x0, 0x1e)
        /build/podman/src/src/github.com/containers/libpod/cmd/podman/libpodruntime/runtime.go:149 +0xbf9
github.com/containers/libpod/cmd/podman/libpodruntime.GetRuntime(...)
        /build/podman/src/src/github.com/containers/libpod/cmd/podman/libpodruntime/runtime.go:26
main.setupRootless(0x5556455dea00, 0xc00032c430, 0x0, 0x1, 0x0, 0x0)
        /build/podman/src/src/github.com/containers/libpod/cmd/podman/main_local.go:116 +0x144
main.before(0x5556455dea00, 0xc00032c430, 0x0, 0x1, 0x1, 0xc000010578)
        /build/podman/src/src/github.com/containers/libpod/cmd/podman/main.go:106 +0x6e
main.glob..func68(0x5556455dea00, 0xc00032c430, 0x0, 0x1, 0x0, 0x0)
        /build/podman/src/src/github.com/containers/libpod/cmd/podman/main.go:75 +0x4b
github.com/containers/libpod/vendor/github.com/spf13/cobra.(*Command).execute(0x5556455dea00, 0xc0000b2170, 0x1, 0x1, 0x5556455dea00, 0xc0000b2170)
        /build/podman/src/src/github.com/containers/libpod/vendor/github.com/spf13/cobra/command.go:741 +0x572
github.com/containers/libpod/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0x5556455e01c0, 0xc0000b2cf0, 0x7ffc40dd54fb, 0x6)
        /build/podman/src/src/github.com/containers/libpod/vendor/github.com/spf13/cobra/command.go:852 +0x2ee
github.com/containers/libpod/vendor/github.com/spf13/cobra.(*Command).Execute(...)
        /build/podman/src/src/github.com/containers/libpod/vendor/github.com/spf13/cobra/command.go:800
main.main()
        /build/podman/src/src/github.com/containers/libpod/cmd/podman/main.go:142 +0x8a

Additional environment details (AWS, VirtualBox, physical, etc.):

> uname -a
Linux adel-pc 4.9.178-1-MANJARO #1 SMP PREEMPT Tue May 21 19:31:19 UTC 2019 x86_64 GNU/Linux

With sudo the command sudo podman info --debug works and outputs:

adel@adel-pc:~            
> sudo podman info --debug
debug:                    
  compiler: gc                          
  git commit: ""
  go version: go1.12.4       
  podman version: 1.3.0        
host:                      
  BuildahVersion: 1.8.2    
  Conmon:    
    package: Unknown
    path: /usr/bin/conmon             
    version: 'conmon version , commit: 8fba206232c249a8fc4e2fac1469fb2fddbf5cf7'
  Distribution:
    distribution: manjaro
    version: unknown
  MemFree: 3711102976
  MemTotal: 7782268928
  OCIRuntime:
    package: Unknown
    path: /usr/bin/runc
    version: |-
      runc version 1.0.0-rc8
      commit: 425e105d5a03fabd737a126ad93d62a9eeede87f
      spec: 1.0.1-dev
  SwapFree: 9448923136
  SwapTotal: 9448923136
  arch: amd64
  cpus: 4
  hostname: adel-pc
  kernel: 4.9.178-1-MANJARO
  os: linux
  rootless: false
  uptime: 42m 34.67s
registries:
  blocked: null
  insecure: null
  search:
  - docker.io
  - registry.fedoraproject.org
  - quay.io
  - registry.access.redhat.com
  - registry.centos.org
store:
  ConfigFile: /etc/containers/storage.conf
  ContainerStore:
    number: 0
  GraphDriverName: overlay
  GraphOptions:
  - overlay.mountopt=nodev
  GraphRoot: /var/lib/containers/storage
  GraphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  ImageStore:
    number: 0
  RunRoot: /var/run/containers/storage
  VolumePath: /var/lib/containers/storage/volumes
kinbug
Source
picture adel-mamin

All 17 comments

Hi @adel-mamin, thanks a lot for opening the issue. Which kind of filesystem is your home directory on?

vrothberg picture vrothberg on 28 May 2019

I believe it is ext4

adel-mamin picture adel-mamin on 28 May 2019

@adel-mamin, is it on NFS? I suspect it's on NFS since the filelocks are not working there, which is inherently important for the storage. If that's the case, we can edit ~/.config/containers/storage.conf and make the graph_root point to a non-NFS directory.

vrothberg picture vrothberg on 28 May 2019

It is not on NFS. It is on my local disk:

> stat -f -L -c %T /home/adel
ext2/ext3

> stat -f -L -c %T /home/adel/.local/
ext2/ext3
adel-mamin picture adel-mamin on 28 May 2019

Thanks for clarifying. I checked the code and believe there's a bug.

vrothberg picture vrothberg on 28 May 2019

@adel-mamin, if possible, could you compile https://github.com/containers/libpod/pull/3208 and test if that fixes your issue?

vrothberg picture vrothberg on 28 May 2019

@vrothberg I can try, if you point me to the build instructions.

adel-mamin picture adel-mamin on 28 May 2019

@adel-mamin, you can checkout the pull request in your local git-tree via:

git fetch -f -u $remote pull/3208/head:pull-request-3208

Note that $remote is the git-remote pointing to github.com/containers/libpod which is origin if you just clone this repository. Once that's done, you can follow https://github.com/containers/libpod/blob/master/install.md#building-from-scratch for building podman.

vrothberg picture vrothberg on 28 May 2019

FWIW, I have tried building and installing podman from master SHA1 bc7afd6d.
Here is what I get:

adel@adel-pc:~/src/libpod  
> podman --version
podman version 1.3.2-dev

adel@adel-pc:~/src/libpod  
> podman info
cannot clone: Invalid argument
Error: could not get runtime: cannot re-exec process

adel@adel-pc:~/src/libpod  
> sudo podman info
host:
  BuildahVersion: 1.9.0-dev
  Conmon:
    package: Unknown
    path: /usr/bin/conmon
    version: 'conmon version , commit: 8fba206232c249a8fc4e2fac1469fb2fddbf5cf7'
  Distribution:
    distribution: manjaro
    version: unknown
  MemFree: 4104130560
  MemTotal: 7782268928
  OCIRuntime:
    package: Unknown
    path: /usr/bin/runc
    version: |-
      runc version 1.0.0-rc8
      commit: 425e105d5a03fabd737a126ad93d62a9eeede87f
      spec: 1.0.1-dev
  SwapFree: 9448923136
  SwapTotal: 9448923136
  arch: amd64
  cpus: 4
  hostname: adel-pc
  kernel: 4.9.178-1-MANJARO
  os: linux
  rootless: false
  uptime: 23m 30.31s
registries:
  blocked: null
  insecure: null
  search:
  - docker.io
  - registry.fedoraproject.org
  - quay.io
  - registry.access.redhat.com
  - registry.centos.org
store:
  ConfigFile: /etc/containers/storage.conf
  ContainerStore:
    number: 0
  GraphDriverName: overlay
  GraphOptions:
  - overlay.mountopt=nodev
  GraphRoot: /var/lib/containers/storage
  GraphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  ImageStore:
    number: 0
  RunRoot: /var/run/containers/storage
  VolumePath: /var/lib/containers/storage/volumes

Here is the build log:

adel@adel-pc:~/src/libpod  
> make -j4
mkdir -p "/home/adel/src/libpod/_output/src/github.com/containers"
go generate ./cmd/podman/varlink/...
ln -sf "/home/adel/src/libpod" "/home/adel/src/libpod/_output/src/github.com/containers"
ln -sf "/home/adel/src/libpod/vendor/github.com/varlink" "/home/adel/src/libpod/_output/src/github.com/varlink"
touch .gopathok
go build -ldflags ' -X github.com/containers/libpod/libpod.gitCommit=bc7afd6d71da4173e4894ff352667a25987fa2ea -X github.com/containers/libpod/libpod.buildInfo=1559065847' -tags "   ostree selinux systemd exclude_graphdriver_devicemapper seccomp varlink" -o bin/podman github.com/containers/libpod/cmd/podman
go build -ldflags ' -X github.com/containers/libpod/libpod.gitCommit=bc7afd6d71da4173e4894ff352667a25987fa2ea -X github.com/containers/libpod/libpod.buildInfo=1559065847' -tags "   ostree selinux systemd exclude_graphdriver_devicemapper seccomp varlink remoteclient" -o bin/podman-remote github.com/containers/libpod/cmd/podman
adel-mamin picture adel-mamin on 28 May 2019

FWIW, I have tried building and installing podman from master SHA1 bc7afd6d.
Here is what I get:

Thanks a ton for checking. I'll reopen. Note that I can't reproduce this at all.

adel@adel-pc:~/src/libpod  
> podman --version
podman version 1.3.2-dev

adel@adel-pc:~/src/libpod  
> podman info
cannot clone: Invalid argument
Error: could not get runtime: cannot re-exec process

@giuseppe, do you have a suspicion what could have gone south?

vrothberg picture vrothberg on 28 May 2019

Error: could not get runtime: cannot re-exec process:

I fear there is no support for user namespaces in the kernel.

What do you see with ls /proc/self/ns; cat /proc/self/uid_map ?

giuseppe picture giuseppe on 28 May 2019
👍1 
> ls /proc/self/ns; cat /proc/self/uid_map
cgroup  ipc  mnt  net  pid  uts
cat: /proc/self/uid_map: No such file or directory
adel-mamin picture adel-mamin on 29 May 2019

@giuseppe's suspicion is correct, the system does not have user namespace support. I guess that every distro configures that a bit different, but here's a link how to enable it in Manjaro: https://forum.manjaro.org/t/how-to-permanently-enable-user-namespaces-in-manjaro-linux/71614/2

vrothberg picture vrothberg on 29 May 2019
👍1

I updated the Linux kernel to 4.19.45-1-MANJARO and enabled the user namespaces by
sysctl kernel.unprivileged_userns_clone=1
Now podman info works as expected.
I am closing the issue.
Thank you!

adel-mamin picture adel-mamin on 29 May 2019
👍1

Awesome, thanks a lot!

vrothberg picture vrothberg on 29 May 2019

@adel-mamin Would you mind opening a PR for adding information on MANJARO on install.md. So future users would not hit the issues you have had.

rhatdan picture rhatdan on 29 May 2019

@rhatdan I've added https://github.com/containers/libpod/pull/3242. Please take a look.

adel-mamin picture adel-mamin on 31 May 2019
Was this page helpful?
0 / 5 - 0 ratings

Related issues

yufeifly picture yufeifly  路  4Comments
cruwe picture cruwe  路  3Comments
daltschu22 picture daltschu22  路  4Comments
dmesser picture dmesser  路  3Comments
juansuerogit picture juansuerogit  路  5Comments