Podman: "podman start" not working

Created on 15 Nov 2019 · 19Comments · Source: containers/podman

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

On Ubuntu 18.04 in AWS, when stopping and starting a standard container, it seems to always show an error during "start".

Error: unable to start container "<container id>": container create failed (no logs from conmon): EOF

Steps to reproduce the issue:

Launch a small instance in AWS with Ubuntu 18.04.
Connect and switch to the root user.

sudo su -

Create install.sh:

set -x
sudo apt-get install -y \
  btrfs-tools \
  git \
  golang-go \
  go-md2man \
  iptables \
  libassuan-dev \
  libc6-dev \
  libdevmapper-dev \
  libglib2.0-dev \
  libgpgme-dev \
  libgpg-error-dev \
  libprotobuf-dev \
  libprotobuf-c0-dev \
  libseccomp-dev \
  libselinux1-dev \
  libsystemd-dev \
  pkg-config \
  runc \
  uidmap

sudo apt-get update -qq
sudo apt-get install -qq -y software-properties-common uidmap
sudo add-apt-repository -y ppa:projectatomic/ppa
sudo apt-get update -qq
sudo apt-get -qq -y install podman
sudo mkdir -p /etc/containers
echo -e "[registries.search]\nregistries = ['docker.io', 'quay.io']" | sudo tee /etc/containers/registries.conf

Run the installation:

chmod 755 install.sh
./install.sh

Download and run a container:

podman pull docker.io/nginx
podman run -d docker.io/library/nginx

View that it's running:

podman ps

Stop the container:

podman stop 571c0  (or podman stop <container id>)

The 'stop' is successful, and shows the id.

Start the container:

podman start 571c0 (or podman start <container id>)

Describe the results you received:

Error: unable to start container "571c0": container create failed (no logs from conmon): EOF

Describe the results you expected:

Just the id of the started container:

571c0

Additional information you deem important (e.g. issue happens only occasionally):

With those steps, the issue seems to be consistent and reproducible.

Output of podman version:

# podman version
Version:            1.6.2
RemoteAPI Version:  1
Go Version:         go1.10.4
OS/Arch:            linux/amd64

Output of podman info --debug:

# podman info --debug
debug:
  compiler: gc
  git commit: ""
  go version: go1.10.4
  podman version: 1.6.2
host:
  BuildahVersion: 1.11.3
  CgroupVersion: v1
  Conmon:
    package: 'conmon: /usr/bin/conmon'
    path: /usr/bin/conmon
    version: 'conmon version 2.0.3, commit: unknown'
  Distribution:
    distribution: ubuntu
    version: "18.04"
  MemFree: 1668878336
  MemTotal: 4135120896
  OCIRuntime:
    name: runc
    package: 'containerd.io: /usr/bin/runc'
    path: /usr/bin/runc
    version: |-
      runc version 1.0.0-rc8+dev
      commit: 3e425f80a8c931f88e6d94a8c831b9d5aa481657
      spec: 1.0.1-dev
  SwapFree: 0
  SwapTotal: 0
  arch: amd64
  cpus: 2
  eventlogger: journald
  hostname: ip-172-31-13-241
  kernel: 4.15.0-1051-aws
  os: linux
  rootless: false
  uptime: 22m 1.82s
registries:
  blocked: null
  insecure: null
  search:
  - docker.io
  - quay.io
store:
  ConfigFile: /etc/containers/storage.conf
  ContainerStore:
    number: 1
  GraphDriverName: overlay
  GraphOptions: {}
  GraphRoot: /var/lib/containers/storage
  GraphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  ImageStore:
    number: 1
  RunRoot: /var/run/containers/storage
  VolumePath: /var/lib/containers/storage/volumes

Package info (e.g. output of rpm -q podman or apt list podman):

# apt list podman
Listing... Done
podman/bionic,now 1.6.2-1~ubuntu18.04~ppa1 amd64 [installed]

Additional environment details (AWS, VirtualBox, physical, etc.):

AWS

kinbug

Source

sdarwin

Most helpful comment

We are preparing for a 1.6.4 release soon. That will include the patch that resolves the issue.

mheon on 5 Dec 2019

👍4 ❤1

All 19 comments

Can you run with --log-level=debug and paste the result, as well as any logs in syslog from conmon?

mheon on 15 Nov 2019

@mheon - very good idea - the conmon logs in /var/log/syslog shows the error:

Nov 15 22:50:24 ip-172-31-13-241 conmon: conmon 8e7b417faf5b76195947 <ninfo>: attach sock path: /var/run/libpod/socket/8e7b417faf5b761959470e1d08a460d4ebb7ae525d8eee88a7812e5e79475679/attach
Nov 15 22:50:24 ip-172-31-13-241 conmon: conmon 8e7b417faf5b76195947 <ninfo>: addr{sun_family=AF_UNIX, sun_path=/var/run/libpod/socket/8e7b417faf5b761959470e1d08a460d4ebb7ae525d8eee88a7812e5e79475679/attach}
Nov 15 22:50:24 ip-172-31-13-241 conmon: conmon 8e7b417faf5b76195947 <ninfo>: terminal_ctrl_fd: 16
Nov 15 22:50:24 ip-172-31-13-241 conmon: conmon 8e7b417faf5b76195947 <error>: Failed to mkfifo at /var/lib/containers/storage/overlay-containers/8e7b417faf5b761959470e1d08a460d4ebb7ae525d8eee88a7812e5e79475679/userdata/winsz: File exists

it says "File exists".

for completeness, here is the debug output of "podman start" also:

# podman start --log-level=debug 8e7
DEBU[0000] using conmon: "/usr/bin/conmon"              
DEBU[0000] Initializing boltdb state at /var/lib/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /var/lib/containers/storage 
DEBU[0000] Using run root /var/run/containers/storage   
DEBU[0000] Using static dir /var/lib/containers/storage/libpod 
DEBU[0000] Using tmp dir /var/run/libpod                
DEBU[0000] Using volume path /var/lib/containers/storage/volumes 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] cached value indicated that overlay is supported 
DEBU[0000] cached value indicated that metacopy is not being used 
DEBU[0000] cached value indicated that native-diff is usable 
DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=true, usingMetacopy=false 
DEBU[0000] Initializing event backend journald          
DEBU[0000] using runtime "/usr/bin/runc"                
WARN[0000] Error initializing configured OCI runtime crun: no valid executable found for OCI runtime crun: invalid argument 
INFO[0000] Found CNI network podman (type=bridge) at /etc/cni/net.d/87-podman-bridge.conflist 
DEBU[0000] overlay: mount_data=lowerdir=/var/lib/containers/storage/overlay/l/KPI72ZD43ORLU4PQBNTLGRGN7H:/var/lib/containers/storage/overlay/l/RCXFOQ5ZL5GLPMNTWGC2I6WBM3:/var/lib/containers/storage/overlay/l/EPS6NXK3IVOVGAQDSO76LGUASR,upperdir=/var/lib/containers/storage/overlay/5e913f3998bab492c31746f4c1d2fb3bed4f346ba12647a2caad3446c9c8039e/diff,workdir=/var/lib/containers/storage/overlay/5e913f3998bab492c31746f4c1d2fb3bed4f346ba12647a2caad3446c9c8039e/work 
DEBU[0000] mounted container "8e7b417faf5b761959470e1d08a460d4ebb7ae525d8eee88a7812e5e79475679" at "/var/lib/containers/storage/overlay/5e913f3998bab492c31746f4c1d2fb3bed4f346ba12647a2caad3446c9c8039e/merged" 
DEBU[0000] Created root filesystem for container 8e7b417faf5b761959470e1d08a460d4ebb7ae525d8eee88a7812e5e79475679 at /var/lib/containers/storage/overlay/5e913f3998bab492c31746f4c1d2fb3bed4f346ba12647a2caad3446c9c8039e/merged 
DEBU[0000] Made network namespace at /var/run/netns/cni-0631d181-0a0d-0cb2-1516-0e2ccbad1a85 for container 8e7b417faf5b761959470e1d08a460d4ebb7ae525d8eee88a7812e5e79475679 
INFO[0000] Got pod network &{Name:interesting_lederberg Namespace:interesting_lederberg ID:8e7b417faf5b761959470e1d08a460d4ebb7ae525d8eee88a7812e5e79475679 NetNS:/var/run/netns/cni-0631d181-0a0d-0cb2-1516-0e2ccbad1a85 Networks:[] RuntimeConfig:map[podman:{IP: PortMappings:[] Bandwidth:<nil> IpRanges:[]}]} 
INFO[0000] About to add CNI network cni-loopback (type=loopback) 
INFO[0000] Got pod network &{Name:interesting_lederberg Namespace:interesting_lederberg ID:8e7b417faf5b761959470e1d08a460d4ebb7ae525d8eee88a7812e5e79475679 NetNS:/var/run/netns/cni-0631d181-0a0d-0cb2-1516-0e2ccbad1a85 Networks:[] RuntimeConfig:map[podman:{IP: PortMappings:[] Bandwidth:<nil> IpRanges:[]}]} 
INFO[0000] About to add CNI network podman (type=bridge) 
DEBU[0000] [0] CNI result: Interfaces:[{Name:cni-podman0 Mac:ce:da:85:19:99:d2 Sandbox:} {Name:veth5bc236d0 Mac:2a:a1:47:79:f5:6c Sandbox:} {Name:eth0 Mac:5a:94:c2:a6:f8:ed Sandbox:/var/run/netns/cni-0631d181-0a0d-0cb2-1516-0e2ccbad1a85}], IP:[{Version:4 Interface:0xc42052cca8 Address:{IP:10.88.0.10 Mask:ffff0000} Gateway:10.88.0.1}], Routes:[{Dst:{IP:0.0.0.0 Mask:00000000} GW:<nil>}], DNS:{Nameservers:[] Domain: Search:[] Options:[]} 
INFO[0000] No non-localhost DNS nameservers are left in resolv.conf. Using default external servers: [nameserver 8.8.8.8 nameserver 8.8.4.4] 
INFO[0000] IPv6 enabled; Adding default IPv6 external servers: [nameserver 2001:4860:4860::8888 nameserver 2001:4860:4860::8844] 
DEBU[0000] /etc/system-fips does not exist on host, not mounting FIPS mode secret 
DEBU[0000] Setting CGroups for container 8e7b417faf5b761959470e1d08a460d4ebb7ae525d8eee88a7812e5e79475679 to machine.slice:libpod:8e7b417faf5b761959470e1d08a460d4ebb7ae525d8eee88a7812e5e79475679 
DEBU[0000] reading hooks from /usr/share/containers/oci/hooks.d 
DEBU[0000] reading hooks from /etc/containers/oci/hooks.d 
DEBU[0000] Created OCI spec for container 8e7b417faf5b761959470e1d08a460d4ebb7ae525d8eee88a7812e5e79475679 at /var/lib/containers/storage/overlay-containers/8e7b417faf5b761959470e1d08a460d4ebb7ae525d8eee88a7812e5e79475679/userdata/config.json 
DEBU[0000] /usr/bin/conmon messages will be logged to syslog 
DEBU[0000] running conmon: /usr/bin/conmon               args="[--api-version 1 -s -c 8e7b417faf5b761959470e1d08a460d4ebb7ae525d8eee88a7812e5e79475679 -u 8e7b417faf5b761959470e1d08a460d4ebb7ae525d8eee88a7812e5e79475679 -r /usr/bin/runc -b /var/lib/containers/storage/overlay-containers/8e7b417faf5b761959470e1d08a460d4ebb7ae525d8eee88a7812e5e79475679/userdata -p /var/run/containers/storage/overlay-containers/8e7b417faf5b761959470e1d08a460d4ebb7ae525d8eee88a7812e5e79475679/userdata/pidfile -l k8s-file:/var/lib/containers/storage/overlay-containers/8e7b417faf5b761959470e1d08a460d4ebb7ae525d8eee88a7812e5e79475679/userdata/ctr.log --exit-dir /var/run/libpod/exits --socket-dir-path /var/run/libpod/socket --log-level debug --syslog --conmon-pidfile /var/run/containers/storage/overlay-containers/8e7b417faf5b761959470e1d08a460d4ebb7ae525d8eee88a7812e5e79475679/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /var/lib/containers/storage --exit-command-arg --runroot --exit-command-arg /var/run/containers/storage --exit-command-arg --log-level --exit-command-arg error --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /var/run/libpod --exit-command-arg --runtime --exit-command-arg runc --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --events-backend --exit-command-arg journald --exit-command-arg container --exit-command-arg cleanup --exit-command-arg 8e7b417faf5b761959470e1d08a460d4ebb7ae525d8eee88a7812e5e79475679]"
INFO[0000] Running conmon under slice machine.slice and unitName libpod-conmon-8e7b417faf5b761959470e1d08a460d4ebb7ae525d8eee88a7812e5e79475679.scope 
WARN[0000] Failed to add conmon to systemd sandbox cgroup: Unit libpod-conmon-8e7b417faf5b761959470e1d08a460d4ebb7ae525d8eee88a7812e5e79475679.scope already exists. 
DEBU[0000] Cleaning up container 8e7b417faf5b761959470e1d08a460d4ebb7ae525d8eee88a7812e5e79475679 
DEBU[0000] Tearing down network namespace at /var/run/netns/cni-0631d181-0a0d-0cb2-1516-0e2ccbad1a85 for container 8e7b417faf5b761959470e1d08a460d4ebb7ae525d8eee88a7812e5e79475679 
INFO[0000] Got pod network &{Name:interesting_lederberg Namespace:interesting_lederberg ID:8e7b417faf5b761959470e1d08a460d4ebb7ae525d8eee88a7812e5e79475679 NetNS:/var/run/netns/cni-0631d181-0a0d-0cb2-1516-0e2ccbad1a85 Networks:[] RuntimeConfig:map[podman:{IP: PortMappings:[] Bandwidth:<nil> IpRanges:[]}]} 
INFO[0000] About to del CNI network podman (type=bridge) 
DEBU[0000] unmounted container "8e7b417faf5b761959470e1d08a460d4ebb7ae525d8eee88a7812e5e79475679" 
ERRO[0000] unable to start container "8e7": container create failed (no logs from conmon): EOF

sdarwin on 15 Nov 2019

@sdarwin can we also get a podman info ? Might help understanding all the versions involved. also @haircommander ptal, thoughts?

baude on 17 Nov 2019

I think the PR to clean up winsz might fix this as well

On Sun, Nov 17, 2019, 10:58 Brent Baude notifications@github.com wrote:

@sdarwin https://github.com/sdarwin can we also get a podman info ?
Might help understanding all the versions involved. also @haircommander
https://github.com/haircommander ptal, thoughts?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/containers/libpod/issues/4522?email_source=notifications&email_token=AB3AOCC733OY6Q6GBONFHK3QUFSZDA5CNFSM4JOBR2NKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEEIPNHA#issuecomment-554759836,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AB3AOCAID7WFE7KUUBOHPITQUFSZDANCNFSM4JOBR2NA
.

mheon on 17 Nov 2019

I think the PR to clean up winsz might fix this as well

Yes.

can we also get a podman info ?

# podman info
host:
  BuildahVersion: 1.11.3
  CgroupVersion: v1
  Conmon:
    package: 'conmon: /usr/bin/conmon'
    path: /usr/bin/conmon
    version: 'conmon version 2.0.3, commit: unknown'
  Distribution:
    distribution: ubuntu
    version: "18.04"
  MemFree: 703287296
  MemTotal: 2088759296
  OCIRuntime:
    name: runc
    package: 'cri-o-runc: /usr/lib/cri-o-runc/sbin/runc'
    path: /usr/lib/cri-o-runc/sbin/runc
    version: 'runc version spec: 1.0.1-dev'
  SwapFree: 0
  SwapTotal: 0
  arch: amd64
  cpus: 1
  eventlogger: journald
  hostname: jenkins
  kernel: 4.15.0-1054-aws
  os: linux
  rootless: false
  uptime: 4m 2.76s
registries:
  blocked: null
  insecure: null
  search: null
store:
  ConfigFile: /etc/containers/storage.conf
  ContainerStore:
    number: 8
  GraphDriverName: overlay
  GraphOptions: {}
  GraphRoot: /var/lib/containers/storage
  GraphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  ImageStore:
    number: 5
  RunRoot: /var/run/containers/storage
  VolumePath: /var/lib/containers/storage/volumes

sdarwin on 17 Nov 2019

https://github.com/containers/libpod/pull/4521 merged, are you able to verify with it @sdarwin ?

haircommander on 17 Nov 2019

Fixed.

sdarwin on 18 Nov 2019

Any estimate on when this fix will be released? Not being able to start containers a second time after stopping them is kind of hard to work around.

DaanDeMeyer on 25 Nov 2019

@mheon will know when the next release happens.

vrothberg on 25 Nov 2019

We should have podman 1.6.3 released ASAP or something newer. 1.6.3 is already set up to be used on RHEL8.2, I believe.

rhatdan on 25 Nov 2019

I'll try and get a 1.6.4 out this week for 8.2 and Fedora. We need a few more patches that haven't been written yet for CGroups v2.

mheon on 25 Nov 2019

👍1

I am getting the same error on podman container start ${container_name} on version 1.6.3

The log obtained from podman --log-level=debug container start rdis 2> podman.log is podman.log and the command used to create the container was

podman run --name rdis -p 6379:6379 -e REDIS_PASSWORD=${REDIS_PASSWORD} -d bitnami/redis:latest

I am also not able to start any of my other containers (these were all previously working containers) the output of podman info is podman_info.log
Any help or work around will be extremely appreciated. Sorry in advance if this is the wrong place for this, I did not want to open a new issue for this because this thread was fairly new and seemed relevant.

L0stLink on 3 Dec 2019

This is also happening to me on Arch Linux with Podman 1.6.3 and conmon 2.0.3.

Steps to reproduce

➜  # ~ podman run -it alpine
/ # exit

➜  # ~ podman ps -a
CONTAINER ID  IMAGE                            COMMAND  CREATED         STATUS                     PORTS  NAMES
8946589d4831  docker.io/library/alpine:latest  /bin/sh  49 seconds ago  Exited (0) 47 seconds ago         beautiful_mendel

➜  # ~ podman start --log-level=debug 8946589d4831
(log output truncated)
ERRO[0000] unable to start container "8946589d4831": container create failed (no logs from conmon): EOF

podman info

host:
  BuildahVersion: 1.12.0-dev
  CgroupVersion: v1
  Conmon:
    package: Unknown
    path: /usr/bin/conmon
    version: 'conmon version 2.0.3, commit: eb5fa88c26fde5ce1e3f8a1d2a8a9498b2d7dbe6'
  Distribution:
    distribution: arch
    version: unknown
  MemFree: 3226333184
  MemTotal: 8313151488
  OCIRuntime:
    name: runc
    package: Unknown
    path: /usr/bin/runc
    version: |-
      runc version 1.0.0-rc9
      commit: d736ef14f0288d6993a1845745d6756cfc9ddd5a
      spec: 1.0.1-dev
  SwapFree: 4294963200
  SwapTotal: 4294963200
  arch: amd64
  cpus: 4
  eventlogger: journald
  hostname: <hostname>
  kernel: 5.4.1-arch1-1
  os: linux
  rootless: false
  uptime: 5h 36m 20.37s (Approximately 0.21 days)
registries:
  blocked: null
  insecure: null
  search:
  - docker.io
  - registry.fedoraproject.org
  - quay.io
  - registry.access.redhat.com
  - registry.centos.org
store:
  ConfigFile: /etc/containers/storage.conf
  ContainerStore:
    number: 3
  GraphDriverName: overlay
  GraphOptions:
    overlay.mountopt: nodev
  GraphRoot: /var/lib/containers/storage
  GraphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "true"
  ImageStore:
    number: 1
  RunRoot: /var/run/containers/storage
  VolumePath: /var/lib/containers/storage/volumes

Complete log output.

miquecg on 5 Dec 2019

We are preparing for a 1.6.4 release soon. That will include the patch that resolves the issue.

mheon on 5 Dec 2019

👍4 ❤1

Thank you for the update! for now I can create a new container, mount the same volumes (this is using postgres:alpine image from docker) and run it just fine — have not tried stopping it, I just hibernate the system between sessions and am able to get work done.

L0stLink on 6 Dec 2019

Would someone please be so kind to publish a new release to the ppa for *buntu ?
PPA is still on the buggy 1.6.2 (1.6.2-1~ubuntu18.04~ppa1) version

https://launchpad.net/~projectatomic/+archive/ubuntu/ppa

Thnx