Podman: Error when forwarding SIGTERM to container with systemd

[EmilienM]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Originally reported here:
https://bugzilla.redhat.com/show_bug.cgi?id=1691408

Error when stopping a container managed by systemd:

Mar 21 18:08:03 undercloud.localdomain systemd[1]: Stopping heat_engine container...
Mar 21 18:08:38 undercloud.localdomain systemd[1]: heat_engine container is not active.
Mar 21 18:09:01 undercloud.localdomain podman[253956]: d6c494d9657c73e5fa9ac946136bf085ad84be8d17db26725ca54a8e8cec759f
Mar 21 18:09:01 undercloud.localdomain podman[26123]: time="2019-03-21T18:09:01Z" level=error msg="Error forwarding signal 15 to container d6c494d9657c73e5fa9ac946136bf085ad8
4be8d17db26725ca54a8e8cec759f: can only kill running containers: container state improper"

It always happens with the same container (heat-engine).

Steps to reproduce the issue:

1. Create a systemd unit file to control a container:

[Unit]
Description=heat_engine container
After=paunch-container-shutdown.service
Wants=
[Service]
Restart=always
ExecStart=/usr/bin/podman start -a heat_engine
ExecStop=/usr/bin/podman stop -t 60 heat_engine
KillMode=process
[Install]
WantedBy=multi-user.target
````

2. Start the container with systemd.

3. Stop the container

**Describe the results you received:**

Stopping the container with systemd produces an error:
[![asciicast](https://asciinema.org/a/iiWwN1HEmJHDU0tkjaYXDHaLn.svg)](https://asciinema.org/a/iiWwN1HEmJHDU0tkjaYXDHaLn)

Stopping the container with podman CLI works fine:
[![asciicast](https://asciinema.org/a/F4lZLyDd6bhOfgoHSFuuOkPGc.svg)](https://asciinema.org/a/F4lZLyDd6bhOfgoHSFuuOkPGc)

**Describe the results you expected:**
The container should be stopped without error.

**Additional information you deem important (e.g. issue happens only occasionally):**

**Output of `podman version`:**

Version: 1.0.2-dev
Go Version: go1.11.5
OS/Arch: linux/amd64

**Output of `podman info --debug`:**

debug:
compiler: gc
git commit: ""
go version: go1.11.5
podman version: 1.0.2-dev
host:
BuildahVersion: 1.6-dev
Conmon:
package: podman-1.0.0-2.git921f98f.module+el8+2785+ff8a053f.x86_64
path: /usr/libexec/podman/conmon
version: 'conmon version 1.14.0-dev, commit: be8255a19cda8a598d76dfa49e16e337769d4528-dirty'
Distribution:
distribution: '"rhel"'
version: "8.0"
MemFree: 7279849472
MemTotal: 23926374400
OCIRuntime:
package: runc-1.0.0-54.rc5.dev.git2abd837.module+el8+2769+577ad176.x86_64
path: /usr/bin/runc
version: 'runc version spec: 1.0.0'
SwapFree: 10722734080
SwapTotal: 10737414144
arch: amd64
cpus: 4
hostname: undercloud.localdomain
kernel: 4.18.0-80.el8.x86_64
os: linux
rootless: true
uptime: 9h 55m 45.58s (Approximately 0.38 days)
insecure registries:
registries:

• 192.168.24.1:8787
• 192.168.24.3:8787
• brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888
  registries:
  registries:
• registry.redhat.io
• quay.io
• docker.io
  store:
  ConfigFile: /home/stack/.config/containers/storage.conf
  ContainerStore:
  number: 0
  GraphDriverName: overlay
  GraphOptions:
• overlay.mount_program=/usr/bin/fuse-overlayfs
  GraphRoot: /home/stack/.local/share/containers/storage
  GraphStatus:
  Backing Filesystem: xfs
  Native Overlay Diff: "false"
  Supports d_type: "true"
  ImageStore:
  number: 0
  RunRoot: /run/user/1000
  ```

Additional environment details (AWS, VirtualBox, physical, etc.):
Running Podman on RHEL8.

[giuseppe]

this looks like a race between ExecStop= and the command specified in ExecStart=.

The ExecStop= is faster and the container is killed and cleaned up, but the podman started by ExecStart= is still running and systemd kills it as soon as the ExecStop= finished.

Since you are relying on Podman for managing the containers/processes I think you'll just need to use KillMode=none

[EmilienM]

Resolved per @giuseppe comment.

Thanks :-)