Podman: systemd doesn't autostart/can't find pods or containers from a non-root user account

[Rafficer]

Is this a BUG REPORT or FEATURE REQUEST?

/kind bug

Description

Rootless pods or containers don't start on boot with unit files generated by podman generate systemd -f --name <podname>

Steps to reproduce the issue:

1. Create a pod with containers (not as root)
   $ podman pod create --name testpod $ podman container run -d --pod testpod nginx:latest

2. Generate the systemd files

   $ podman generate systemd -f --name testpod
   

3. Copy these files to /etc/systemd/system/

   $ sudo cp pod-testpod.service container-containername.service /etc/systemd/system/
   

4. Enable the service

   $ sudo systemctl daemon-reload
   $ sudo systemctl enable pod-testpod.service
   

5. Reboot

   $ sudo shutdown -r now
   

Describe the results you received:

The containers aren't running ($ podman containers ls) and the service is in a failed state. journalctl -u pod-testpod.service shows the following error:

podman[1064]: Error: unable to find container 952a9ebc3b47-infra: no container with name or ID 952a9ebc3b47-infra found: no such container

Describe the results you expected:

The pods and container are running

Additional information you deem important (e.g. issue happens only occasionally):

It makes sense that this happens, as the service is executed by root, but the containers live in a normal user environment. I've previously also tried adding User= and Group= directives to the unit files, but this didn't help either. How can I make sure that systemd finds my containers when they aren't living in root's environment, but in a user environment?

Output of podman version:

Version:            1.8.0
RemoteAPI Version:  1
Go Version:         go1.13.6
OS/Arch:            linux/amd64

Output of podman info --debug:

debug:
  compiler: gc
  git commit: ""
  go version: go1.13.6
  podman version: 1.8.0
host:
  BuildahVersion: 1.13.1
  CgroupVersion: v2
  Conmon:
    package: conmon-2.0.10-2.fc31.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.10, commit: 6b526d9888abb86b9e7de7dfdeec0da98ad32ee0'
  Distribution:
    distribution: fedora
    version: "31"
  IDMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  MemFree: 1584246784
  MemTotal: 2077593600
  OCIRuntime:
    name: crun
    package: crun-0.12.2.1-1.fc31.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 0.12.2.1
      commit: cd7cea7114db5f6aa35fbb69fa307c19c2728a31
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  SwapFree: 1719660544
  SwapTotal: 1719660544
  arch: amd64
  cpus: 2
  eventlogger: journald
  hostname: testserver
  kernel: 5.5.8-200.fc31.x86_64
  os: linux
  rootless: true
  slirp4netns:
    Executable: /usr/bin/slirp4netns
    Package: slirp4netns-0.4.0-20.1.dev.gitbbd6f25.fc31.x86_64
    Version: |-
      slirp4netns version 0.4.0-beta.3+dev
      commit: bbd6f25c70d5db2a1cd3bfb0416a8db99a75ed7e
  uptime: 4m 41.08s
registries:
  search:
  - docker.io
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - registry.centos.org
  - quay.io
store:
  ConfigFile: /home/myusername/.config/containers/storage.conf
  ContainerStore:
    number: 5
  GraphDriverName: overlay
  GraphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-0.7.5-2.fc31.x86_64
      Version: |-
        fusermount3 version: 3.6.2
        fuse-overlayfs: version 0.7.5
        FUSE library version 3.6.2
        using FUSE kernel interface version 7.29
  GraphRoot: /home/myusername/.local/share/containers/storage
  GraphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  ImageStore:
    number: 4
  RunRoot: /run/user/1000/containers
  VolumePath: /home/myusername/.local/share/containers/storage/volumes

Package info (e.g. output of rpm -q podman or apt list podman):

podman-1.8.0-2.fc31.x86_64

Additional environment details (AWS, VirtualBox, physical, etc.):
Fedora Server 31 on Proxmox. Also happened in CentOS 8.1, with an older Version of podman, though.

[vrothberg]

Hi @Rafficer, thanks for opening the issue! It should be fixed with Podman v1.8.1. Once the package is updated, can you re-try and report if it works?

[Rafficer]

Hey, thanks for the quick reply. I've set up Fedora Rawhide to test it there with v1.8.1, but sadly I get the same outcome.

[mheon]

@vrothberg This looks like a root/rootless thing - container was created as a normal user, but it's being managed by root systemd.

@Rafficer Root and rootless podman do not share containers (for security and technical reasons), so this is somewhat expected. If you have a systemd user session you can autostart there with systemctl --user or it may be possible to set the user in the unit file being run as root so that it is run as your user.

[Rafficer]

@mheon Thanks for your reply.

Yes, as I've said in the "Additional information you deem important" section, it does make sense. However, I couldn't get it to work with the User= flag and couldn't find any documentation on how to run it as non-root user. That's why I've opened the issue.

[mheon]

@vrothberg Do we have any documentation on Podman + Systemd as rootless?

[vrothberg]

Good catch, @mheon!

@vrothberg Do we have any documentation on Podman + Systemd as rootless?

I will open an issue to improve documentation in the podman-generate-systemd man page.

[vrothberg]

Closing this issue. The .service files must be copied to $HOME/.config/systemd/user for rootless.

[Rafficer]

I've gotten it to work now. I'll briefly document them here in case someone else stumbles upon it as well:

None of these commands are executed as root.

1. Normally generate the files

   podman generate systemd -f --name testpod

2. Create the systemd user directory

   mkdir -p ~/.config/systemd/user

3. Copy the files over to that directory

   cp pod-testpod.service container-containername.service ~/.config/systemd/user/

4. Edit the ~/.config/systemd/user/pod-testpod.service file (or the container file if you're not using pods) and replace WantedBy=multi-user.target with WantedBy=default.target at the bottom. This is necessary as the user instance of systemd doesn't have a multi-user.target.

5. Enable the service

   systemctl --user daemon-reload
systemctl --user enable pod-testpod.service

6. Enable lingering for that user so the service starts at boot

   loginctl enable-linger $USER

That's it. Rebooting should now autostart the containers.

---

@vrothberg Thanks for your help again :)

Just one more idea, it might be cool to somehow make it work that you don't have to replace the multi-user.target line. That podman automatically detects this. After all, a non-root user can't generate files that are then used by the root account anyway.

[vrothberg]

Just one more idea, it might be cool to somehow make it work that you don't have to replace the multi-user.target line. That podman automatically detects this. After all, a non-root user can't generate files that are then used by the root account anyway.

Podman v1.8.1 generates both:

[Install]
WantedBy=multi-user.target default.target

This allows for running it both as root and non-root.

[rhatdan]

Should we add a --install flag to the code to automatically install the service once it is created, To eliminate the work that @Rafficer had to do, to figure out how to get this to work.

[vrothberg]

Should we add a --install flag to the code to automatically install the service once it is created, To eliminate the work that @Rafficer had to do, to figure out how to get this to work.

I think improving the man pages should suffice. Doing a cp is easy to script and there are more steps needed in order to run the service (e.g., reloading the daemon etc.).