Podman: Error: could not get runtime: operation not permitted with stable podman image

[orf]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Using the quay.io/podman/stable image on MacOS or Kubernetes results in Error: could not get runtime: operation not permitted

Steps to reproduce the issue:

1. Run docker run -it quay.io/podman/stable podman info

Describe the results you received:

Error: could not get runtime: operation not permitted

Describe the results you expected:

It to work

Additional information you deem important (e.g. issue happens only occasionally):

It appears to be an issue with the fuse vfs. If I do docker run -it quay.io/podman/stable bash, then execute:

rm -rf /var/lib/shared/overlay-images && rm -rf /var/lib/shared/overlay-layers
rm -rf /var/lib/shared/overlay-images/images.lock /var/lib/shared/overlay-layers/layers.lock
rm -f /etc/containers/storage.conf

Then podman no longer exits with the error.

Output of podman version:

Version:            1.4.2
RemoteAPI Version:  1
Go Version:         go1.12.5
OS/Arch:            linux/amd64

Output of podman info --debug:

$docker run -it quay.io/podman/stable podman --log-level=debug info                                                   
DEBU[0000] Initializing boltdb state at /var/lib/containers/storage/libpod/bolt_state.db
DEBU[0000] Using graph driver overlay
DEBU[0000] Using graph root /var/lib/containers/storage
DEBU[0000] Using run root /var/run/containers/storage
DEBU[0000] Using static dir /var/lib/containers/storage/libpod
DEBU[0000] Using tmp dir /var/run/libpod
DEBU[0000] Using volume path /var/lib/containers/storage/volumes
DEBU[0000] Set libpod namespace to ""
DEBU[0000] [graphdriver] trying provided driver "overlay"
DEBU[0000] overlay: imagestore=/var/lib/shared
DEBU[0000] overlay: mount_program=/usr/bin/fuse-overlayfs
ERRO[0000] could not get runtime: operation not permitted

[mheon]

This is a pretty terrible error, regardless of where it's coming from - we need to get this wrapped so we can present a more reasonable explanation as to where it's coming from

[TomSweeneyRedHat]

FYI @giuseppe

[giuseppe]

we don't have really anything in place in Podman for creating a user namespace when root has limited capabilities.
Creating the user namespace won't be difficult, but we need to ensure we will still refer to the correct config files. Differently from Buildah, we will need to ensure all containers are created from the same user namespace, so we need a pause process for root as well.

I'd suggest just to use vfs for now, @rhatdan what do you think?

[rhatdan]

Yes, I am not really sure of the use case of running podman in a locked down container anyways.
Seems to me the container should be privileged if you are running it to easily launch containers or demonstrate the features of podman.

[orf]

What about a simple case where you want to push and pull an image, or do any number of things that doesn't involve running a container? Podman is pretty great in this regard.

[rhatdan]

Ok, but skopeo would also be good in that category, perhaps better.

[imcsk8]

Is this a valid use case?

[orf]

True, Skopeo is a better use case for re-tagging and pushing, you're right.

In any case, it's quite a confusing error that's not great right off the bat. If you cannot really support podman running in a locked down container it should error with a slightly more informative message!

[rhatdan]

@TomSweeneyRedHat PTAL at this, should work. Does this work if podman is used to launch command?

[TomSweeneyRedHat]

@rhatdan I'm not 100% sure what you mean by "podman is used to launch command". If I use an installed version of Podman, then podman info as a root or non-root user works just fine. If I add --privileged to the command in this issue, then I see the output as expected from info, but I do get a soft error that I didn't expect.

# podman run --privileged -it quay.io/podman/stable sudo podman info
ERRO[0000] unable to write system event: "write unixgram @00048->/run/systemd/journal/socket: sendmsg: no such file or directory" 
host:
  BuildahVersion: 1.9.0
  Conmon:
    package: podman-1.4.4-4.fc30.x86_64
    path: /usr/libexec/podman/conmon
    version: 'conmon version 1.0.0-dev, commit: 164df8af4e62dc759c312eab4b97ea9fb6b5f1fc'
  Distribution:
    distribution: fedora
    version: "30"
...  {removed rest of the happy output for brevity}

I'm not sure what's up with the system event, that's a new one to me. I think we still need to keep the --privileged flag in play though.

[TomSweeneyRedHat]

In regards to the soft error prior, there doesn't appear to be a /run/systemd/journal file being created in the container and I think should be. @mheon do you know if this is a current systemd woe?

[kbaegis]

In general, several of us not using systemd would appreciate decoupling.

[baude]

podman just needs to be compiled without systemd support ... simple as that.

[rhatdan]

Shouldn't podman fall back to the non journal method of events, if journal does not exists? Or is this configured in libpod.conf to use the journal?

[mheon]

We're explicitly configured from libpod.conf, but we could rewrite this to fall back to non-journal if not present, I think

[smekkley]

@TomSweeneyRedHat is right. I think this is related to #3917 .
The parent container engine needs --privileged to run podman inside.
I don't know what the actual requirement is. I tried with readonly mount of /dev with privileged, but it still worked.
--cap-add ALL alone also couldn't make it work.
I've tried with some other less privileged command options, but I get

Error: error creating libpod runtime: operation not permitted

I also got error with creating libpod directory in /sys/fs/cgroup

buildah has --isolation=chroot option to mitigate this. Using vfs, no host mount nor privileges is needed. It would be great if there was seamless way to use this option with podman, or somehow make it work without privileged options.

[cwansart]

If @TomSweeneyRedHat changes are in the current quay.io/podman/stable image inside a Pod in an minikube Kubernetes cluster I still get the same error as above.

When running docker run -it quay.io/podman/stable podman --log-level=debug info locally on macOS it still crashes with could not get runtime: operation not permitted. Only if I add --privileged it works fine.

There is one thing I still wanted to try: adding privileged to the image in my Kubernetes cluster. But using the privileged parameter shouldn't be the solution, should it?

[TomSweeneyRedHat]

@rhatdan, got $.02?

[rhatdan]

I have been experimenting with running podman within a locked down container, and I have made some progress. Hope to get back to it soon. But the bottom line is for now it can not be done.
Needs to disable lots of security features.

Hopefully we can fix these up and get it working. I believe it is a solvable problem. But for now you need to run with privileged.

[github-actions[bot]]

This issue had no activity for 30 days. In the absence of activity or the "do-not-close" label, the issue will be automatically closed within 7 days.

[rhatdan]

We are still working on this. We have gotten it to work, but need to get the container.conf work done to make it a reasonable solution, so that we don't need to include a huge Command line in podman.

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[vrothberg]

The containers.conf work is still on going: https://github.com/containers/libpod/pull/4698

[vrothberg]

Closing in favor of https://github.com/containers/libpod/issues/4131 (they're essentially the same).