Podman: Port rebinding not working properly for rootless containers within pods?

[andrew-kennedy]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Rootless podman doesn't seem to be able to respect --publish parameter on containers associated with rootless pods.

Steps to reproduce the issue:

1. Create a rootless pod with podman pod create

2. Create a new container with podman create, rebinding an existing port, for example --publish 5000:3000.

3. Other containers in that pod will not be able to access that container at localhost:5000. If the publish flag is omitted, they are available at localhost:3000 for other containers in the pod as you would expect.

Describe the results you received:
Containers in a rootless pod could not connect to other containers at either the original port or the remapped port specified in the --publish hostport:containerport argument.

Describe the results you expected:
Rootless containers sharing the same pod could have their ports remapped within the context of that pod. This makes it useful when dealing with existing docker images that listen on say, port 80, so that one can rebind them within the pod to a higher port. Currently this only works when they are not in a pod.

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

Version:      2.0.3
API Version:  1
Go Version:   go1.14.3
Built:        Wed Dec 31 16:00:00 1969
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.15.0
  cgroupVersion: v2
  conmon:
    package: conmon-2.0.19-1.fc32.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.19, commit: 5dce9767526ed27f177a8fa3f281889ad509fea7'
  cpus: 24
  distribution:
    distribution: fedora
    version: "32"
  eventLogger: file
  hostname: yggdrasil.ak.codes
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.7.10-201.fc32.x86_64
  linkmode: dynamic
  memFree: 3707359232
  memTotal: 33604173824
  ociRuntime:
    name: crun
    package: crun-0.14.1-1.fc32.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 0.14.1
      commit: 598ea5e192ca12d4f6378217d3ab1415efeddefa
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  rootless: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.4-1.fc32.x86_64
    version: |-
      slirp4netns version 1.1.4
      commit: b66ffa8e262507e37fca689822d23430f3357fe8
      libslirp: 4.3.1
      SLIRP_CONFIG_VERSION_MAX: 2
  swapFree: 8109944832
  swapTotal: 8589930496
  uptime: 55h 55m 9.62s (Approximately 2.29 days)
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - registry.centos.org
  - docker.io
store:
  configFile: /home/andrew/.config/containers/storage.conf
  containerStore:
    number: 15
    paused: 0
    running: 14
    stopped: 1
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.1.2-1.fc32.x86_64
      Version: |-
        fusermount3 version: 3.9.1
        fuse-overlayfs: version 1.1.0
        FUSE library version 3.9.1
        using FUSE kernel interface version 7.31
  graphRoot: /home/andrew/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 41
  runRoot: /run/user/1000/containers
  volumePath: /home/andrew/.local/share/containers/storage/volumes
version:
  APIVersion: 1
  Built: 0
  BuiltTime: Wed Dec 31 16:00:00 1969
  GitCommit: ""
  GoVersion: go1.14.3
  OsArch: linux/amd64
  Version: 2.0.3```

**Package info (e.g. output of `rpm -q podman` or `apt list podman`):**

podman-2.0.3-1.fc32.x86_64
```

Additional environment details (AWS, VirtualBox, physical, etc.):

[mheon]

The 5000 in 5000:3000 is a host port - that is, a port on the host the container is running on. It's meant for external access to the container from other systems.

This has no effect within the pod. localhost in the pod points to the pod's network, not the host's network; so it's completely expected that you would only be able to connect to port 3000 within the pod - as that's the port your service is running on. The --publish flag has no effect on the internal network of pods or containers; it just forwards external traffic to them.

[mheon]

(The documentation on how this works is sparse to non-existent, which is something that definitely needs to be improved)

[andrew-kennedy]

So, is there no way to rebind a container's port within the context of the pod's generated networking interface?

[andrew-kennedy]

If so that's really frustrating, heavily limits the usefulness of pods if it basically means I'd have to change any image's exposed ports inside the images themselves. Makes it really hard to group two images in a pod that by default have colliding ports.

[mheon]

Networking within a pod acts like networking on a normal, fat host with
multiple different services running on it - there's a shared public network
interface, and a shared set of ports available to bind. No more than one
app within the pod can own port 80, or 443, etc. This isn't really a Podman
limitation - the kernel will prevent you from binding more than one app to
the port. This is a necessary consequence of sharing a network namespace
between containers in the pod, which is the only way we can achieve
inter-container communication for rootless containers at present. Rootless
networking is an area we'd like to improve, but our lack of privileges
makes our options somewhat limited.

The general recommendation we have for things like webapps is to run them
on 8080, 8081, 8082, etc and then have a reverse proxy that is actually
exposed. This works pretty well, albeit with considerable additional
effort. For other things like databases, where a proxy solution may not be
feasible, there is still the option of running them on different ports, and
then exposing each port.

On Fri, Jul 31, 2020 at 8:22 PM Andrew Kennedy [e-mail]
wrote:

If so that's really frustrating, heavily limits the usefulness of pods if
it basically means I'd have to change any image's exposed ports inside the
images themselves. Makes it really hard to group two images in a pod that
by default have colliding ports.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/containers/podman/issues/7175#issuecomment-667436236,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AB3AOCGO7CK5XIWCNSOBWXTR6NN5RANCNFSM4PQ3VWMA
.

[andrew-kennedy]

Right, so I know that only one container can be bound to any one port inside the pod. What I want is for podman to let me "override" the ports the container image "ships with" with my own ports. Many distinct PHP based docker image services by default all bind to the same ports, because they expect you to be using them in a context where you always use the --publish flag and can thus ensure no port binding collision on the host. I want a way to avoid port binding collision in pods regardless of the images I run, without changing those images. Currently there seems to be no simple way to do this.

[andrew-kennedy]

Basically I want like a --pod-publish flag that lets me easily avoid port collisions.

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[andrew-kennedy]

Is there no planned way to remap ports within the context of a pod? This seems like a really necessary feature for containers that weren't necessarily already intended to be in the same "port namespace". Otherwise good luck trying to easily use containers like Organizr in the same pod as other containers that contain some kind of PHP web service like Heimdall because they will always conflict on the default port, and changing the default port sometimes requires direct image edits and rebuilds. Podman needs to provide a better way, within the scope of pods.

[rhatdan]

@giuseppe @AkihiroSuda @mheon Any comment on this?

[mheon]

The only way I can think of doing this is Seccomp notifier syscall interception/hijacking - we could intercept bind calls, strip application-requested port numbers, and slot in our own randomly-assigned number. Potential problems: We'd need to report that number back to the user somehow - podman pod inspect would work but the script would need to give Podman pointers as to what PID had been given what port number. Complicating matters is that we'd need the same notifier script for every container in the pod.

If I had to implement this, I'd do it as a pod-level flag, --randomize-port, accepting a port or range of ports. Any application that tries to bind to something in that range will receive a random unused port instead.

[andrew-kennedy]

@mheon I'd be okay with that. Anything that saves me container image edits and/or stops forcing me to not use a pod would be welcome. So would your proposed future method be that I'd put ports I know will conflict in the --randomize-ports flag and then use podman inspect to actually find those ports to be able to reference them from other containers? And they'd persist between restarts of the pod?

[mheon]

Persistence across restart wasn't something I originally thought about, but it should be possible, we'd just have to store which container ID corresponded to which port.

Otherwise, yes, that sounds like the flow I had in mind.

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@andrew-kennedy @mheon Did anyone ever work on this? Seems like you came ot agreement, and then progress stalled?

[mheon]

We need support for Seccomp notifier in Common first (in progress but not
merged last I checked). Once that is done I can look into it.

On Sat, Oct 10, 2020, 07:24 Daniel J Walsh [e-mail] wrote:

@andrew-kennedy https://github.com/andrew-kennedy @mheon
https://github.com/mheon Did anyone ever work on this? Seems like you
came ot agreement, and then progress stalled?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/containers/podman/issues/7175#issuecomment-706533445,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AB3AOCDFADGYZ2UUUIGQIDTSKA75BANCNFSM4PQ3VWMA
.

[giuseppe]

I am not sure what the expectation is for --publish 5000:3000. I'd expect the host port 5000 is forwarded to the pod network namespace port 3000. If we want to support rebinding ports inside of the pod itself, we could create a network namespace for the pod and then another network namespace for the container and setup a veth between the two namespaces. That is possible, but seems quite complex to deal with, and IMO it should be a different option than --publish

We recently added podman network for rootless containers. Is it something that could help with your use case? You could setup a network for the containers to communicate with each other, and at the same time without sharing the network namespace (pod create --network='shared-net' --share="cgroup,ipc" ...).

[rhatdan]

Since we got no further feedback, I am closing. Reopen if this is still an issue.