Podman: Network namespace silently stops port sharing.

[bbros-dev]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Launching a container with a network namespace that connects to Wireguard, silently prevents ports being shared with the host.

Steps to reproduce the issue:

1. Create network namespace wg-cntnr per WireGuard instructions

# ip netns add wg-cntnr
# ip link add wg0 type wireguard
# ip link set wg0 netns wg-cntnr
# ip -n wg-cntnr addr add 192.168.4.33/32 dev wg0
# ip netns exec wg-cntnr wg setconf wg0 /etc/wireguard/wg0.conf
# ip -n wg-cntnr link set wg0 up
# ip -n wg-cntnr route add default dev wg0

1. Test that Podman shares ports without network namespace. In a terminal shell run the following:

sudo podman run -it --rm -p 8080:8080 busybox sh -c "while true ; do echo 'Hello World' | nc -lp 8080; done"

Open another terminal shell and run:

$ socat TCP4:localhost:8080 -
Hello World

1. Test that Podman silently stops sharing ports with the network namespace wg-cntnr. In a terminal shell run the following:

sudo podman run --network=ns:/var/run/netns/wg-cntnr -it --rm -p 8080:8080 busybox sh -c "while true ; do echo 'Hello World' | nc -lp 8080; done"

Open another terminal shell and run:

$ socat TCP4:localhost:8080 -
2020/10/15 22:43:14 socat[8473] E connect(5, AF=2 127.0.0.1:8080, 16): Connection refused

Describe the results you received:

Using the network namespace wg-cntnr, the second terminal shell returns:

2020/10/15 22:43:14 socat[8473] E connect(5, AF=2 127.0.0.1:8080, 16): Connection refused

Describe the results you expected:

If port sharing is prevbented by the use of a network namespace, the Podman command should error out or warn, rather than silently consuming the --publish port data.

If port sharing should work when using the network namespace wg-cntnr, the second terminal shell should return:

Hello World

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

$ podman version
Version:      2.1.1
API Version:  2.0.0
Go Version:   go1.15.2
Built:        Thu Jan  1 10:00:00 1970
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.16.1
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: 'conmon: /usr/libexec/podman/conmon'
    path: /usr/libexec/podman/conmon
    version: 'conmon version 2.0.20, commit: '
  cpus: 2
  distribution:
    distribution: ubuntu
    version: "18.04"
  eventLogger: journald
  hostname: desktop.local.lan
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.4.0-47-generic
  linkmode: dynamic
  memFree: 354873344
  memTotal: 16652054528
  ociRuntime:
    name: runc
    package: 'runc: /usr/sbin/runc'
    path: /usr/sbin/runc
    version: 'runc version spec: 1.0.1-dev'
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  rootless: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: 'slirp4netns: /usr/bin/slirp4netns'
    version: |-
      slirp4netns version 1.1.4
      commit: unknown
      libslirp: 4.3.1-git
      SLIRP_CONFIG_VERSION_MAX: 3
  swapFree: 4293640192
  swapTotal: 4294963200
  uptime: 199h 12m 4.9s (Approximately 8.29 days)
registries:
  search:
  - docker.io
store:
  configFile: /home/hedge/.config/containers/storage.conf
  containerStore:
    number: 10
    paused: 0
    running: 0
    stopped: 10
  graphDriverName: vfs
  graphOptions: {}
  graphRoot: /home/hedge/.local/share/containers/storage
  graphStatus: {}
  imageStore:
    number: 23
  runRoot: /run/user/1000/containers
  volumePath: /home/hedge/.local/share/containers/storage/volumes
version:
  APIVersion: 2.0.0
  Built: 0
  BuiltTime: Thu Jan  1 10:00:00 1970
  GitCommit: ""
  GoVersion: go1.15.2
  OsArch: linux/amd64
  Version: 2.1.1

Package info (e.g. output of rpm -q podman or apt list podman):

$ apt list podman
Listing... Done
podman/unknown,now 2.1.1~2 amd64 [installed]

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide?

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):

Physical

[mheon]

To clarify - does accessing the port never work, or does it briefly work and then cease working?

[bbros-dev]

It never works with the network namespace (update: adding --cap-add all does not help).
With the network namespace:

$ socat -d -d TCP4:localhost:8080 -
2020/10/16 06:42:59 socat[23443] N opening connection to AF=2 127.0.0.1:8080
2020/10/16 06:42:59 socat[23443] E connect(5, AF=2 127.0.0.1:8080, 16): Connection refused
2020/10/16 06:42:59 socat[23443] N exit(1)

[bbros-dev]

Hmm, as best we can tell this is the only workaround:

$ sudo podman run --network=ns:/var/run/netns/wg-cntnr -it --rm -p 8080:8080 --name bb2 busybox sh
/ # nc localhost 8080
Hello World

Any other workaround suggestions to get the port data shared?

[bbros-dev]

Noting a use case for issue #5069 :
Turns out curl can use a unix-socket... its not uncommon to see in the wild snippets such as

curl --unix-socket /var/run/docker.sock http://localhost/images/json
# and
echo -e "GET /images/json HTTP/1.1\r\n\r\n" | socat unix-connect:/var/run/docker.sock STDIO

It appears that the only possible workaround for this issue would have been via #5069.

Any other workaround suggestions to get the port data shared?

We think the answer is: no workarounds are possible.

[bbros-dev]

Extending the curl --unix-socket ... idea above you can workaround this issue:

1. Inside the container: Configure the service/application to redirect TCP port to a socket
2. Run container with network namespace and shared folder --network=ns:/var/run/netns/wg-cntnr --volume /host/app/:/cnt/app/
3. Outside the container redirect the socket to a local TCP port:

Example on the host:

HOST_PORT=3333
SHARED_SOCKET=/host/app/gitea.sock
SOCAT_LOG=/host/app/log/socat-$HOST_PORT.log
socat -d -d -lf $SOCAT_LOG \
      TCP4-LISTEN:$HOST_PORT,reuseaddr,fork \
      UNIX-CONNECT:$SHARED_SOCKET

[DazAh]

@bbros-dev Another workaround I found was to use the veth device type to create a link between the default and container namespace.

ip link add veth0 type veth peer name veth1
ip link set veth1 netns wg-cntnr
ip -n wg-cntnr addr add 10.1.1.2/24 dev veth1
ip -n wg-cntnr link set up veth1
ip addr add 10.1.1.1/24 dev veth0
ip link set up veth0

curl 10.1.1.2

More reading:
https://lwn.net/Articles/580893/
https://man7.org/linux/man-pages/man4/veth.4.html

[bbros-dev]

Interesting, thanks for sharing. Am deep elsewhere in our code base so can't go back to this.

It isn't clear where the port ~sharing~ mapping takes place, e.g. say curl localhost:8080 on the host hits curl 10.1.1.2:80 inside wg-cntnr?
If an example like that can be made to work then this becomes a documentation issue, where your example would be the fix - IMO.

[DazAh]

Interesting, thanks for sharing. Am deep elsewhere in our code base so can't go back to this.

It isn't clear where the port sharing takes place, e.g. say curl localhost:8080 on the host hits curl 10.1.1.2:80 inside wg-cntnr?
If an example like that can be made to work then this becomes a documentation issue, where your example would be the fix - IMO.

If I am following your question correctly - no port sharing actually takes place still, you would have to hit the peered veth device on your host (eg 10.1.1.2) using the containers local port. This requires your application to be listening on the peered veth1 device inside the container.

sudo podman run -ti --rm --network=ns:/var/run/netns/wg-vpn --dns=1.1.1.1 -d -p 8080:80 --name=demo nginxdemos/hello
curl 10.1.1.2:8080
curl: (7) Failed to connect to 10.1.1.2 port 8080: Connection refused
curl 10.1.1.2:80
<!DOCTYPE html>
.....

[bbros-dev]

This requires your application to be listening on the peered veth1 device inside the container.

If I understand correctly, this requirement is the wrinkle in this setup. The application in the namespace is listening on a WG interface as part of an OAuth2 dance with a remote partner.
I believe we could set this up with the veth device pair and VxLAN but haven't the time to sort all that out, so the socat workaround was our quick and dirty ;)

[mheon]

We really need to make more network options conflict - Podman can easily detect a configuration like this where the -p flag is doing nothing and emit a warning or error (Docker only logs a warning for cases like this, but I feel like it's more of an error).

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@mheon Did you ever make this change?

[mheon]

Yes - we now warn (but do not error) in circumstances where -p is specified but not honored.