Podman: CNI: Make firewalld rules --permanent

[Schrottfresse]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

I configured CNI to use firewalld as firewall backend and added my podman network interface to the trusted zone. When I'm reloading firewalld (firewall-cmd --reload) all my podman related rules go missing. I'd like to have the ability to make the podman firewalld rules --permanent.
(I'm starting my containers as root.)

Steps to reproduce the issue:

1. Start container

2. firewall-cmd --reload

3. Rules are missing.

Describe the results you received:

# podman start pihole
5d95bb331f8d59c4d7d5da9af98570b3f82dd933aaf22126650e31741f225a6a
# podman ps
CONTAINER ID  IMAGE                                   COMMAND  CREATED            STATUS             PORTS  NAMES
5d95bb331f8d  docker.io/pihole/pihole:latest                   About an hour ago  Up 49 seconds ago         pihole
# firewall-cmd --zone=trusted --list-all
trusted (active)
  target: ACCEPT
  icmp-block-inversion: no
  interfaces:
  sources: 172.16.21.4/32 fd00:c001::9e/128
  services:
  ports:
  protocols:
  masquerade: no
  forward-ports:
  source-ports:
  icmp-blocks:
  rich rules:
# firewall-cmd --reload
success
# firewall-cmd --zone=trusted --list-all
trusted
  target: ACCEPT
  icmp-block-inversion: no
  interfaces:
  sources:
  services:
  ports:
  protocols:
  masquerade: no
  forward-ports:
  source-ports:
  icmp-blocks:
  rich rules:

Describe the results you expected:

Rules don't go missing.

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

# podman version
Version:            1.6.4
RemoteAPI Version:  1
Go Version:         go1.12.12
OS/Arch:            linux/amd64

Output of podman info --debug:

# podman info --debug
debug:
  compiler: gc
  git commit: ""
  go version: go1.12.12
  podman version: 1.6.4
host:
  BuildahVersion: 1.12.0-dev
  CgroupVersion: v1
  Conmon:
    package: conmon-2.0.6-1.module_el8.1.0+272+3e64ee36.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.6, commit: 7a4f0dd7b20a3d4bf9ef3e5cbfac05606b08eac0'
  Distribution:
    distribution: '"centos"'
    version: "8"
  MemFree: 32140668928
  MemTotal: 33369112576
  OCIRuntime:
    name: runc
    package: runc-1.0.0-64.rc9.module_el8.1.0+272+3e64ee36.x86_64
    path: /usr/bin/runc
    version: 'runc version spec: 1.0.1-dev'
  SwapFree: 16835932160
  SwapTotal: 16835932160
  arch: amd64
  cpus: 4
  eventlogger: journald
  hostname: carter
  kernel: 4.18.0-147.5.1.el8_1.x86_64
  os: linux
  rootless: false
  uptime: 1h 35m 6.12s (Approximately 0.04 days)
registries:
  blocked: null
  insecure: null
  search:
  - registry.access.redhat.com
  - registry.fedoraproject.org
  - registry.centos.org
  - docker.io
store:
  ConfigFile: /etc/containers/storage.conf
  ContainerStore:
    number: 2
  GraphDriverName: overlay
  GraphOptions: {}
  GraphRoot: /var/lib/containers/storage
  GraphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  ImageStore:
    number: 3
  RunRoot: /var/run/containers/storage
  VolumePath: /var/lib/containers/storage/volumes

Package info (e.g. output of rpm -q podman or apt list podman):

# rpm -q podman
podman-1.6.4-2.module_el8.1.0+272+3e64ee36.x86_64

Additional environment details (AWS, VirtualBox, physical, etc.):

physical server

[mheon]

I don't think we can do this because we want the rules to disappear after reboot.

We've talked about this before, and the best we think we can do is offer a command that forces network reconfiguration for all running containers, recreating the rules.

[baude]

@mheon doesnt this also make some assumptions about the network stack being exactly the same each time as well? Given that we have zero interaction with firewalld itself, this seems nearly impossible.

would it be possible to have something like a run-hook option where an executable could be run once the container state hit running or some such? of course, this could be done nicely with systemd as well and is maybe the more appropriate approach.

[mheon]

If we do it any later, we end up with a race where traffic after the container started but before the network started will be lost.

We can probably preserves IP address and MAC address on container recreation, even for non-static IP and MAC (we have a mechanism built-in already for doing this, originally for checkpoint/restore). I think we can probably duplicate the network configuration "well enough" with this (it works for checkpoint/restore on a different system, which seems promising)

[ttys3]

I've encountered with this problem too.

sudo podman run --name ngxdemo -p 8086:80 nginxdemos/hello

❯ sudo podman inspect ngxdemo | grep IPAddress
            "SecondaryIPAddresses": null,
            "IPAddress": "10.88.1.5",

❯ sudo firewall-cmd --zone=trusted --list-all 
trusted (active)
  target: ACCEPT
  icmp-block-inversion: no
  interfaces: 
  sources: 10.88.1.2/32 10.88.1.1/32 10.88.1.4/32 10.88.1.5/32
  services: 
  ports: 
  protocols: 
  masquerade: no
  forward-ports: 
  source-ports: 
  icmp-blocks: 
  rich rules: 

reload firewalld rules:

sudo firewall-cmd --reload

sources empty now:

❯ sudo firewall-cmd --zone=trusted --list-all                  
trusted
  target: ACCEPT
  icmp-block-inversion: no
  interfaces: 
  sources: 
  services: 
  ports: 
  protocols: 
  masquerade: no
  forward-ports: 
  source-ports: 
  icmp-blocks: 
  rich rules: 

try restart one of the container:

❯ sudo podman restart ngxdemo
Error: cannot listen on the TCP port: listen tcp4 :8086: bind: address already in use

try restart again, this time it will success:

❯ sudo podman restart ngxdemo
c7a0efe0c7dc540717c301216f01fef7f9eba12315418c16ed554dd8733e2add

and got back:

❯ sudo podman inspect ngxdemo | grep IPAddress
            "SecondaryIPAddresses": null,
            "IPAddress": "10.88.1.6",

❯ sudo firewall-cmd --zone=trusted --list-all  
trusted (active)
  target: ACCEPT
  icmp-block-inversion: no
  interfaces: 
  sources: 10.88.1.4/32 10.88.1.1/32 10.88.1.6/32
  services: 
  ports: 
  protocols: 
  masquerade: no
  forward-ports: 
  source-ports: 
  icmp-blocks: 
  rich rules: 

---

ps: I even use firewalld as CNI firewall plugin backend, not the default iptables.

❯ cat /etc/cni/net.d/87-podman-bridge.conflist
{
    "cniVersion": "0.4.0",
    "name": "podman",
    "plugins": [
    {
            "type": "bridge",
            "bridge": "cni-podman0",
            "isGateway": true,
            "ipMasq": true,
            "ipam": {
        "type": "host-local",
        "routes": [
            {
            "dst": "0.0.0.0/0"
            }
        ],
        "ranges": [
            [
            {
                "subnet": "10.88.0.0/16",
                "gateway": "10.88.0.1"
            }
            ]
        ]
            }
    },
    {
            "type": "portmap",
            "capabilities": {
        "portMappings": true
            }
    },
    {
            "type": "firewall",
        "backend": "firewalld"
    }
    ]
}

[abalage]

Although this is really annoying, as I recall Docker + firewalld worked the very same way.
I always had to restart docker daemon to recreate its iptables chains and rules after I did a 'firewall-cmd --reload'. I might recall that containers had to be redeployed as well.

For me what worked is to use a container specific OS (like OpenSUSE MicroOS, Fedora CoreOS) explicitly dedicated for containers and leave network and firewall management to podman alone. I can use firewall rules outside of this VM.

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@mheon Any movement on this one?

[mheon]

No. I do want to add a command to do this, but there is no way we'll have time until Podmanv2 is done.

[KlausThornProgrammfabrik]

To my understanding, firewalld and nftables were designed to solve this exact type of problem:

• does not assume complete control of firewall backend
• won’t delete firewall rules installed by other tools or users

... from: https://developers.redhat.com/blog/2018/08/10/firewalld-the-future-is-nftables/

I guess podman/CNI would have to use nftables and create own tables. And leave firewalld to the user's local customization, only. Then firewalld-cmd --reload will just affect the user's stuff.

Have you investigated this? Are there plans for this?

[dcode]

I just ran into this as well and thinking through it, it makes sense. I'm not familiar enough with the intimate details, but seems like CNI doesn't have any sort of daemon (sort of the point, I guess), would it be possible to use a dbus-activated service (via systemd) that could examine the state of containers and recreate the appropriate entries when firewalld reloads, e.g. when Reloaded signal is emitted on org.fedoraproject.FirewallD1

[sergeiwaigant]

Same here with Podman 1.6.4 on RHEL7.7 EUS.
I don't understand why this issue is not being tackled.
With this situation we would need to disable Firewalld at all to use Podman containers as services - or is there any other workaround to overcome that problem?

We are using systems maintained by central team with Puppet and they run firewalld --reload every 30 minutes, this is breaking the iptables and the services in the containers are not working anymore.

[mheon]

As said before, --permanent is not an option because it has undesirable side effects - we do not want rules to outlive the container that created them, as would be the case after reboots. We are considering a podman system command that will recreate rules after a firewall reload, which I suppose you could run after your firewalld --reload though there would be an interruption in connectivity.

Can I ask why you are reloading firewalld on such a regular basis? I don't think I've heard of anyone doing that before, and it's definitely not something I think we (or the CNI maintainers) have considered.

[sergeiwaigant]

OK. Let's say it differently.
Any reload, which can happen from tme to time even manualy, should not result in any kind of unavailablity of a service running in a container.
If that should be normal, then any kind of system services running in container are just useless.

[EssGeeEich]

What about adding a --persistent flag for the podman run command?

[mheon]

That doesn't solve the fundamental problem that the rules will survive reboot, and the CNI plugins do not deal well with reboots - we'll leak rules if any container is still running when the system is shut down, and that will break things badly when the IP in question is reused for a fresh container. Podman itself does not directly create the rules, so I can't hack around this during our usual post-reboot cleanup because I don't know what rules need to be changed. The CNI plugins are theoretically safe to call on an already-dead container after a reboot to finalize cleanup of things like these firewall rules, but this has never actually worked for us.

[rhatdan]

@mccv1r0 @mheon Any new ideas on this one? Seems to be an issue with CNI Plugins.

[Masber]

as a workaround, I managed to go over this issue by moving Centos 8 firewall configuration from nftables to iptables

[mnsmithuk]

I found the best workaround for now was to run :

firewall-cmd --permanent --zone=trusted --add-interface=cni-podman0

but not sure if this introduces a security issue in anyway.

[mnsmithuk]

My bad. It does not work because podman populates trusted sources not trusted interfaces. On a reboot the source disappears and the conatiner is unreachable. I thought the workaround I suggested had fixed this yesterday but just confirmed it is actually still in the same position.

[thkukuk]

Can I ask why you are reloading firewalld on such a regular basis?

I see the same problem that the podman rules go away regular. The reason for the firewalld reload here are intrusion detection tools, which reconfigure the firewall according to entries they find in the logs.

[mchouque]

Regarding the reload thing, on Fedora, NetworkManager has this postinstall line:

test -f /usr/bin/firewall-cmd && firewall-cmd --reload --quiet || :

Meaning when you upgrade the package, you lose rules created by podman.

[rhatdan]

I could swear I saw a pull request that was reloading the network to fix this? But I can not find it right now.
@Luap99 @mheon @zhangguanzhang @baude Was I dreaming?

[mheon]

@Luap99 was working on it. I believe it merged.

On Thu, Dec 24, 2020 at 08:02 Daniel J Walsh [e-mail]
wrote:

I could swear I saw a pull request that was reloading the network to fix
this? But I can not find it right now.
@Luap99 https://github.com/Luap99 @mheon https://github.com/mheon
@zhangguanzhang https://github.com/zhangguanzhang @baude
https://github.com/baude Was I dreaming?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/containers/podman/issues/5431#issuecomment-750876746,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AB3AOCC6EKQA3YUHJM4YI3LSWM3WBANCNFSM4LEOX5XQ
.

[Luap99]

Yes it was #8571. We now have the podman network reload command to recreate the iptables rules. This does not make the rules permanent but if they are deleted after a firewall reload you can run this command with the --all option to recreate the rules without having to restart containers.

[rhatdan]

Well that is the best we got for now. Is this documented in the troublshooting.md? It would be nice to get a quick blog out on this.