Kind: No DNS resolution with Kind running in rootful Podman

[BobyMCbobs]

What happened:
I'm running Kind in rootful Podman and am having issues pulling images.
It appears that there's no DNS working.

What you expected to happen:

• containers to be able to pull
• DNS to resolve

How to reproduce it (as minimally and precisely as possible):

1. Bring up a Pod

$ kubectl run nginx --image nginx
pod/nginx created

1. Show Pod info

$ kubectl describe pod nginx
...
  Warning  Failed     4s    kubelet, kind-control-plane  Failed to pull image "nginx": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/library/nginx:latest": failed to resolve reference "docker.io/library/nginx:latest": failed to do request: Head https://registry-1.docker.io/v2/library/nginx/manifests/latest: dial tcp: lookup registry-1.docker.io on 1.1.1.1:53: read udp 10.88.0.6:54866->1.1.1.1:53: read: no route to host
...

1. ping a domain from an existing Pod

$ kubectl -n local-path-storage exec -it deployment/local-path-provisioner -- ping dns.google
ping: bad address 'dns.google'
command terminated with exit code 1

1. ping an address from an existing Pod

$ kubectl -n local-path-storage exec -it deployment/local-path-provisioner -- ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: seq=0 ttl=117 time=31.139 ms

So there appears to be internet, but no DNS.

Anything else we need to know?:

When running Podman as root, it appears to have DNS resolution

$ sudo podman run -it --rm alpine:3.12 ping dns.google
PING dns.google (8.8.4.4): 56 data bytes
64 bytes from 8.8.4.4: seq=0 ttl=42 time=31.008 ms

Environment:

• kind version: (use kind version): kind v0.9.0 go1.15.2 linux/amd64
• Kubernetes version: (use kubectl version): Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.0", GitCommit:"9e991415386e4cf155a24b1da15becaa390438d8", GitTreeState:"clean", BuildDate:"2020-03-25T14:58:59Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"linux/amd64"}
• Podman version: (use podman info):

version:
  APIVersion: 2.0.0
  Built: 1602087680
  BuiltTime: Thu Oct  8 05:21:20 2020
  GitCommit: ""
  GoVersion: go1.15.2
  OsArch: linux/amd64
  Version: 2.1.1

• OS (e.g. from /etc/os-release):

NAME=Fedora
VERSION="33.20201030.0 (Silverblue)"
ID=fedora
VERSION_ID=33
VERSION_CODENAME=""
PLATFORM_ID="platform:f33"
PRETTY_NAME="Fedora 33.20201030.0 (Silverblue)"
...

[BobyMCbobs]

/assign @aojea

[aojea]

I can' t reproduce it in my Fedora 32 and podman podman 2.1.1.

kind]$ sudo kubectl get pods -o wide
NAME    READY   STATUS    RESTARTS   AGE   IP           NODE                 NOMINATED NODE   READINESS GATES
nginx   1/1     Running   0          40s   10.244.0.5   kind-control-plane   <none>           <none>

usual suspects are firewalld ot iptables mode
https://kind.sigs.k8s.io/docs/user/known-issues/#fedora32-firewalld

[BobyMCbobs]

@aojea thank you for following up!

I installed a fresh copy of Fedora Silverblue 33 in Boxes.
Downloaded _kind_ (applied a setuid for root on it) and _kubectl_.
Brought up the cluster and went through the original steps in this ticket but was unable to replicate my issue.
I haven't tweaked any system settings.

It appears to be using _nftables_ also:

$ sudo cat /etc/firewalld/firewalld.conf | grep Firewall
# FirewallBackend
FirewallBackend=nftables

[BobyMCbobs]

OK I found the issue @aojea.

There's an interference with my _firewalld_ config.
After systemctl disable --now firewalld, things appear to be working now:

kubectl get pods -A -w
NAMESPACE            NAME                                         READY   STATUS              RESTARTS   AGE
default              nginx                                        0/1     ContainerCreating   0          1s
kube-system          coredns-f9fd979d6-54xxz                      1/1     Running             0          75s
kube-system          coredns-f9fd979d6-pw88w                      1/1     Running             0          75s
kube-system          etcd-kind-control-plane                      1/1     Running             0          79s
kube-system          kindnet-7k7kd                                1/1     Running             0          75s
kube-system          kube-apiserver-kind-control-plane            1/1     Running             0          79s
kube-system          kube-controller-manager-kind-control-plane   1/1     Running             0          78s
kube-system          kube-proxy-ltfnn                             1/1     Running             0          75s
kube-system          kube-scheduler-kind-control-plane            1/1     Running             0          78s
local-path-storage   local-path-provisioner-78776bfc44-pfwnj      1/1     Running             0          75s
default              nginx                                        1/1     Running             0          12s

I'll review my _firewalld_ configuration.

[BenTheElder]

Sounds like https://kind.sigs.k8s.io/docs/user/known-issues/#fedora32-firewalld but with podman as well

[aojea]

/close
Thank for verifying ... _iptables and DNS... it's always iptables and DNS ..._ :rofl:

[k8s-ci-robot]

@aojea: Closing this issue.

In response to this:

/close
Thank for verifying ... _iptables and DNS... it's always iptables and DNS ..._ :rofl:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[BobyMCbobs]

Sounds like https://kind.sigs.k8s.io/docs/user/known-issues/#fedora32-firewalld but with podman as well

@BenTheElder

Thank for verifying ... _iptables and DNS... it's always iptables and DNS ..._

@aojea

Thank you for following up, I wasn't sure that this was a duplicate issue - sorry about that.

I switched to _iptables_ as per the _known-issues_ page. Working fine now!

[BenTheElder]

We might need to update the known issues page.

I'm pretty certain I have iptables-nft working on my corp workstation, but I'm not sure if firewalld etc. are involved actually. (it's dockerd on ~debian).

[aojea]

hi @BobyMCbobs , I just found your blog https://blog.calebwoodbine.com/kind-on-fedora-33/ and KIND should be independent of the iptables backend, it should work for iptables-nft (nftables) and iptables-legacy,.
Previous issues were with docker, because it is incompatible with firewalld as you can see in the known issues

https://kind.sigs.k8s.io/docs/user/known-issues/#fedora32-firewalld

On Fedora 32 firewalld moved to nftables backend by default. This seems to be incompatible with Docker, leading to KIND cluster nodes not being able to reach each other.

But if you say that you are running with podman I need to go deeper to understand better the problem
/reopen
/assign

@BobyMCbobs do you still have an environment to reproduce this?
if affirmative, can you paste the iptables-legacy-save and iptables-nft-save output in an environment without DNS working?

[k8s-ci-robot]

@aojea: Reopened this issue.

In response to this:

hi @BobyMCbobs , I just found your blog https://blog.calebwoodbine.com/kind-on-fedora-33/ and I KIND should be independent of the iptables backend, it should work for iptables-nft (nftables) and iptables-legacy,.
Previous issues were with docker, because it is incompatible with firewalld as you can see in the known issues

https://kind.sigs.k8s.io/docs/user/known-issues/#fedora32-firewalld

On Fedora 32 firewalld moved to nftables backend by default. This seems to be incompatible with Docker, leading to KIND cluster nodes not being able to reach each other.

But if you say that you are running with podman I need to go deeper to understand better the problem
/reopen
/assign

@BobyMCbobs do you still have an environment to reproduce this?
if affirmtive, can you paste the iptables-legacy-save and iptables-nft-save output in an environment without DNS working?

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[BobyMCbobs]

@BobyMCbobs do you still have an environment to reproduce this?
if affirmative, can you paste the iptables-legacy-save and iptables-nft-save output in an environment without DNS working?

@aojea,
Thank you for following up.

I switched back to nftables for this output and made sure the same errors occurred (including the same image pull cause of DNS).

iptables-legacy-save output:

# Generated by iptables-save v1.8.5 on Wed Dec  2 21:39:32 2020
*nat
:PREROUTING ACCEPT [174:24468]
:INPUT ACCEPT [6:1262]
:OUTPUT ACCEPT [254:16284]
:POSTROUTING ACCEPT [254:16284]
:CNI-DN-dc6a4d127af830bf71995 - [0:0]
:CNI-HOSTPORT-DNAT - [0:0]
:CNI-HOSTPORT-MASQ - [0:0]
:CNI-HOSTPORT-SETMARK - [0:0]
:CNI-dc6a4d127af830bf71995c2e - [0:0]
-A PREROUTING -m addrtype --dst-type LOCAL -j CNI-HOSTPORT-DNAT
-A OUTPUT -m addrtype --dst-type LOCAL -j CNI-HOSTPORT-DNAT
-A POSTROUTING -m comment --comment "CNI portfwd requiring masquerade" -j CNI-HOSTPORT-MASQ
-A POSTROUTING -s 10.88.0.21/32 -m comment --comment "name: \"podman\" id: \"8e2af17f9660b35f50e4c07ca1ab897b1e2844b7d1007578be466c73dc2c217d\"" -j CNI-dc6a4d127af830bf71995c2e
-A CNI-DN-dc6a4d127af830bf71995 -s 10.88.0.0/16 -d 127.0.0.1/32 -p tcp -m tcp --dport 40603 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-dc6a4d127af830bf71995 -s 127.0.0.1/32 -d 127.0.0.1/32 -p tcp -m tcp --dport 40603 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-dc6a4d127af830bf71995 -d 127.0.0.1/32 -p tcp -m tcp --dport 40603 -j DNAT --to-destination 10.88.0.21:6443
-A CNI-HOSTPORT-DNAT -p tcp -m comment --comment "dnat name: \"podman\" id: \"8e2af17f9660b35f50e4c07ca1ab897b1e2844b7d1007578be466c73dc2c217d\"" -m multiport --dports 40603 -j CNI-DN-dc6a4d127af830bf71995
-A CNI-HOSTPORT-MASQ -m mark --mark 0x2000/0x2000 -j MASQUERADE
-A CNI-HOSTPORT-SETMARK -m comment --comment "CNI portfwd masquerade mark" -j MARK --set-xmark 0x2000/0x2000
-A CNI-dc6a4d127af830bf71995c2e -d 10.88.0.0/16 -m comment --comment "name: \"podman\" id: \"8e2af17f9660b35f50e4c07ca1ab897b1e2844b7d1007578be466c73dc2c217d\"" -j ACCEPT
-A CNI-dc6a4d127af830bf71995c2e ! -d 224.0.0.0/4 -m comment --comment "name: \"podman\" id: \"8e2af17f9660b35f50e4c07ca1ab897b1e2844b7d1007578be466c73dc2c217d\"" -j MASQUERADE
COMMIT
# Completed on Wed Dec  2 21:39:32 2020
# Generated by iptables-save v1.8.5 on Wed Dec  2 21:39:32 2020
*mangle
:PREROUTING ACCEPT [1026:291610]
:INPUT ACCEPT [858:268404]
:FORWARD ACCEPT [131:7001]
:OUTPUT ACCEPT [1014:125876]
:POSTROUTING ACCEPT [1086:132101]
COMMIT
# Completed on Wed Dec  2 21:39:32 2020
# Generated by iptables-save v1.8.5 on Wed Dec  2 21:39:32 2020
*raw
:PREROUTING ACCEPT [1026:291610]
:OUTPUT ACCEPT [1014:125876]
COMMIT
# Completed on Wed Dec  2 21:39:32 2020
# Generated by iptables-save v1.8.5 on Wed Dec  2 21:39:32 2020
*security
:INPUT ACCEPT [838:267436]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [1014:125876]
COMMIT
# Completed on Wed Dec  2 21:39:32 2020
# Generated by iptables-save v1.8.5 on Wed Dec  2 21:39:32 2020
*filter
:INPUT ACCEPT [794:259911]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [944:118347]
:CNI-ADMIN - [0:0]
:CNI-FORWARD - [0:0]
-A FORWARD -m comment --comment "CNI firewall plugin rules" -j CNI-FORWARD
-A CNI-FORWARD -m comment --comment "CNI firewall plugin admin overrides" -j CNI-ADMIN
-A CNI-FORWARD -d 10.88.0.21/32 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A CNI-FORWARD -s 10.88.0.21/32 -j ACCEPT
COMMIT
# Completed on Wed Dec  2 21:39:32 2020

Finally iptables-nft-save output outputs:

Unknown arguments found on commandline

A fresh Fedora Silverblue 33 ISO will render the same results to my testing. Set up is same instructions as my post and the steps in this issue.

[aojea]

@dcbw @squeed it seems that the cni plugins are using the legacy iptables instead of the nft version, that is the same one used by firewalld, causing the environment to fail ^^^
Do the CNI plugins check the iptables version to use? how can we sync it with firewalld?

[squeed]

Hmm. CNI binaries just use whatever /bin/iptables they find - how are you packaging them?

[BenTheElder]

does firewalld use iptables-nft? or just nft directly?

kind is doing ~the same update-alternatives trick from kube-proxy, counting
the rules in iptables and iptables-nft and selecting the one with the most
rules.

docker injects rules inside the container using the host iptables.
podman I have no idea, but I'm guessing it doesn't do that, so we don't
have anything to go on and default to legacy.

On Wed, Dec 2, 2020 at 1:51 AM Casey Callendrello [e-mail]
wrote:

Hmm. CNI binaries just use whatever /bin/iptables they find - how are you
packaging them?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/kubernetes-sigs/kind/issues/1939#issuecomment-737119459,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAHADK3PEUJQWPHXPY5B2HLSSYE3HANCNFSM4T5OXRJA
.

[aojea]

Hmm. CNI binaries just use whatever /bin/iptables they find - how are you packaging them?

They CNI plugin package come from fedora repos, podman just use them for the network iirc

[BobyMCbobs]

does firewalld use iptables-nft? or just nft directly?

@BenTheElder, here's the _FirewallBackend_ section from my _/etc/firewalld/firewalld.conf_ file

# FirewallBackend
# Selects the firewall backend implementation.
# Choices are:
#   - nftables (default)
#   - iptables (iptables, ip6tables, ebtables and ipset)
# FirewallBackend=nftables
FirewallBackend=iptables

However, there isn't an _nftables_ program installed - so I believe it's using _iptables-nft_ since it's installed

[BenTheElder]

Discussed more with Antonio -- There's two potential issues here:

• We need everything on the host to line up and use iptables-legacy or iptables-nft (this is hopefully "just" a configuration issue on your machine)
• Once the host is lined up, the node tries to detect which to use, but podman isn't writing any rules (reasonably) so we don't currently detect what the host is using and default to legacy. It's not obvious how we'd best solve that one even once the first one is correct

I'd been referring to the latter mostly, but I think you all are discussing the former 😅

[squeed]

This is, sadly, a common battle in this world, since there is no kernel-level flag that indicates which mode of iptables is in use. The "official" API is: where does /sbin/iptables symlink.

When we need to run iptables in a container, we generally bind-mount / to /host and then chroot. We then patch things up in the dockerfile with a simple script.

For example, see https://github.com/openshift/sdn/blob/master/images/iptables-scripts/iptables

[BenTheElder]

Interesting approach, thanks. I'm not sure how comfortable I'd be having kind always bind mount / into the container ... 🤔 (versus kube-proxy which is actually supposed to manage host network anyhow).

The kubernetes approach (which we currently use) is pretty nice, but requires some existing rules of some sort in the namespace unfortunately 😞

[squeed]

I just realized this is all moot - we're talking about CNI plugins executed by podman on the real node to bring up the kind "node" containers. Not the CNI executed by containerd / cri-o inside the kind nodes.

Scanning the issue, I suspect it's because we have some bits inside the CNI plugins that tweak iptables, and the node in question has firewalld with raw nftables. We support iptables-nft, but not nft itself.

[BenTheElder]

Scanning the issue, I suspect it's because we have some bits inside the CNI plugins that tweak iptables, and the node in question has firewalld with raw nftables. We support iptables-nft, but not nft itself.

Fedora seems to be pushing for firewalld / nft to be the default going forward, is there any hope for fixing this?

As is we point users to:
https://kind.sigs.k8s.io/docs/user/known-issues/#fedora32-firewalld
(should probably be renamed "firewalld nft" ...)

[squeed]

Fedora seems to be pushing for firewalld / nft to be the default going forward, is there any hope for fixing this?

Definitely! We probably just need to teak the "default" CNI configuration used by podman.