Rke: [CentOS 8][Debian 10] No network connectivity outside of pod or to other pods with Canal and Calico

[papanito]

I've successfully installed kubernetes 1.16 with rke 0.3.2 however, there is no connection from the pods to outside nor to other pods.

RKE version:

v0.3.2

Docker version: (docker version,docker info preferred)

vdocker info
Client:
 Debug Mode: false

Server:
 Containers: 31
  Running: 22
  Paused: 0
  Stopped: 9
 Images: 25
 Server Version: 18.09.1
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: c4446665cb9c30056f4998ed953e6d4ff22c7c39
 runc version: 4fc53a81fb7c994640722ac585fa9ca548971871
 init version: fec3683
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 4.18.0-80.11.2.el8_0.x86_64
 Operating System: CentOS Linux 8 (Core)
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 31.19GiB
 Name: node002
 ID: EDFH:4JPU:JZ26:T5CH:N2TS:FUBG:QKIC:I7WA:C5A5:LN4A:424G:VFAX
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
 Product License: Community Engine

Operating system and kernel: (cat /etc/os-release, uname -r preferred)

4.18.0-80.11.2.el8_0.x86_64 (CentOS 8)

Type/provider of hosts: (VirtualBox/Bare-metal/AWS/GCE/DO)

Bare-metal

cluster.yml file:

# 
# Cluster Config
# 
enable_cluster_alerting: true
enable_cluster_monitoring: true
enable_network_policy: false
local_cluster_auth_endpoint:
  enabled: true

nodes:
  - address: y.y.y.8
    internal_address: 192.168.100.1
    user: ansible
    role: [controlplane,worker,etcd]
  - address: x.x.x.14
    internal_address: 192.168.100.2
    user: ansible
    role: [controlplane,worker,etcd]

cluster_name: dev-cluster
kubernetes_version: "v1.16.2-rancher1-1" # rke config --list-version --all

# 
# Rancher Config
# 
rancher_kubernetes_engine_config:
  addon_job_timeout: 90
  authentication:
    strategy: x509
  ignore_docker_version: true

dns:
  provider: coredns
  upstreamnameservers:
  - 1.1.1.1
  - 8.8.8.8
  - 8.8.4.4
  - 213.133.100.100
  - 213.133.99.99
  - 213.133.98.98
  #- 2a01:4f8:0:1::add:1010
  #- 2a01:4f8:0:1::add:9898
  #- 2a01:4f8:0:1::add:9999

network:
  plugin: canal
  options:
      #canal_iface: enp4s0.4000
      canal_flannel_backend_type: vxlan

ingress:
    provider: nginx

services:
  etcd:
    snapshot: true
    creation: 6h
    retention: 48h
  kube-api:
    service_cluster_ip_range: 10.43.0.0/16
  kube-controller:
    cluster_cidr: 10.42.0.0/16
    service_cluster_ip_range: 10.43.0.0/16
  kubelet:
    cluster_domain: cluster.local
    cluster_dns_server: 10.43.0.10

Steps to Reproduce:

1. install new cluster
   rke up --config config.yaml
2. Run multitool container
   kubectl run multitool --image=praqma/network-multitool --restart Never kubectl exec -it multitool -- bash

Results:
connection from pod

bash-5.0# traceroute 172.217.168.14
traceroute to 172.217.168.14 (172.217.168.14), 30 hops max, 46 byte packets
 1  x.x.x.14 (x.x.x.14)  0.012 ms  0.098 ms  0.004 ms
 2  *  *  *
 3  *  *  *
 4  *  *  *
 5  *c^C

bash-5.0# curl 172.217.168.14
curl: (7) Failed to connect to 172.217.168.14 port 80: Operation timed out

connection from node

The connection on the hosts seems fine

[user@node001 ~]$ ping google.com
PING google.com(fra16s25-in-x0e.1e100.net (2a00:1450:4001:825::200e)) 56 data bytes
64 bytes from fra16s25-in-x0e.1e100.net (2a00:1450:4001:825::200e): icmp_seq=1 ttl=57 time=4.84 ms
64 bytes from fra16s25-in-x0e.1e100.net (2a00:1450:4001:825::200e): icmp_seq=2 ttl=57 time=4.86 ms
^C
--- google.com ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 2ms
rtt min/avg/max/mdev = 4.836/4.850/4.864/0.014 ms

[user@node001 ~]$ curl 172.217.168.14
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>301 Moved</TITLE></HEAD><BODY>
<H1>301 Moved</H1>
The document has moved
<A HREF="http://www.google.com/">here</A>.
</BODY></HTML>

Here my ip route output on the node

default via x.x.x.1 dev enp4s0 proto static metric 100 
x.x.x.1 dev enp4s0 proto static scope link metric 100 
x.x.x.14 dev enp4s0 proto kernel scope link src x.x.x.14 metric 100 
10.42.0.3 dev calie0ef90acfad scope link 
10.42.0.4 dev cali6d09fa47963 scope link 
10.42.0.5 dev calicfe92e547cd scope link 
10.42.0.6 dev cali83e218e70ab scope link 
10.42.1.0/24 via 10.42.1.0 dev flannel.1 onlink 
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown 
192.168.100.0/24 dev enp4s0.4000 proto kernel scope link src 192.168.100.2 metric 400 

The k8s nodes themselves to not run firewalld and I checked several things which seem fine

selinux disabled

[user@node001 ~]$ cat /etc/sysconfig/selinux

# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#     enforcing - SELinux security policy is enforced.
#     permissive - SELinux prints warnings instead of enforcing.
#     disabled - No SELinux policy is loaded.
# SELINUX=enforcing
SELINUX=disabled
# SELINUXTYPE= can take one of these three values:
#     targeted - Targeted processes are protected,
#     minimum - Modification of targeted policy. Only selected processes are protected. 
#     mls - Multi Level Security protection.
SELINUXTYPE=targeted

sysctl

[user@node001 ~]$ sysctl net.bridge
net.bridge.bridge-nf-call-arptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
....
[user@node001 ~]$ sysctl net.ipv4.ip_forward
net.ipv4.ip_forward = 1
[user@node001 ~]$ sysctl net.ipv6.ip_forward
net.ipv6.conf.all.forwarding = 1

iptables

sudo iptables -L -v -n
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain FORWARD (policy DROP 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
 145K 6575K DOCKER-USER  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
 145K 6575K DOCKER-ISOLATION-STAGE-1  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
  190 18756 ACCEPT     all  --  *      docker0  0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
   63  3780 DOCKER     all  --  *      docker0  0.0.0.0/0            0.0.0.0/0           
  296 15245 ACCEPT     all  --  docker0 !docker0  0.0.0.0/0            0.0.0.0/0           
    0     0 ACCEPT     all  --  docker0 docker0  0.0.0.0/0            0.0.0.0/0           

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain DOCKER (1 references)
 pkts bytes target     prot opt in     out     source               destination         

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
 pkts bytes target     prot opt in     out     source               destination         
  296 15245 DOCKER-ISOLATION-STAGE-2  all  --  docker0 !docker0  0.0.0.0/0            0.0.0.0/0           
 145K 6575K RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Maybe also useful to know, I use a multitool container for debugging, this is the config

 kubectl get pods -o wide
NAME        READY   STATUS    RESTARTS   AGE   IP          NODE        NOMINATED NODE   READINESS GATES
multitool   1/1     Running   1          20h   10.42.0.4   x.x.x.14    <none>           <none>

and this is the ip addr list in the container

bash-5.0# ip addr list
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
3: eth0@if106528: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether ca:49:03:ec:75:43 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.42.0.4/32 scope global eth0
       valid_lft forever preferred_lft forever
4: tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000
    link/ipip 0.0.0.0 brd 0.0.0.0

[papanito]

Switching* to calico results in the same problem

*) complete teardown of cluster, cleaning nodes and spin up new cluster

[ekarlso]

I am trying on a brand new Centos 8 installed cluster to run Canal but my canal instances on the nodes are not coming up.

2019-12-01 00:25:38.163 [WARNING][8696] table.go 797: Retrying... error=exit status 1 ipVersion=0x4 table="filter"                                                                                                
2019-12-01 00:25:38.173 [WARNING][8696] table.go 1030: Failed to execute ip(6)tables-restore command error=exit status 1 errorOutput="iptables-restore: line 81 failed\n" input="*filter\n:cali-OUTPUT - -\n:cali-pro-kns.cattle-system - -\n:cali-fw-calif681d912619 - -\n:cali-from-wl-dispatch - -\n:cali-to-wl-dispatch - -\n:cali-INPUT - -\n:cali-failsafe-in - -\n:cali-to-host-endpoint - -\n:cali-from-hep-forward - -\n:cali-FORWARD - -\n:cali-pri-_XnQ5h_hZf854SLqzqE - -\n:cali-pro-_XnQ5h_hZf854SLqzqE - -\n:cali-from-host-endpoint - -\n:cali-to-hep-forward - -\n:cali-wl-to-host - -\n:cali-failsafe-out - -\n:cali-pri-kns.cattle-system - -\n:cali-tw-calif681d912619 - -\n-A cali-INPUT -m comment --comment \"cali:FewJpBykm9iJ-YNH\" --in-interface cali+ --goto cali-wl-to-host\n-A cali-INPUT -m comment --comment \"cali:hder3ARWznqqv8Va\" -m mark --mark 0x10000/0x10000 --jump ACCEPT\n-A cali-INPUT -m comment --comment \"cali:xgOu2uJft6H9oDGF\" --jump MARK --set-mark 0/0xf0000\n-A cali-INPUT -m comment --comment \"cali:_-d-qojMfHM6NwBo\" --jump cali-from-host-endpoint\n-A cali-INPUT -m comment --comment \"cali:LqmE76MP94lZTGhA\" -m comment --comment \"Host endpoint policy accepted packet.\" -m mark --mark 0x10000/0x10000 --jump ACCEPT\n-A cali-failsafe-in -m comment --comment \"cali:wWFQM43tJU7wwnFZ\" -p tcp -m multiport --destination-ports 22 --jump ACCEPT\n-A cali-failsafe-in -m comment --comment \"cali:LwNV--R8MjeUYacw\" -p udp -m multiport --destination-ports 68
--jump ACCEPT\n-A cali-failsafe-in -m comment --comment \"cali:QOO5NUOqOSS1_Iw0\" -p tcp -m multiport --destination-ports 179 --jump ACCEPT\n-A cali-failsafe-in -m comment --comment \"cali:cwZWoBSwVeIAZmVN\" -p
tcp -m multiport --destination-ports 2379 --jump ACCEPT\n-A cali-failsafe-in -m comment --comment \"cali:7FbNXT91kugE_upR\" -p tcp -m multiport --destination-ports 2380 --jump ACCEPT\n-A cali-failsafe-in -m comment --comment \"cali:ywE9WYUBEpve70WT\" -p tcp -m multiport --destination-ports 6666 --jump ACCEPT\n-A cali-failsafe-in -m comment --comment \"cali:l-WQSVBf_lygPR0J\" -p tcp -m multiport --destination-ports 6667 --jump ACCEPT\n-A cali-FORWARD -m comment --comment \"cali:vjrMJCRpqwy5oRoX\" --jump MARK --set-mark 0/0xe0000\n-A cali-FORWARD -m comment --comment \"cali:A_sPAO0mcxbT9mOV\" -m mark --mark 0/0x10000 --jump cali-from-hep-forward\n-A cali-FORWARD -m comment --comment \"cali:8ZoYfO5HKXWbB3pk\" --in-interface cali+ --jump cali-from-wl-dispatch\n-A cali-FORWARD -m comment --comment \"cali:jdEuaPBe14V2hutn\" --out-interface cali+ --jump cali-to-wl-dispatch\n-A cali-FORWARD -m comment --comment \"cali:12bc6HljsMKsmfr-\" --jump cali-to-hep-forward\n-A cali-FORWARD -m comment --comment \"cali:MH9kMp5aNICL-Olv\" -m comment --comment
\"Policy explicitly accepted packet.\" -m mark --mark 0x10000/0x10000 --jump ACCEPT\n-A cali-pri-kns.cattle-system -m comment --comment \"cali:blfKjcY1bW5P59PS\" --jump MARK --set-mark 0x10000/0x10000\n-A cali-pri-kns.cattle-system -m comment --comment \"cali:cOr8yOvbzAjvJk4K\" -m mark --mark 0x10000/0x10000 --jump RETURN\n-A cali-tw-calif681d912619 -m comment --comment \"cali:B7uEGrk9xwBVhMk4\" -m conntrack --ctstate
RELATED,ESTABLISHED --jump ACCEPT\n-A cali-tw-calif681d912619 -m comment --comment \"cali:GndLzXIjxwKzvmLx\" -m conntrack --ctstate INVALID --jump DROP\n-A cali-tw-calif681d912619 -m comment --comment \"cali:rwSfJISxsKRZQwze\" --jump MARK --set-mark 0/0x10000\n-A cali-tw-calif681d912619 -m comment --comment \"cali:P65cGkixdG-HMcCm\" --jump cali-pri-kns.cattle-system\n-A cali-tw-calif681d912619 -m comment --comment \"cali:4f9QMVITo7_f04lP\" -m comment --comment \"Return if profile accepted\" -m mark --mark 0x10000/0x10000 --jump RETURN\n-A cali-tw-calif681d912619 -m comment --comment \"cali:s_dx3YzQTEIqOgNx\" --jump cali-pri-_XnQ5h_hZf854SLqzqE\n-A cali-tw-calif681d912619 -m comment --comment \"cali:JbwwEaclY9B_wUIr\" -m comment --comment \"Return if profile accepted\" -m mark --mark 0x10000/0x10000 --jump RETURN\n-A cali-tw-calif681d912619 -m comment --comment \"cali:mLQz2ruAgH0vwXy5\" -m comment --comment \"Drop if no profiles matched\" --jump DROP\n-A cali-wl-to-host -m comment --comment \"cali:Ee9Sbo10IpVujdIY\" --jump cali-from-wl-dispatch\n-A cali-wl-to-host -m comment --comment \"cali:nSZbcOoG1xPONxb8\" -m comment --comment \"Configured DefaultEndpointToHostAction\" --jump ACCEPT\n-A cali-failsafe-out -m comment --comment \"cali:82hjfji-wChFhAqL\" -p udp -m multiport --destination-ports 53 --jump ACCEPT\n-A cali-failsafe-out -m comment --comment \"cali:TNM3RfEjbNr72hgH\" -p udp -m multiport --destination-ports 67 --jump ACCEPT\n-A cali-failsafe-out -m comment --comment \"cali:ycxKitIl4u3dK0HR\" -p tcp -m multiport --destination-ports 179 --jump ACCEPT\n-A cali-failsafe-out -m comment --comment \"cali:hxjEWyxdkXXkdvut\" -p tcp -m multiport --destination-ports 2379 --jump ACCEPT\n-A cali-failsafe-out -m comment --comment \"cali:cA_GLtruuvG88KiO\" -p tcp -m multiport --destination-ports 2380 --jump ACCEPT\n-A cali-failsafe-out -m comment --comment \"cali:Sb1hkLYFMrKS6r01\" -p tcp -m multiport --destination-ports 6666 --jump ACCEPT\n-A cali-failsafe-out -m comment --comment \"cali:UwLSebGONJUG4yG-\" -p tcp -m multiport --destination-ports 6667 --jump ACCEPT\n-A cali-fw-calif681d912619 -m comment --comment \"cali:P1bwMnOb_-3OiXDO\" -m conntrack --ctstate RELATED,ESTABLISHED --jump ACCEPT\n-A cali-fw-calif681d912619 -m comment --comment \"cali:4tyPTVV1X7ZLonHT\" -m conntrack --ctstate INVALID --jump DROP\n-A cali-fw-calif681d912619 -m comment --comment \"cali:PsZD-B0eDVy2ELCn\" --jump MARK --set-mark 0/0x10000\n-A cali-fw-calif681d912619 -m comment --comment \"cali:NB-MLy7B6EiMqlh7\" -m
comment --comment \"Drop VXLAN encapped packets originating in pods\" -p 17 -m multiport --destination-ports 4789 -m u32 --u32 \"0>>22&0x3C@12>>8=0x1000\" --jump DROP\n-A cali-fw-calif681d912619 -m comment --comment \"cali:_gjkNyGLSYJELMtr\" -m comment --comment \"Drop IPinIP encapped packets originating in pods\" -p 4 --jump DROP\n-A cali-fw-calif681d912619 -m comment --comment \"cali:RC2x855oEoYZOQ8l\" --jump cali-pro-kns.cattle-system\n-A cali-fw-calif681d912619 -m comment --comment \"cali:oiifS9Y5mHjbvqDU\" -m comment --comment \"Return if profile accepted\" -m mark --mark 0x10000/0x10000 --jump RETURN\n-A cali-fw-calif681d912619 -m comment --comment \"cali:PeFeZBf_XZL5RVPg\" --jump cali-pro-_XnQ5h_hZf854SLqzqE\n-A cali-fw-calif681d912619 -m comment --comment \"cali:8FxrLzfk0OiOWcA5\" -m comment --comment \"Return if profile accepted\" -m mark --mark 0x10000/0x10000 --jump RETURN\n-A cali-fw-calif681d912619 -m comment --comment \"cali:ExvN2YpzFPl27A8n\" -m comment --comment \"Drop if no profiles matched\" --jump DROP\n-A cali-from-wl-dispatch -m comment --comment \"cali:0zKaZFhNkKbT6WTl\" --in-interface calif681d912619 --goto cali-fw-calif681d912619\n-A cali-from-wl-dispatch -m comment --comment \"cali:-tMWQXr3kwR69xxP\" -m comment --comment
\"Unknown interface\" --jump DROP\n-A cali-to-wl-dispatch -m comment --comment \"cali:NGJcHtUIoILmwDoo\" --out-interface calif681d912619 --goto cali-tw-calif681d912619\n-A cali-to-wl-dispatch -m comment --comment \"cali:cjscRHXHZm3-MZHr\" -m comment --comment \"Unknown interface\" --jump DROP\n-A cali-OUTPUT -m comment --comment \"cali:Mq1_rAdXXH3YkrzW\" -m mark --mark 0x10000/0x10000 --jump ACCEPT\n-A cali-OUTPUT -m comment --comment \"cali:69FkRTJDvD5Vu6Vl\" --out-interface cali+ --jump RETURN\n-A cali-OUTPUT -m comment --comment \"cali:Fskumj4SGQtDV6GC\" --jump MARK --set-mark 0/0xf0000\n-A cali-OUTPUT -m comment --comment \"cali:8rXMdo5sNesjJxGc\" --jump cali-to-host-endpoint\n-A cali-OUTPUT -m comment --comment \"cali:Ja-pnrHi-PrNKxgd\" -m comment --comment \"Host endpoint policy accepted packet.\" -m mark --mark 0x10000/0x10000 --jump ACCEPT\n-A cali-pro-kns.cattle-system -m comment --comment \"cali:IClMGDKmI4RBpktd\" --jump MARK --set-mark 0x10000/0x10000\n-A cali-pro-kns.cattle-system -m comment --comment \"cali:iBJqPq9boKtL_Qr-\"
-m mark --mark 0x10000/0x10000 --jump RETURN\n-I INPUT -m comment --comment \"cali:Cz_u1IQiXIMmKD4c\" --jump cali-INPUT\n-I FORWARD -m comment --comment \"cali:wUHhoiAYhphO9Mso\" --jump cali-FORWARD\n-I OUTPUT -m comment --comment \"cali:tVnHkvAo15HuiPy0\" --jump cali-OUTPUT\nCOMMIT\n" ipVersion=0x4 output="" table="filter"                                                                                                
2019-12-01 00:25:38.174 [WARNING][8696] table.go 794: Failed to program iptables, will retry error=exit status 1 ipVersion=0x4 table="filter"                                                                     
2019-12-01 00:25:38.238 [WARNING][8696] table.go 797: Retrying... error=exit status 1 ipVersion=0x4 table="filter"                                                                                                
2019-12-01 00:25:38.245 [WARNING][8696] table.go 1030: Failed to execute ip(6)tables-restore command error=exit status 1 errorOutput="iptables-restore: line 81 failed\n" input="*filter\n:cali-from-hep-forward -
-\n:cali-INPUT - -\n:cali-failsafe-in - -\n:cali-to-host-endpoint - -\n:cali-from-host-endpoint - -\n:cali-to-hep-forward - -\n:cali-FORWARD - -\n:cali-pri-_XnQ5h_hZf854SLqzqE - -\n:cali-pro-_XnQ5h_hZf854SLqzqE
- -\n:cali-tw-calif681d912619 - -\n:cali-wl-to-host - -\n:cali-failsafe-out - -\n:cali-pri-kns.cattle-system - -\n:cali-from-wl-dispatch - -\n:cali-to-wl-dispatch - -\n:cali-OUTPUT - -\n:cali-pro-kns.cattle-system - -\n:cali-fw-calif681d912619 - -\n-A cali-FORWARD -m comment --comment \"cali:vjrMJCRpqwy5oRoX\" --jump MARK --set-mark 0/0xe0000\n-A cali-FORWARD -m comment --comment \"cali:A_sPAO0mcxbT9mOV\" -m mark --mark 0/0x10000 --jump cali-from-hep-forward\n-A cali-FORWARD -m comment --comment \"cali:8ZoYfO5HKXWbB3pk\" --in-interface cali+ --jump cali-from-wl-dispatch\n-A cali-FORWARD -m comment --comment \"cali:jdEuaPBe14V2hutn\" --out-interface cali+ --jump cali-to-wl-dispatch\n-A cali-FORWARD -m comment --comment \"cali:12bc6HljsMKsmfr-\" --jump cali-to-hep-forward\n-A cali-FORWARD -m comment --comment \"cali:MH9kMp5aNICL-Olv\" -m comment --comment \"Policy explicitly accepted packet.\" -m mark --mark 0x10000/0x10000 --jump ACCEPT\n-A cali-wl-to-host -m comment --comment \"cali:Ee9Sbo10IpVujdIY\" --jump cali-from-wl-dispatch\n-A cali-wl-to-host -m comment --comment \"cali:nSZbcOoG1xPONxb8\" -m comment --comment \"Configured DefaultEndpointToHostAction\" --jump ACCEPT\n-A cali-failsafe-out -m comment --comment \"cali:82hjfji-wChFhAqL\" -p udp -m multiport --destination-ports 53 --jump ACCEPT\n-A cali-failsafe-out -m comment --comment \"cali:TNM3RfEjbNr72hgH\" -p udp -m multiport --destination-ports 67 --jump ACCEPT\n-A cali-failsafe-out -m comment --comment \"cali:ycxKitIl4u3dK0HR\" -p tcp -m multiport --destination-ports 179 --jump ACCEPT\n-A cali-failsafe-out -m comment --comment \"cali:hxjEWyxdkXXkdvut\" -p tcp -m multiport --destination-ports 2379 --jump ACCEPT\n-A cali-failsafe-out -m comment --comment \"cali:cA_GLtruuvG88KiO\" -p tcp -m multiport --destination-ports 2380 --jump ACCEPT\n-A cali-failsafe-out -m comment --comment \"cali:Sb1hkLYFMrKS6r01\" -p tcp -m multiport --destination-ports 6666 --jump ACCEPT\n-A cali-failsafe-out -m comment --comment \"cali:UwLSebGONJUG4yG-\" -p tcp -m multiport --destination-ports 6667 --jump ACCEPT\n-A cali-pri-kns.cattle-system -m comment --comment \"cali:blfKjcY1bW5P59PS\" --jump MARK --set-mark 0x10000/0x10000\n-A cali-pri-kns.cattle-system -m comment --comment \"cali:cOr8yOvbzAjvJk4K\" -m mark --mark 0x10000/0x10000 --jump RETURN\n-A cali-tw-calif681d912619 -m comment --comment \"cali:B7uEGrk9xwBVhMk4\" -m conntrack --ctstate RELATED,ESTABLISHED --jump ACCEPT\n-A cali-tw-calif681d912619 -m comment --comment \"cali:GndLzXIjxwKzvmLx\" -m
conntrack --ctstate INVALID --jump DROP\n-A cali-tw-calif681d912619 -m comment --comment \"cali:rwSfJISxsKRZQwze\" --jump MARK --set-mark 0/0x10000\n-A cali-tw-calif681d912619 -m comment --comment \"cali:P65cGkixdG-HMcCm\" --jump cali-pri-kns.cattle-system\n-A cali-tw-calif681d912619 -m comment --comment \"cali:4f9QMVITo7_f04lP\" -m comment --comment \"Return if profile accepted\" -m mark --mark 0x10000/0x10000 --jump
RETURN\n-A cali-tw-calif681d912619 -m comment --comment \"cali:s_dx3YzQTEIqOgNx\" --jump cali-pri-_XnQ5h_hZf854SLqzqE\n-A cali-tw-calif681d912619 -m comment --comment \"cali:JbwwEaclY9B_wUIr\" -m comment --comment \"Return if profile accepted\" -m mark --mark 0x10000/0x10000 --jump RETURN\n-A cali-tw-calif681d912619 -m comment --comment \"cali:mLQz2ruAgH0vwXy5\" -m comment --comment \"Drop if no profiles matched\" --jump DROP\n-A cali-OUTPUT -m comment --comment \"cali:Mq1_rAdXXH3YkrzW\" -m mark --mark 0x10000/0x10000 --jump ACCEPT\n-A cali-OUTPUT -m comment --comment \"cali:69FkRTJDvD5Vu6Vl\" --out-interface cali+ --jump RETURN\n-A cali-OUTPUT -m comment --comment \"cali:Fskumj4SGQtDV6GC\" --jump MARK --set-mark 0/0xf0000\n-A cali-OUTPUT -m comment --comment \"cali:8rXMdo5sNesjJxGc\" --jump cali-to-host-endpoint\n-A cali-OUTPUT -m
comment --comment \"cali:Ja-pnrHi-PrNKxgd\" -m comment --comment \"Host endpoint policy accepted packet.\" -m mark --mark 0x10000/0x10000 --jump ACCEPT\n-A cali-pro-kns.cattle-system -m comment --comment \"cali:IClMGDKmI4RBpktd\" --jump MARK --set-mark 0x10000/0x10000\n-A cali-pro-kns.cattle-system -m comment --comment \"cali:iBJqPq9boKtL_Qr-\" -m mark --mark 0x10000/0x10000 --jump RETURN\n-A cali-fw-calif681d912619 -m comment --comment \"cali:P1bwMnOb_-3OiXDO\" -m conntrack --ctstate RELATED,ESTABLISHED --jump ACCEPT\n-A cali-fw-calif681d912619 -m comment --comment \"cali:4tyPTVV1X7ZLonHT\" -m conntrack --ctstate INVALID --jump DROP\n-A cali-fw-calif681d912619 -m comment --comment \"cali:PsZD-B0eDVy2ELCn\" --jump MARK --set-mark 0/0x10000\n-A cali-fw-calif681d912619 -m comment --comment \"cali:NB-MLy7B6EiMqlh7\" -m comment --comment \"Drop VXLAN encapped packets originating in pods\" -p 17 -m multiport --destination-ports 4789 -m u32 --u32 \"0>>22&0x3C@12>>8=0x1000\" --jump DROP\n-A cali-fw-calif681d912619 -m comment --comment \"cali:_gjkNyGLSYJELMtr\" -m comment --comment \"Drop IPinIP encapped packets originating in pods\" -p 4 --jump DROP\n-A cali-fw-calif681d912619 -m comment --comment \"cali:RC2x855oEoYZOQ8l\" --jump cali-pro-kns.cattle-system\n-A cali-fw-calif681d912619 -m comment --comment \"cali:oiifS9Y5mHjbvqDU\" -m comment --comment \"Return if profile accepted\" -m mark --mark 0x10000/0x10000 --jump RETURN\n-A cali-fw-calif681d912619 -m comment --comment \"cali:PeFeZBf_XZL5RVPg\" --jump cali-pro-_XnQ5h_hZf854SLqzqE\n-A cali-fw-calif681d912619 -m comment --comment \"cali:8FxrLzfk0OiOWcA5\" -m comment --comment \"Return if profile accepted\" -m mark
--mark 0x10000/^C

[johnjcool]

the problem is centos8 switched from iptables to nftables.

fixed it by adding FELIX_IPTABLESBACKEND=NFT to calico-node like mentioned in this thread https://github.com/projectcalico/calico/issues/2322

[papanito]

@johnjcool you mean by adding an environment variable in the pod?

[johnjcool]

Yes, on DeamonSet.

[papanito]

By change do you know how the rke config.yaml should look like?

[papanito]

I've updated the dameonset

....
    Environment:
      FELIX_IPTABLESBACKEND:              NFT
....

and new pods were created

calico-node-82s74                          1/1     Running     0          70s
calico-node-qv7fg                          1/1     Running     0          48s

However, when I issue a ping from my multitoolcontainer the ping still fails

bash-5.0# ping google.com
PING google.com (216.58.207.78) 56(84) bytes of data.
^C
--- google.com ping statistics ---
7 packets transmitted, 0 received, 100% packet loss, time 144ms

However, name resolution now works!

[longwuyuan]

I changed default iptables in /etc/alternatives to point to iptables-legacy as a resolution. The problem occurred on all latest combo of k8s, calico on debian10 ;

% sudo docker images | grep calico
[sudo] password for me: 
calico/node                                                      v3.13.3             3efc460414d9        3 days ago          261MB
calico/pod2daemon-flexvol                                        v3.13.3             d8e1bc26a77b        3 days ago          112MB
calico/cni                                                       v3.13.3             8229c7314d00        3 days ago          224MB
calico/kube-controllers                                          v3.13.3             15858f141bbf        3 days ago          56.6MB
calico/node                                                      v3.11.2             81f501755bb9        3 months ago        255MB
calico/cni                                                       v3.11.2             c317181e3b59        3 months ago        204MB
calico/pod2daemon-flexvol                                        v3.11.2             f69bca7e2325        3 months ago        111MB
calico/kube-controllers                                          v3.11.2             9e897df2f2af        3 months ago        52.5MB
% kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.2", GitCommit:"52c56ce7a8272c798dbc29846288d7cd9fbae032", GitTreeState:"clean", BuildDate:"2020-04-16T11:54:15Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
ssdnodes0% kubectl version
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.2", GitCommit:"52c56ce7a8272c798dbc29846288d7cd9fbae032", GitTreeState:"clean", BuildDate:"2020-04-16T11:56:40Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.2", GitCommit:"52c56ce7a8272c798dbc29846288d7cd9fbae032", GitTreeState:"clean", BuildDate:"2020-04-16T11:48:36Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
% 

[stale[bot]]

This issue/PR has been automatically marked as stale because it has not had activity (commit/comment/label) for 60 days. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.