K3s: Traefik HTTP and HTTS CrashLoopBackOff

[saphoooo]

Describe the bug

When installing K3s with canal, traefik pod with HTTP and HTTPS failed to start :

k3s kubectl logs -n kube-system svclb-traefik-58ddb6cc55-z8nd2 -c https
+ trap exit TERM INT
+ cat /proc/sys/net/ipv4/ip_forward
+ '[' 0 '!=' 1 ]
+ exit 1

To Reproduce
Steps to reproduce the behavior:

sudo k3s server --cluster-cidr 10.244.0.0/16 --disable-agent
sudo k3s agent --no-flannel --server https://192.168.99.131:6443 --token $(sudo cat /var/lib/rancher/k3s/server/node-token)
k3s kubectl apply -f https://docs.projectcalico.org/v3.3/getting-started/kubernetes/installation/hosted/canal/rbac.yaml
k3s kubectl apply -f https://docs.projectcalico.org/v3.3/getting-started/kubernetes/installation/hosted/canal/canal.yaml
k3s kubectl get po -n kube-system -w

[erikwilson]

It looks like ip forwarding is disabled for your system @saphoooo. Does enabling it help?

[saphoooo]

No, it doesn't. It's the subject of this report.

[Indexyz]

@erikwilson I have same problem when I create LoadBalancer Service

# sysctl net.ipv4.ip_forward
net.ipv4.ip_forward = 1
crictl exec -it pod bash
(In Pod) #  sysctl net.ipv4.ip_forward
net.ipv4.ip_forward = 0

It's looks like ip_forward different between host and pod?

k3s version v0.4.0 (7822549f)
Using custom network cni (Calico)

[manics]

I'm seeing the same, k3s 0.6.0-rc3. Here's a Vagrantfile to reproduce:

# -*- mode: ruby -*-
# vi: set ft=ruby :

Vagrant.configure("2") do |config|
  config.vm.box = "centos/7"
  #config.vm.box = "ubuntu/bionic64"
  #config.vm.box = "generic/ubuntu1804"
  config.vm.hostname = "k3s"

  config.vm.provider "virtualbox" do |vb|
    vb.memory = "3172"
    vb.cpus = 3
  end
  config.vm.provider "libvirt" do |lv|
    lv.memory = "3172"
    lv.cpus = 3
  end

  config.vm.provision "shell", inline: <<-SHELL
    set -eu
    if [ -f /etc/centos-release ]; then yum install -y -q policycoreutils-python; fi
    curl -sfL https://get.k3s.io -o install-k3s.sh
    INSTALL_K3S_VERSION=v0.6.0-rc3 INSTALL_K3S_EXEC=--no-flannel sh install-k3s.sh

    export PATH=/usr/local/bin:$PATH
    sleep 10

    curl -sSfLO https://docs.projectcalico.org/v3.7/manifests/calico.yaml
    # Adjust Calico pod network to match k3s default
    sed -i.bak -e 's%192.168.0.0/16%10.42.0.0/16%' calico.yaml
    kubectl apply -f calico.yaml

    while [ `kubectl get nodes -o jsonpath='{range .items[*]}{@.metadata.name}:{range @.status.conditions[*]}{@.type}={@.status};{end}{end}' | grep 'Ready=True' | wc -l` -lt 1 ];
      do echo -n .; sleep 1;
    done

    SHELL
end

$ vagrant up
...

$ vagrant ssh

$ sudo /usr/local/bin/kubectl -n kube-system get pods
NAME                                       READY   STATUS             RESTARTS   AGE
calico-kube-controllers-8646dd497f-qpmv2   1/1     Running            0          2m8s
calico-node-r4lgs                          1/1     Running            0          2m8s
coredns-695688789-7s2q7                    1/1     Running            0          2m8s
helm-install-traefik-44gmf                 0/1     Completed          0          2m8s
svclb-traefik-sl72j                        0/2     CrashLoopBackOff   6          80s
traefik-55bd9646fc-8rbxq                   1/1     Running            0          80s

$ sudo /usr/local/bin/kubectl -n kube-system logs svclb-traefik-sl72j lb-port-80
+ trap exit TERM INT
+ cat /proc/sys/net/ipv4/ip_forward
+ '[' 0 '!=' 1 ]
+ exit 1

$ cat /proc/sys/net/ipv4/ip_forward
1

$ sudo /usr/local/bin/kubectl run -i --tty busybox --image=busybox --restart=Never --rm -- sh -c 'cat /proc/sys/net/ipv4/ip_forward'
0
pod "busybox" deleted

[wilsonianb]

This appears to be due to the fact that calico disables ip forwarding by default
https://github.com/projectcalico/cni-plugin/pull/381
It looks like I was able to get the svclb-traefik pod and other LoadBalancer services working on k3s with canal by adding

          "container_settings": {
            "allow_ip_forwarding": true
          },

to the cni_network_config (under "plugins") in https://docs.projectcalico.org/v3.7/manifests/canal.yaml

https://github.com/projectcalico/cni-plugin/pull/501
https://docs.projectcalico.org/master/reference/cni-plugin/configuration (see Container Settings)

[btashton]

I'm running into this same issue, except that I am using flanel and not adding canal. My host shows ip forwarding enabled, but the containers do not.

root@qemux86-64:~# cat /proc/sys/net/ipv4/ip_forward
1
root@qemux86-64:~# k3s kubectl -n kube-system logs svclb-traefik-8t7kc -p --all-containers
+ trap exit TERM INT
+ cat /proc/sys/net/ipv4/ip_forward
+ '[' 0 '!=' 1 ]
+ exit 1
+ trap exit TERM INT
+ cat /proc/sys/net/ipv4/ip_forward
+ '[' 0 '!=' 1 ]
+ exit 1
+ trap exit TERM INT
+ cat /proc/sys/net/ipv4/ip_forward
+ '[' 0 '!=' 1 ]
+ exit 1

[alexellis]

@wilsonianb do you think this might get upstreamed at some point? I saw it was still in your patches here