K3s: Doc about changing CNI

[valentin2105]

Hi,

I would thank you about k3s which seem a pretty cool lightweight k8s implementation.

I seen in the doc the part about changing CNI, but on the server side, it seem it force to use flannel.
It's not possible to disable flannel via the install script.

I wondering use Calico to enable dual-stack (v4/v6) with k3s, any idea how achieve that and stop the use of flannel ?

I've seen some Range are hardcoded in k3s :

pkg/daemons/control/server.go:          _, clusterIPNet, _ := net.ParseCIDR("10.42.0.0/16")

Thanks for any road to follow.

[ibuildthecloud]

@vaibhavsood Trying disabling flannel per the docs and also the cluster cidr can be set with k3s server --cluster-cidr X.X.X.X/X. Let me know if that works, we really haven't fully tested third party CNI drivers, but we very much need them to work. So any feedback or issues are appreciated.

[valentin2105]

@ibuildthecloud Thanks for pointing that,

It seem that k3s does'nt use ETCD at all ? I requiere it for Calico.

[epicfilemcnulty]

@ibuildthecloud specifying --cluster-cidr x.x.x.x/x does not seem to work:

root@k8s-master-dev:~$ /usr/local/bin/k3s server --cluster-cidr 192.168.0.0/16
INFO[2019-03-05T21:56:55.807592266Z] Starting k3s v0.1.0 (91251aa)                
INFO[2019-03-05T21:56:55.808984449Z] Running kube-apiserver --watch-cache=false --cert-dir /var/lib/rancher/k3s/server/tls/temporary-certs --allow-privileged=true --authorization-mode Node,RBAC --service-account-signing-key-file /var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range 10.43.0.0/16 --advertise-port 6445 --advertise-address 127.0.0.1 --insecure-port 0 --secure-port 6444 --bind-address 127.0.0.1 --tls-cert-file /var/lib/rancher/k3s/server/tls/localhost.crt --tls-private-key-file /var/lib/rancher/k3s/server/tls/localhost.key --service-account-key-file /var/lib/rancher/k3s/server/tls/service.key --service-account-issuer k3s --api-audiences unknown --basic-auth-file /var/lib/rancher/k3s/server/cred/passwd --kubelet-client-certificate /var/lib/rancher/k3s/server/tls/token-node.crt --kubelet-client-key /var/lib/rancher/k3s/server/tls/token-node.key 
INFO[2019-03-05T21:56:55.972630074Z] Running kube-scheduler --kubeconfig /var/lib/rancher/k3s/server/cred/kubeconfig-system.yaml --port 0 --secure-port 0 --leader-elect=false 
INFO[2019-03-05T21:56:55.973624160Z] Running kube-controller-manager --kubeconfig /var/lib/rancher/k3s/server/cred/kubeconfig-system.yaml --service-account-private-key-file /var/lib/rancher/k3s/server/tls/service.key --allocate-node-cidrs --cluster-cidr 10.42.0.0/16 --root-ca-file /var/lib/rancher/k3s/server/tls/token-ca.crt --port 0 --secure-port 0 --leader-elect=false 
INFO[2019-03-05T21:56:56.179101139Z] Listening on :6443        

[epicfilemcnulty]

I can confirm that --cluster-cidr is working in v0.2.0-rc5.
Also, for what it's worth, if you want to use cilium for networking with k3s, you'll need to change this https://github.com/cilium/cilium/blob/master/examples/kubernetes/1.13/cilium-containerd.yaml#L334 line in official cilium manifest to /run/k3s/containerd/containerd.sock.

[ritarya]

@valentin2105 Did you managed to have calico on k3s? Is it works fine ?

[ctr49]

@ibuildthecloud I tried disabling flannel and specifying --service-cidr and --cluster-cidr but it seems custom configs in /var/lib/rancher/k3s/agent/etc/cni/net.d/ are not picked up.

I'm trying to use the CNI bridge plugin (https://github.com/containernetworking/plugins/tree/master/plugins/main/bridge)
installing k3s with

INSTALL_K3S_EXEC="server --no-flannel --service-cidr 10.1.0.0/16 --cluster-cidr=10.2.0.0/16"

and a config like

{
  "name": "kubepods",
  "type": "bridge",
  "bridge": "kubepods",
  "isDefaultGateway": true,
  "forceAddress": false,
  "ipMasq": true,
  "hairpinMode": true,
  "ipam": {
    "type": "host-local",
    "ranges": [
      {
        "subnet": "10.1.0.0/16",
        "rangeStart": "10.1.8.2",
        "rangeEnd": "10.1.11.254",
        "gateway": "10.1.8.1"
      }
    ],
    "dataDir": "/tmp/container-ipam-state",
    "resolvConf": "/etc/resolv.conf"
  }
}

as /var/lib/rancher/k3s/agent/etc/cni/net.d/11-bridge-kubepods.conf
results in

k3s[26449]: E1015 14:17:50.792650   26449 kubelet.go:2173] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized

Are there any other parameter that need to be passed to have k3s pick the CNI configs up from the default config dir?

[ctr49]

Looking at containerd.log I found that it is looking in /etc/cni/net.d/ instead of /var/lib/rancher/k3s/agent/etc/cni/net.d/ for configs (although the bundled flannel config is written into the latter). However, if I put my own config in /etc/cni/net.d it seems the binary is expected in /opt/cni/bin, while it is actually in /var/lib/rancher/k3s/data//bin
@ibuildthecloud is this expected behaviour? Is it save to create a symlink from /opt/cni to /var/lib/rancher/k3s/data//? If yes is the id static or where I look it up for automated generation of the symlink?

[identitymonk]

Hi,

I got the same error as @valentin2105 and @ctr49 when trying to migrate my K3s cluster from flannel to Calico.

In fact it does not seem that error is in calico-kube-controllers. If it complains in the log on not finding calico bin in /opt/cni/bin it is because calico-node did not finish its task properly. In fact this pod use a container for IPAM migration and running the logs here is what I found:

kubectl logs <name of pod type calico-node> -c upgrade-ipam -n kube-system

2019-11-04 23:14:37.760 [INFO][1] migrate.go 72: retrieving node for IPIP tunnel address
2019-11-04 23:14:37.764 [INFO][1] migrate.go 148: checking if host-local IPAM data dir dir is empty...
2019-11-04 23:14:37.764 [INFO][1] migrate.go 171: host-local IPAM data dir is not empty, migrating...
2019-11-04 23:14:37.764 [INFO][1] migrate.go 179: setting datastore readiness to false
2019-11-04 23:14:37.780 [INFO][1] migrate.go 200: successfully set datastore readiness to false
2019-11-04 23:14:37.780 [INFO][1] migrate.go 203: removing cni binaries...
2019-11-04 23:14:37.780 [INFO][1] migrate.go 208: successfully removed cni binary! binary="/host/opt/cni/bin/calico"
2019-11-04 23:14:37.780 [INFO][1] migrate.go 208: successfully removed cni binary! binary="/host/opt/cni/bin/host-local"
2019-11-04 23:14:42.781 [INFO][1] migrate.go 215: acquiring lock on host-local IPAM
2019-11-04 23:14:42.781 [INFO][1] migrate.go 223: successfully acquired lock on host-local IPAM
2019-11-04 23:14:42.781 [INFO][1] migrate.go 236: mapping pod ips to pods...
2019-11-04 23:14:42.791 [INFO][1] migrate.go 246: mapping in pod IP="10.42.0.3" namespace="kube-system" pod="helm-install-traefik-4vd5z"
2019-11-04 23:14:42.791 [INFO][1] migrate.go 246: mapping in pod IP="10.42.0.4" namespace="kube-system" pod="coredns-66f496764-dn9kn"
2019-11-04 23:14:42.791 [INFO][1] migrate.go 246: mapping in pod IP="10.42.0.5" namespace="kube-system" pod="traefik-d869575c8-zscz2"
2019-11-04 23:14:42.791 [INFO][1] migrate.go 246: mapping in pod IP="" namespace="kube-system" pod="calico-kube-controllers-6d85fdfbd8-gvqhb"
2019-11-04 23:14:42.791 [INFO][1] migrate.go 246: mapping in pod IP="10.8.2.111" namespace="kube-system" pod="calico-node-mn29n"
2019-11-04 23:14:42.791 [INFO][1] migrate.go 246: mapping in pod IP="10.8.2.111" namespace="kube-system" pod="canal-9m2x2"
2019-11-04 23:14:42.791 [INFO][1] migrate.go 246: mapping in pod IP="" namespace="kube-system" pod="sh-5f4cc76d6c-pwm4q"
2019-11-04 23:14:42.792 [INFO][1] migrate.go 246: mapping in pod IP="" namespace="kube-system" pod="test-685ddd4774-gq9tz"
2019-11-04 23:14:42.792 [INFO][1] migrate.go 246: mapping in pod IP="" namespace="kube-system" pod="local-path-provisioner-58fb86bdfd-76btn"
2019-11-04 23:14:42.792 [INFO][1] migrate.go 246: mapping in pod IP="10.42.0.3" namespace="kube-system" pod="svclb-traefik-smrmc"
2019-11-04 23:14:42.792 [INFO][1] migrate.go 249: successfully mapped pod ips to pods!
2019-11-04 23:14:42.792 [INFO][1] migrate.go 252: reading files from host-local IPAM data dir...
2019-11-04 23:14:42.792 [INFO][1] migrate.go 261: processing file... file="10.42.0.3"
2019-11-04 23:14:42.792 [INFO][1] migrate.go 305: assinging pod IP to Calico IPAM... file="10.42.0.3"
2019-11-04 23:14:42.792 [INFO][1] ipam.go 583: Assigning IP 10.42.0.3 to host: localhost.localdomain
2019-11-04 23:14:42.795 [INFO][1] migrate.go 226: releasing lock on host-local backend...
2019-11-04 23:14:42.795 [INFO][1] migrate.go 230: successfully released lock on host-local backend!
2019-11-04 23:14:42.795 [ERROR][1] ipam_plugin.go 95: failed to migrate ipam, retrying... error=failed to assign IP to calico backend: The provided IP address is not in a configured pool
node="localhost.localdomain"
2019-11-04 23:14:43.796 [INFO][1] migrate.go 65: checking host-local IPAM data dir dir existence...
2019-11-04 23:14:43.796 [INFO][1] migrate.go 72: retrieving node for IPIP tunnel address
2019-11-04 23:14:43.800 [INFO][1] migrate.go 148: checking if host-local IPAM data dir dir is empty...
2019-11-04 23:14:43.800 [INFO][1] migrate.go 171: host-local IPAM data dir is not empty, migrating...
2019-11-04 23:14:43.800 [INFO][1] migrate.go 179: setting datastore readiness to false
2019-11-04 23:14:43.817 [INFO][1] migrate.go 200: successfully set datastore readiness to false
2019-11-04 23:14:43.817 [INFO][1] migrate.go 203: removing cni binaries...
2019-11-04 23:14:43.817 [INFO][1] migrate.go 208: successfully removed cni binary! binary="/host/opt/cni/bin/calico"
2019-11-04 23:14:43.817 [INFO][1] migrate.go 208: successfully removed cni binary! binary="/host/opt/cni/bin/host-local"
2019-11-04 23:14:48.818 [INFO][1] migrate.go 215: acquiring lock on host-local IPAM

So it seems that it does not succeed in migrating the IP of each pod deployed over flannel to the calico CNI. Delete each pod one by one unlock the situation. But then it is blocking to traefik LB.

At this point, I decided to drop off migrating my cluster and to start from a fresh cluster. Here is the procedure for installing calico v0.3.1 over K3s v0.10.2:

install.k3s.sh --write-kubeconfig-mode 644 --docker --no-flannel
curl https://docs.projectcalico.org/v3.10/manifests/calico.yaml -O
POD_CIDR=$(kubectl get nodes -o jsonpath='{.items[*].spec.podCIDR}') sed -i -e "s?192.168.0.0/16?$POD_CIDR?g" calico.yaml
export DATASTORE_TYPE=kubernetes
export KUBECONFIG=/etc/rancher/k3s/k3s.yaml

Now before you apply calico, you must activate ip forwarding which is disabled in calico by default. This can be done by adding the container_settings parameter thanks to this doc. You network config should look like:

  cni_network_config: |-
    {
      "name": "k8s-pod-network",
      "cniVersion": "0.3.1",
      "plugins": [
        {
          "type": "calico",
          "log_level": "info",
          "datastore_type": "kubernetes",
          "nodename": "__KUBERNETES_NODE_NAME__",
          "mtu": __CNI_MTU__,
          "ipam": {
              "type": "calico-ipam"
          },
          "container_settings": {
              "allow_ip_forwarding": true
          },
          "policy": {
              "type": "k8s"
          },
          "kubernetes": {
              "kubeconfig": "__KUBECONFIG_FILEPATH__"
          }
        },
        {
          "type": "portmap",
          "snat": true,
          "capabilities": {"portMappings": true}
        }
      ]
    }

Now you can apply it with kubectl apply -f calico.yaml and everything should be running fine.

If IP forwarding is not set properly, you should see error in kubectl logs -n kube-system <svclb-traefik pod name> -c https:

/proc/sys/net/ipv4/ip_forward is read only
+ trap exit TERM INT
+ cat /proc/sys/net/ipv4/ip_forward
+ '[' 0 '!=' 1 ]
+ exit 1

Jeff
PS: this put me on the path of resolution

[marshallford]

While trying to install Calico 3.12 with k3s v1.17.2+k3s1 I've run into a couple of problems and I'm not sure what to troubleshoot next. As per the docs I've added the allow_ip_forwarding snippet to the CNI config and changed the pod CIDR to 10.42.0.0/16 to match the k3s defaults. After applying the calico yaml I had to run kubectl taint nodes --all node.kubernetes.io/not-ready:NoSchedule- in order to get the calico-kube-controllers-XXX scheduled (the single node was in a not ready state because of the lack of a network). After removing the taint the pod becomes stuck in ContainerCreating with the error message: network is not ready: runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized. My current theory is that this is all caused by the fact that this cluster is on a single node.

EDIT: My issue was with the default psp bindings that I had created.

[cjellick]

As far as I can tell, there is no bug here, but configuring calico on k3s requires some pretty specific configuration that we should consider testing and documenting. I'd like to do this in order to have some known reference implementations on swapping out CNI.

[jawabuu]

Hi,

I got the same error as @valentin2105 and @ctr49 when trying to migrate my K3s cluster from flannel to Calico.

In fact it does not seem that error is in calico-kube-controllers. If it complains in the log on not finding calico bin in /opt/cni/bin it is because calico-node did not finish its task properly. In fact this pod use a container for IPAM migration and running the logs here is what I found:

kubectl logs <name of pod type calico-node> -c upgrade-ipam -n kube-system

2019-11-04 23:14:37.760 [INFO][1] migrate.go 72: retrieving node for IPIP tunnel address
2019-11-04 23:14:37.764 [INFO][1] migrate.go 148: checking if host-local IPAM data dir dir is empty...
2019-11-04 23:14:37.764 [INFO][1] migrate.go 171: host-local IPAM data dir is not empty, migrating...
2019-11-04 23:14:37.764 [INFO][1] migrate.go 179: setting datastore readiness to false
2019-11-04 23:14:37.780 [INFO][1] migrate.go 200: successfully set datastore readiness to false
2019-11-04 23:14:37.780 [INFO][1] migrate.go 203: removing cni binaries...
2019-11-04 23:14:37.780 [INFO][1] migrate.go 208: successfully removed cni binary! binary="/host/opt/cni/bin/calico"
2019-11-04 23:14:37.780 [INFO][1] migrate.go 208: successfully removed cni binary! binary="/host/opt/cni/bin/host-local"
2019-11-04 23:14:42.781 [INFO][1] migrate.go 215: acquiring lock on host-local IPAM
2019-11-04 23:14:42.781 [INFO][1] migrate.go 223: successfully acquired lock on host-local IPAM
2019-11-04 23:14:42.781 [INFO][1] migrate.go 236: mapping pod ips to pods...
2019-11-04 23:14:42.791 [INFO][1] migrate.go 246: mapping in pod IP="10.42.0.3" namespace="kube-system" pod="helm-install-traefik-4vd5z"
2019-11-04 23:14:42.791 [INFO][1] migrate.go 246: mapping in pod IP="10.42.0.4" namespace="kube-system" pod="coredns-66f496764-dn9kn"
2019-11-04 23:14:42.791 [INFO][1] migrate.go 246: mapping in pod IP="10.42.0.5" namespace="kube-system" pod="traefik-d869575c8-zscz2"
2019-11-04 23:14:42.791 [INFO][1] migrate.go 246: mapping in pod IP="" namespace="kube-system" pod="calico-kube-controllers-6d85fdfbd8-gvqhb"
2019-11-04 23:14:42.791 [INFO][1] migrate.go 246: mapping in pod IP="10.8.2.111" namespace="kube-system" pod="calico-node-mn29n"
2019-11-04 23:14:42.791 [INFO][1] migrate.go 246: mapping in pod IP="10.8.2.111" namespace="kube-system" pod="canal-9m2x2"
2019-11-04 23:14:42.791 [INFO][1] migrate.go 246: mapping in pod IP="" namespace="kube-system" pod="sh-5f4cc76d6c-pwm4q"
2019-11-04 23:14:42.792 [INFO][1] migrate.go 246: mapping in pod IP="" namespace="kube-system" pod="test-685ddd4774-gq9tz"
2019-11-04 23:14:42.792 [INFO][1] migrate.go 246: mapping in pod IP="" namespace="kube-system" pod="local-path-provisioner-58fb86bdfd-76btn"
2019-11-04 23:14:42.792 [INFO][1] migrate.go 246: mapping in pod IP="10.42.0.3" namespace="kube-system" pod="svclb-traefik-smrmc"
2019-11-04 23:14:42.792 [INFO][1] migrate.go 249: successfully mapped pod ips to pods!
2019-11-04 23:14:42.792 [INFO][1] migrate.go 252: reading files from host-local IPAM data dir...
2019-11-04 23:14:42.792 [INFO][1] migrate.go 261: processing file... file="10.42.0.3"
2019-11-04 23:14:42.792 [INFO][1] migrate.go 305: assinging pod IP to Calico IPAM... file="10.42.0.3"
2019-11-04 23:14:42.792 [INFO][1] ipam.go 583: Assigning IP 10.42.0.3 to host: localhost.localdomain
2019-11-04 23:14:42.795 [INFO][1] migrate.go 226: releasing lock on host-local backend...
2019-11-04 23:14:42.795 [INFO][1] migrate.go 230: successfully released lock on host-local backend!
2019-11-04 23:14:42.795 [ERROR][1] ipam_plugin.go 95: failed to migrate ipam, retrying... error=failed to assign IP to calico backend: The provided IP address is not in a configured pool
node="localhost.localdomain"
2019-11-04 23:14:43.796 [INFO][1] migrate.go 65: checking host-local IPAM data dir dir existence...
2019-11-04 23:14:43.796 [INFO][1] migrate.go 72: retrieving node for IPIP tunnel address
2019-11-04 23:14:43.800 [INFO][1] migrate.go 148: checking if host-local IPAM data dir dir is empty...
2019-11-04 23:14:43.800 [INFO][1] migrate.go 171: host-local IPAM data dir is not empty, migrating...
2019-11-04 23:14:43.800 [INFO][1] migrate.go 179: setting datastore readiness to false
2019-11-04 23:14:43.817 [INFO][1] migrate.go 200: successfully set datastore readiness to false
2019-11-04 23:14:43.817 [INFO][1] migrate.go 203: removing cni binaries...
2019-11-04 23:14:43.817 [INFO][1] migrate.go 208: successfully removed cni binary! binary="/host/opt/cni/bin/calico"
2019-11-04 23:14:43.817 [INFO][1] migrate.go 208: successfully removed cni binary! binary="/host/opt/cni/bin/host-local"
2019-11-04 23:14:48.818 [INFO][1] migrate.go 215: acquiring lock on host-local IPAM

So it seems that it does not succeed in migrating the IP of each pod deployed over flannel to the calico CNI. Delete each pod one by one unlock the situation. But then it is blocking to traefik LB.

At this point, I decided to drop off migrating my cluster and to start from a fresh cluster. Here is the procedure for installing calico v0.3.1 over K3s v0.10.2:

install.k3s.sh --write-kubeconfig-mode 644 --docker --no-flannel
curl https://docs.projectcalico.org/v3.10/manifests/calico.yaml -O
POD_CIDR=$(kubectl get nodes -o jsonpath='{.items[*].spec.podCIDR}') sed -i -e "s?192.168.0.0/16?$POD_CIDR?g" calico.yaml
export DATASTORE_TYPE=kubernetes
export KUBECONFIG=/etc/rancher/k3s/k3s.yaml

Now before you apply calico, you must activate ip forwarding which is disabled in calico by default. This can be done by adding the container_settings parameter thanks to this doc. You network config should look like:

  cni_network_config: |-
    {
      "name": "k8s-pod-network",
      "cniVersion": "0.3.1",
      "plugins": [
        {
          "type": "calico",
          "log_level": "info",
          "datastore_type": "kubernetes",
          "nodename": "__KUBERNETES_NODE_NAME__",
          "mtu": __CNI_MTU__,
          "ipam": {
              "type": "calico-ipam"
          },
          "container_settings": {
              "allow_ip_forwarding": true
          },
          "policy": {
              "type": "k8s"
          },
          "kubernetes": {
              "kubeconfig": "__KUBECONFIG_FILEPATH__"
          }
        },
        {
          "type": "portmap",
          "snat": true,
          "capabilities": {"portMappings": true}
        }
      ]
    }

Now you can apply it with kubectl apply -f calico.yaml and everything should be running fine.

If IP forwarding is not set properly, you should see error in kubectl logs -n kube-system <svclb-traefik pod name> -c https:

/proc/sys/net/ipv4/ip_forward is read only
+ trap exit TERM INT
+ cat /proc/sys/net/ipv4/ip_forward
+ '[' 0 '!=' 1 ]
+ exit 1

Jeff
PS: this put me on the path of resolution

Hey @identitymonk .
What steps did you follow when trying to migrate from flannel to calico?
I have found this resource
https://docs.projectcalico.org/getting-started/kubernetes/flannel/migration-from-flannel
but the job does not run. Logs from the container are
Daemonset kube-flannel-ds-amd64 not exists, no migration process is needed

[jawabuu]

@ibuildthecloud How would I uninstall or disable embedded flannel CNI especially for a case where I do not have ssh access to the node. I can only use kubectl.

[brandond]

@jawabuu you need to pass arguments to the k3s server when it is started, which would require access to the host to edit the systemd unit or rc script.

[jawabuu]

Hey @brandond I meant a k3s cluster that is already running with flannel as the CNI. How would I change the CNI for such a cluster?
This works for 'stock' k8s
https://docs.projectcalico.org/getting-started/kubernetes/flannel/migration-from-flannel

[brandond]

@jawabuu you would need to add --flannel-backend=none to the k3s server arguments, restart all the servers, then install an alternative CNI.

[jawabuu]

Thanks @brandond , this instruction is for an already running k3s cluster?

[brandond]

Yes, although I haven't tested it myself. I would also expect things to be pretty broken after you do that, until you get the new CNI up and running.

[jawabuu]

This is fine, of the top of your head would there be another way to disable the default flannel?

[brandond]

No?

[jawabuu]

Alright, I'll test this out and see if it works

[jawabuu]

@vaibhavsood Trying disabling flannel per the docs and also the cluster cidr can be set with k3s server --cluster-cidr X.X.X.X/X. Let me know if that works, we really haven't fully tested third party CNI drivers, but we very much need them to work. So any feedback or issues are appreciated.

Hey @ibuildthecloud I have published a project https://github.com/jawabuu/kloud-3s that uses k3s and have successfully tested with flannel, calico, cilium and weave while managing to preserve source ip whenever needed.
Could this be of help in documenting k3s against 3rd party CNI's?

[anasalloush]

@vaibhavsood Trying disabling flannel per the docs and also the cluster cidr can be set with k3s server --cluster-cidr X.X.X.X/X. Let me know if that works, we really haven't fully tested third party CNI drivers, but we very much need them to work. So any feedback or issues are appreciated.

Hey @ibuildthecloud I have published a project https://github.com/jawabuu/kloud-3s that uses k3s and have successfully tested with flannel, calico, cilium and weave while managing to preserve source ip whenever needed.
Could this be of help in documenting k3s against 3rd party CNI's?

Hi @jawabuu
I was trying to set K3s with Calico, which requires Etcd as far as I know.
I came across your project while looking for help.
Did you manage to deploy K3s + Calico without Etcd?
My goal is to install Istio on K3s, but injecting the side car of istio requires a CNI other than flannel, thats why am looking for Calico/K3s installation, but so far didnt find any successful ones.

[jawabuu]

⁣Hey, Yes it does deploy calico just fine without etcd.

On 10 Jun 2020, 21:29, at 21:29, anasalloush [e-mail] wrote:

@vaibhavsood Trying disabling flannel per the docs and also the
cluster cidr can be set with k3s server --cluster-cidr X.X.X.X/X. Let
me know if that works, we really haven't fully tested third party CNI
drivers, but we very much need them to work. So any feedback or issues
are appreciated.

Hey @ibuildthecloud I have published a project
https://github.com/jawabuu/kloud-3s that uses k3s and have successfully
tested with flannel, calico, cilium and weave while managing to
preserve source ip whenever needed.
Could this be of help in documenting k3s against 3rd party CNI's?

Hi @jawabuu
I was trying to set K3s with Calico, which requires Etcd as far as I
know.
I came across your project while looking for help.
Did you manage to deploy K3s + Calico without Etcd?
My goal is to install Istio on K3s, but injecting the side care of
istio requires a CNI other than flannel, thats why am looking for
Calico/K3s installation, but so far didnt find any successful ones.

--
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
https://github.com/rancher/k3s/issues/54#issuecomment-642183780

[anasalloush]

⁣Hey, Yes it does deploy calico just fine without etcd.
…
On 10 Jun 2020, 21:29, at 21:29, anasalloush @.*> wrote: > > @vaibhavsood Trying disabling flannel per the docs and also the cluster cidr can be set with k3s server --cluster-cidr X.X.X.X/X. Let me know if that works, we really haven't fully tested third party CNI drivers, but we very much need them to work. So any feedback or issues are appreciated. > > Hey @ibuildthecloud I have published a project https://github.com/jawabuu/kloud-3s that uses k3s and have successfully tested with flannel, calico, cilium and weave while managing to preserve source ip whenever needed. > Could this be of help in documenting k3s against 3rd party CNI's? Hi @jawabuu I was trying to set K3s with Calico, which requires Etcd as far as I know. I came across your project while looking for help. Did you manage to deploy K3s + Calico without Etcd? My goal is to install Istio on K3s, but injecting the side care of istio requires a CNI other than flannel, thats why am looking for Calico/K3s installation, but so far didnt find any successful ones. -- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: #54 (comment)

thanks ..
Is there a deploy option for on-prem? or it is just for cloud environment?

[jawabuu]

Currently only the stated cloud providers are supported but the principle is the same and should work for on-prem.
The options to get k3s working with calico are,

            - name: IP_AUTODETECTION_METHOD
              value: "interface=${interface}"
            # Enable IPIP
            - name: CALICO_IPV4POOL_IPIP
              value: "Never"
            # Enable IPIP
            - name: CALICO_IPV4POOL_VXLAN
              value: "Always" 
           - name: CALICO_IPV4POOL_CIDR
              value: "${calico_cidr}"

These are envs for calico.yaml
interface is the interface connecting your nodes.
calico_cidr is the k3s cluster cidr

[anasalloush]

Currently only the stated cloud providers are supported but the principle is the same and should work for on-prem.
The options to get k3s working with calico are,

            - name: IP_AUTODETECTION_METHOD
              value: "interface=${interface}"
            # Enable IPIP
            - name: CALICO_IPV4POOL_IPIP
              value: "Never"
            # Enable IPIP
            - name: CALICO_IPV4POOL_VXLAN
              value: "Always" 
           - name: CALICO_IPV4POOL_CIDR
              value: "${calico_cidr}"

These are envs for calico.yaml
interface is the interface connecting your nodes.
calico_cidr is the k3s cluster cidr

Thanks, I will check it out..