Calico: Add non-cluster host to Kubernets Calico network

[AlejoAsd]

Context

I am trying to establish a connection between a docker container running on a non-cluster host and a pod inside a Kubernetes cluster running Calico. I have followed the instructions in About non-cluster hosts and Docker container install to add the non-cluster host to the Calico network, but I ran into problems when launching the calico/node container.

My setup consists of an unmanaged single-host Kubernetes cluster running in an AWS EC2 instance, and an Elastic Beanstalk (EB) docker application. Calico is running using Kubernetes as its store and has IP-in-IP encapsulation enabled for cross subnet traffic. The EB application creates a single EC2 instance that needs to communicate with several pods in the cluster. The application and the pods in the cluster are in the same AWS VPC, but may be in different subnets.

Expected Behavior

Running a calico/node docker container in a non-cluster host should add the host to the Calico network.

Current Behavior

I ran the following command in the EB application instance to launch the calico/node container and add the host to the Calico network:

 sudo docker run --rm --net=host --privileged \
  --name="calico-node" \
  -e DATASTORE_TYPE="kubernetes" \
  -e KUBECONFIG="/tmp/kube_config" \
  -e CALICO_NODENAME="test" \
  -e CALICO_NETWORKING_BACKEND="bird" \
  -v /var/log/calico:/var/log/calico \
  -v /run/docker/plugins:/run/docker/plugins \
  -v /lib/modules:/lib/modules \
  -v /var/run/calico:/var/run/calico \
  -v /etc/pki:/pki \
  -v /home/ec2-user/.kube/config:/tmp/kube_config \
  calico/node:v3.13.3

Which returned the following error message:

2020-05-13 22:47:33.819 [INFO][8] startup.go 309: Early log level set to info
2020-05-13 22:47:33.823 [INFO][8] startup.go 329: Using HOSTNAME environment (lowercase) for node name
2020-05-13 22:47:33.823 [INFO][8] startup.go 337: Determined node name: ip-172-30-39-148
2020-05-13 22:47:33.831 [INFO][8] startup.go 106: Skipping datastore connection test
2020-05-13 22:47:33.866 [INFO][8] startup.go 420: Building new node resource Name="ip-172-30-39-148"
2020-05-13 22:47:33.867 [INFO][8] startup.go 435: Initialize BGP data
2020-05-13 22:47:33.867 [INFO][8] startup.go 671: Using autodetected IPv4 address on interface eth0: 172.30.39.148/20
2020-05-13 22:47:33.868 [INFO][8] startup.go 505: Node IPv4 changed, will check for conflicts
2020-05-13 22:47:33.871 [INFO][8] startup.go 734: No AS number configured on node resource, using global value
2020-05-13 22:47:33.871 [WARNING][8] startup_linux.go 47: Expected /var/lib/calico to be mounted into the container but it wasn't present. Node name may not be detected properly
2020-05-13 22:47:33.875 [WARNING][8] node.go 64: Operation Create is not supported on Node type
2020-05-13 22:47:33.879 [ERROR][8] startup.go 217: Unable to set node resource configuration error=operation Create is not supported on Node(ip-A-B-C-D)
2020-05-13 22:47:33.879 [WARNING][8] startup.go 1214: Terminating
Calico node failed to start

A quick search led me to issues #3397 and #3407, which if I understood correctly indicate that my use case is not supported. Could you please confirm this? If that is the case, I am not sure I understand what use cases the calico/node docker installation is meant for.

I created this issue to confirm that I haven't missed anything and that this is the right way to go about things.

If this is an issue of my specific setup, please let me know so that I can understand how to change it.

Thank you for your time!

Possible Solution

Not a solution, but if this use case is indeed not supported, the documentation should be updated to reflect this.

Steps to Reproduce (for bugs)

1. Create single-host Kubernetes cluster with Calico in an AWS EC2 instance with kubeadm.
2. Launch an additional AWS EC2 instance and try to run a calico/node container.

Your Environment

• Calico version 3.13.3
• Orchestrator version (e.g. kubernetes, mesos, rkt): Kubernetes 1.14.10
• Operating System and version: Amazon Linux AMI (Based on CentOS 6)
• Link to your project (optional):

[AlejoAsd]

Would changing the Calico store from Kubernetes to etcd allow adding the node to the network?

[AlejoAsd]

Ended up using Calico for policy only. Leaving this issue open to discuss documentation updates.

[caseydavenport]

@AlejoAsd sorry for delay on response here - agree we should make this work.

Using Calico in etcd mode should make this scenario work. In fact, this is the primary use-case for using etcd directly instead of Kubernetes mode.

However, I think we should enhance Calico to make this work without the need for etcd mode. The main issue is that in k8s mode we assume all nodes are k8s nodes, and use the k8s node API to get node information.

We'll need to either tweak the code to support running when a k8s node doesn't exist, or you could create dummy nodes in the k8s API to represent these external nodes.

[AlejoAsd]

After opening this issue and reading the documentation more thoroughly I found out that etcd indeed had support for creating nodes automatically, while k8s didn't.

The main issue is that in k8s mode we assume all nodes are k8s nodes, and use the k8s node API to get node information.

While it would certainly be nice to have k8s support, I think this is fine for now as you at least have a working alternative.

To give this some closure, we should update the documentation to better reflect that k8s does not have support for nodes out of the cluster just yet. I can do that.

I plan on editing the following pages:

• About non-cluster hosts
• Docker container install

Let me know if there's anywhere else this should be mentioned.

[pshizhsysu]

1、create a virtual node in k8s

$ kubectl create -f - <<EOF
> apiVersion: v1
> kind: Node
> metadata:
>   name: peng03
> EOF 

$ kubectl get node
NAME     STATUS     ROLES    AGE   VERSION
peng01   Ready      master   61d   v1.17.3
peng02   Ready      <none>   26d   v1.17.3
peng03   NotReady   <none>   28s 

2、then install calico on non-cluster-host（docker mode）, it works

$ systemctl start calico

$ docker ps
CONTAINER ID        IMAGE                 COMMAND             CREATED             STATUS              PORTS               NAMES
75d0f627c632        calico/node:v3.16.1   "start_runit"       2 minutes ago      Up 2 minutes                           calico-node

$ route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.2.2     0.0.0.0         UG    100    0        0 ens33
172.17.0.0      0.0.0.0         255.255.0.0     U     0      0        0 docker0
172.26.14.192   192.168.2.102   255.255.255.192 UG    0      0        0 tunl0
172.26.71.192   192.168.2.101   255.255.255.192 UG    0      0        0 tunl0
172.26.176.0    0.0.0.0         255.255.255.192 U     0      0        0 *
192.168.2.0     0.0.0.0         255.255.255.0   U     100    0        0 ens33