Weave: Replace (default) Flannel with Weave Net (on vanilla k8s installed using tectonic-installer)

[trescst]

What you expected to happen?

I am trying to replace the default Flannel (CNI) with Weave (CNI) on a k8s cluster that was installed using the tectonic-installer (Terraform edition).

I made some changes to the Terraform code:

• opened the ports required by Weave (6783-6784 both TCP & UDP)
• replaced the default kube-flannel.yml (DaemonSet) with a kube-weave.yml alternative (some minor changes: IPALLOC_RANGE variable range, and I'm mounting /etc/kubernetes instead of /etc/)

I expected this to work, however it does not...

What happened?

Originally I assumed that my cluster worked, all nodes are healthy:

NAME                                        STATUS    AGE       VERSION
ip-10-0-24-3.us-east-2.compute.internal     Ready     25m       v1.6.7+coreos.0
ip-10-0-38-230.us-east-2.compute.internal   Ready     25m       v1.6.7+coreos.0
ip-10-0-54-94.us-east-2.compute.internal    Ready     25m       v1.6.7+coreos.0
ip-10-0-6-183.us-east-2.compute.internal    Ready     26m       v1.6.7+coreos.0
ip-10-0-75-162.us-east-2.compute.internal   Ready     25m       v1.6.7+coreos.0
ip-10-0-94-16.us-east-2.compute.internal    Ready     25m       v1.6.7+coreos.0

However I notice that some pods (API, controller & DNS) remain stuck in ContainerCreating state:

NAME                                                               READY     STATUS              RESTARTS   AGE
kube-apiserver-bd4p0                                               1/1       Running             0          26m
kube-apiserver-h9gth                                               1/1       Running             0          26m
kube-apiserver-pnj1v                                               1/1       Running             0          26m
kube-controller-manager-3857991999-0g3md                           0/1       ContainerCreating   0          26m
kube-controller-manager-3857991999-4z66p                           0/1       ContainerCreating   0          26m
kube-controller-manager-3857991999-ll9nj                           0/1       ContainerCreating   0          26m
kube-dns-1102003091-gg9p3                                          0/3       ContainerCreating   0          26m
kube-proxy-09rww                                                   1/1       Running             0          26m
kube-proxy-0qhn6                                                   1/1       Running             0          26m
kube-proxy-2k37v                                                   1/1       Running             0          26m
kube-proxy-986kb                                                   1/1       Running             0          26m
kube-proxy-gp8cm                                                   1/1       Running             0          26m
kube-proxy-p95rx                                                   1/1       Running             0          26m
kube-scheduler-151842939-84vxl                                     0/1       ContainerCreating   0          26m
kube-scheduler-151842939-q6zrw                                     0/1       ContainerCreating   0          26m
kube-scheduler-151842939-sjjp4                                     0/1       ContainerCreating   0          26m
pod-checkpointer-qw8kf                                             1/1       Running             0          26m
pod-checkpointer-qw8kf-ip-10-0-6-183.us-east-2.compute.internal    1/1       Running             0          26m
pod-checkpointer-snn20                                             1/1       Running             0          26m
pod-checkpointer-snn20-ip-10-0-38-230.us-east-2.compute.internal   1/1       Running             0          25m
pod-checkpointer-tnrwm                                             1/1       Running             0          26m
pod-checkpointer-tnrwm-ip-10-0-24-3.us-east-2.compute.internal     1/1       Running             0          25m
weave-net-47630                                                    2/2       Running             0          26m
weave-net-92j13                                                    2/2       Running             0          26m
weave-net-k2q6p                                                    2/2       Running             1          26m
weave-net-pwtkt                                                    2/2       Running             0          26m
weave-net-wfwkc                                                    2/2       Running             1          26m
weave-net-zdrw2                                                    2/2       Running             0          26m

When I describe the pods that are in the ContainerCreating state I see:

Events:
  FirstSeen LastSeen    Count   From                            SubObjectPath   Type        Reason          Message
  --------- --------    -----   ----                            -------------   --------    ------          -------
  29m       29m     5   default-scheduler                           Warning     FailedScheduling    no nodes available to schedule pods
  29m       29m     1   default-scheduler                           Normal      Scheduled       Successfully assigned kube-scheduler-151842939-84vxl to ip-10-0-6-183.us-east-2.compute.internal
  29m       29m     1   kubelet, ip-10-0-6-183.us-east-2.compute.internal           Warning     FailedSync      Error syncing pod, skipping: failed to "CreatePodSandbox" for "kube-scheduler-151842939-84vxl_kube-system(2a5a2f4a-6887-11e7-a58c-0208e7ad5c41)" with CreatePodSandboxError: "CreatePodSandbox for pod \"kube-scheduler-151842939-84vxl_kube-system(2a5a2f4a-6887-11e7-a58c-0208e7ad5c41)\" failed: rpc error: code = 2 desc = NetworkPlugin cni failed to set up pod \"kube-scheduler-151842939-84vxl_kube-system\" network: failed to find plugin \"weave-net\" in path [/opt/cni/bin /opt/weave-net/bin]"

  29m   10s 134 kubelet, ip-10-0-6-183.us-east-2.compute.internal       Normal  SandboxChanged  Pod sandbox changed, it will be killed and re-created.
  29m   10s 134 kubelet, ip-10-0-6-183.us-east-2.compute.internal       Warning FailedSync  Error syncing pod, skipping: failed to "KillPodSandbox" for "2a5a2f4a-6887-11e7-a58c-0208e7ad5c41" with KillPodSandboxError: "rpc error: code = 2 desc = NetworkPlugin cni failed to teardown pod \"kube-scheduler-151842939-84vxl_kube-system\" network: failed to find plugin \"weave-net\" in path [/opt/cni/bin /opt/weave-net/bin]"

However, the binaries seem to be all present:

ip-10-0-24-3 ~ # ls -al /opt/cni/bin/
total 9852
drwxr-xr-x. 2 root root     4096 Jul 14 11:27 .
drwxr-xr-x. 3 root root     4096 Jul 14 11:27 ..
lrwxrwxrwx. 1 root root       18 Jul 14 11:27 weave-ipam -> weave-plugin-2.0.1
lrwxrwxrwx. 1 root root       18 Jul 14 11:27 weave-net -> weave-plugin-2.0.1
-rwxr-xr-x. 1 root root 10057600 Jul 14 11:27 weave-plugin-2.0.1

Does anybody have idea what is going wrong? I already opened an issue on cores/tectonic-installer, but it was closed because Weave is currently not supported. If I get it working I will try to have it pushed into tectonic-installer...

Or does anybody have a clue where I have to start looking?

Anything else we need to know?

Running on AWS (irrelevant to my issue I assume)

Versions:

Weave: 2.0.1
Kubernetes: 1.6.7
Docker: 1.12.6

Logs:

Weave

INFO: 2017/07/14 11:30:21.903923 ->[10.0.54.94:44821] connection accepted
INFO: 2017/07/14 11:30:21.930890 ->[10.0.54.94:44821|e6:7b:21:b5:43:93(ip-10-0-54-94.us-east-2.compute.internal)]: connection ready; using protocol version 2
INFO: 2017/07/14 11:30:21.930972 overlay_switch ->[e6:7b:21:b5:43:93(ip-10-0-54-94.us-east-2.compute.internal)] using fastdp
INFO: 2017/07/14 11:30:21.931002 ->[10.0.54.94:44821|e6:7b:21:b5:43:93(ip-10-0-54-94.us-east-2.compute.internal)]: connection added (new peer)
INFO: 2017/07/14 11:30:21.939706 ->[10.0.54.94:44821|e6:7b:21:b5:43:93(ip-10-0-54-94.us-east-2.compute.internal)]: connection fully established
INFO: 2017/07/14 11:30:21.961965 EMSGSIZE on send, expecting PMTU update (IP packet was 60028 bytes, payload was 60020 bytes)
INFO: 2017/07/14 11:30:21.967475 sleeve ->[10.0.54.94:6783|e6:7b:21:b5:43:93(ip-10-0-54-94.us-east-2.compute.internal)]: Effective MTU verified at 8939
INFO: 2017/07/14 11:30:22.038867 Discovered remote MAC 96:27:aa:29:cf:51 at e6:7b:21:b5:43:93(ip-10-0-54-94.us-east-2.compute.internal)
INFO: 2017/07/14 11:30:22.571458 Discovered remote MAC e6:7b:21:b5:43:93 at e6:7b:21:b5:43:93(ip-10-0-54-94.us-east-2.compute.internal)
INFO: 2017/07/14 11:30:22.699666 Discovered remote MAC 76:56:f1:65:f5:1c at e6:7b:21:b5:43:93(ip-10-0-54-94.us-east-2.compute.internal)
INFO: 2017/07/14 11:52:22.104038 Expired MAC be:eb:da:bd:b7:21 at 32:b1:7a:04:fc:d7(ip-10-0-6-183.us-east-2.compute.internal)
INFO: 2017/07/14 11:53:21.463600 Expired MAC 92:d5:3a:6d:8e:10 at 2a:ff:96:9c:32:c0(ip-10-0-24-3.us-east-2.compute.internal)
INFO: 2017/07/14 11:56:21.464164 Expired MAC 1a:99:af:bd:22:b9 at 0a:0f:63:90:ce:d6(ip-10-0-75-162.us-east-2.compute.internal)
INFO: 2017/07/14 11:57:21.464386 Expired MAC 8a:53:2d:ad:aa:b3 at 32:2b:2e:d3:01:16(ip-10-0-38-230.us-east-2.compute.internal)
INFO: 2017/07/14 11:57:21.464415 Expired MAC 9e:32:c9:5d:dc:15 at da:31:df:78:78:5c(ip-10-0-94-16.us-east-2.compute.internal)
INFO: 2017/07/14 11:58:07.827715 Discovered local MAC 92:d5:3a:6d:8e:10
INFO: 2017/07/14 11:58:21.464603 Expired MAC 96:27:aa:29:cf:51 at e6:7b:21:b5:43:93(ip-10-0-54-94.us-east-2.compute.internal)

Network:

ip-10-0-24-3 ~ # ip route
default via 10.0.16.1 dev eth0  proto dhcp  src 10.0.24.3  metric 1024
10.0.16.0/20 dev eth0  proto kernel  scope link  src 10.0.24.3
10.0.16.1 dev eth0  proto dhcp  scope link  src 10.0.24.3  metric 1024
10.2.0.0/16 dev weave  proto kernel  scope link  src 10.2.240.0
172.17.0.0/16 dev docker0  proto kernel  scope link  src 172.17.0.1

ip-10-0-24-3 ~ # ip -4 -o addr
1: lo    inet 127.0.0.1/8 scope host lo\       valid_lft forever preferred_lft forever
2: eth0    inet 10.0.24.3/20 brd 10.0.31.255 scope global dynamic eth0\       valid_lft 3147sec preferred_lft 3147sec
3: docker0    inet 172.17.0.1/16 scope global docker0\       valid_lft forever preferred_lft forever
24: weave    inet 10.2.240.0/16 scope global weave\       valid_lft forever preferred_lft forever

[mikebryant]

See https://github.com/weaveworks/weave/issues/2613#issuecomment-277988259

In particular we have it working by adding this to the kubelet.service:

                      --volume=opt-cni,kind=host,source=/opt/cni,readOnly=true \
                      --mount volume=opt-cni,target=/opt/weave-net \

so that the rkt hyperkube image can find the weave cni plugin in the right place

[trescst]

@mikebryant thanks a bunch! Ran into another tiny issue (had to add ExecStartPre=/bin/mkdir -p /opt/cni to kubelet.service as well), but got working now!