Weave: Nodes are attempting to claim same IP range

Created on 6 Jun 2018  ·  30Comments  ·  Source: weaveworks/weave

What you expected to happen?

Nodes should claim their own distinct IP ranges.

What happened?

It seems that two nodes have (or are attempting to) claim the same IP address range.

The node ip-10-83-42-111.ec2.internal claimed the 100.105.68.0 IP range:

...
{
    "Token": "100.105.68.0",
    "Size": 1024,
    "Peer": "12:84:ba:e1:47:79",
    "Nickname": "ip-10-83-42-111.ec2.internal",
    "IsKnownPeer": true,
    "Version": 1451
}
...

However, the node ip-10-83-54-199.ec2.internal is _also_ attempting to claim the 100.105.68.0 IP range:

$ ./weave --local status connections
-> 10.83.80.134:6783     failed      Received update for IP range I own at 100.105.68.0 v592: incoming message says owner 12:84:ba:e1:47:79 v1460, retry: 2018-06-05 11:15:31.000788717 +0000 UTC m=+1034.303622109
-> 10.83.98.46:6783      failed      Received update for IP range I own at 100.105.68.0 v592: incoming message says owner 12:84:ba:e1:47:79 v1460, retry: 2018-06-05 11:11:04.422577727 +0000 UTC m=+767.725411089
-> 10.83.126.221:6783    failed      Received update for IP range I own at 100.105.68.0 v592: incoming message says owner 12:84:ba:e1:47:79 v1460, retry: 2018-06-05 11:11:33.742999475 +0000 UTC m=+797.045832929
-> 10.83.85.183:6783     failed      Received update for IP range I own at 100.105.68.0 v592: incoming message says owner 12:84:ba:e1:47:79 v1460, retry: 2018-06-05 11:08:47.877010584 +0000 UTC m=+631.179843953
-> 10.83.60.42:6783      failed      Received update for IP range I own at 100.105.68.0 v592: incoming message says owner 12:84:ba:e1:47:79 v1460, retry: 2018-06-05 11:09:09.444156631 +0000 UTC m=+652.746990055
-> 10.83.78.215:6783     failed      Received update for IP range I own at 100.105.68.0 v592: incoming message says owner 12:84:ba:e1:47:79 v1460, retry: 2018-06-05 11:11:40.747663238 +0000 UTC m=+804.050496608
-> 10.83.124.188:6783    failed      Received update for IP range I own at 100.105.68.0 v592: incoming message says owner 12:84:ba:e1:47:79 v1460, retry: 2018-06-05 11:10:38.016883177 +0000 UTC m=+741.319716603
-> 10.83.42.111:6783     failed      Received update for IP range I own at 100.105.68.0 v592: incoming message says owner 12:84:ba:e1:47:79 v1460, retry: 2018-06-05 11:14:14.702258004 +0000 UTC m=+958.005091377
-> 10.83.109.188:6783    failed      Received update for IP range I own at 100.105.68.0 v592: incoming message says owner 12:84:ba:e1:47:79 v1460, retry: 2018-06-05 11:11:40.776242153 +0000 UTC m=+804.079075513
-> 10.83.36.17:6783      failed      Received update for IP range I own at 100.105.68.0 v592: incoming message says owner 12:84:ba:e1:47:79 v1460, retry: 2018-06-05 11:10:19.676858864 +0000 UTC m=+722.979692293
-> 10.83.59.237:6783     failed      Received update for IP range I own at 100.105.68.0 v592: incoming message says owner 12:84:ba:e1:47:79 v1460, retry: 2018-06-05 11:08:27.22810513 +0000 UTC m=+610.530938541
-> 10.83.41.74:6783      failed      Received update for IP range I own at 100.105.68.0 v592: incoming message says owner 12:84:ba:e1:47:79 v1460, retry: 2018-06-05 11:11:44.830033868 +0000 UTC m=+808.132867307
-> 10.83.54.199:6783     failed      cannot connect to ourself, retry: never
-> 10.83.87.92:6783      failed      Received update for IP range I own at 100.105.68.0 v592: incoming message says owner 12:84:ba:e1:47:79 v1460, retry: 2018-06-05 11:10:32.774426814 +0000 UTC m=+736.077260261
-> 10.83.76.130:6783     failed      Received update for IP range I own at 100.105.68.0 v592: incoming message says owner 12:84:ba:e1:47:79 v1460, retry: 2018-06-05 11:10:48.434728192 +0000 UTC m=+751.737561589

Restarting the weave pods on either ip-10-83-42-111.ec2.internal or ip-10-83-54-199.ec2.internal does not change the situation, likely due to the the fact that state is persisted on disk at /var/lib/weave.

During the same time period, there was another issue with weave I noticed. I'm not sure if it is related but some nodes appeared to be attempting to remove a non-existent peer:

DEBU: 2018/06/05 11:20:29.571705 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:20:29.571731 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:20:29.571739 [kube-peers] Existing annotation 46:f4:4b:41:dd:11

According to our logs, this error started on 2018/05/16 (~21 days ago). Around this time we were having issues with weave consuming a large amount of memory.

How to reproduce it?

Not sure if it is easily reproducible, but it occurred while restarting all weave pods in the cluster, with a 15 second sleep between each restart.

There is a possibility that two pods started at the same time due to scheduling delays.

Anything else we need to know?

Cloud provider: AWS, managed by Kops

Versions:

$ ./weave --local version
weave 2.3.0

$ docker version
Client:
 Version:      17.03.2-ce
 API version:  1.27
 Go version:   go1.7.5
 Git commit:   f5ec1e2
 Built:        Tue Jun 27 02:09:56 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.03.2-ce
 API version:  1.27 (minimum version 1.12)
 Go version:   go1.7.5
 Git commit:   f5ec1e2
 Built:        Tue Jun 27 02:09:56 2017
 OS/Arch:      linux/amd64
 Experimental: false

$ uname -a
Linux ip-10-83-54-199 4.4.115-k8s #1 SMP Thu Feb 8 15:37:40 UTC 2018 x86_64 GNU/Linux

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.13", GitCommit:"290fb182489a396dce5d136451388f9b12f29c94", GitTreeState:"clean", BuildDate:"2018-05-15T18:16:09Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.13", GitCommit:"290fb182489a396dce5d136451388f9b12f29c94", GitTreeState:"clean", BuildDate:"2018-05-15T18:08:47Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

$ kops version
Version 1.9.1

Logs:

Click the black arrows to reveal the details!

From the node with the issue (ip-10-83-54-199.ec2.internal):

Weave IPAM

$ ./weave --local status ipam
6a:5c:7c:af:e3:2e(ip-10-83-54-199.ec2.internal)   297040 IPs (14.2% of total) (4 active)
46:f4:4b:41:dd:11(ip-10-83-124-112.ec2.internal)    65536 IPs (03.1% of total) - unreachable!
fe:6a:75:3b:98:1f(ip-10-83-123-141.ec2.internal)     8192 IPs (00.4% of total) - unreachable!
7a:54:95:96:77:8e(ip-10-83-98-15.ec2.internal)    65536 IPs (03.1% of total) - unreachable!
c2:35:ec:05:d8:a5()                      65536 IPs (03.1% of total) - unreachable!
1a:cd:c1:53:24:bf(ip-10-83-36-17.ec2.internal)    16384 IPs (00.8% of total) - unreachable!
1e:dd:1b:3a:d3:11(ip-10-83-109-170.ec2.internal)    65536 IPs (03.1% of total) - unreachable!
3e:06:82:a1:9c:ec(ip-10-83-103-30.ec2.internal)    66048 IPs (03.1% of total) - unreachable!
1e:1c:57:4a:9e:66(ip-10-83-52-111.ec2.internal)     8192 IPs (00.4% of total) - unreachable!
fe:9a:a2:ec:04:20(ip-10-83-117-100.ec2.internal)    65536 IPs (03.1% of total) - unreachable!
2e:60:25:ef:21:f3(ip-10-83-125-194.ec2.internal)    32768 IPs (01.6% of total) - unreachable!
22:e1:23:b2:2f:8c(ip-10-83-59-237.ec2.internal)     2048 IPs (00.1% of total) - unreachable!
92:6b:5f:82:d0:24(ip-10-83-41-63.ec2.internal)    98304 IPs (04.7% of total) - unreachable!
6e:89:89:29:40:32(ip-10-83-98-46.ec2.internal)    16384 IPs (00.8% of total) - unreachable!
fa:ea:3e:94:41:e0(ip-10-83-85-183.ec2.internal)    16384 IPs (00.8% of total) - unreachable!
16:67:3c:ec:79:79(ip-10-83-47-111.ec2.internal)    16384 IPs (00.8% of total) - unreachable!
ba:f1:81:93:ab:b4(ip-10-83-32-140.ec2.internal)    32768 IPs (01.6% of total) - unreachable!
8a:9b:87:49:9d:5b(ip-10-83-67-78.ec2.internal)     8192 IPs (00.4% of total) - unreachable!
9a:55:19:b7:97:7c(ip-10-83-76-130.ec2.internal)     2048 IPs (00.1% of total) - unreachable!
f2:39:2f:d4:16:d7(ip-10-83-72-203.ec2.internal)    30720 IPs (01.5% of total) - unreachable!
ee:c9:d9:9b:bd:ec(ip-10-83-78-215.ec2.internal)     4608 IPs (00.2% of total) - unreachable!
9e:d3:fb:04:82:af(ip-10-83-109-104.ec2.internal)    16384 IPs (00.8% of total) - unreachable!
7e:44:ff:00:f5:f9(ip-10-83-60-42.ec2.internal)     4096 IPs (00.2% of total) - unreachable!
8e:71:01:6f:cd:9f(ip-10-83-73-41.ec2.internal)    16384 IPs (00.8% of total) - unreachable!
86:a9:66:cb:4a:c6(ip-10-83-85-200.ec2.internal)    16384 IPs (00.8% of total) - unreachable!
f6:b4:71:ad:ba:22(ip-10-83-108-91.ec2.internal)    65536 IPs (03.1% of total) - unreachable!
f2:11:ce:93:da:0b(ip-10-83-34-95.ec2.internal)    32768 IPs (01.6% of total) - unreachable!
0e:8c:f8:33:67:7f(ip-10-83-87-92.ec2.internal)    16384 IPs (00.8% of total) - unreachable!
6e:1b:67:ca:e9:72(ip-10-83-101-85.ec2.internal)    65536 IPs (03.1% of total) - unreachable!
7a:fb:6b:30:aa:2a(ip-10-83-106-172.ec2.internal)    32768 IPs (01.6% of total) - unreachable!
72:bf:6c:d7:7a:03(ip-10-83-85-193.ec2.internal)    32768 IPs (01.6% of total) - unreachable!
3a:47:b8:a9:34:ef(ip-10-83-126-221.ec2.internal)    19456 IPs (00.9% of total) - unreachable!
3a:5f:b6:6e:dd:bb(ip-10-83-80-134.ec2.internal)    15360 IPs (00.7% of total) - unreachable!
fe:f3:20:f4:5a:52(ip-10-83-60-125.ec2.internal)    65536 IPs (03.1% of total) - unreachable!
12:ea:66:1b:70:3d(ip-10-83-38-5.ec2.internal)     8192 IPs (00.4% of total) - unreachable!
ae:9e:35:14:6c:0c(ip-10-83-109-171.ec2.internal)    32768 IPs (01.6% of total) - unreachable!
86:d8:10:87:a0:b1(ip-10-83-126-77.ec2.internal)    32768 IPs (01.6% of total) - unreachable!
26:00:36:0b:01:55(ip-10-83-51-193.ec2.internal)    32768 IPs (01.6% of total) - unreachable!
92:55:13:0d:72:f7(ip-10-83-96-158.ec2.internal)     2048 IPs (00.1% of total) - unreachable!
66:15:ce:c7:af:ab(ip-10-83-96-156.ec2.internal)   131072 IPs (06.2% of total) - unreachable!
9a:5d:d4:e9:78:29(ip-10-83-109-188.ec2.internal)     1024 IPs (00.0% of total) - unreachable!
8e:02:27:40:f8:fd(ip-10-83-106-134.ec2.internal)    15872 IPs (00.8% of total) - unreachable!
36:35:70:69:df:eb(ip-10-83-50-197.ec2.internal)    32768 IPs (01.6% of total) - unreachable!
4a:d5:9f:aa:e3:99(ip-10-83-124-188.ec2.internal)     8192 IPs (00.4% of total) - unreachable!
02:77:dd:50:7b:3a(ip-10-83-63-111.ec2.internal)    32768 IPs (01.6% of total) - unreachable!
16:d8:40:86:d6:bf(ip-10-83-41-74.ec2.internal)     6576 IPs (00.3% of total) - unreachable!
16:e5:a9:4e:45:a4(ip-10-83-115-236.ec2.internal)   327680 IPs (15.6% of total) - unreachable!
9a:8b:af:32:3b:0d(ip-10-83-81-239.ec2.internal)    16384 IPs (00.8% of total) - unreachable!
62:03:63:9c:81:8c(ip-10-83-124-69.ec2.internal)    32768 IPs (01.6% of total) - unreachable!

Weave connections

$ ./weave --local status connections
-> 10.83.80.134:6783     failed      Received update for IP range I own at 100.105.68.0 v592: incoming message says owner 12:84:ba:e1:47:79 v1460, retry: 2018-06-05 11:15:31.000788717 +0000 UTC m=+1034.303622109
-> 10.83.98.46:6783      failed      Received update for IP range I own at 100.105.68.0 v592: incoming message says owner 12:84:ba:e1:47:79 v1460, retry: 2018-06-05 11:11:04.422577727 +0000 UTC m=+767.725411089
-> 10.83.126.221:6783    failed      Received update for IP range I own at 100.105.68.0 v592: incoming message says owner 12:84:ba:e1:47:79 v1460, retry: 2018-06-05 11:11:33.742999475 +0000 UTC m=+797.045832929
-> 10.83.85.183:6783     failed      Received update for IP range I own at 100.105.68.0 v592: incoming message says owner 12:84:ba:e1:47:79 v1460, retry: 2018-06-05 11:08:47.877010584 +0000 UTC m=+631.179843953
-> 10.83.60.42:6783      failed      Received update for IP range I own at 100.105.68.0 v592: incoming message says owner 12:84:ba:e1:47:79 v1460, retry: 2018-06-05 11:09:09.444156631 +0000 UTC m=+652.746990055
-> 10.83.78.215:6783     failed      Received update for IP range I own at 100.105.68.0 v592: incoming message says owner 12:84:ba:e1:47:79 v1460, retry: 2018-06-05 11:11:40.747663238 +0000 UTC m=+804.050496608
-> 10.83.124.188:6783    failed      Received update for IP range I own at 100.105.68.0 v592: incoming message says owner 12:84:ba:e1:47:79 v1460, retry: 2018-06-05 11:10:38.016883177 +0000 UTC m=+741.319716603
-> 10.83.42.111:6783     failed      Received update for IP range I own at 100.105.68.0 v592: incoming message says owner 12:84:ba:e1:47:79 v1460, retry: 2018-06-05 11:14:14.702258004 +0000 UTC m=+958.005091377
-> 10.83.109.188:6783    failed      Received update for IP range I own at 100.105.68.0 v592: incoming message says owner 12:84:ba:e1:47:79 v1460, retry: 2018-06-05 11:11:40.776242153 +0000 UTC m=+804.079075513
-> 10.83.36.17:6783      failed      Received update for IP range I own at 100.105.68.0 v592: incoming message says owner 12:84:ba:e1:47:79 v1460, retry: 2018-06-05 11:10:19.676858864 +0000 UTC m=+722.979692293
-> 10.83.59.237:6783     failed      Received update for IP range I own at 100.105.68.0 v592: incoming message says owner 12:84:ba:e1:47:79 v1460, retry: 2018-06-05 11:08:27.22810513 +0000 UTC m=+610.530938541
-> 10.83.41.74:6783      failed      Received update for IP range I own at 100.105.68.0 v592: incoming message says owner 12:84:ba:e1:47:79 v1460, retry: 2018-06-05 11:11:44.830033868 +0000 UTC m=+808.132867307
-> 10.83.54.199:6783     failed      cannot connect to ourself, retry: never
-> 10.83.87.92:6783      failed      Received update for IP range I own at 100.105.68.0 v592: incoming message says owner 12:84:ba:e1:47:79 v1460, retry: 2018-06-05 11:10:32.774426814 +0000 UTC m=+736.077260261
-> 10.83.76.130:6783     failed      Received update for IP range I own at 100.105.68.0 v592: incoming message says owner 12:84:ba:e1:47:79 v1460, retry: 2018-06-05 11:10:48.434728192 +0000 UTC m=+751.737561589

Weave logs

$ kubectl logs -f -n=kube-system weave-net-5qbw9  -c=weave
DEBU: 2018/06/05 11:15:30.215989 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:30.215999 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:30.415837 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:30.415863 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:30.415872 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:30.616648 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:30.616686 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:30.616699 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:30.816246 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:30.816273 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:30.816284 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
INFO: 2018/06/05 11:15:31.001044 ->[10.83.80.134:6783] attempting connection
INFO: 2018/06/05 11:15:31.002845 ->[10.83.80.134:6783|3a:5f:b6:6e:dd:bb(ip-10-83-80-134.ec2.internal)]: connection ready; using protocol version 2
INFO: 2018/06/05 11:15:31.002919 overlay_switch ->[3a:5f:b6:6e:dd:bb(ip-10-83-80-134.ec2.internal)] using fastdp
INFO: 2018/06/05 11:15:31.002973 ->[10.83.80.134:6783|3a:5f:b6:6e:dd:bb(ip-10-83-80-134.ec2.internal)]: connection added (new peer)
INFO: 2018/06/05 11:15:31.013028 ->[10.83.80.134:6783|3a:5f:b6:6e:dd:bb(ip-10-83-80-134.ec2.internal)]: connection shutting down due to error: Received update for IP range I own at 100.105.68.0 v592: incoming message says owner 12:84:ba:e1:47:79 v1460
INFO: 2018/06/05 11:15:31.013113 ->[10.83.80.134:6783|3a:5f:b6:6e:dd:bb(ip-10-83-80-134.ec2.internal)]: connection deleted
INFO: 2018/06/05 11:15:31.013140 Removed unreachable peer 12:84:ba:e1:47:79(ip-10-83-42-111.ec2.internal)
INFO: 2018/06/05 11:15:31.013151 Removed unreachable peer 7e:44:ff:00:f5:f9(ip-10-83-60-42.ec2.internal)
INFO: 2018/06/05 11:15:31.013160 Removed unreachable peer 9a:55:19:b7:97:7c(ip-10-83-76-130.ec2.internal)
INFO: 2018/06/05 11:15:31.013168 Removed unreachable peer 3a:5f:b6:6e:dd:bb(ip-10-83-80-134.ec2.internal)
INFO: 2018/06/05 11:15:31.013176 Removed unreachable peer 22:e1:23:b2:2f:8c(ip-10-83-59-237.ec2.internal)
INFO: 2018/06/05 11:15:31.013185 Removed unreachable peer 0e:8c:f8:33:67:7f(ip-10-83-87-92.ec2.internal)
INFO: 2018/06/05 11:15:31.013192 Removed unreachable peer ee:c9:d9:9b:bd:ec(ip-10-83-78-215.ec2.internal)
INFO: 2018/06/05 11:15:31.013198 Removed unreachable peer 1a:cd:c1:53:24:bf(ip-10-83-36-17.ec2.internal)
INFO: 2018/06/05 11:15:31.013204 Removed unreachable peer 9a:5d:d4:e9:78:29(ip-10-83-109-188.ec2.internal)
INFO: 2018/06/05 11:15:31.013209 Removed unreachable peer 16:d8:40:86:d6:bf(ip-10-83-41-74.ec2.internal)
INFO: 2018/06/05 11:15:31.013215 Removed unreachable peer 4a:d5:9f:aa:e3:99(ip-10-83-124-188.ec2.internal)
INFO: 2018/06/05 11:15:31.013221 Removed unreachable peer fa:ea:3e:94:41:e0(ip-10-83-85-183.ec2.internal)
INFO: 2018/06/05 11:15:31.013226 Removed unreachable peer 3a:47:b8:a9:34:ef(ip-10-83-126-221.ec2.internal)
INFO: 2018/06/05 11:15:31.013232 Removed unreachable peer 6e:89:89:29:40:32(ip-10-83-98-46.ec2.internal)
DEBU: 2018/06/05 11:15:31.016697 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:31.016753 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:31.016777 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:31.215970 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:31.215996 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:31.216007 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:31.416197 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:31.416222 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:31.416231 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:31.616070 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:31.616096 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:31.616105 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:31.815906 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:31.815931 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:31.815941 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:32.015766 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:32.015791 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:32.015801 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:32.216196 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:32.216221 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:32.216231 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:32.415785 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:32.415811 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:32.415821 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:32.615892 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:32.615919 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:32.615928 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:32.815841 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:32.815867 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:32.815877 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:33.016143 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:33.016172 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:33.016182 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:33.216024 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:33.216049 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:33.216060 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:33.416010 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:33.416037 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:33.416046 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:33.616145 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:33.616173 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:33.616182 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:33.815812 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:33.815836 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:33.815847 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:34.017516 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:34.017542 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:34.017552 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:34.216132 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:34.216156 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:34.216167 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:34.415847 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:34.415872 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:34.415882 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:34.615669 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:34.615696 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:34.615706 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:34.815762 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:34.815788 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:34.815798 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:35.015882 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:35.015909 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:35.015919 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:35.215745 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:35.215770 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:35.215780 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:35.415768 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:35.415793 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:35.415803 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:35.615818 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:35.615843 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:35.615853 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:35.815760 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:35.815785 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:35.815796 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:36.015913 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:36.015940 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:36.015950 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:36.215677 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:36.215705 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:36.215715 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:36.415952 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:36.415978 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:36.415988 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:36.616091 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:36.616116 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:36.616127 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:36.816059 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:36.816086 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:36.816099 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:37.016210 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:37.016237 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:37.016246 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:37.216211 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:37.216237 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:37.216247 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:37.415755 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:37.415781 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:37.415791 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:37.615952 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:37.615978 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:37.615988 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:37.815813 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:37.815838 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:37.815848 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:38.016128 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:38.016157 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:38.016167 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:38.216355 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:38.216380 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:38.216391 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:38.415765 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:38.415788 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:38.415798 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:38.616028 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:38.616055 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:38.616065 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:38.818819 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:38.818845 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:38.818854 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:39.016368 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:39.016394 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:39.016404 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:39.215962 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:39.215986 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:39.215995 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:39.415718 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:39.415745 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:39.415754 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:39.616077 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:39.616102 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:39.616117 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:39.815972 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:39.815998 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:39.816008 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:40.016211 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:40.016238 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:40.016248 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:40.216860 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:40.216886 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:40.216897 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:40.416162 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:40.416187 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:40.416197 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:40.616059 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:40.616083 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:40.616093 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:40.816285 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:40.816312 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:40.816322 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:41.016070 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:41.016098 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:41.016109 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:41.216637 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:41.216663 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:41.216673 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:41.416124 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:41.416188 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:41.416220 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:41.616358 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:41.616383 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:41.616393 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:41.815816 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:41.815841 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:41.815850 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:42.016532 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:42.016558 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:42.016568 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:42.217878 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:42.217905 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:42.217919 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:42.415987 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:42.416019 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:42.416033 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:42.616205 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:42.616230 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:42.616240 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:42.816255 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:42.816280 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:42.816290 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:43.016187 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:43.016212 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:43.016223 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:43.216480 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:43.216503 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:43.216513 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:43.416036 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:43.416063 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:43.416074 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:43.615932 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:15:43.615960 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:15:43.615971 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:15:43.815733 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]


From a node that has already claimed the IP range (ip-10-83-42-111.ec2.internal):

Weave IPAM

$ ./weave --local status ipam
12:84:ba:e1:47:79(ip-10-83-42-111.ec2.internal)     1024 IPs (00.0% of total) (61 active)
16:d8:40:86:d6:bf(ip-10-83-41-74.ec2.internal)   334256 IPs (15.9% of total)
3a:47:b8:a9:34:ef(ip-10-83-126-221.ec2.internal)    19504 IPs (00.9% of total)
1a:cd:c1:53:24:bf(ip-10-83-36-17.ec2.internal)   507856 IPs (24.2% of total)
22:e1:23:b2:2f:8c(ip-10-83-59-237.ec2.internal)   190464 IPs (09.1% of total)
fa:ea:3e:94:41:e0(ip-10-83-85-183.ec2.internal)    16384 IPs (00.8% of total)
6a:5c:7c:af:e3:2e(ip-10-83-54-199.ec2.internal)   296016 IPs (14.1% of total) - unreachable!
6e:89:89:29:40:32(ip-10-83-98-46.ec2.internal)    16388 IPs (00.8% of total)
ee:c9:d9:9b:bd:ec(ip-10-83-78-215.ec2.internal)     4608 IPs (00.2% of total)
4a:d5:9f:aa:e3:99(ip-10-83-124-188.ec2.internal)     8192 IPs (00.4% of total)
9a:55:19:b7:97:7c(ip-10-83-76-130.ec2.internal)   313344 IPs (14.9% of total)
3a:5f:b6:6e:dd:bb(ip-10-83-80-134.ec2.internal)   367612 IPs (17.5% of total)
0e:8c:f8:33:67:7f(ip-10-83-87-92.ec2.internal)    16384 IPs (00.8% of total)
9a:5d:d4:e9:78:29(ip-10-83-109-188.ec2.internal)     1024 IPs (00.0% of total)
7e:44:ff:00:f5:f9(ip-10-83-60-42.ec2.internal)     4096 IPs (00.2% of total)

Weave connections

$ ./weave --local status connections
-> 10.83.41.74:6783      established fastdp 16:d8:40:86:d6:bf(ip-10-83-41-74.ec2.internal) mtu=1376
-> 10.83.126.221:6783    established fastdp 3a:47:b8:a9:34:ef(ip-10-83-126-221.ec2.internal) mtu=1376
-> 10.83.76.130:6783     established fastdp 9a:55:19:b7:97:7c(ip-10-83-76-130.ec2.internal) mtu=1376
-> 10.83.109.188:6783    established fastdp 9a:5d:d4:e9:78:29(ip-10-83-109-188.ec2.internal) mtu=1376
-> 10.83.60.42:6783      established fastdp 7e:44:ff:00:f5:f9(ip-10-83-60-42.ec2.internal) mtu=1376
-> 10.83.98.46:6783      established fastdp 6e:89:89:29:40:32(ip-10-83-98-46.ec2.internal) mtu=1376
-> 10.83.85.183:6783     established fastdp fa:ea:3e:94:41:e0(ip-10-83-85-183.ec2.internal) mtu=1376
-> 10.83.124.188:6783    established fastdp 4a:d5:9f:aa:e3:99(ip-10-83-124-188.ec2.internal) mtu=1376
-> 10.83.36.17:6783      established fastdp 1a:cd:c1:53:24:bf(ip-10-83-36-17.ec2.internal) mtu=1376
-> 10.83.78.215:6783     established fastdp ee:c9:d9:9b:bd:ec(ip-10-83-78-215.ec2.internal) mtu=1376
-> 10.83.80.134:6783     established fastdp 3a:5f:b6:6e:dd:bb(ip-10-83-80-134.ec2.internal) mtu=1376
-> 10.83.59.237:6783     established fastdp 22:e1:23:b2:2f:8c(ip-10-83-59-237.ec2.internal) mtu=1376
-> 10.83.87.92:6783      established fastdp 0e:8c:f8:33:67:7f(ip-10-83-87-92.ec2.internal) mtu=1376
-> 10.83.42.111:6783     failed      cannot connect to ourself, retry: never
-> 10.83.54.199:6783     failed      read tcp4 10.83.42.111:48865->10.83.54.199:6783: read: connection reset by peer, retry: 2018-06-05 11:15:00.712417081 +0000 UTC m=+836.618070318

Weave logs

$ kubectl logs -f -n=kube-system weave-net-tk4np -c=weave
DEBU: 2018/06/05 11:20:20.171724 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:20:20.373726 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:20:20.373752 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:20:20.373759 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:20:20.571557 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:20:20.571584 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:20:20.571591 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
INFO: 2018/06/05 11:20:20.701216 Removed unreachable peer 6a:5c:7c:af:e3:2e(ip-10-83-54-199.ec2.internal)
DEBU: 2018/06/05 11:20:20.771492 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:20:20.771516 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:20:20.771525 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:20:20.971641 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:20:20.971671 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:20:20.971684 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:20:21.171472 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:20:21.171499 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:20:21.171506 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:20:21.371635 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:20:21.371660 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:20:21.371668 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:20:21.571635 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:20:21.571663 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:20:21.571673 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:20:21.771057 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:20:21.771085 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:20:21.771092 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:20:21.971717 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:20:21.971743 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:20:21.971751 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:20:22.171426 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:20:22.171453 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:20:22.171463 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:20:22.371484 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:20:22.371510 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:20:22.371520 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:20:22.571555 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:20:22.571581 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:20:22.571589 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
INFO: 2018/06/05 11:20:22.743656 Discovered remote MAC 5a:d8:6a:42:96:2e at 9a:5d:d4:e9:78:29(ip-10-83-109-188.ec2.internal)
DEBU: 2018/06/05 11:20:22.771318 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:20:22.771341 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:20:22.771352 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:20:22.971001 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:20:22.971027 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:20:22.971035 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:20:23.171592 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:20:23.171617 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:20:23.171625 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:20:23.371614 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:20:23.371644 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:20:23.371652 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:20:23.574286 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:20:23.574312 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:20:23.574321 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:20:23.771439 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:20:23.771463 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:20:23.771471 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:20:23.971544 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:20:23.971571 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:20:23.971579 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:20:24.171478 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:20:24.171503 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:20:24.171512 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:20:24.371619 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:20:24.371645 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:20:24.371653 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:20:24.571288 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:20:24.571313 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:20:24.571321 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:20:24.771667 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:20:24.771712 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:20:24.771726 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:20:24.971541 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:20:24.971567 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:20:24.971575 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:20:25.171645 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:20:25.171675 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:20:25.171690 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:20:25.371589 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:20:25.371618 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:20:25.371629 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:20:25.571137 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:20:25.571170 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:20:25.571181 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:20:25.771402 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:20:25.771430 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:20:25.771438 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:20:25.971431 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:20:25.971459 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:20:25.971468 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:20:26.171483 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:20:26.171511 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:20:26.171521 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:20:26.371503 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:20:26.371531 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:20:26.371542 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:20:26.571542 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:20:26.571571 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:20:26.571581 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:20:26.771073 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:20:26.771102 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:20:26.771113 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:20:26.971435 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:20:26.971462 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:20:26.971469 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
INFO: 2018/06/05 11:20:27.070323 Removed unreachable peer 6a:5c:7c:af:e3:2e(ip-10-83-54-199.ec2.internal)
DEBU: 2018/06/05 11:20:27.171473 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:20:27.171498 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:20:27.171506 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:20:27.371505 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:20:27.371530 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:20:27.371538 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:20:27.574060 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:20:27.574088 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:20:27.574097 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:20:27.771457 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:20:27.771483 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:20:27.771492 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:20:27.972061 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:20:27.972087 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:20:27.972095 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:20:28.171557 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:20:28.171583 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:20:28.171592 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:20:28.371123 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:20:28.371149 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:20:28.371157 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:20:28.571669 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:20:28.571695 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:20:28.571703 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
INFO: 2018/06/05 11:20:28.677615 Discovered remote MAC ea:2d:28:ff:48:cf at ee:c9:d9:9b:bd:ec(ip-10-83-78-215.ec2.internal)
DEBU: 2018/06/05 11:20:28.771496 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:20:28.771524 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:20:28.771533 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:20:28.971517 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:20:28.971545 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:20:28.971552 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:20:29.171549 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:20:29.171575 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:20:29.171584 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:20:29.371325 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:20:29.371351 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:20:29.371413 [kube-peers] Existing annotation 46:f4:4b:41:dd:11
DEBU: 2018/06/05 11:20:29.571705 [kube-peers] Nodes that have disappeared: map[ip-10-83-124-112.ec2.internal:{46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}]
DEBU: 2018/06/05 11:20:29.571731 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}
DEBU: 2018/06/05 11:20:29.571739 [kube-peers] Existing annotation 46:f4:4b:41:dd:11

statinvestigating statneed-more-info
Source
picture zacblazic

Most helpful comment

@bboreham I don't want to celebrate just yet but we've taken the following steps are seeing positive results:

  1. Upgraded to 2.4.0 to get the fix @brb mentions in https://github.com/weaveworks/weave/issues/3310#issuecomment-396558236, then ...
  2. Cleared any unreachable! on the ipam-list.
  3. Edited the peer-list annotation on the weave-net configmap to make sure the entries match with the ipam-list (there were entries on the peer-list that were not on the ipam-list as @zacblazic explained in https://github.com/weaveworks/weave/issues/3310#issuecomment-416756507)
  4. Cleared state of any pods exhibiting this behaviour ("Received update for IP range I own") i.e. remove db in /var/lib/weave on host and delete the pod.

On node deletion I can the see entry remains unreachable on the ipam-list (as I expect) but when a new node comes up, there's cleanup (this wasn't happening before).

I want to give it a week and keep checking before I celebrate!

picture itskingori on 30 Aug 2018
👍2 ❤1 🎉1

All 30 comments

@zacblazic Thanks for the issue.

Regarding the first issue, do you have by any chance full Weave Net logs from ip-10-83-42-111.ec2.internal and ip-10-83-54-199.ec2.internal before the restart?

The second issue is going to be fixed in #3317.

brb picture brb on 12 Jun 2018

@brb I'll try get them for you, no guarantees though.

We're seeing this happen more and more often now, seemingly random but cannot confirm the catalyst at this time.

Regarding the second issue, is there a way to manually remove the annotation or clear this state? It's clearly something that persists across the weave cluster as fresh nodes are having the same problem.

zacblazic picture zacblazic on 13 Jun 2018

@brb I work with Zac, and our cluster is getting unstable at least once a day. We fix it by finding out the bad weave pod and clearing it's state. This only started happening after upgrading to 2.3.0. We never had this issue with 2.2.0.

This is the result of connection status on all weave pods 👇 ... you can see it's pretty bad (although better than what we've seen recently).

$ ./weave.sh sandbox "status connections"
Running 'weave --local status connections' on weave-net-2md4g
----------------------------------------------
Defaulting container name to weave.
Use 'kubectl describe pod/weave-net-2md4g -n kube-system' to see all of the containers in this pod.
<- 10.83.109.188:64322   established fastdp 9a:5d:d4:e9:78:29(ip-10-83-109-188.ec2.internal) mtu=1376
<- 10.83.59.237:25989    established fastdp 22:e1:23:b2:2f:8c(ip-10-83-59-237.ec2.internal) mtu=1376
-> 10.83.42.111:6783     established fastdp 12:84:ba:e1:47:79(ip-10-83-42-111.ec2.internal) mtu=1376
<- 10.83.108.123:44036   established fastdp fa:17:fc:80:d1:6c(ip-10-83-108-123.ec2.internal) mtu=1376
-> 10.83.41.74:6783      established fastdp 16:d8:40:86:d6:bf(ip-10-83-41-74.ec2.internal) mtu=1376
<- 10.83.80.99:44626     established fastdp 2a:d6:a8:e7:2e:de(ip-10-83-80-99.ec2.internal) mtu=1376
<- 10.83.87.92:14742     established fastdp 0e:8c:f8:33:67:7f(ip-10-83-87-92.ec2.internal) mtu=1376
-> 10.83.78.215:6783     established fastdp ee:c9:d9:9b:bd:ec(ip-10-83-78-215.ec2.internal) mtu=1376
<- 10.83.125.214:18042   established fastdp 16:fb:51:d2:5c:25(ip-10-83-125-214.ec2.internal) mtu=1376
-> 10.83.60.42:6783      established fastdp 7e:44:ff:00:f5:f9(ip-10-83-60-42.ec2.internal) mtu=1376
<- 10.83.76.130:18782    established fastdp 9a:55:19:b7:97:7c(ip-10-83-76-130.ec2.internal) mtu=1376
<- 10.83.61.125:27523    established fastdp 46:1b:b4:25:cf:87(ip-10-83-61-125.ec2.internal) mtu=1376
-> 10.83.45.86:6783      established fastdp b2:8f:dd:99:f4:ae(ip-10-83-45-86.ec2.internal) mtu=1376
-> 10.83.114.127:6783    established fastdp d6:4a:51:67:64:13(ip-10-83-114-127.ec2.internal) mtu=1376
-> 10.83.88.180:6783     failed      cannot connect to ourself, retry: never
-> 10.83.98.46:6783      failed      Inconsistent entries for 100.97.0.36: owned by 3a:47:b8:a9:34:ef but incoming message says 2a:20:75:a8:38:e9, retry: 2018-06-14 11:31:33.965456343 +0000 UTC m=+81967.687532890
-> 10.83.126.221:6783    retrying    dial tcp4 :0->10.83.126.221:6783: connect: connection timed out
-> 10.83.58.17:6783      failed      Received update for IP range I own at 100.126.32.0 v31: incoming message says owner b2:30:8c:a3:99:1d v32, retry: 2018-06-14 11:31:05.706477947 +0000 UTC m=+81939.428554426
Running 'weave --local status connections' on weave-net-5kvfv
----------------------------------------------
Defaulting container name to weave.
Use 'kubectl describe pod/weave-net-5kvfv -n kube-system' to see all of the containers in this pod.
<- 10.83.125.214:48907   established fastdp 16:fb:51:d2:5c:25(ip-10-83-125-214.ec2.internal) mtu=1376
-> 10.83.80.99:6783      established fastdp 2a:d6:a8:e7:2e:de(ip-10-83-80-99.ec2.internal) mtu=1376
-> 10.83.45.86:6783      established fastdp b2:8f:dd:99:f4:ae(ip-10-83-45-86.ec2.internal) mtu=1376
-> 10.83.88.180:6783     established fastdp ae:d0:02:a5:ff:b4(ip-10-83-88-180.ec2.internal) mtu=1376
<- 10.83.60.42:37129     established fastdp 7e:44:ff:00:f5:f9(ip-10-83-60-42.ec2.internal) mtu=1376
<- 10.83.78.215:51406    established fastdp ee:c9:d9:9b:bd:ec(ip-10-83-78-215.ec2.internal) mtu=1376
<- 10.83.76.130:52366    established fastdp 9a:55:19:b7:97:7c(ip-10-83-76-130.ec2.internal) mtu=1376
<- 10.83.109.188:11699   established fastdp 9a:5d:d4:e9:78:29(ip-10-83-109-188.ec2.internal) mtu=1376
<- 10.83.59.237:62120    established fastdp 22:e1:23:b2:2f:8c(ip-10-83-59-237.ec2.internal) mtu=1376
-> 10.83.114.127:6783    established fastdp d6:4a:51:67:64:13(ip-10-83-114-127.ec2.internal) mtu=1376
<- 10.83.41.74:32446     established fastdp 16:d8:40:86:d6:bf(ip-10-83-41-74.ec2.internal) mtu=1376
-> 10.83.42.111:6783     established fastdp 12:84:ba:e1:47:79(ip-10-83-42-111.ec2.internal) mtu=1376
-> 10.83.108.123:6783    established fastdp fa:17:fc:80:d1:6c(ip-10-83-108-123.ec2.internal) mtu=1376
-> 10.83.61.125:6783     established fastdp 46:1b:b4:25:cf:87(ip-10-83-61-125.ec2.internal) mtu=1376
-> 10.83.58.17:6783      failed      Inconsistent entries for 100.96.36.0: owned by fa:17:fc:80:d1:6c but incoming message says 6e:89:89:29:40:32, retry: 2018-06-14 11:27:33.772854715 +0000 UTC m=+83875.969683927
-> 10.83.126.221:6783    failed      dial tcp4 :0->10.83.126.221:6783: connect: connection timed out, retry: 2018-06-14 11:29:24.549128843 +0000 UTC m=+83986.745958356
-> 10.83.87.92:6783      failed      cannot connect to ourself, retry: never
-> 10.83.98.46:6783      failed      Inconsistent entries for 100.105.64.0: owned by fa:17:fc:80:d1:6c but incoming message says 6e:89:89:29:40:32, retry: 2018-06-14 11:25:42.160799739 +0000 UTC m=+83764.357628911
Running 'weave --local status connections' on weave-net-5l8f9
----------------------------------------------
Defaulting container name to weave.
Use 'kubectl describe pod/weave-net-5l8f9 -n kube-system' to see all of the containers in this pod.
-> 10.83.87.92:6783      established fastdp 0e:8c:f8:33:67:7f(ip-10-83-87-92.ec2.internal) mtu=1376
<- 10.83.41.74:26540     established fastdp 16:d8:40:86:d6:bf(ip-10-83-41-74.ec2.internal) mtu=1376
<- 10.83.88.180:44552    established fastdp ae:d0:02:a5:ff:b4(ip-10-83-88-180.ec2.internal) mtu=1376
-> 10.83.80.99:6783      established fastdp 2a:d6:a8:e7:2e:de(ip-10-83-80-99.ec2.internal) mtu=1376
-> 10.83.45.86:6783      established fastdp b2:8f:dd:99:f4:ae(ip-10-83-45-86.ec2.internal) mtu=1376
-> 10.83.61.125:6783     established fastdp 46:1b:b4:25:cf:87(ip-10-83-61-125.ec2.internal) mtu=1376
<- 10.83.109.188:11537   established fastdp 9a:5d:d4:e9:78:29(ip-10-83-109-188.ec2.internal) mtu=1376
<- 10.83.125.214:19043   established fastdp 16:fb:51:d2:5c:25(ip-10-83-125-214.ec2.internal) mtu=1376
-> 10.83.114.127:6783    established fastdp d6:4a:51:67:64:13(ip-10-83-114-127.ec2.internal) mtu=1376
-> 10.83.78.215:6783     established fastdp ee:c9:d9:9b:bd:ec(ip-10-83-78-215.ec2.internal) mtu=1376
-> 10.83.42.111:6783     established fastdp 12:84:ba:e1:47:79(ip-10-83-42-111.ec2.internal) mtu=1376
<- 10.83.59.237:17338    established fastdp 22:e1:23:b2:2f:8c(ip-10-83-59-237.ec2.internal) mtu=1376
<- 10.83.76.130:31475    established fastdp 9a:55:19:b7:97:7c(ip-10-83-76-130.ec2.internal) mtu=1376
<- 10.83.108.123:33698   established fastdp fa:17:fc:80:d1:6c(ip-10-83-108-123.ec2.internal) mtu=1376
-> 10.83.98.46:6783      failed      Inconsistent entries for 100.105.64.0: owned by fa:17:fc:80:d1:6c but incoming message says 6e:89:89:29:40:32, retry: 2018-06-14 11:28:21.515968018 +0000 UTC m=+83917.194931761
-> 10.83.126.221:6783    failed      dial tcp4 :0->10.83.126.221:6783: connect: connection timed out, retry: 2018-06-14 11:31:59.350542407 +0000 UTC m=+84135.029506466
-> 10.83.58.17:6783      failed      Inconsistent entries for 100.96.36.0: owned by fa:17:fc:80:d1:6c but incoming message says 6e:89:89:29:40:32, retry: 2018-06-14 11:25:51.38209743 +0000 UTC m=+83767.061061136
-> 10.83.60.42:6783      failed      cannot connect to ourself, retry: never
Running 'weave --local status connections' on weave-net-62x5t
----------------------------------------------
Defaulting container name to weave.
Use 'kubectl describe pod/weave-net-62x5t -n kube-system' to see all of the containers in this pod.
-> 10.83.42.111:6783     established fastdp 12:84:ba:e1:47:79(ip-10-83-42-111.ec2.internal) mtu=1376
-> 10.83.59.237:6783     established fastdp 22:e1:23:b2:2f:8c(ip-10-83-59-237.ec2.internal) mtu=1376
-> 10.83.109.188:6783    established fastdp 9a:5d:d4:e9:78:29(ip-10-83-109-188.ec2.internal) mtu=1376
<- 10.83.76.130:52288    established fastdp 9a:55:19:b7:97:7c(ip-10-83-76-130.ec2.internal) mtu=1376
<- 10.83.78.215:29365    established fastdp ee:c9:d9:9b:bd:ec(ip-10-83-78-215.ec2.internal) mtu=1376
-> 10.83.60.42:6783      established fastdp 7e:44:ff:00:f5:f9(ip-10-83-60-42.ec2.internal) mtu=1376
-> 10.83.61.125:6783     established fastdp 46:1b:b4:25:cf:87(ip-10-83-61-125.ec2.internal) mtu=1376
-> 10.83.114.127:6783    established fastdp d6:4a:51:67:64:13(ip-10-83-114-127.ec2.internal) mtu=1376
-> 10.83.125.214:6783    established fastdp 16:fb:51:d2:5c:25(ip-10-83-125-214.ec2.internal) mtu=1376
-> 10.83.45.86:6783      established fastdp b2:8f:dd:99:f4:ae(ip-10-83-45-86.ec2.internal) mtu=1376
-> 10.83.108.123:6783    established fastdp fa:17:fc:80:d1:6c(ip-10-83-108-123.ec2.internal) mtu=1376
-> 10.83.87.92:6783      established fastdp 0e:8c:f8:33:67:7f(ip-10-83-87-92.ec2.internal) mtu=1376
-> 10.83.80.99:6783      established fastdp 2a:d6:a8:e7:2e:de(ip-10-83-80-99.ec2.internal) mtu=1376
<- 10.83.88.180:54210    established fastdp ae:d0:02:a5:ff:b4(ip-10-83-88-180.ec2.internal) mtu=1376
-> 10.83.41.74:6783      failed      cannot connect to ourself, retry: never
-> 10.83.58.17:6783      failed      Inconsistent entries for 100.96.36.0: owned by fa:17:fc:80:d1:6c but incoming message says 6e:89:89:29:40:32, retry: 2018-06-14 11:29:29.912737519 +0000 UTC m=+78654.224234518
-> 10.83.108.81:6783     failed      dial tcp4 :0->10.83.108.81:6783: connect: connection timed out, retry: 2018-06-14 11:26:12.325025858 +0000 UTC m=+78456.636522850
-> 10.83.98.46:6783      failed      Inconsistent entries for 100.105.64.0: owned by fa:17:fc:80:d1:6c but incoming message says 6e:89:89:29:40:32, retry: 2018-06-14 11:26:29.340993969 +0000 UTC m=+78473.652490924
-> 10.83.89.115:6783     failed      dial tcp4 :0->10.83.89.115:6783: connect: connection timed out, retry: 2018-06-14 11:28:42.789700924 +0000 UTC m=+78607.101198116
Running 'weave --local status connections' on weave-net-88667
----------------------------------------------
Defaulting container name to weave.
Use 'kubectl describe pod/weave-net-88667 -n kube-system' to see all of the containers in this pod.
-> 10.83.45.86:6783      established fastdp b2:8f:dd:99:f4:ae(ip-10-83-45-86.ec2.internal) mtu=1376
-> 10.83.114.127:6783    established fastdp d6:4a:51:67:64:13(ip-10-83-114-127.ec2.internal) mtu=1376
-> 10.83.108.123:6783    established fastdp fa:17:fc:80:d1:6c(ip-10-83-108-123.ec2.internal) mtu=1376
<- 10.83.60.42:17711     established fastdp 7e:44:ff:00:f5:f9(ip-10-83-60-42.ec2.internal) mtu=1376
-> 10.83.61.125:6783     established fastdp 46:1b:b4:25:cf:87(ip-10-83-61-125.ec2.internal) mtu=1376
<- 10.83.78.215:19858    established fastdp ee:c9:d9:9b:bd:ec(ip-10-83-78-215.ec2.internal) mtu=1376
<- 10.83.109.188:65431   established fastdp 9a:5d:d4:e9:78:29(ip-10-83-109-188.ec2.internal) mtu=1376
<- 10.83.41.74:17723     established fastdp 16:d8:40:86:d6:bf(ip-10-83-41-74.ec2.internal) mtu=1376
<- 10.83.125.214:39423   established fastdp 16:fb:51:d2:5c:25(ip-10-83-125-214.ec2.internal) mtu=1376
-> 10.83.76.130:6783     established fastdp 9a:55:19:b7:97:7c(ip-10-83-76-130.ec2.internal) mtu=1376
-> 10.83.88.180:6783     established fastdp ae:d0:02:a5:ff:b4(ip-10-83-88-180.ec2.internal) mtu=1376
<- 10.83.87.92:34053     established fastdp 0e:8c:f8:33:67:7f(ip-10-83-87-92.ec2.internal) mtu=1376
<- 10.83.80.99:28473     established fastdp 2a:d6:a8:e7:2e:de(ip-10-83-80-99.ec2.internal) mtu=1376
<- 10.83.59.237:35314    established fastdp 22:e1:23:b2:2f:8c(ip-10-83-59-237.ec2.internal) mtu=1376
-> 10.83.42.111:6783     failed      cannot connect to ourself, retry: never
-> 10.83.98.46:6783      failed      Inconsistent entries for 100.105.64.0: owned by fa:17:fc:80:d1:6c but incoming message says 6e:89:89:29:40:32, retry: 2018-06-14 11:29:55.896806219 +0000 UTC m=+84130.047212383
-> 10.83.58.17:6783      failed      Inconsistent entries for 100.96.36.0: owned by fa:17:fc:80:d1:6c but incoming message says 6e:89:89:29:40:32, retry: 2018-06-14 11:25:49.94123336 +0000 UTC m=+83884.091639521
-> 10.83.126.221:6783    failed      dial tcp4 :0->10.83.126.221:6783: connect: connection timed out, retry: 2018-06-14 11:29:06.362327232 +0000 UTC m=+84080.512733531
Running 'weave --local status connections' on weave-net-8wdpt
----------------------------------------------
Defaulting container name to weave.
Use 'kubectl describe pod/weave-net-8wdpt -n kube-system' to see all of the containers in this pod.
-> 10.83.108.123:6783    established fastdp fa:17:fc:80:d1:6c(ip-10-83-108-123.ec2.internal) mtu=1376
-> 10.83.42.111:6783     established fastdp 12:84:ba:e1:47:79(ip-10-83-42-111.ec2.internal) mtu=1376
<- 10.83.125.214:28764   established fastdp 16:fb:51:d2:5c:25(ip-10-83-125-214.ec2.internal) mtu=1376
-> 10.83.114.127:6783    established fastdp d6:4a:51:67:64:13(ip-10-83-114-127.ec2.internal) mtu=1376
<- 10.83.76.130:15202    established fastdp 9a:55:19:b7:97:7c(ip-10-83-76-130.ec2.internal) mtu=1376
-> 10.83.109.188:6783    established fastdp 9a:5d:d4:e9:78:29(ip-10-83-109-188.ec2.internal) mtu=1376
<- 10.83.88.180:54577    established fastdp ae:d0:02:a5:ff:b4(ip-10-83-88-180.ec2.internal) mtu=1376
-> 10.83.87.92:6783      established fastdp 0e:8c:f8:33:67:7f(ip-10-83-87-92.ec2.internal) mtu=1376
<- 10.83.61.125:55102    established fastdp 46:1b:b4:25:cf:87(ip-10-83-61-125.ec2.internal) mtu=1376
<- 10.83.60.42:48370     established fastdp 7e:44:ff:00:f5:f9(ip-10-83-60-42.ec2.internal) mtu=1376
-> 10.83.41.74:6783      established fastdp 16:d8:40:86:d6:bf(ip-10-83-41-74.ec2.internal) mtu=1376
<- 10.83.80.99:61954     established fastdp 2a:d6:a8:e7:2e:de(ip-10-83-80-99.ec2.internal) mtu=1376
-> 10.83.59.237:6783     established fastdp 22:e1:23:b2:2f:8c(ip-10-83-59-237.ec2.internal) mtu=1376
<- 10.83.45.86:44301     established fastdp b2:8f:dd:99:f4:ae(ip-10-83-45-86.ec2.internal) mtu=1376
-> 10.83.98.46:6783      failed      Received update for IP range I own at 100.113.2.0 v15: incoming message says owner 6e:89:89:29:40:32 v16, retry: 2018-06-14 11:32:31.770019755 +0000 UTC m=+84291.863032889
-> 10.83.126.221:6783    failed      dial tcp4 :0->10.83.126.221:6783: connect: connection timed out, retry: 2018-06-14 11:29:00.021782688 +0000 UTC m=+84080.114795935
-> 10.83.58.17:6783      failed      Received update for IP range I own at 100.113.2.0 v15: incoming message says owner 6e:89:89:29:40:32 v16, retry: 2018-06-14 11:25:37.719566741 +0000 UTC m=+83877.812579878
-> 10.83.78.215:6783     failed      cannot connect to ourself, retry: never
Running 'weave --local status connections' on weave-net-9mr87
----------------------------------------------
Defaulting container name to weave.
Use 'kubectl describe pod/weave-net-9mr87 -n kube-system' to see all of the containers in this pod.
<- 10.83.109.188:18863   established fastdp 9a:5d:d4:e9:78:29(ip-10-83-109-188.ec2.internal) mtu=1376
-> 10.83.114.127:6783    established fastdp d6:4a:51:67:64:13(ip-10-83-114-127.ec2.internal) mtu=1376
<- 10.83.41.74:10803     established fastdp 16:d8:40:86:d6:bf(ip-10-83-41-74.ec2.internal) mtu=1376
-> 10.83.45.86:6783      established fastdp b2:8f:dd:99:f4:ae(ip-10-83-45-86.ec2.internal) mtu=1376
<- 10.83.108.123:49548   established fastdp fa:17:fc:80:d1:6c(ip-10-83-108-123.ec2.internal) mtu=1376
-> 10.83.80.99:6783      established fastdp 2a:d6:a8:e7:2e:de(ip-10-83-80-99.ec2.internal) mtu=1376
<- 10.83.59.237:25988    established fastdp 22:e1:23:b2:2f:8c(ip-10-83-59-237.ec2.internal) mtu=1376
-> 10.83.125.214:6783    established fastdp 16:fb:51:d2:5c:25(ip-10-83-125-214.ec2.internal) mtu=1376
-> 10.83.60.42:6783      established fastdp 7e:44:ff:00:f5:f9(ip-10-83-60-42.ec2.internal) mtu=1376
-> 10.83.42.111:6783     established fastdp 12:84:ba:e1:47:79(ip-10-83-42-111.ec2.internal) mtu=1376
-> 10.83.78.215:6783     established fastdp ee:c9:d9:9b:bd:ec(ip-10-83-78-215.ec2.internal) mtu=1376
<- 10.83.61.125:47415    established fastdp 46:1b:b4:25:cf:87(ip-10-83-61-125.ec2.internal) mtu=1376
-> 10.83.87.92:6783      established fastdp 0e:8c:f8:33:67:7f(ip-10-83-87-92.ec2.internal) mtu=1376
-> 10.83.88.180:6783     established fastdp ae:d0:02:a5:ff:b4(ip-10-83-88-180.ec2.internal) mtu=1376
-> 10.83.58.17:6783      failed      Received update for IP range I own at 100.96.0.0 v9741: incoming message says owner 6e:89:89:29:40:32 v9743, retry: 2018-06-14 11:28:21.015301913 +0000 UTC m=+84025.243656987
-> 10.83.76.130:6783     failed      cannot connect to ourself, retry: never
-> 10.83.98.46:6783      failed      Received update for IP range I own at 100.96.0.0 v9741: incoming message says owner 46:1b:b4:25:cf:87 v9744, retry: 2018-06-14 11:27:52.330933768 +0000 UTC m=+83996.559288868
-> 10.83.126.221:6783    retrying    dial tcp4 :0->10.83.126.221:6783: connect: connection timed out
Running 'weave --local status connections' on weave-net-9vdqx
----------------------------------------------
Defaulting container name to weave.
Use 'kubectl describe pod/weave-net-9vdqx -n kube-system' to see all of the containers in this pod.
<- 10.83.59.237:31036    established fastdp 22:e1:23:b2:2f:8c(ip-10-83-59-237.ec2.internal) mtu=1376
<- 10.83.108.123:14174   established fastdp fa:17:fc:80:d1:6c(ip-10-83-108-123.ec2.internal) mtu=1376
<- 10.83.41.74:58313     established fastdp 16:d8:40:86:d6:bf(ip-10-83-41-74.ec2.internal) mtu=1376
-> 10.83.114.127:6783    established fastdp d6:4a:51:67:64:13(ip-10-83-114-127.ec2.internal) mtu=1376
<- 10.83.78.215:40928    established fastdp ee:c9:d9:9b:bd:ec(ip-10-83-78-215.ec2.internal) mtu=1376
-> 10.83.45.86:6783      established fastdp b2:8f:dd:99:f4:ae(ip-10-83-45-86.ec2.internal) mtu=1376
-> 10.83.87.92:6783      established fastdp 0e:8c:f8:33:67:7f(ip-10-83-87-92.ec2.internal) mtu=1376
<- 10.83.88.180:55270    established fastdp ae:d0:02:a5:ff:b4(ip-10-83-88-180.ec2.internal) mtu=1376
-> 10.83.60.42:6783      established fastdp 7e:44:ff:00:f5:f9(ip-10-83-60-42.ec2.internal) mtu=1376
-> 10.83.42.111:6783     established fastdp 12:84:ba:e1:47:79(ip-10-83-42-111.ec2.internal) mtu=1376
-> 10.83.76.130:6783     established fastdp 9a:55:19:b7:97:7c(ip-10-83-76-130.ec2.internal) mtu=1376
-> 10.83.80.99:6783      established fastdp 2a:d6:a8:e7:2e:de(ip-10-83-80-99.ec2.internal) mtu=1376
<- 10.83.61.125:30354    established fastdp 46:1b:b4:25:cf:87(ip-10-83-61-125.ec2.internal) mtu=1376
<- 10.83.125.214:30387   established fastdp 16:fb:51:d2:5c:25(ip-10-83-125-214.ec2.internal) mtu=1376
-> 10.83.58.17:6783      failed      Inconsistent entries for 100.96.36.0: owned by fa:17:fc:80:d1:6c but incoming message says 6e:89:89:29:40:32, retry: 2018-06-14 11:25:38.789173427 +0000 UTC m=+83754.709480414
-> 10.83.98.46:6783      failed      Inconsistent entries for 100.105.64.0: owned by fa:17:fc:80:d1:6c but incoming message says 6e:89:89:29:40:32, retry: 2018-06-14 11:25:48.485915806 +0000 UTC m=+83764.406222788
-> 10.83.126.221:6783    failed      dial tcp4 :0->10.83.126.221:6783: connect: no route to host, retry: 2018-06-14 11:30:05.84487768 +0000 UTC m=+84021.765184757
-> 10.83.109.188:6783    failed      cannot connect to ourself, retry: never
Running 'weave --local status connections' on weave-net-h5wcj
----------------------------------------------
Defaulting container name to weave.
Use 'kubectl describe pod/weave-net-h5wcj -n kube-system' to see all of the containers in this pod.
-> 10.83.88.180:6783     failed      read tcp4 10.83.98.46:53004->10.83.88.180:6783: read: connection reset by peer, retry: 2018-06-14 11:29:16.377667883 +0000 UTC m=+2054.107086596
-> 10.83.58.17:6783      failed      Inconsistent entries for 100.97.0.36: owned by 2a:20:75:a8:38:e9 but incoming message says 3a:47:b8:a9:34:ef, retry: 2018-06-14 11:32:45.741249992 +0000 UTC m=+2263.470668683
-> 10.83.76.130:6783     failed      read tcp4 10.83.98.46:38697->10.83.76.130:6783: read: connection reset by peer, retry: 2018-06-14 11:26:47.594652351 +0000 UTC m=+1905.324071028
-> 10.83.78.215:6783     failed      read tcp4 10.83.98.46:47933->10.83.78.215:6783: read: connection reset by peer, retry: 2018-06-14 11:26:03.4515573 +0000 UTC m=+1861.180976012
-> 10.83.42.111:6783     failed      Inconsistent entries for 100.105.64.0: owned by 6e:89:89:29:40:32 but incoming message says fa:17:fc:80:d1:6c, retry: 2018-06-14 11:29:50.44202945 +0000 UTC m=+2088.171448117
-> 10.83.59.237:6783     failed      Inconsistent entries for 100.105.64.0: owned by 6e:89:89:29:40:32 but incoming message says fa:17:fc:80:d1:6c, retry: 2018-06-14 11:32:47.025322481 +0000 UTC m=+2264.754741227
-> 10.83.60.42:6783      failed      Inconsistent entries for 100.105.64.0: owned by 6e:89:89:29:40:32 but incoming message says fa:17:fc:80:d1:6c, retry: 2018-06-14 11:29:21.857391059 +0000 UTC m=+2059.586809785
-> 10.83.87.92:6783      failed      Inconsistent entries for 100.105.64.0: owned by 6e:89:89:29:40:32 but incoming message says fa:17:fc:80:d1:6c, retry: 2018-06-14 11:27:04.77568889 +0000 UTC m=+1922.505107616
-> 10.83.108.123:6783    failed      Inconsistent entries for 100.105.64.0: owned by 6e:89:89:29:40:32 but incoming message says fa:17:fc:80:d1:6c, retry: 2018-06-14 11:27:16.422466316 +0000 UTC m=+1934.151885043
-> 10.83.98.46:6783      failed      cannot connect to ourself, retry: never
-> 10.83.125.214:6783    failed      write tcp4 10.83.98.46:21423->10.83.125.214:6783: write: connection reset by peer, retry: 2018-06-14 11:30:01.332060558 +0000 UTC m=+2099.061479252
-> 10.83.109.188:6783    failed      Inconsistent entries for 100.105.64.0: owned by 6e:89:89:29:40:32 but incoming message says fa:17:fc:80:d1:6c, retry: 2018-06-14 11:29:39.671812835 +0000 UTC m=+2077.401231586
-> 10.83.41.74:6783      failed      Inconsistent entries for 100.105.64.0: owned by 6e:89:89:29:40:32 but incoming message says fa:17:fc:80:d1:6c, retry: 2018-06-14 11:28:26.938954228 +0000 UTC m=+2004.668372957
-> 10.83.114.127:6783    failed      write tcp4 10.83.98.46:11946->10.83.114.127:6783: write: connection reset by peer, retry: 2018-06-14 11:27:33.910939201 +0000 UTC m=+1951.640357936
-> 10.83.45.86:6783      failed      Inconsistent entries for 100.105.64.0: owned by 6e:89:89:29:40:32 but incoming message says fa:17:fc:80:d1:6c, retry: 2018-06-14 11:30:01.670949968 +0000 UTC m=+2099.400368672
-> 10.83.61.125:6783     failed      Inconsistent entries for 100.97.0.36: owned by 2a:20:75:a8:38:e9 but incoming message says 3a:47:b8:a9:34:ef, retry: 2018-06-14 11:27:49.079178243 +0000 UTC m=+1966.808596976
-> 10.83.80.99:6783      failed      read tcp4 10.83.98.46:52649->10.83.80.99:6783: read: connection reset by peer, retry: 2018-06-14 11:28:16.808492709 +0000 UTC m=+1994.537911427
Running 'weave --local status connections' on weave-net-l2c8d
----------------------------------------------
Defaulting container name to weave.
Use 'kubectl describe pod/weave-net-l2c8d -n kube-system' to see all of the containers in this pod.
-> 10.83.59.237:6783     failed      Inconsistent entries for 100.96.36.0: owned by 6e:89:89:29:40:32 but incoming message says fa:17:fc:80:d1:6c, retry: 2018-06-14 11:30:27.339833685 +0000 UTC m=+1858.591352797
-> 10.83.42.111:6783     failed      Inconsistent entries for 100.96.36.0: owned by 6e:89:89:29:40:32 but incoming message says fa:17:fc:80:d1:6c, retry: 2018-06-14 11:26:25.688472069 +0000 UTC m=+1616.939991047
-> 10.83.61.125:6783     failed      Received update for IP range I own at 100.126.0.0 v10066: incoming message says owner ae:d0:02:a5:ff:b4 v10401, retry: 2018-06-14 11:33:22.334129662 +0000 UTC m=+2033.585648696
-> 10.83.41.74:6783      failed      Inconsistent entries for 100.96.36.0: owned by 6e:89:89:29:40:32 but incoming message says fa:17:fc:80:d1:6c, retry: 2018-06-14 11:26:14.442909091 +0000 UTC m=+1605.694428105
-> 10.83.76.130:6783     failed      read tcp4 10.83.58.17:40573->10.83.76.130:6783: read: connection reset by peer, retry: 2018-06-14 11:28:35.847360488 +0000 UTC m=+1747.098879463
-> 10.83.78.215:6783     failed      read tcp4 10.83.58.17:11003->10.83.78.215:6783: read: connection reset by peer, retry: 2018-06-14 11:27:56.413148287 +0000 UTC m=+1707.664667257
-> 10.83.98.46:6783      failed      Inconsistent entries for 100.97.0.36: owned by 3a:47:b8:a9:34:ef but incoming message says 2a:20:75:a8:38:e9, retry: 2018-06-14 11:28:28.70787515 +0000 UTC m=+1739.959394166
-> 10.83.80.99:6783      failed      Inconsistent entries for 100.96.36.0: owned by 6e:89:89:29:40:32 but incoming message says fa:17:fc:80:d1:6c, retry: 2018-06-14 11:28:58.681237192 +0000 UTC m=+1769.932756205
-> 10.83.114.127:6783    failed      Inconsistent entries for 100.96.36.0: owned by 6e:89:89:29:40:32 but incoming message says fa:17:fc:80:d1:6c, retry: 2018-06-14 11:30:57.939384677 +0000 UTC m=+1889.190903646
-> 10.83.58.17:6783      failed      cannot connect to ourself, retry: never
-> 10.83.60.42:6783      failed      read tcp4 10.83.58.17:35738->10.83.60.42:6783: read: connection reset by peer, retry: 2018-06-14 11:26:09.202236771 +0000 UTC m=+1600.453755744
-> 10.83.45.86:6783      failed      Inconsistent entries for 100.96.36.0: owned by 6e:89:89:29:40:32 but incoming message says fa:17:fc:80:d1:6c, retry: 2018-06-14 11:28:39.650034941 +0000 UTC m=+1750.901554017
-> 10.83.109.188:6783    failed      Inconsistent entries for 100.96.36.0: owned by 6e:89:89:29:40:32 but incoming message says fa:17:fc:80:d1:6c, retry: 2018-06-14 11:30:40.313034644 +0000 UTC m=+1871.564553678
-> 10.83.87.92:6783      failed      Inconsistent entries for 100.96.36.0: owned by 6e:89:89:29:40:32 but incoming message says fa:17:fc:80:d1:6c, retry: 2018-06-14 11:28:55.560646201 +0000 UTC m=+1766.812165217
-> 10.83.108.123:6783    failed      Inconsistent entries for 100.96.36.0: owned by 6e:89:89:29:40:32 but incoming message says fa:17:fc:80:d1:6c, retry: 2018-06-14 11:29:02.675724678 +0000 UTC m=+1773.927243704
-> 10.83.125.214:6783    failed      Inconsistent entries for 100.96.36.0: owned by 6e:89:89:29:40:32 but incoming message says fa:17:fc:80:d1:6c, retry: 2018-06-14 11:29:00.585894378 +0000 UTC m=+1771.837413407
-> 10.83.88.180:6783     failed      Received update for IP range I own at 100.126.0.0 v10066: incoming message says owner ae:d0:02:a5:ff:b4 v10401, retry: 2018-06-14 11:31:11.480243818 +0000 UTC m=+1902.731762985
Running 'weave --local status connections' on weave-net-mzm7x
----------------------------------------------
Defaulting container name to weave.
Use 'kubectl describe pod/weave-net-mzm7x -n kube-system' to see all of the containers in this pod.
<- 10.83.60.42:29859     established fastdp 7e:44:ff:00:f5:f9(ip-10-83-60-42.ec2.internal) mtu=1376
<- 10.83.109.188:51219   established fastdp 9a:5d:d4:e9:78:29(ip-10-83-109-188.ec2.internal) mtu=1376
<- 10.83.59.237:10622    established fastdp 22:e1:23:b2:2f:8c(ip-10-83-59-237.ec2.internal) mtu=1376
-> 10.83.61.125:6783     established fastdp 46:1b:b4:25:cf:87(ip-10-83-61-125.ec2.internal) mtu=1376
<- 10.83.41.74:20246     established fastdp 16:d8:40:86:d6:bf(ip-10-83-41-74.ec2.internal) mtu=1376
-> 10.83.114.127:6783    established fastdp d6:4a:51:67:64:13(ip-10-83-114-127.ec2.internal) mtu=1376
-> 10.83.108.123:6783    established fastdp fa:17:fc:80:d1:6c(ip-10-83-108-123.ec2.internal) mtu=1376
<- 10.83.87.92:36680     established fastdp 0e:8c:f8:33:67:7f(ip-10-83-87-92.ec2.internal) mtu=1376
<- 10.83.125.214:34279   established fastdp 16:fb:51:d2:5c:25(ip-10-83-125-214.ec2.internal) mtu=1376
-> 10.83.42.111:6783     established fastdp 12:84:ba:e1:47:79(ip-10-83-42-111.ec2.internal) mtu=1376
-> 10.83.78.215:6783     established fastdp ee:c9:d9:9b:bd:ec(ip-10-83-78-215.ec2.internal) mtu=1376
<- 10.83.76.130:31920    established fastdp 9a:55:19:b7:97:7c(ip-10-83-76-130.ec2.internal) mtu=1376
-> 10.83.45.86:6783      established fastdp b2:8f:dd:99:f4:ae(ip-10-83-45-86.ec2.internal) mtu=1376
<- 10.83.88.180:21758    established fastdp ae:d0:02:a5:ff:b4(ip-10-83-88-180.ec2.internal) mtu=1376
-> 10.83.126.221:6783    failed      dial tcp4 :0->10.83.126.221:6783: connect: connection timed out, retry: 2018-06-14 11:30:28.945977082 +0000 UTC m=+84156.011272833
-> 10.83.98.46:6783      failed      Inconsistent entries for 100.105.64.0: owned by fa:17:fc:80:d1:6c but incoming message says 6e:89:89:29:40:32, retry: 2018-06-14 11:29:19.584910211 +0000 UTC m=+84086.650205957
-> 10.83.58.17:6783      failed      Inconsistent entries for 100.96.36.0: owned by fa:17:fc:80:d1:6c but incoming message says 6e:89:89:29:40:32, retry: 2018-06-14 11:31:58.47465015 +0000 UTC m=+84245.539945759
-> 10.83.80.99:6783      failed      cannot connect to ourself, retry: never
Running 'weave --local status connections' on weave-net-nh6p4
----------------------------------------------
Defaulting container name to weave.
Use 'kubectl describe pod/weave-net-nh6p4 -n kube-system' to see all of the containers in this pod.
-> 10.83.88.180:6783     established fastdp ae:d0:02:a5:ff:b4(ip-10-83-88-180.ec2.internal) mtu=1376
<- 10.83.41.74:46144     established fastdp 16:d8:40:86:d6:bf(ip-10-83-41-74.ec2.internal) mtu=1376
-> 10.83.87.92:6783      established fastdp 0e:8c:f8:33:67:7f(ip-10-83-87-92.ec2.internal) mtu=1376
-> 10.83.109.188:6783    established fastdp 9a:5d:d4:e9:78:29(ip-10-83-109-188.ec2.internal) mtu=1376
<- 10.83.108.123:30342   established fastdp fa:17:fc:80:d1:6c(ip-10-83-108-123.ec2.internal) mtu=1376
<- 10.83.78.215:44092    established fastdp ee:c9:d9:9b:bd:ec(ip-10-83-78-215.ec2.internal) mtu=1376
-> 10.83.45.86:6783      established fastdp b2:8f:dd:99:f4:ae(ip-10-83-45-86.ec2.internal) mtu=1376
-> 10.83.61.125:6783     established fastdp 46:1b:b4:25:cf:87(ip-10-83-61-125.ec2.internal) mtu=1376
<- 10.83.125.214:25229   established fastdp 16:fb:51:d2:5c:25(ip-10-83-125-214.ec2.internal) mtu=1376
-> 10.83.42.111:6783     established fastdp 12:84:ba:e1:47:79(ip-10-83-42-111.ec2.internal) mtu=1376
-> 10.83.60.42:6783      established fastdp 7e:44:ff:00:f5:f9(ip-10-83-60-42.ec2.internal) mtu=1376
-> 10.83.114.127:6783    established fastdp d6:4a:51:67:64:13(ip-10-83-114-127.ec2.internal) mtu=1376
-> 10.83.76.130:6783     established fastdp 9a:55:19:b7:97:7c(ip-10-83-76-130.ec2.internal) mtu=1376
-> 10.83.80.99:6783      established fastdp 2a:d6:a8:e7:2e:de(ip-10-83-80-99.ec2.internal) mtu=1376
-> 10.83.98.46:6783      failed      Inconsistent entries for 100.105.64.0: owned by fa:17:fc:80:d1:6c but incoming message says 6e:89:89:29:40:32, retry: 2018-06-14 11:29:15.966023365 +0000 UTC m=+83969.896079258
-> 10.83.59.237:6783     failed      cannot connect to ourself, retry: never
-> 10.83.58.17:6783      failed      write tcp4 10.83.59.237:31865->10.83.58.17:6783: write: connection reset by peer, retry: 2018-06-14 11:29:29.378304351 +0000 UTC m=+83983.308360281
-> 10.83.126.221:6783    failed      dial tcp4 :0->10.83.126.221:6783: connect: connection timed out, retry: 2018-06-14 11:30:15.868698158 +0000 UTC m=+84029.798754184
Running 'weave --local status connections' on weave-net-qbl5b
----------------------------------------------
Defaulting container name to weave.
Use 'kubectl describe pod/weave-net-qbl5b -n kube-system' to see all of the containers in this pod.
<- 10.83.109.188:28757   established fastdp 9a:5d:d4:e9:78:29(ip-10-83-109-188.ec2.internal) mtu=1376
<- 10.83.59.237:44165    established fastdp 22:e1:23:b2:2f:8c(ip-10-83-59-237.ec2.internal) mtu=1376
<- 10.83.76.130:61619    established fastdp 9a:55:19:b7:97:7c(ip-10-83-76-130.ec2.internal) mtu=1376
<- 10.83.108.123:39192   established fastdp fa:17:fc:80:d1:6c(ip-10-83-108-123.ec2.internal) mtu=1376
-> 10.83.88.180:6783     established fastdp ae:d0:02:a5:ff:b4(ip-10-83-88-180.ec2.internal) mtu=1376
-> 10.83.45.86:6783      established fastdp b2:8f:dd:99:f4:ae(ip-10-83-45-86.ec2.internal) mtu=1376
<- 10.83.42.111:19399    established fastdp 12:84:ba:e1:47:79(ip-10-83-42-111.ec2.internal) mtu=1376
<- 10.83.87.92:56853     established fastdp 0e:8c:f8:33:67:7f(ip-10-83-87-92.ec2.internal) mtu=1376
-> 10.83.78.215:6783     established fastdp ee:c9:d9:9b:bd:ec(ip-10-83-78-215.ec2.internal) mtu=1376
<- 10.83.125.214:23211   established fastdp 16:fb:51:d2:5c:25(ip-10-83-125-214.ec2.internal) mtu=1376
<- 10.83.41.74:22143     established fastdp 16:d8:40:86:d6:bf(ip-10-83-41-74.ec2.internal) mtu=1376
<- 10.83.80.99:46426     established fastdp 2a:d6:a8:e7:2e:de(ip-10-83-80-99.ec2.internal) mtu=1376
<- 10.83.60.42:51528     established fastdp 7e:44:ff:00:f5:f9(ip-10-83-60-42.ec2.internal) mtu=1376
<- 10.83.61.125:12704    established fastdp 46:1b:b4:25:cf:87(ip-10-83-61-125.ec2.internal) mtu=1376
-> 10.83.98.46:6783      failed      Inconsistent entries for 100.105.64.0: owned by fa:17:fc:80:d1:6c but incoming message says 6e:89:89:29:40:32, retry: 2018-06-14 11:27:22.117036827 +0000 UTC m=+84070.657088638
-> 10.83.114.127:6783    failed      cannot connect to ourself, retry: never
-> 10.83.126.221:6783    failed      dial tcp4 :0->10.83.126.221:6783: connect: no route to host, retry: 2018-06-14 11:27:19.348685417 +0000 UTC m=+84067.888737367
-> 10.83.58.17:6783      failed      Inconsistent entries for 100.96.36.0: owned by fa:17:fc:80:d1:6c but incoming message says 6e:89:89:29:40:32, retry: 2018-06-14 11:27:07.322506294 +0000 UTC m=+84055.862558090
Running 'weave --local status connections' on weave-net-qh7s5
----------------------------------------------
Defaulting container name to weave.
Use 'kubectl describe pod/weave-net-qh7s5 -n kube-system' to see all of the containers in this pod.
-> 10.83.114.127:6783    established fastdp d6:4a:51:67:64:13(ip-10-83-114-127.ec2.internal) mtu=1376
-> 10.83.59.237:6783     established fastdp 22:e1:23:b2:2f:8c(ip-10-83-59-237.ec2.internal) mtu=1376
-> 10.83.60.42:6783      established fastdp 7e:44:ff:00:f5:f9(ip-10-83-60-42.ec2.internal) mtu=1376
-> 10.83.80.99:6783      established fastdp 2a:d6:a8:e7:2e:de(ip-10-83-80-99.ec2.internal) mtu=1376
<- 10.83.108.123:54931   established fastdp fa:17:fc:80:d1:6c(ip-10-83-108-123.ec2.internal) mtu=1376
-> 10.83.88.180:6783     established fastdp ae:d0:02:a5:ff:b4(ip-10-83-88-180.ec2.internal) mtu=1376
<- 10.83.41.74:21263     established fastdp 16:d8:40:86:d6:bf(ip-10-83-41-74.ec2.internal) mtu=1376
-> 10.83.61.125:6783     established fastdp 46:1b:b4:25:cf:87(ip-10-83-61-125.ec2.internal) mtu=1376
<- 10.83.76.130:45151    established fastdp 9a:55:19:b7:97:7c(ip-10-83-76-130.ec2.internal) mtu=1376
-> 10.83.109.188:6783    established fastdp 9a:5d:d4:e9:78:29(ip-10-83-109-188.ec2.internal) mtu=1376
-> 10.83.45.86:6783      established fastdp b2:8f:dd:99:f4:ae(ip-10-83-45-86.ec2.internal) mtu=1376
-> 10.83.42.111:6783     established fastdp 12:84:ba:e1:47:79(ip-10-83-42-111.ec2.internal) mtu=1376
-> 10.83.78.215:6783     established fastdp ee:c9:d9:9b:bd:ec(ip-10-83-78-215.ec2.internal) mtu=1376
-> 10.83.87.92:6783      established fastdp 0e:8c:f8:33:67:7f(ip-10-83-87-92.ec2.internal) mtu=1376
-> 10.83.98.46:6783      failed      Inconsistent entries for 100.105.64.0: owned by fa:17:fc:80:d1:6c but incoming message says 6e:89:89:29:40:32, retry: 2018-06-14 11:27:37.770269076 +0000 UTC m=+83869.405886652
-> 10.83.125.214:6783    failed      cannot connect to ourself, retry: never
-> 10.83.126.221:6783    failed      dial tcp4 :0->10.83.126.221:6783: connect: no route to host, retry: 2018-06-14 11:27:04.881744793 +0000 UTC m=+83836.517362462
-> 10.83.58.17:6783      failed      Inconsistent entries for 100.96.36.0: owned by fa:17:fc:80:d1:6c but incoming message says 6e:89:89:29:40:32, retry: 2018-06-14 11:32:41.52100929 +0000 UTC m=+84173.156626951
Running 'weave --local status connections' on weave-net-w9w9s
----------------------------------------------
Defaulting container name to weave.
Use 'kubectl describe pod/weave-net-w9w9s -n kube-system' to see all of the containers in this pod.
<- 10.83.109.188:48502   established fastdp 9a:5d:d4:e9:78:29(ip-10-83-109-188.ec2.internal) mtu=1376
<- 10.83.87.92:45114     established fastdp 0e:8c:f8:33:67:7f(ip-10-83-87-92.ec2.internal) mtu=1376
<- 10.83.60.42:17872     established fastdp 7e:44:ff:00:f5:f9(ip-10-83-60-42.ec2.internal) mtu=1376
<- 10.83.59.237:27796    established fastdp 22:e1:23:b2:2f:8c(ip-10-83-59-237.ec2.internal) mtu=1376
<- 10.83.42.111:13949    established fastdp 12:84:ba:e1:47:79(ip-10-83-42-111.ec2.internal) mtu=1376
-> 10.83.78.215:6783     established fastdp ee:c9:d9:9b:bd:ec(ip-10-83-78-215.ec2.internal) mtu=1376
<- 10.83.88.180:24330    established fastdp ae:d0:02:a5:ff:b4(ip-10-83-88-180.ec2.internal) mtu=1376
<- 10.83.61.125:45596    established fastdp 46:1b:b4:25:cf:87(ip-10-83-61-125.ec2.internal) mtu=1376
<- 10.83.108.123:57311   established fastdp fa:17:fc:80:d1:6c(ip-10-83-108-123.ec2.internal) mtu=1376
<- 10.83.114.127:64346   established fastdp d6:4a:51:67:64:13(ip-10-83-114-127.ec2.internal) mtu=1376
<- 10.83.125.214:12791   established fastdp 16:fb:51:d2:5c:25(ip-10-83-125-214.ec2.internal) mtu=1376
<- 10.83.41.74:56660     established fastdp 16:d8:40:86:d6:bf(ip-10-83-41-74.ec2.internal) mtu=1376
<- 10.83.80.99:54719     established fastdp 2a:d6:a8:e7:2e:de(ip-10-83-80-99.ec2.internal) mtu=1376
<- 10.83.76.130:58535    established fastdp 9a:55:19:b7:97:7c(ip-10-83-76-130.ec2.internal) mtu=1376
-> 10.83.98.46:6783      failed      Inconsistent entries for 100.105.64.0: owned by fa:17:fc:80:d1:6c but incoming message says 6e:89:89:29:40:32, retry: 2018-06-14 11:26:41.285648311 +0000 UTC m=+84034.787572182
-> 10.83.126.221:6783    failed      dial tcp4 :0->10.83.126.221:6783: connect: connection timed out, retry: 2018-06-14 11:29:43.146954682 +0000 UTC m=+84216.648878636
-> 10.83.45.86:6783      failed      cannot connect to ourself, retry: never
-> 10.83.58.17:6783      failed      Inconsistent entries for 100.96.36.0: owned by fa:17:fc:80:d1:6c but incoming message says 6e:89:89:29:40:32, retry: 2018-06-14 11:30:13.058231491 +0000 UTC m=+84246.560155285
Running 'weave --local status connections' on weave-net-wvlnc
----------------------------------------------
Defaulting container name to weave.
Use 'kubectl describe pod/weave-net-wvlnc -n kube-system' to see all of the containers in this pod.
-> 10.83.45.86:6783      established fastdp b2:8f:dd:99:f4:ae(ip-10-83-45-86.ec2.internal) mtu=1376
-> 10.83.76.130:6783     established fastdp 9a:55:19:b7:97:7c(ip-10-83-76-130.ec2.internal) mtu=1376
-> 10.83.78.215:6783     established fastdp ee:c9:d9:9b:bd:ec(ip-10-83-78-215.ec2.internal) mtu=1376
<- 10.83.125.214:39681   established fastdp 16:fb:51:d2:5c:25(ip-10-83-125-214.ec2.internal) mtu=1376
<- 10.83.80.99:37240     established fastdp 2a:d6:a8:e7:2e:de(ip-10-83-80-99.ec2.internal) mtu=1376
-> 10.83.59.237:6783     established fastdp 22:e1:23:b2:2f:8c(ip-10-83-59-237.ec2.internal) mtu=1376
-> 10.83.88.180:6783     established fastdp ae:d0:02:a5:ff:b4(ip-10-83-88-180.ec2.internal) mtu=1376
-> 10.83.41.74:6783      established fastdp 16:d8:40:86:d6:bf(ip-10-83-41-74.ec2.internal) mtu=1376
<- 10.83.87.92:55814     established fastdp 0e:8c:f8:33:67:7f(ip-10-83-87-92.ec2.internal) mtu=1376
<- 10.83.114.127:12361   established fastdp d6:4a:51:67:64:13(ip-10-83-114-127.ec2.internal) mtu=1376
-> 10.83.109.188:6783    established fastdp 9a:5d:d4:e9:78:29(ip-10-83-109-188.ec2.internal) mtu=1376
-> 10.83.42.111:6783     established fastdp 12:84:ba:e1:47:79(ip-10-83-42-111.ec2.internal) mtu=1376
-> 10.83.60.42:6783      established fastdp 7e:44:ff:00:f5:f9(ip-10-83-60-42.ec2.internal) mtu=1376
<- 10.83.108.123:52011   established fastdp fa:17:fc:80:d1:6c(ip-10-83-108-123.ec2.internal) mtu=1376
-> 10.83.98.46:6783      failed      Inconsistent entries for 100.97.0.36: owned by 3a:47:b8:a9:34:ef but incoming message says 2a:20:75:a8:38:e9, retry: 2018-06-14 11:27:24.566478723 +0000 UTC m=+78938.833355396
-> 10.83.89.115:6783     failed      dial tcp4 :0->10.83.89.115:6783: connect: connection timed out, retry: 2018-06-14 11:31:28.361942045 +0000 UTC m=+79182.628818762
-> 10.83.58.17:6783      failed      read tcp4 10.83.61.125:36602->10.83.58.17:6783: read: connection reset by peer, retry: 2018-06-14 11:34:34.458989108 +0000 UTC m=+79368.725865781
-> 10.83.61.125:6783     failed      cannot connect to ourself, retry: never
-> 10.83.108.81:6783     failed      dial tcp4 :0->10.83.108.81:6783: connect: connection timed out, retry: 2018-06-14 11:28:21.152854771 +0000 UTC m=+78995.419731614
Running 'weave --local status connections' on weave-net-zhrmd
----------------------------------------------
Defaulting container name to weave.
Use 'kubectl describe pod/weave-net-zhrmd -n kube-system' to see all of the containers in this pod.
-> 10.83.60.42:6783      established fastdp 7e:44:ff:00:f5:f9(ip-10-83-60-42.ec2.internal) mtu=1376
-> 10.83.59.237:6783     established fastdp 22:e1:23:b2:2f:8c(ip-10-83-59-237.ec2.internal) mtu=1376
-> 10.83.114.127:6783    established fastdp d6:4a:51:67:64:13(ip-10-83-114-127.ec2.internal) mtu=1376
-> 10.83.88.180:6783     established fastdp ae:d0:02:a5:ff:b4(ip-10-83-88-180.ec2.internal) mtu=1376
<- 10.83.87.92:27093     established fastdp 0e:8c:f8:33:67:7f(ip-10-83-87-92.ec2.internal) mtu=1376
-> 10.83.45.86:6783      established fastdp b2:8f:dd:99:f4:ae(ip-10-83-45-86.ec2.internal) mtu=1376
<- 10.83.78.215:48104    established fastdp ee:c9:d9:9b:bd:ec(ip-10-83-78-215.ec2.internal) mtu=1376
-> 10.83.125.214:6783    established fastdp 16:fb:51:d2:5c:25(ip-10-83-125-214.ec2.internal) mtu=1376
-> 10.83.76.130:6783     established fastdp 9a:55:19:b7:97:7c(ip-10-83-76-130.ec2.internal) mtu=1376
-> 10.83.109.188:6783    established fastdp 9a:5d:d4:e9:78:29(ip-10-83-109-188.ec2.internal) mtu=1376
<- 10.83.80.99:25749     established fastdp 2a:d6:a8:e7:2e:de(ip-10-83-80-99.ec2.internal) mtu=1376
<- 10.83.41.74:20169     established fastdp 16:d8:40:86:d6:bf(ip-10-83-41-74.ec2.internal) mtu=1376
-> 10.83.61.125:6783     established fastdp 46:1b:b4:25:cf:87(ip-10-83-61-125.ec2.internal) mtu=1376
<- 10.83.42.111:22124    established fastdp 12:84:ba:e1:47:79(ip-10-83-42-111.ec2.internal) mtu=1376
-> 10.83.108.123:6783    failed      cannot connect to ourself, retry: never
-> 10.83.98.46:6783      failed      Received update for IP range I own at 100.96.36.0 v4: incoming message says owner 46:1b:b4:25:cf:87 v5, retry: 2018-06-14 11:30:58.746286293 +0000 UTC m=+81575.949094494
-> 10.83.58.17:6783      failed      Inconsistent entries for 100.96.36.0: owned by fa:17:fc:80:d1:6c but incoming message says 6e:89:89:29:40:32, retry: 2018-06-14 11:26:38.271243376 +0000 UTC m=+81315.474051605
itskingori picture itskingori on 14 Jun 2018

@itskingori thanks; we really would like logs (as in kubectl logs ... weave) to debug.

If you have cleared a pod's state and restarted it and then it goes bad within the day then there is a good chance the logs do have the vital information. If the pod has restarted please fetch the previous logs too with kubectl logs -p.

To be clear, in case of message -> 10.83.98.46:6783 failed Received update for IP ... message says owner 46:1b:b4:25:cf:87 v5, we would like the logs for the pod that gives this message and for the pod with ID 46:1b:b4:25:cf:87

bboreham picture bboreham on 14 Jun 2018

@bboreham Got it! Will keep you guys posted.

itskingori picture itskingori on 15 Jun 2018

@bboreham @brb While it hasn't happened again, I've done some investigation and I'd like to pass a theory past you two. Could it be as a result of the pods being abruptly killed .e.g. an OOM?

In the case of https://github.com/weaveworks/weave/issues/3310#issuecomment-397263915, we cleared the state of weave-net-h5wcj and weave-net-l2c8d, then the cluster healed.

You can see that weave-net-h5wcj was killed at around 13:55, weave-net-l2c8d at around 13:59 ... 👇

screen shot 2018-06-15 at 03 54 30

We installed via kops, therefore our resource requests and limits are set to 👇 

resources:
  limits:
    memory: 200Mi
  requests:
    cpu: 50m
    memory: 200Mi

Here are the logs of weave-net-h5wcj at around 13:55 👇

screen shot 2018-06-15 at 04 05 46

And here are the logs of weave-net-l2c8d at around 13:59 👇

screen shot 2018-06-15 at 04 11 24

You can see the logging profile of weave-net-h5wcj around the kill below 👇

screen shot 2018-06-15 at 04 21 42

And the logging profile of weave-net-l2c8d around the kill below 👇

screen shot 2018-06-15 at 04 20 04

So:

  1. Those kills definitely affected weave.
  2. Is there a possibility that the state was affected by the kill?
  3. Should we increase memory assigned to weave?
itskingori picture itskingori on 15 Jun 2018

@bboreham @brb Another theory, could it be that the infinite loop fixed in https://github.com/weaveworks/weave/pull/3317 is causing the weave pod to use more and more memory? See 30 days usage below:

screen shot 2018-06-15 at 10 24 32

My colleague @tsu-shiuan, thinks that increasing memory would be hiding the problem ... and that while the OOMs are likely the cause of the issue described in https://github.com/weaveworks/weave/issues/3310#issuecomment-397486312, that there's an underlying problem causing the increase in memory.

Some points to note:

  1. We upgraded weave to 2.3.0 about 18 days ago i.e. around May 28th.
  2. Before that we have upgraded weave to 2.2.0 about 4 months ago.

The disruption right after the 28th is us rotating nodes, so that's expected ... but you can see after that things are not looking good.

itskingori picture itskingori on 15 Jun 2018
👍1

@zacblazic @itskingori Thanks for the info.

This only started happening after upgrading to 2.3.0. We never had this issue with 2.2.0.

I doubt that any change from 2.2.0 to 2.3.0 could have triggered the problem (maybe "Build Weave Net with Go 1.10.1 #3273", but let's keep it as a last resort). My guess is that the upgrade restarted Weave Net on each node which triggered the faulty reclaim DEBU: 2018/06/05 11:20:29.571731 [kube-peers] Preparing to remove disappeared peer {46:f4:4b:41:dd:11 ip-10-83-124-112.ec2.internal}, as reclaiming happens upon the Weave Net initialization.

Regarding the second issue, is there a way to manually remove the annotation or clear this state?

I don't have a cmd at hand, but try updating the annotation with kubectl annotate configmap weave-net -n=kube-system --overwrite kube-peers.weave.works/peers='$PEERS' where $PEERS is the existing annotation value (you can get it with kubectl get configmap weave-net ... -o yaml) minus the faulty peer. The annotation format is JSON. After the change, you need to restart Weave on nodes which are in the infinite loop.

Anyway, do you have any logs for ip-10-83-124-112.ec2.internal?

Could it be as a result of the pods being abruptly killed .e.g. an OOM?

Might be that boltdb which stores IPAM fails to write to disk as the process gets killed. So, after the restart Weave Net restores a stale data (@bboreham shouldn't the CRDT IPAM ring be able to merge in such situation anyway?).

Interested in your logs after you have reproduced the issue. Just please upload them as txt to make grepping possible. Ta.

brb picture brb on 17 Jun 2018

I can see exactly same kind of error Received update for IP range I own but simply when I try to add another node to a cluster. The strange thing is that it does not happen always.

kuznero picture kuznero on 4 Jul 2018

@kuznero please open your own issue and supply logs.

bboreham picture bboreham on 4 Jul 2018

(@bboreham shouldn't the CRDT IPAM ring be able to merge in such situation anyway?).

@brb it can merge consistent data of any age. The problem here is that the data is inconsistent with what someone else has.

Issue #2084 covers the case where the inconsistency doesn't actually matter.

bboreham picture bboreham on 4 Jul 2018

OK, I didn't read the whole question, sorry.

Could it be as a result of the pods being abruptly killed .e.g. an OOM?

Might be that boltdb which stores IPAM fails to write to disk as the process gets killed. So, after the restart Weave Net restores a stale data

Something like this sequence?

  • a donates IP range r to b
  • a dies and restarts, loads previous state
  • both a and b think they own r

To avoid this, we persist to disk before sending the message from a to b, and that goes through to a fsync syscall.

I guess it's possible there is a bug in the above. Or it's something else.

bboreham picture bboreham on 4 Jul 2018

@zacblazic @itskingori @kuznero What file system (with options) is of the
/var/lib/weave host dir? mount can show this information.

@kuznero Can you easily reproduce the issue?

brb picture brb on 9 Jul 2018

Interested in your logs after you have reproduced the issue. Just please upload them as txt to make grepping possible. Ta.

@bboreham @brb I think I might have the logs of a pod that's just started and is exhibiting this behaviour.

You can see that it started slightly past 23:25 👇

screen shot 2018-08-29 at 00 00 29

And started getting the "Received update for IP range I own" error a little after 👇

screen shot 2018-08-29 at 00 01 00

These are the full logs 👉 weave-net-z5lmp.log

Yet it's connections look ok 👇 

$ kubectl exec -it weave-net-z5lmp -n kube-system /bin/sh
Defaulting container name to weave.
Use 'kubectl describe pod/weave-net-z5lmp -n kube-system' to see all of the containers in this pod.
/home/weave # ./weave --local status connections
-> 10.80.102.195:6783    established fastdp 56:4c:56:6f:f5:90(ip-10-80-102-195.ec2.internal) mtu=1376
<- 10.80.67.160:32281    established fastdp 3e:8f:f3:15:0c:e6(ip-10-80-67-160.ec2.internal) mtu=1376
-> 10.80.45.255:6783     established fastdp 6a:7d:84:55:63:40(ip-10-80-45-255.ec2.internal) mtu=1376
-> 10.80.110.250:6783    established fastdp 02:8f:3d:02:22:b2(ip-10-80-110-250.ec2.internal) mtu=1376
-> 10.80.91.69:6783      established fastdp a2:89:81:f0:f7:cb(ip-10-80-91-69.ec2.internal) mtu=1376
-> 10.80.112.109:6783    established fastdp 72:d3:22:05:8f:08(ip-10-80-112-109.ec2.internal) mtu=1376
-> 10.80.66.231:6783     established fastdp 52:ff:e3:6c:74:1d(ip-10-80-66-231.ec2.internal) mtu=1376
<- 10.80.112.120:28928   established fastdp be:10:d6:f0:eb:68(ip-10-80-112-120.ec2.internal) mtu=1376
-> 10.80.90.166:6783     established fastdp d2:fe:ff:e6:ca:a7(ip-10-80-90-166.ec2.internal) mtu=1376
<- 10.80.61.78:40629     established fastdp c2:05:23:81:b9:b1(ip-10-80-61-78.ec2.internal) mtu=1376
<- 10.80.44.252:13551    established fastdp 32:c1:c8:45:6e:0d(ip-10-80-44-252.ec2.internal) mtu=1376
-> 10.80.53.166:6783     established fastdp 9a:6b:6f:33:1c:60(ip-10-80-53-166.ec2.internal) mtu=1376
<- 10.80.89.245:49733    established fastdp 0a:e3:03:b2:57:8c(ip-10-80-89-245.ec2.internal) mtu=1376
-> 10.80.59.37:6783      established fastdp 9a:49:8b:6b:a8:12(ip-10-80-59-37.ec2.internal) mtu=1376
-> 10.80.67.172:6783     established fastdp 4a:f9:d0:15:61:a7(ip-10-80-67-172.ec2.internal) mtu=1376
-> 10.80.65.216:6783     established fastdp 42:90:28:9a:2c:a2(ip-10-80-65-216.ec2.internal) mtu=1376
<- 10.80.114.8:52196     established fastdp 6e:0e:51:4f:66:9e(ip-10-80-114-8.ec2.internal) mtu=1376
<- 10.80.96.210:25680    established fastdp 96:36:1a:d1:59:07(ip-10-80-96-210.ec2.internal) mtu=1376
-> 10.80.41.90:6783      established fastdp a2:63:98:df:ee:d0(ip-10-80-41-90.ec2.internal) mtu=1376
-> 10.80.120.250:6783    established fastdp e6:19:2c:86:42:27(ip-10-80-120-250.ec2.internal) mtu=1376
-> 10.80.125.5:6783      established fastdp 8e:b1:f9:dc:6b:31(ip-10-80-125-5.ec2.internal) mtu=1376
-> 10.80.54.76:6783      established fastdp b2:ef:a5:21:f9:a3(ip-10-80-54-76.ec2.internal) mtu=1376
-> 10.80.37.248:6783     failed      cannot connect to ourself, retry: never

And ipam status doesn't look too bad 👇 

$ kubectl exec -it weave-net-z5lmp -n kube-system /bin/sh
Defaulting container name to weave.
Use 'kubectl describe pod/weave-net-z5lmp -n kube-system' to see all of the containers in this pod.
/home/weave # ./weave --local status ipam
ca:b9:26:99:63:b4(ip-10-80-37-248.ec2.internal)   135880 IPs (06.5% of total) (4 active)
d6:c3:9a:8e:74:f0(ip-10-80-105-52.ec2.internal)    65536 IPs (03.1% of total) - unreachable!
8a:23:63:a2:3b:04(ip-10-80-112-98.ec2.internal)    32768 IPs (01.6% of total) - unreachable!
0a:ed:0e:50:ed:f9(ip-10-80-108-104.ec2.internal)    65536 IPs (03.1% of total) - unreachable!
6a:7d:84:55:63:40(ip-10-80-45-255.ec2.internal)   108236 IPs (05.2% of total)
a2:89:81:f0:f7:cb(ip-10-80-91-69.ec2.internal)    91695 IPs (04.4% of total)
9a:6b:6f:33:1c:60(ip-10-80-53-166.ec2.internal)    47608 IPs (02.3% of total)
a2:63:98:df:ee:d0(ip-10-80-41-90.ec2.internal)    44136 IPs (02.1% of total)
e2:09:61:37:d0:9e(ip-10-80-46-186.ec2.internal)    65536 IPs (03.1% of total) - unreachable!
5e:80:0c:4b:9a:41(ip-10-80-120-27.ec2.internal)     8192 IPs (00.4% of total) - unreachable!
c2:05:23:81:b9:b1(ip-10-80-61-78.ec2.internal)       83 IPs (00.0% of total)
5a:47:78:0d:74:53(ip-10-80-104-144.ec2.internal)    32768 IPs (01.6% of total) - unreachable!
56:4c:56:6f:f5:90(ip-10-80-102-195.ec2.internal)    88952 IPs (04.2% of total)
52:ff:e3:6c:74:1d(ip-10-80-66-231.ec2.internal)   106296 IPs (05.1% of total)
e6:19:2c:86:42:27(ip-10-80-120-250.ec2.internal)    32768 IPs (01.6% of total)
8e:b1:f9:dc:6b:31(ip-10-80-125-5.ec2.internal)     8192 IPs (00.4% of total)
ba:5f:5d:8e:c5:49(ip-10-80-102-17.ec2.internal)    16384 IPs (00.8% of total) - unreachable!
62:49:f3:21:c1:b3(ip-10-80-57-2.ec2.internal)    32768 IPs (01.6% of total) - unreachable!
7a:26:75:22:4c:a3(ip-10-80-55-0.ec2.internal)    65536 IPs (03.1% of total) - unreachable!
02:f2:c2:33:21:a8(ip-10-80-89-194.ec2.internal)    32768 IPs (01.6% of total) - unreachable!
72:d3:22:05:8f:08(ip-10-80-112-109.ec2.internal)    65536 IPs (03.1% of total)
62:4b:82:bc:ee:4a(ip-10-80-106-92.ec2.internal)    32768 IPs (01.6% of total) - unreachable!
c2:08:11:3c:6b:02(ip-10-80-44-87.ec2.internal)    65536 IPs (03.1% of total) - unreachable!
3e:8f:f3:15:0c:e6(ip-10-80-67-160.ec2.internal)       24 IPs (00.0% of total)
16:f0:47:37:8a:18(ip-10-80-101-176.ec2.internal)    65536 IPs (03.1% of total) - unreachable!
fa:d3:e4:a1:47:54(ip-10-80-74-138.ec2.internal)    32768 IPs (01.6% of total) - unreachable!
0a:e3:03:b2:57:8c(ip-10-80-89-245.ec2.internal)    56116 IPs (02.7% of total)
be:10:d6:f0:eb:68(ip-10-80-112-120.ec2.internal)    89798 IPs (04.3% of total)
6e:0e:51:4f:66:9e(ip-10-80-114-8.ec2.internal)   115398 IPs (05.5% of total)
9a:49:8b:6b:a8:12(ip-10-80-59-37.ec2.internal)    98216 IPs (04.7% of total)
4a:f9:d0:15:61:a7(ip-10-80-67-172.ec2.internal)     8304 IPs (00.4% of total)
42:90:28:9a:2c:a2(ip-10-80-65-216.ec2.internal)    52000 IPs (02.5% of total)
32:c1:c8:45:6e:0d(ip-10-80-44-252.ec2.internal)    16384 IPs (00.8% of total)
ea:0c:50:90:93:c3(ip-10-80-46-212.ec2.internal)    16384 IPs (00.8% of total) - unreachable!
9e:ce:d8:bd:e6:8d(ip-10-80-124-112.ec2.internal)    32768 IPs (01.6% of total) - unreachable!
96:36:1a:d1:59:07(ip-10-80-96-210.ec2.internal)    16384 IPs (00.8% of total)
b2:ef:a5:21:f9:a3(ip-10-80-54-76.ec2.internal)    87747 IPs (04.2% of total)
02:8f:3d:02:22:b2(ip-10-80-110-250.ec2.internal)    65543 IPs (03.1% of total)
fe:0f:36:cd:88:d8(ip-10-80-40-99.ec2.internal)    16384 IPs (00.8% of total) - unreachable!
1a:d9:db:f0:74:a4(ip-10-80-63-127.ec2.internal)    65536 IPs (03.1% of total) - unreachable!
26:be:94:88:07:51(ip-10-80-63-133.ec2.internal)    16384 IPs (00.8% of total) - unreachable!

What file system (with options) is of the
/var/lib/weave host dir? mount can show this information.

Host filesystem 👇 

$ mount | grep /dev/xvda1
/dev/xvda1 on / type ext4 (rw,relatime,data=ordered)
/dev/xvda1 on /var/lib/docker/overlay2 type ext4 (rw,relatime,data=ordered)
/dev/xvda1 on /var/lib/kubelet/pods/a0ea4d18-ab00-11e8-ac1a-06f70fba1e78/volume-subpaths/telegraf-configuration/telegraf/0 type ext4 (ro,relatime,data=ordered)
/dev/xvda1 on /var/lib/kubelet/pods/a0e24f65-ab00-11e8-ac1a-06f70fba1e78/volume-subpaths/filebeat-configuration/filebeat/2 type ext4 (ro,relatime,data=ordered)
itskingori picture itskingori on 28 Aug 2018

I don't have a cmd at hand, but try updating the annotation with kubectl annotate configmap weave-net -n=kube-system --overwrite kube-peers.weave.works/peers='$PEERS' where $PEERS is the existing annotation value (you can get it with kubectl get configmap weave-net ... -o yaml) minus the faulty peer. The annotation format is JSON. After the change, you need to restart Weave on nodes which are in the infinite loop.

@brb Sorry that I'm so late on this, but I've got some interesting findings.

$ kubectl get configmap -n=kube-system -oyaml weave-net
...
{
  "Peers": [
    {
      "PeerName": "da:3e:d4:a5:e9:42",
      "NodeName": "ip-10-80-78-181.ec2.internal"
    },
    {
      "PeerName": "0a:e3:03:b2:57:8c",
      "NodeName": "ip-10-80-89-245.ec2.internal"
    },
    {
      "PeerName": "9a:49:8b:6b:a8:12",
      "NodeName": "ip-10-80-59-37.ec2.internal"
    },
    {
      "PeerName": "56:4c:56:6f:f5:90",
      "NodeName": "ip-10-80-102-195.ec2.internal"
    },
    {
      "PeerName": "9a:6b:6f:33:1c:60",
      "NodeName": "ip-10-80-53-166.ec2.internal"
    },
    {
      "PeerName": "a2:89:81:f0:f7:cb",
      "NodeName": "ip-10-80-91-69.ec2.internal"
    },
    {
      "PeerName": "be:10:d6:f0:eb:68",
      "NodeName": "ip-10-80-112-120.ec2.internal"
    },
    {
      "PeerName": "a2:63:98:df:ee:d0",
      "NodeName": "ip-10-80-41-90.ec2.internal"
    },
    {
      "PeerName": "6e:0e:51:4f:66:9e",
      "NodeName": "ip-10-80-114-8.ec2.internal"
    },
    {
      "PeerName": "6a:7d:84:55:63:40",
      "NodeName": "ip-10-80-45-255.ec2.internal"
    },
    {
      "PeerName": "3e:8f:f3:15:0c:e6",
      "NodeName": "ip-10-80-67-160.ec2.internal"
    },
    {
      "PeerName": "b2:ef:a5:21:f9:a3",
      "NodeName": "ip-10-80-54-76.ec2.internal"
    },
    {
      "PeerName": "52:ff:e3:6c:74:1d",
      "NodeName": "ip-10-80-66-231.ec2.internal"
    },
    {
      "PeerName": "42:90:28:9a:2c:a2",
      "NodeName": "ip-10-80-65-216.ec2.internal"
    },
    {
      "PeerName": "4a:f9:d0:15:61:a7",
      "NodeName": "ip-10-80-67-172.ec2.internal"
    },
    {
      "PeerName": "02:8f:3d:02:22:b2",
      "NodeName": "ip-10-80-110-250.ec2.internal"
    },
    {
      "PeerName": "ca:5d:93:ce:43:59",
      "NodeName": "ip-10-80-103-14.ec2.internal"
    }
  ]
}

I'm not sure how this config map is used by weave, assuming as some sort of state storage, but I expected the number of peers in the annotation to be near to the number of nodes in the cluster, which it wasn't at all. Nodes in cluster at the time was 20 while the number of peers in that list is 17.

Below I've attempted to map the peers in the annotation to nodes in the cluster:

# Missing node included in configmap annotation
ip-10-80-78-181.ec2.internal

# Existing nodes correctly in configmap annotation
ip-10-80-89-245.ec2.internal  (reachable)
ip-10-80-59-37.ec2.internal   (reachable)
ip-10-80-102-195.ec2.internal (reachable)
ip-10-80-53-166.ec2.internal  (reachable)
ip-10-80-91-69.ec2.internal   (reachable)
ip-10-80-112-120.ec2.internal (reachable)
ip-10-80-41-90.ec2.internal   (reachable)
ip-10-80-114-8.ec2.internal   (reachable)
ip-10-80-45-255.ec2.internal  (reachable)
ip-10-80-67-160.ec2.internal  (reachable)
ip-10-80-54-76.ec2.internal   (reachable)
ip-10-80-66-231.ec2.internal  (reachable)
ip-10-80-65-216.ec2.internal  (reachable)
ip-10-80-67-172.ec2.internal  (reachable)
ip-10-80-110-250.ec2.internal (reachable)
ip-10-80-103-14.ec2.internal  (reachable)

# Existing nodes not in configmap annotation
ip-10-80-112-109.ec2.internal  (reachable)
ip-10-80-37-248.ec2.internal   (reachable)
ip-10-80-44-252.ec2.internal   (reachable)
ip-10-80-61-78.ec2.internal    (reachable)

Above we can see that ip-10-80-78-181.ec2.internal is the peer that needs to be removed, similarly to ip-10-83-124-112.ec2.internal in my previous example (though this is a different cluster): 

DEBU: 2018/08/28 21:17:08.953164 [kube-peers] Preparing to remove disappeared peer {da:3e:d4:a5:e9:42 ip-10-80-78-181.ec2.internal} | weave-net-6kkps
DEBU: 2018/08/28 21:17:08.953138 [kube-peers] Nodes that have disappeared: map[ip-10-80-78-181.ec2.internal:{da:3e:d4:a5:e9:42 ip-10-80-78-181.ec2.internal}] | weave-net-6kkps
DEBU: 2018/08/28 21:17:08.753070 [kube-peers] Nodes that have disappeared: map[ip-10-80-78-181.ec2.internal:{da:3e:d4:a5:e9:42 ip-10-80-78-181.ec2.internal}] | weave-net-6kkps
DEBU: 2018/08/28 21:17:08.753098 [kube-peers] Preparing to remove disappeared peer {da:3e:d4:a5:e9:42 ip-10-80-78-181.ec2.internal}

Secondly there are a few nodes that are missing from the peer list on the annotation, is this a bad thing?

I'm going to attempt updating the config map and will report back.

Edit: I'm aware of the fix (https://github.com/weaveworks/weave/pull/3317) added in 2.4.0. We've not upgraded to that version yet, but we will tomorrow.

zacblazic picture zacblazic on 29 Aug 2018

@itsingoria I opened a new issue #3386 to cover the "unreachable peers" part of your comment which is distinct to the subject of this issue.

bboreham picture bboreham on 29 Aug 2018

@bboreham Thank you. Just a note, Zac and I are on the same team. So if you need anything from us, let us know.

itskingori picture itskingori on 29 Aug 2018

@zacblazic

I'm not sure how this config map is used by weave

It's the answer to the question raised at the beginning of #2797 - "How do we even detect that a Weave Net peer originated as a Kubernetes node?". We add each peer to the map on first initialization, and remove it after cleanup.

Nodes in cluster at the time was 20 while the number of peers in that list is 17.

You can pick one of the missing 3 and look in its logfile at startup to see what happened when it should have added itself. There should be a log line like:

INFO: 2018/08/28 20:26:35.537339 [kube-peers] Added myself to peer list &{[{da:3e:d4:a5:e9:42 ip-10-80-78-181.ec2.internal} {0a:e3:03:b2:57:8c ip-10-80-89-245.ec2.internal} {9a:49:8b:6b:a8:12 ip-10-80-59-37.ec2.internal} {56:4c:56:6f:f5:90 ip-10-80-102-195.ec2.internal} {9a:6b:6f:33:1c:60 ip-10-80-53-166.ec2.internal} {a2:89:81:f0:f7:cb ip-10-80-91-69.ec2.internal} {be:10:d6:f0:eb:68 ip-10-80-112-120.ec2.internal} {a2:63:98:df:ee:d0 ip-10-80-41-90.ec2.internal} {6e:0e:51:4f:66:9e ip-10-80-114-8.ec2.internal} {96:be:47:91:4d:4b ip-10-80-91-173.ec2.internal} {6a:7d:84:55:63:40 ip-10-80-45-255.ec2.internal} {3e:8f:f3:15:0c:e6 ip-10-80-67-160.ec2.internal} {b2:ef:a5:21:f9:a3 ip-10-80-54-76.ec2.internal} {52:ff:e3:6c:74:1d ip-10-80-66-231.ec2.internal} {42:90:28:9a:2c:a2 ip-10-80-65-216.ec2.internal} {4a:f9:d0:15:61:a7 ip-10-80-67-172.ec2.internal} {02:8f:3d:02:22:b2 ip-10-80-110-250.ec2.internal} {e6:19:2c:86:42:27 ip-10-80-120-250.ec2.internal} {ca:b9:26:99:63:b4 ip-10-80-37-248.ec2.internal}]}

Secondly there are a few nodes that are missing from the peer list on the annotation, is this a bad thing?

It means they will not be cleaned up when the nodes are deleted from Kubernetes.

(I don't believe any of this is particularly related to "attempting to claim same IP range", which uses a different data structure)

bboreham picture bboreham on 29 Aug 2018

Looking at the unreachable peers from https://github.com/weaveworks/weave/issues/3310#issuecomment-416747709:

d6:c3:9a:8e:74:f0(ip-10-80-105-52.ec2.internal)    65536 IPs (03.1% of total) - unreachable!
8a:23:63:a2:3b:04(ip-10-80-112-98.ec2.internal)    32768 IPs (01.6% of total) - unreachable!
0a:ed:0e:50:ed:f9(ip-10-80-108-104.ec2.internal)    65536 IPs (03.1% of total) - unreachable!
e2:09:61:37:d0:9e(ip-10-80-46-186.ec2.internal)    65536 IPs (03.1% of total) - unreachable!
5e:80:0c:4b:9a:41(ip-10-80-120-27.ec2.internal)     8192 IPs (00.4% of total) - unreachable!
5a:47:78:0d:74:53(ip-10-80-104-144.ec2.internal)    32768 IPs (01.6% of total) - unreachable!
ba:5f:5d:8e:c5:49(ip-10-80-102-17.ec2.internal)    16384 IPs (00.8% of total) - unreachable!
62:49:f3:21:c1:b3(ip-10-80-57-2.ec2.internal)    32768 IPs (01.6% of total) - unreachable!
7a:26:75:22:4c:a3(ip-10-80-55-0.ec2.internal)    65536 IPs (03.1% of total) - unreachable!
02:f2:c2:33:21:a8(ip-10-80-89-194.ec2.internal)    32768 IPs (01.6% of total) - unreachable!
62:4b:82:bc:ee:4a(ip-10-80-106-92.ec2.internal)    32768 IPs (01.6% of total) - unreachable!
c2:08:11:3c:6b:02(ip-10-80-44-87.ec2.internal)    65536 IPs (03.1% of total) - unreachable!
16:f0:47:37:8a:18(ip-10-80-101-176.ec2.internal)    65536 IPs (03.1% of total) - unreachable!
fa:d3:e4:a1:47:54(ip-10-80-74-138.ec2.internal)    32768 IPs (01.6% of total) - unreachable!
ea:0c:50:90:93:c3(ip-10-80-46-212.ec2.internal)    16384 IPs (00.8% of total) - unreachable!
9e:ce:d8:bd:e6:8d(ip-10-80-124-112.ec2.internal)    32768 IPs (01.6% of total) - unreachable!
fe:0f:36:cd:88:d8(ip-10-80-40-99.ec2.internal)    16384 IPs (00.8% of total) - unreachable!
1a:d9:db:f0:74:a4(ip-10-80-63-127.ec2.internal)    65536 IPs (03.1% of total) - unreachable!
26:be:94:88:07:51(ip-10-80-63-133.ec2.internal)    16384 IPs (00.8% of total) - unreachable!

5e:80:0c:4b:9a:41 and fe:0f:36:cd:88:d8 appear in the log, but we can't talk to them because of inconsistent IPAM data:

INFO: 2018/08/28 20:49:48.124055 ->[10.80.120.27:6783|5e:80:0c:4b:9a:41(ip-10-80-120-27.ec2.internal)]: connection shutting down due to error: Received update for IP range I own at 100.96.0.31 v19: incoming message says owner e6:19:2c:86:42:27 v24
[...]
INFO: 2018/08/28 20:36:50.709044 ->[10.80.40.99:6783|fe:0f:36:cd:88:d8(ip-10-80-40-99.ec2.internal)]: connection shutting down due to error: Received update for IP range I own at 100.96.0.31 v19: incoming message says owner e6:19:2c:86:42:27 v20

The others don't appear at all. Perhaps they were cleaned up on a node the other side of the inconsistency partition, thus the knowledge of the cleanup didn't make it across the whole cluster?

bboreham picture bboreham on 29 Aug 2018

I did not see any logs in response to the request at https://github.com/weaveworks/weave/issues/3310#issuecomment-397335691. This is still important to discovering the root cause of this issue.

Also, I'm not sure if we said here what the cleanup procedure is:

To recover, you need to eliminate the IPAM data from the affected nodes and restart. Since you installed via the Kubernetes Addon, this data will be in a file under /var/lib/weave on the node - delete this file and restart the node.

Since you seem to have about 17 nodes that can talk to each other and 3 that can't talk to them, do this just on the 3. Restarting (rebooting) after removing the file ensures there are no IP addresses in use on the node, which makes the procedure safe.

bboreham picture bboreham on 29 Aug 2018

@bboreham I don't want to celebrate just yet but we've taken the following steps are seeing positive results:

  1. Upgraded to 2.4.0 to get the fix @brb mentions in https://github.com/weaveworks/weave/issues/3310#issuecomment-396558236, then ...
  2. Cleared any unreachable! on the ipam-list.
  3. Edited the peer-list annotation on the weave-net configmap to make sure the entries match with the ipam-list (there were entries on the peer-list that were not on the ipam-list as @zacblazic explained in https://github.com/weaveworks/weave/issues/3310#issuecomment-416756507)
  4. Cleared state of any pods exhibiting this behaviour ("Received update for IP range I own") i.e. remove db in /var/lib/weave on host and delete the pod.

On node deletion I can the see entry remains unreachable on the ipam-list (as I expect) but when a new node comes up, there's cleanup (this wasn't happening before).

I want to give it a week and keep checking before I celebrate!

itskingori picture itskingori on 30 Aug 2018
👍2 ❤1 🎉1

Cool! Please let us know, such that we can also proceed with an update if
appropriate.

On Thu, Aug 30, 2018, 18:50 itskingori notifications@github.com wrote:

@bboreham https://github.com/bboreham I don't want to celebrate just
yet but we've taken the following steps are seeing positive results:

  1. Upgraded to 2.4.0 to get the fix @brb https://github.com/brb
    mentions in #3310 (comment)
    https://github.com/weaveworks/weave/issues/3310#issuecomment-396558236,
    then ...
  2. Cleared any unreachable! on the ipam-list.
  3. Edited the peer-list annotation on the weave-net configmap to make
    sure the entries match with the ipam-list (there were entries on the
    peer-list that were not on the ipam-list as @zacblazic
    https://github.com/zacblazic explained in
    #3310 (comment)
    https://github.com/weaveworks/weave/issues/3310#issuecomment-416756507
    )
  4. Cleared state of any pods exhibiting this behaviour ("Received
    update for IP range I own") i.e. remove db in /var/lib/weave on host
    and delete the pod.

On node deletion I can the see entry remains unreachable on the ipam-list
(as I expect) but when a new node comes up, there's cleanup (this wasn't
happening before).

I want to give it a week and keep checking before I celebrate!

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/weaveworks/weave/issues/3310#issuecomment-417388958,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AApv1BnA8UcMf0MyWWqnpGaxAW_l-Co3ks5uWBfIgaJpZM4UccSf
.

Raffo picture Raffo on 30 Aug 2018

@Raffo if your issue at #3384 is still continuing please post the logs of a weave container starting up. That way we can find out if you have the same issue and the same steps are appropriate, or if you have a different issue. #3384 certainly lacks the key symptom of this issue, "Received update for IP range I own"

bboreham picture bboreham on 30 Aug 2018

Yes absolutely, I'm keeping an eye on it and will update my issue when I'll have more data.

Raffo picture Raffo on 31 Aug 2018

In #3392 we've found one scenario which leads to the reported error. Could you all run kubectl get nodes -o wide?

brb picture brb on 4 Sep 2018

@brb Even though my issue is #3384, just in case you wanted to know what I saw as well, I'm sure that the IPs were there, but I don't have the output anymore.

Raffo picture Raffo on 4 Sep 2018

@brb About 6 days after https://github.com/weaveworks/weave/issues/3310#issuecomment-417388958, our clusters are still looking healthy. And this is despite scaling up and down quite a bit ... 👇

screen shot 2018-09-04 at 16 36 38

We can verify that cleanup is taking place ... 👇 

 kubectl exec -it weave-net-897xt /bin/sh -n kube-system
Defaulting container name to weave.
Use 'kubectl describe pod/weave-net-897xt -n kube-system' to see all of the containers in this pod.
76:7f:2a:86:53:7a(ip-10-80-106-202.ec2.internal)    32768 IPs (01.6% of total) (15 active)
6e:0e:51:4f:66:9e(ip-10-80-114-8.ec2.internal)   345486 IPs (16.5% of total)
6a:7d:84:55:63:40(ip-10-80-45-255.ec2.internal)    71362 IPs (03.4% of total)
9a:6b:6f:33:1c:60(ip-10-80-53-166.ec2.internal)    31224 IPs (01.5% of total)
ba:a1:15:9f:64:7b(ip-10-80-113-52.ec2.internal)    16384 IPs (00.8% of total)
c6:51:3e:e0:73:85(ip-10-80-58-36.ec2.internal)    65536 IPs (03.1% of total)
ca:38:9f:69:92:76(ip-10-80-94-19.ec2.internal)    71340 IPs (03.4% of total)
02:8f:3d:02:22:b2(ip-10-80-110-250.ec2.internal)    32776 IPs (01.6% of total)
e2:b7:e3:40:a8:f9(ip-10-80-36-237.ec2.internal)    40872 IPs (01.9% of total)
c2:43:d1:36:2e:44(ip-10-80-46-144.ec2.internal)   528477 IPs (25.2% of total)
ce:81:1f:3d:d1:86(ip-10-80-105-153.ec2.internal)    65536 IPs (03.1% of total)
a2:89:81:f0:f7:cb(ip-10-80-91-69.ec2.internal)    42543 IPs (02.0% of total)
42:90:28:9a:2c:a2(ip-10-80-65-216.ec2.internal)    35616 IPs (01.7% of total)
a2:63:98:df:ee:d0(ip-10-80-41-90.ec2.internal)    27752 IPs (01.3% of total)
72:d3:22:05:8f:08(ip-10-80-112-109.ec2.internal)   143472 IPs (06.8% of total)
3e:8f:f3:15:0c:e6(ip-10-80-67-160.ec2.internal)   131096 IPs (06.3% of total)
be:10:d6:f0:eb:68(ip-10-80-112-120.ec2.internal)    44742 IPs (02.1% of total)
b2:ef:a5:21:f9:a3(ip-10-80-54-76.ec2.internal)    34498 IPs (01.6% of total)
52:ff:e3:6c:74:1d(ip-10-80-66-231.ec2.internal)    24376 IPs (01.2% of total)
52:cb:29:b2:39:4e(ip-10-80-90-159.ec2.internal)   311296 IPs (14.8% of total)

We'll continue monitoring. In case it happens again we'll check kubectl get nodes -o wide for internal IP as mentioned in https://github.com/weaveworks/weave/issues/3392.

itskingori picture itskingori on 4 Sep 2018
👍1

I did not see any logs in response to the request at #3310 (comment). This is still important to discovering the root cause of this issue.

Also, I'm not sure if we said here what the cleanup procedure is:

To recover, you need to eliminate the IPAM data from the affected nodes and restart. Since you installed via the Kubernetes Addon, this data will be in a file under /var/lib/weave on the node - delete this file and restart the node.

Since you seem to have about 17 nodes that can talk to each other and 3 that can't talk to them, do this just on the 3. Restarting (rebooting) after removing the file ensures there are no IP addresses in use on the node, which makes the procedure safe.

I was facing issues where few nodes were not able to access internet while others were working fine. Deleting /var/lib/weave/weave-netdata.db file and reboot did worked fine but again after 3-4hrs, the issue recreated. I again deleted the file and rebooted and it worked fine again. Why is the file creating issues. Thanks

abhinavrawat27 picture abhinavrawat27 on 22 Nov 2018

Linking #1962 which would make this condition less fatal.

bboreham picture bboreham on 18 Mar 2019

I'm going to close this as fixed in 2.6.0 - you may need to reboot a node which is actually using an IP in conflict with another node, but all the un-allocated space should get resolved automatically.

bboreham picture bboreham on 11 Nov 2019
Was this page helpful?
0 / 5 - 0 ratings

Related issues

emfrias picture emfrias  ·  6Comments
jeroenjacobs79 picture jeroenjacobs79  ·  7Comments
philicious picture philicious  ·  11Comments
bergtwvd picture bergtwvd  ·  11Comments
brb picture brb  ·  3Comments