NodePort services to be working on all nodes in my Kubernetes cluster.
I have four nodes in my test cluster. I am running the guestbook sample app, and pods are currently running on two of the four nodes. The guestbook is exposed with a NodePort service. If I test the service on each of the four nodes via the node port, it only loads on the nodes where the pod is actually running. Nodes without a local pod do not load and cause messages to be printed in the Weave NPC logs, like:
WARN: 2017/06/08 03:20:10.853205 TCP connection from 10.73.99.230:52927 to 10.47.160.3:3000 blocked by Weave NPC.
TODO: verify if this is happening on a new cluster with weave 1.9.7 (not one upgraded from 1.9.4 to 1.9.7)
Running on AWS, k8s configured with kops 1.6.0 then upgraded with my test branch from jordanjennings/kops:weave-upgrade where I'm testing out weave 1.9.7. Take a look the weave file I'm using there and let me know if there's anything that ought to be changed:
Also, this doesn't appear to be the NPC host name issue that 1.9.7 was released to fix:
kubelet host name override and docker logs for weave-npc match up:
$ ps aux | grep kubelet
root 1542 1.8 1.1 341492 84508 ? Ssl 02:46 2:07 /usr/local/bin/kubelet --allow-privileged=true --babysit-daemons=true --cgroup-root=/ --cloud-provider=aws --cluster-dns=100.64.0.10 --cluster-domain=cluster.local --enable-debugging-handlers=true --eviction-hard=memory.available<100Mi,nodefs.available<10%,nodefs.inodesFree<5%,imagefs.available<10%,imagefs.inodesFree<5% --hostname-override=ip-10-253-89-155.ec2.internal --kubeconfig=/var/lib/kubelet/kubeconfig --network-plugin=cni --node-labels=kubernetes.io/role=node,node-role.kubernetes.io/node= --non-masquerade-cidr=100.64.0.0/10 --pod-manifest-path=/etc/kubernetes/manifests --register-schedulable=true --require-kubeconfig=true --v=2 --cni-bin-dir=/opt/cni/bin/ --cni-conf-dir=/etc/cni/net.d/
$ sudo docker logs k8s_weave-npc_weave-net-qckkr_kube-system_b79658bc-4bf4-11e7-abcd-124fa73331d0_0
INFO: 2017/06/08 02:47:11.729761 Starting Weaveworks NPC 1.9.7; node name "ip-10-253-89-155.ec2.internal"
$ sudo docker version
Client:
Version: 1.12.6
API version: 1.24
Go version: go1.6.4
Git commit: 78d1802
Built: Tue Jan 10 20:17:57 2017
OS/Arch: linux/amd64
Server:
Version: 1.12.6
API version: 1.24
Go version: go1.6.4
Git commit: 78d1802
Built: Tue Jan 10 20:17:57 2017
OS/Arch: linux/amd64
$ uname -a
Linux ip-10-253-88-79 4.4.65-k8s #1 SMP Tue May 2 15:48:24 UTC 2017 x86_64 GNU/Linux
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.4", GitCommit:"d6f433224538d4f9ca2f7ae19b252e6fcb66a3ae", GitTreeState:"clean", BuildDate:"2017-05-19T18:44:27Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.4", GitCommit:"d6f433224538d4f9ca2f7ae19b252e6fcb66a3ae", GitTreeState:"clean", BuildDate:"2017-05-19T18:33:17Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"}
Nothing is printed in the weave logs when this issue happens, only weave NPC prints things as mentioned above.
$ ip route
default via 10.253.88.1 dev eth0
10.32.0.0/12 dev weave proto kernel scope link src 10.45.240.0
10.253.88.0/23 dev eth0 proto kernel scope link src 10.253.88.79
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1
$ ip -4 -o addr
1: lo inet 127.0.0.1/8 scope host lo\ valid_lft forever preferred_lft forever
2: eth0 inet 10.253.88.79/23 brd 10.253.89.255 scope global eth0\ valid_lft forever preferred_lft forever
3: docker0 inet 172.17.0.1/16 scope global docker0\ valid_lft forever preferred_lft forever
6: weave inet 10.45.240.0/12 scope global weave\ valid_lft forever preferred_lft forever
md5-e064ee890df89110fd6f45f5e6392f6b
$ sudo iptables-save
# Generated by iptables-save v1.4.21 on Thu Jun 8 03:58:48 2017
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [6:374]
:POSTROUTING ACCEPT [6:374]
:DOCKER - [0:0]
:KUBE-MARK-DROP - [0:0]
:KUBE-MARK-MASQ - [0:0]
:KUBE-NODEPORTS - [0:0]
:KUBE-POSTROUTING - [0:0]
:KUBE-SEP-2OJF3WUJVZEREVQI - [0:0]
:KUBE-SEP-5LQ6DLGJQD2E5OCY - [0:0]
:KUBE-SEP-CHK35S75JUICWURY - [0:0]
:KUBE-SEP-JBRJUTILVRH2HTWT - [0:0]
:KUBE-SEP-K26IETW3F4EKGCGU - [0:0]
:KUBE-SEP-MEUIDWM46ELHJEDL - [0:0]
:KUBE-SEP-NDJS6ELXGZQYPH2P - [0:0]
:KUBE-SEP-Q25PNZOYDWPJJ25Q - [0:0]
:KUBE-SEP-TJ2FLHKDTXDMCJQZ - [0:0]
:KUBE-SEP-YMPDTV2NQHVZCX47 - [0:0]
:KUBE-SEP-ZTCLQC67JNFJUL3S - [0:0]
:KUBE-SERVICES - [0:0]
:KUBE-SVC-7GF4BJM3Z6CMNVML - [0:0]
:KUBE-SVC-7QHHX5235AO5L5HA - [0:0]
:KUBE-SVC-AGR3D4D4FQNH4O33 - [0:0]
:KUBE-SVC-ERIFXISQEP7F7OF4 - [0:0]
:KUBE-SVC-NPX46M4PTMTKRN6Y - [0:0]
:KUBE-SVC-TCOU7JCQXEZGVUNU - [0:0]
:WEAVE - [0:0]
-A PREROUTING -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -m comment --comment "kubernetes postrouting rules" -j KUBE-POSTROUTING
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A POSTROUTING -j WEAVE
-A DOCKER -i docker0 -j RETURN
-A KUBE-MARK-DROP -j MARK --set-xmark 0x8000/0x8000
-A KUBE-MARK-MASQ -j MARK --set-xmark 0x4000/0x4000
-A KUBE-NODEPORTS -p tcp -m comment --comment "default/guestbook:" -m tcp --dport 31292 -j KUBE-MARK-MASQ
-A KUBE-NODEPORTS -p tcp -m comment --comment "default/guestbook:" -m tcp --dport 31292 -j KUBE-SVC-7QHHX5235AO5L5HA
-A KUBE-POSTROUTING -m comment --comment "kubernetes service traffic requiring SNAT" -m mark --mark 0x4000/0x4000 -j MASQUERADE
-A KUBE-SEP-2OJF3WUJVZEREVQI -s 10.47.64.3/32 -m comment --comment "default/redis-slave:" -j KUBE-MARK-MASQ
-A KUBE-SEP-2OJF3WUJVZEREVQI -p tcp -m comment --comment "default/redis-slave:" -m tcp -j DNAT --to-destination 10.47.64.3:6379
-A KUBE-SEP-5LQ6DLGJQD2E5OCY -s 10.47.160.3/32 -m comment --comment "default/guestbook:" -j KUBE-MARK-MASQ
-A KUBE-SEP-5LQ6DLGJQD2E5OCY -p tcp -m comment --comment "default/guestbook:" -m tcp -j DNAT --to-destination 10.47.160.3:3000
-A KUBE-SEP-CHK35S75JUICWURY -s 10.47.64.1/32 -m comment --comment "default/guestbook:" -j KUBE-MARK-MASQ
-A KUBE-SEP-CHK35S75JUICWURY -p tcp -m comment --comment "default/guestbook:" -m tcp -j DNAT --to-destination 10.47.64.1:3000
-A KUBE-SEP-JBRJUTILVRH2HTWT -s 10.47.64.4/32 -m comment --comment "kube-system/kube-dns:dns" -j KUBE-MARK-MASQ
-A KUBE-SEP-JBRJUTILVRH2HTWT -p udp -m comment --comment "kube-system/kube-dns:dns" -m udp -j DNAT --to-destination 10.47.64.4:53
-A KUBE-SEP-K26IETW3F4EKGCGU -s 10.253.89.10/32 -m comment --comment "default/kubernetes:https" -j KUBE-MARK-MASQ
-A KUBE-SEP-K26IETW3F4EKGCGU -p tcp -m comment --comment "default/kubernetes:https" -m recent --set --name KUBE-SEP-K26IETW3F4EKGCGU --mask 255.255.255.255 --rsource -m tcp -j DNAT --to-destination 10.253.89.10:443
-A KUBE-SEP-MEUIDWM46ELHJEDL -s 10.47.64.4/32 -m comment --comment "kube-system/kube-dns:dns-tcp" -j KUBE-MARK-MASQ
-A KUBE-SEP-MEUIDWM46ELHJEDL -p tcp -m comment --comment "kube-system/kube-dns:dns-tcp" -m tcp -j DNAT --to-destination 10.47.64.4:53
-A KUBE-SEP-NDJS6ELXGZQYPH2P -s 10.47.160.2/32 -m comment --comment "default/redis-slave:" -j KUBE-MARK-MASQ
-A KUBE-SEP-NDJS6ELXGZQYPH2P -p tcp -m comment --comment "default/redis-slave:" -m tcp -j DNAT --to-destination 10.47.160.2:6379
-A KUBE-SEP-Q25PNZOYDWPJJ25Q -s 10.47.64.2/32 -m comment --comment "default/redis-master:" -j KUBE-MARK-MASQ
-A KUBE-SEP-Q25PNZOYDWPJJ25Q -p tcp -m comment --comment "default/redis-master:" -m tcp -j DNAT --to-destination 10.47.64.2:6379
-A KUBE-SEP-TJ2FLHKDTXDMCJQZ -s 10.47.160.4/32 -m comment --comment "kube-system/kube-dns:dns" -j KUBE-MARK-MASQ
-A KUBE-SEP-TJ2FLHKDTXDMCJQZ -p udp -m comment --comment "kube-system/kube-dns:dns" -m udp -j DNAT --to-destination 10.47.160.4:53
-A KUBE-SEP-YMPDTV2NQHVZCX47 -s 10.45.208.1/32 -m comment --comment "default/guestbook:" -j KUBE-MARK-MASQ
-A KUBE-SEP-YMPDTV2NQHVZCX47 -p tcp -m comment --comment "default/guestbook:" -m tcp -j DNAT --to-destination 10.45.208.1:3000
-A KUBE-SEP-ZTCLQC67JNFJUL3S -s 10.47.160.4/32 -m comment --comment "kube-system/kube-dns:dns-tcp" -j KUBE-MARK-MASQ
-A KUBE-SEP-ZTCLQC67JNFJUL3S -p tcp -m comment --comment "kube-system/kube-dns:dns-tcp" -m tcp -j DNAT --to-destination 10.47.160.4:53
-A KUBE-SERVICES ! -s 100.96.0.0/11 -d 100.64.0.10/32 -p udp -m comment --comment "kube-system/kube-dns:dns cluster IP" -m udp --dport 53 -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d 100.64.0.10/32 -p udp -m comment --comment "kube-system/kube-dns:dns cluster IP" -m udp --dport 53 -j KUBE-SVC-TCOU7JCQXEZGVUNU
-A KUBE-SERVICES ! -s 100.96.0.0/11 -d 100.64.0.10/32 -p tcp -m comment --comment "kube-system/kube-dns:dns-tcp cluster IP" -m tcp --dport 53 -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d 100.64.0.10/32 -p tcp -m comment --comment "kube-system/kube-dns:dns-tcp cluster IP" -m tcp --dport 53 -j KUBE-SVC-ERIFXISQEP7F7OF4
-A KUBE-SERVICES ! -s 100.96.0.0/11 -d 100.67.195.178/32 -p tcp -m comment --comment "default/guestbook: cluster IP" -m tcp --dport 3000 -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d 100.67.195.178/32 -p tcp -m comment --comment "default/guestbook: cluster IP" -m tcp --dport 3000 -j KUBE-SVC-7QHHX5235AO5L5HA
-A KUBE-SERVICES ! -s 100.96.0.0/11 -d 100.64.0.1/32 -p tcp -m comment --comment "default/kubernetes:https cluster IP" -m tcp --dport 443 -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d 100.64.0.1/32 -p tcp -m comment --comment "default/kubernetes:https cluster IP" -m tcp --dport 443 -j KUBE-SVC-NPX46M4PTMTKRN6Y
-A KUBE-SERVICES ! -s 100.96.0.0/11 -d 100.65.212.198/32 -p tcp -m comment --comment "default/redis-master: cluster IP" -m tcp --dport 6379 -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d 100.65.212.198/32 -p tcp -m comment --comment "default/redis-master: cluster IP" -m tcp --dport 6379 -j KUBE-SVC-7GF4BJM3Z6CMNVML
-A KUBE-SERVICES ! -s 100.96.0.0/11 -d 100.71.146.49/32 -p tcp -m comment --comment "default/redis-slave: cluster IP" -m tcp --dport 6379 -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d 100.71.146.49/32 -p tcp -m comment --comment "default/redis-slave: cluster IP" -m tcp --dport 6379 -j KUBE-SVC-AGR3D4D4FQNH4O33
-A KUBE-SERVICES -m comment --comment "kubernetes service nodeports; NOTE: this must be the last rule in this chain" -m addrtype --dst-type LOCAL -j KUBE-NODEPORTS
-A KUBE-SVC-7GF4BJM3Z6CMNVML -m comment --comment "default/redis-master:" -j KUBE-SEP-Q25PNZOYDWPJJ25Q
-A KUBE-SVC-7QHHX5235AO5L5HA -m comment --comment "default/guestbook:" -m statistic --mode random --probability 0.33332999982 -j KUBE-SEP-YMPDTV2NQHVZCX47
-A KUBE-SVC-7QHHX5235AO5L5HA -m comment --comment "default/guestbook:" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-5LQ6DLGJQD2E5OCY
-A KUBE-SVC-7QHHX5235AO5L5HA -m comment --comment "default/guestbook:" -j KUBE-SEP-CHK35S75JUICWURY
-A KUBE-SVC-AGR3D4D4FQNH4O33 -m comment --comment "default/redis-slave:" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-NDJS6ELXGZQYPH2P
-A KUBE-SVC-AGR3D4D4FQNH4O33 -m comment --comment "default/redis-slave:" -j KUBE-SEP-2OJF3WUJVZEREVQI
-A KUBE-SVC-ERIFXISQEP7F7OF4 -m comment --comment "kube-system/kube-dns:dns-tcp" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-ZTCLQC67JNFJUL3S
-A KUBE-SVC-ERIFXISQEP7F7OF4 -m comment --comment "kube-system/kube-dns:dns-tcp" -j KUBE-SEP-MEUIDWM46ELHJEDL
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -m recent --rcheck --seconds 10800 --reap --name KUBE-SEP-K26IETW3F4EKGCGU --mask 255.255.255.255 --rsource -j KUBE-SEP-K26IETW3F4EKGCGU
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -j KUBE-SEP-K26IETW3F4EKGCGU
-A KUBE-SVC-TCOU7JCQXEZGVUNU -m comment --comment "kube-system/kube-dns:dns" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-TJ2FLHKDTXDMCJQZ
-A KUBE-SVC-TCOU7JCQXEZGVUNU -m comment --comment "kube-system/kube-dns:dns" -j KUBE-SEP-JBRJUTILVRH2HTWT
-A WEAVE -s 10.32.0.0/12 -d 224.0.0.0/4 -j RETURN
-A WEAVE ! -s 10.32.0.0/12 -d 10.32.0.0/12 -j MASQUERADE
-A WEAVE -s 10.32.0.0/12 ! -d 10.32.0.0/12 -j MASQUERADE
COMMIT
# Completed on Thu Jun 8 03:58:48 2017
# Generated by iptables-save v1.4.21 on Thu Jun 8 03:58:48 2017
*filter
:INPUT ACCEPT [100:58714]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [100:61093]
:DOCKER - [0:0]
:DOCKER-ISOLATION - [0:0]
:KUBE-FIREWALL - [0:0]
:KUBE-SERVICES - [0:0]
:WEAVE-NPC - [0:0]
:WEAVE-NPC-DEFAULT - [0:0]
:WEAVE-NPC-INGRESS - [0:0]
-A INPUT -j KUBE-FIREWALL
-A INPUT -d 172.17.0.1/32 -i docker0 -p tcp -m tcp --dport 6783 -j DROP
-A INPUT -d 172.17.0.1/32 -i docker0 -p udp -m udp --dport 6783 -j DROP
-A INPUT -d 172.17.0.1/32 -i docker0 -p udp -m udp --dport 6784 -j DROP
-A INPUT -i docker0 -p udp -m udp --dport 53 -j ACCEPT
-A INPUT -i docker0 -p tcp -m tcp --dport 53 -j ACCEPT
-A FORWARD -i docker0 -o weave -j DROP
-A FORWARD -j DOCKER-ISOLATION
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A FORWARD -o weave -j WEAVE-NPC
-A FORWARD -o weave -m state --state NEW -j NFLOG --nflog-group 86
-A FORWARD -o weave -j DROP
-A FORWARD -i weave ! -o weave -j ACCEPT
-A FORWARD -o weave -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A OUTPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A OUTPUT -j KUBE-FIREWALL
-A DOCKER-ISOLATION -j RETURN
-A KUBE-FIREWALL -m comment --comment "kubernetes firewall for dropping marked packets" -m mark --mark 0x8000/0x8000 -j DROP
-A WEAVE-NPC -m state --state RELATED,ESTABLISHED -j ACCEPT
-A WEAVE-NPC -d 224.0.0.0/4 -j ACCEPT
-A WEAVE-NPC -m state --state NEW -j WEAVE-NPC-DEFAULT
-A WEAVE-NPC -m state --state NEW -j WEAVE-NPC-INGRESS
-A WEAVE-NPC -m set --match-set weave-local-pods src -m set ! --match-set weave-local-pods dst -j ACCEPT
-A WEAVE-NPC-DEFAULT -m set --match-set weave-k?Z;25^M}|1s7P3|H9i;*;MhG dst -j ACCEPT
-A WEAVE-NPC-DEFAULT -m set --match-set weave-4vtqMI+kx/2]jD%_c0S%thO%V dst -j ACCEPT
-A WEAVE-NPC-DEFAULT -m set --match-set weave-iuZcey(5DeXbzgRFs8Szo]+@p dst -j ACCEPT
COMMIT
# Completed on Thu Jun 8 03:58:48 2017
@jordanjennings, many thanks for the detailed report 馃憤
It will take a bit of time to investigate it, but at first sight, this looks like a duplicate of #2997.
cc me
We managed to replicate this and I will have a PR shortly
For the record:
1) Setup
host0 (10.128.0.6): master
host1 (10.128.0.5): worker
host2 (10.128.0.4): not part of Kubernetes cluster
2) Created a pod with label name=nginx
host0$ kubectl run nginx --image=nginx --replicas=1 -l name=nginx
3) Confirmed the label can be used to select the pod
host0$ kubectl get po -l name=nginx -o wide
NAME READY STATUS RESTARTS AGE IP NODE
nginx-159377361-kwx5t 1/1 Running 0 1h 10.32.0.3 host1
4) curl 10.32.0.3:80
host0: OK
host1: OK
host2: timed out
as expected.
5) Created a NodePort service to expose the nginx pod to host2
host0$ kubectl create -f - <<EOF
apiVersion: v1
kind: Service
metadata:
name: nginxsvc
labels:
app: nginx
spec:
type: NodePort
ports:
- port: 80
name: http
nodePort: 30080
- port: 443
name: https
nodePort: 30443
selector:
name: nginx
EOF
6) The NodePort service did have Endpoints and a ClusterIP, therefore looked correctly configured
host0$ kubectl get endpoints nginxsvc
NAME ENDPOINTS AGE
nginxsvc 10.32.0.3:443,10.32.0.3:80 41m
host0$ kubectl get service nginxsvc
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
nginxsvc 10.103.52.53 <nodes> 80:30080/TCP,443:30443/TCP 41m
md5-d83982cc175d57dd556d31c42c38dbb7
-A WEAVE-NPC -m state --state RELATED,ESTABLISHED -j ACCEPT
-A WEAVE-NPC -d 224.0.0.0/4 -j ACCEPT
-A WEAVE-NPC -m state --state NEW -j WEAVE-NPC-DEFAULT
-A WEAVE-NPC -m state --state NEW -j WEAVE-NPC-INGRESS
-A WEAVE-NPC -m set --match-set weave-local-pods src -m set ! --match-set weave-local-pods dst -j ACCEPT
-A WEAVE-NPC-DEFAULT -m set --match-set weave-iuZcey(5DeXbzgRFs8Szo]+@p dst -j ACCEPT
-A WEAVE-NPC-DEFAULT -m set --match-set weave-k?Z;25^M}|1s7P3|H9i;*;MhG dst -j ACCEPT
-A WEAVE-NPC-DEFAULT -m set --match-set weave-4vtqMI+kx/2]jD%_c0S%thO%V dst -j ACCEPT
md5-c8d90d41112e014ed644532138179e09
-A WEAVE-NPC -m set --match-set weave-local-pods src -m set ! --match-set weave-local-pods dst -j ACCEPT
md5-1a017603118edfa6ddee9548cbb87b1a
-A WEAVE-NPC -m set --match-set weave-local-pods src -m set ! --match-set weave-local-pods dst -j ACCEPT
md5-ac3ec8d64db497fa123b65b4938e840e
-A WEAVE-NPC -m set ! --match-set weave-local-pods dst -j ACCEPT
I tested this with Weave Net 1.9.4 and got identical results, so I'm going to mark the original issue as fixed by #3014, and any remaining issues were pre-existing and still have to be debugged.
Note the fix is in 1.9.8 just released
Most helpful comment
Note the fix is in 1.9.8 just released