Weave: Cannot connect to external IPs from inside kubernetes pod

Created on 8 Apr 2017  路  7Comments  路  Source: weaveworks/weave

My CentOS machine is in network using the 10.24.xxx.xxx subnet and gateway & DNS is 10.24.0.1.
I have my DHCP services disabled for this vlan. I'm running kubernetes 1.5.6 and installed weave plugin using kubectl apply -f https://git.io/weave-kube.

Issue is - I cannot access outside world from the pod. Request to google.com from the pod fails. 

kubectl exec busybox -- nslookup google.com
nslookup: can't resolve 'google.com'
Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

kubectl exec busybox -- nslookup kubernetes
Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

Name:      kubernetes
Address 1: 10.96.0.1 kubernetes.default.svc.cluster.local

kube-dns logs

I0407 22:50:39.438721       1 dns.go:274] New service: kubernetes
I0407 22:50:39.438799       1 dns.go:384] Added SRV record &{Host:kubernetes.default.svc.cluster.local. Port:443 Priority:10 Weight:10 Text: Mail:false Ttl:30 TargetStrip:0 Group: Key:}
I0407 22:50:39.438875       1 dns.go:274] New service: falkonryagent
I0407 22:50:39.439007       1 dns.go:384] Added SRV record &{Host:falkonryagent.default.svc.cluster.local. Port:9090 Priority:10 Weight:10 Text: Mail:false Ttl:30 TargetStrip:0 Group: Key:}
I0407 22:53:05.167558       1 logs.go:41] skydns: failure to forward request "read udp 10.32.0.4:44828->10.24.0.1:53: i/o timeout"
I0407 22:53:05.167633       1 logs.go:41] skydns: failure to forward request "read udp 10.32.0.4:59553->10.24.0.1:53: i/o timeout"
I0407 22:53:49.006206       1 logs.go:41] skydns: failure to forward request "read udp 10.32.0.4:37769->10.24.0.1:53: i/o timeout"
I0407 22:53:59.016166       1 logs.go:41] skydns: failure to forward request "read udp 10.32.0.4:35858->10.24.0.1:53: i/o timeout"
I0407 22:54:09.022056       1 logs.go:41] skydns: failure to forward request "read udp 10.32.0.4:58424->10.24.0.1:53: i/o timeout"
I0407 22:54:19.032227       1 logs.go:41] skydns: failure to forward request "read udp 10.32.0.4:37994->10.24.0.1:53: i/o timeout"
I0407 22:54:34.047816       1 logs.go:41] skydns: failure to forward request "read udp 10.32.0.4:43757->10.24.0.1:53: i/o timeout"
I0407 22:54:44.053430       1 logs.go:41] skydns: failure to forward request "read udp 10.32.0.4:59304->10.24.0.1:53: i/o timeout"
I0407 22:54:54.063791       1 logs.go:41] skydns: failure to forward request "read udp 10.32.0.4:40765->10.24.0.1:53: i/o timeout"
I0407 22:55:04.074410       1 logs.go:41] skydns: failure to forward request "read udp 10.32.0.4:46593->10.24.0.1:53: i/o timeout"

Kube-dns deployment

apiVersion: v1
kind: Pod
metadata:
  annotations:
    kubernetes.io/created-by: |
      {"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"ReplicaSet","namespace":"kube-system","name":"kube-dns-2924299975","uid":"ad719ba3-1bdd-11e7-b25a-ecb1d7858fe2","apiVersion":"extensions","resourceVersion":"211"}}
    scheduler.alpha.kubernetes.io/affinity: '{"nodeAffinity":{"requiredDuringSchedulingIgnoredDuringExecution":{"nodeSelectorTerms":[{"matchExpressions":[{"key":"beta.kubernetes.io/arch","operator":"In","values":["amd64"]}]}]}}}'
    scheduler.alpha.kubernetes.io/tolerations: '[{"key":"dedicated","value":"master","effect":"NoSchedule"}]'
  creationTimestamp: 2017-04-07T22:00:56Z
  generateName: kube-dns-2924299975-
  labels:
    component: kube-dns
    k8s-app: kube-dns
    kubernetes.io/cluster-service: "true"
    name: kube-dns
    pod-template-hash: "2924299975"
    tier: node
  name: kube-dns-2924299975-7c05q
  namespace: kube-system
  ownerReferences:
  - apiVersion: extensions/v1beta1
    controller: true
    kind: ReplicaSet
    name: kube-dns-2924299975
    uid: ad719ba3-1bdd-11e7-b25a-ecb1d7858fe2
  resourceVersion: "728"
  selfLink: /api/v1/namespaces/kube-system/pods/kube-dns-2924299975-7c05q
  uid: ad74255b-1bdd-11e7-b25a-ecb1d7858fe2
spec:
  containers:
  - args:
    - --domain=cluster.local
    - --dns-port=10053
    - --config-map=kube-dns
    - --v=2
    env:
    - name: PROMETHEUS_PORT
      value: "10055"
    image: gcr.io/google_containers/kubedns-amd64:1.9
    imagePullPolicy: IfNotPresent
    livenessProbe:
      failureThreshold: 5
      httpGet:
        path: /healthz-kubedns
        port: 8080
        scheme: HTTP
      initialDelaySeconds: 60
      periodSeconds: 10
      successThreshold: 1
      timeoutSeconds: 5
    name: kube-dns
    ports:
    - containerPort: 10053
      name: dns-local
      protocol: UDP
    - containerPort: 10053
      name: dns-tcp-local
      protocol: TCP
    - containerPort: 10055
      name: metrics
      protocol: TCP
    readinessProbe:
      failureThreshold: 3
      httpGet:
        path: /readiness
        port: 8081
        scheme: HTTP
      initialDelaySeconds: 3
      periodSeconds: 10
      successThreshold: 1
      timeoutSeconds: 5
    resources:
      limits:
        memory: 170Mi
      requests:
        cpu: 100m
        memory: 70Mi
    terminationMessagePath: /dev/termination-log
    volumeMounts:
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: default-token-h9j3t
      readOnly: true
  - args:
    - --cache-size=1000
    - --no-resolv
    - --server=127.0.0.1#10053
    - --log-facility=-
    image: gcr.io/google_containers/kube-dnsmasq-amd64:1.4
    imagePullPolicy: IfNotPresent
    livenessProbe:
      failureThreshold: 5
      httpGet:
        path: /healthz-dnsmasq
        port: 8080
        scheme: HTTP
      initialDelaySeconds: 60
      periodSeconds: 10
      successThreshold: 1
      timeoutSeconds: 5
    name: dnsmasq
    ports:
    - containerPort: 53
      name: dns
      protocol: UDP
    - containerPort: 53
      name: dns-tcp
      protocol: TCP
    resources:
      requests:
        cpu: 150m
        memory: 10Mi
    terminationMessagePath: /dev/termination-log
    volumeMounts:
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: default-token-h9j3t
      readOnly: true
  - args:
    - --v=2
    - --logtostderr
    image: gcr.io/google_containers/dnsmasq-metrics-amd64:1.0
    imagePullPolicy: IfNotPresent
    livenessProbe:
      failureThreshold: 5
      httpGet:
        path: /metrics
        port: 10054
        scheme: HTTP
      initialDelaySeconds: 60
      periodSeconds: 10
      successThreshold: 1
      timeoutSeconds: 5
    name: dnsmasq-metrics
    ports:
    - containerPort: 10054
      name: metrics
      protocol: TCP
    resources:
      requests:
        memory: 10Mi
    securityContext:
      runAsUser: 0
    terminationMessagePath: /dev/termination-log
    volumeMounts:
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: default-token-h9j3t
      readOnly: true
  - args:
    - --cmd=nslookup kubernetes.default.svc.cluster.local 127.0.0.1 >/dev/null
    - --url=/healthz-dnsmasq
    - --cmd=nslookup kubernetes.default.svc.cluster.local 127.0.0.1:10053 >/dev/null
    - --url=/healthz-kubedns
    - --port=8080
    - --quiet
    image: gcr.io/google_containers/exechealthz-amd64:1.2
    imagePullPolicy: IfNotPresent
    name: healthz
    ports:
    - containerPort: 8080
      protocol: TCP
    resources:
      limits:
        memory: 50Mi
      requests:
        cpu: 10m
        memory: 50Mi
    terminationMessagePath: /dev/termination-log
    volumeMounts:
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: default-token-h9j3t
      readOnly: true
  dnsPolicy: Default
  nodeName: c4n1
  restartPolicy: Always
  securityContext: {}
  serviceAccount: default
  serviceAccountName: default
  terminationGracePeriodSeconds: 30
  volumes:
  - name: default-token-h9j3t
    secret:
      defaultMode: 420
      secretName: default-token-h9j3t
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: 2017-04-07T22:00:57Z
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: 2017-04-07T22:05:57Z
    status: "True"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: 2017-04-07T22:00:57Z
    status: "True"
    type: PodScheduled
  containerStatuses:
  - containerID: docker://c4d632b69413a807cc8b4341c814f910bec2650c0a251d616a057c8d99a54a1c
    image: gcr.io/google_containers/kube-dnsmasq-amd64:1.4
    imageID: docker-pullable://gcr.io/google_containers/kube-dnsmasq-amd64@sha256:a722df15c0cf87779aad8ba2468cf072dd208cb5d7cfcaedd90e66b3da9ea9d2
    lastState: {}
    name: dnsmasq
    ready: true
    restartCount: 0
    state:
      running:
        startedAt: 2017-04-07T22:05:43Z
  - containerID: docker://3dd3c8b40eef9d366c7d5d5589af4b3cdcef5c59f9bb422fb740c37a21b1425a
    image: gcr.io/google_containers/dnsmasq-metrics-amd64:1.0
    imageID: docker-pullable://gcr.io/google_containers/dnsmasq-metrics-amd64@sha256:4063e37fd9b2fd91b7cc5392ed32b30b9c8162c4c7ad2787624306fc133e80a9
    lastState: {}
    name: dnsmasq-metrics
    ready: true
    restartCount: 0
    state:
      running:
        startedAt: 2017-04-07T22:05:48Z
  - containerID: docker://84fedf199e252e6513822f2e1010f12130c1ae22c2e48f56260bbb3ae4b03b5b
    image: gcr.io/google_containers/exechealthz-amd64:1.2
    imageID: docker-pullable://gcr.io/google_containers/exechealthz-amd64@sha256:503e158c3f65ed7399f54010571c7c977ade7fe59010695f48d9650d83488c0a
    lastState: {}
    name: healthz
    ready: true
    restartCount: 0
    state:
      running:
        startedAt: 2017-04-07T22:05:51Z
  - containerID: docker://a21dda95b5a2817a85b282ce98c05e35911798137268e377fc0152813e595c49
    image: gcr.io/google_containers/kubedns-amd64:1.9
    imageID: docker-pullable://gcr.io/google_containers/kubedns-amd64@sha256:3d3d67f519300af646e00adcf860b2f380d35ed4364e550d74002dadace20ead
    lastState: {}
    name: kube-dns
    ready: true
    restartCount: 0
    state:
      running:
        startedAt: 2017-04-07T22:05:39Z
  hostIP: 10.24.10.4
  phase: Running
  podIP: 10.32.0.4
  startTime: 2017-04-07T22:00:57Z
question
Source
picture phagunbaya

Most helpful comment

@bboreham I deleted the cluster by running kubeadm reset and deleted /opt/bni/bin/weave-* then restarted the machine before creating new cluster by kubeadm init. Now it's fine and I can make outbound requests. Thanks for helping me debug this.

picture phagunbaya on 8 Apr 2017
👍2

All 7 comments

Should I set "IPALLOC_RANGE" in weave yaml ? if yes then to what IP range ?

phagunbaya picture phagunbaya on 8 Apr 2017

There are several things that can cause this; no sign that setting IPALLOC_RANGE would help.

Can you run the following on the host and post the results please: 

ip addr
ip route
sudo iptables-save
bboreham picture bboreham on 8 Apr 2017

@bboreham
ip addr

[root@c4n1 ~]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether ec:b1:d7:85:8f:e2 brd ff:ff:ff:ff:ff:ff
    inet 10.24.10.4/16 brd 10.24.255.255 scope global eno1
       valid_lft forever preferred_lft forever
    inet6 fe80::d54e:4e88:2b9c:15b4/64 scope link 
       valid_lft forever preferred_lft forever
3: eno1d1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether ec:b1:d7:85:8f:e3 brd ff:ff:ff:ff:ff:ff
4: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN 
    link/ether 02:42:4d:44:7d:05 brd ff:ff:ff:ff:ff:ff
    inet 172.17.50.1/24 scope global docker0
       valid_lft forever preferred_lft forever
    inet6 fe80::42:4dff:fe44:7d05/64 scope link 
       valid_lft forever preferred_lft forever
5: datapath: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1376 qdisc noqueue state UNKNOWN qlen 1000
    link/ether 5e:16:1c:7b:26:b8 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::5c16:1cff:fe7b:26b8/64 scope link 
       valid_lft forever preferred_lft forever
7: weave: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1376 qdisc noqueue state UP qlen 1000
    link/ether ce:a2:c9:c2:ef:82 brd ff:ff:ff:ff:ff:ff
    inet 10.32.0.1/12 scope global weave
       valid_lft forever preferred_lft forever
    inet 10.32.0.2/12 scope global secondary weave
       valid_lft forever preferred_lft forever
    inet 10.32.0.3/12 scope global secondary weave
       valid_lft forever preferred_lft forever
    inet 10.32.0.4/12 scope global secondary weave
       valid_lft forever preferred_lft forever
    inet6 fe80::cca2:c9ff:fec2:ef82/64 scope link 
       valid_lft forever preferred_lft forever
8: dummy0: <BROADCAST,NOARP> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether ca:cb:81:77:d3:5d brd ff:ff:ff:ff:ff:ff
10: vethwe-datapath@vethwe-bridge: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1376 qdisc noqueue master datapath state UP qlen 1000
    link/ether 16:76:cb:b7:e9:c9 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::1476:cbff:feb7:e9c9/64 scope link 
       valid_lft forever preferred_lft forever
11: vethwe-bridge@vethwe-datapath: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1376 qdisc noqueue master weave state UP qlen 1000
    link/ether 7a:94:98:41:fb:23 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::7894:98ff:fe41:fb23/64 scope link 
       valid_lft forever preferred_lft forever
53: vxlan-6784: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65470 qdisc noqueue master datapath state UNKNOWN qlen 1000
    link/ether 82:f7:d5:83:b0:7f brd ff:ff:ff:ff:ff:ff
55: vethweplec0a6e1@if54: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1376 qdisc noqueue master weave state UP 
    link/ether ee:5d:63:0f:1f:db brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet6 fe80::ec5d:63ff:fe0f:1fdb/64 scope link 
       valid_lft forever preferred_lft forever
57: vethweplbc5e6af@if56: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1376 qdisc noqueue master weave state UP 
    link/ether 82:2b:3c:a9:1f:e0 brd ff:ff:ff:ff:ff:ff link-netnsid 1
    inet6 fe80::802b:3cff:fea9:1fe0/64 scope link 
       valid_lft forever preferred_lft forever

ip route

[root@c4n1 ~]# ip route
default via 10.24.0.1 dev eno1  proto static  metric 100 
10.24.0.0/16 dev eno1  proto kernel  scope link  src 10.24.10.4  metric 100 
10.32.0.0/12 dev weave  proto kernel  scope link  src 10.32.0.1 
172.17.50.0/24 dev docker0  proto kernel  scope link  src 172.17.50.1

sudo iptables-save

[root@c4n1 ~]# iptables-save 
# Generated by iptables-save v1.4.21 on Sat Apr  8 03:39:52 2017
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [17:1020]
:POSTROUTING ACCEPT [17:1020]
:DOCKER - [0:0]
:KUBE-MARK-DROP - [0:0]
:KUBE-MARK-MASQ - [0:0]
:KUBE-NODEPORTS - [0:0]
:KUBE-POSTROUTING - [0:0]
:KUBE-SEP-5CGXH7DASKB62Q4K - [0:0]
:KUBE-SEP-AR5TRSQMIM2F553H - [0:0]
:KUBE-SEP-WRE3MG3RYMTYWFJZ - [0:0]
:KUBE-SERVICES - [0:0]
:KUBE-SVC-ERIFXISQEP7F7OF4 - [0:0]
:KUBE-SVC-NPX46M4PTMTKRN6Y - [0:0]
:KUBE-SVC-TCOU7JCQXEZGVUNU - [0:0]
-A PREROUTING -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -m comment --comment "kubernetes postrouting rules" -j KUBE-POSTROUTING
-A POSTROUTING -s 172.17.50.0/24 ! -o docker0 -j MASQUERADE
-A DOCKER -i docker0 -j RETURN
-A KUBE-MARK-DROP -j MARK --set-xmark 0x8000/0x8000
-A KUBE-MARK-MASQ -j MARK --set-xmark 0x4000/0x4000
-A KUBE-POSTROUTING -m comment --comment "kubernetes service traffic requiring SNAT" -m mark --mark 0x4000/0x4000 -j MASQUERADE
-A KUBE-SEP-5CGXH7DASKB62Q4K -s 10.32.0.5/32 -m comment --comment "kube-system/kube-dns:dns-tcp" -j KUBE-MARK-MASQ
-A KUBE-SEP-5CGXH7DASKB62Q4K -p tcp -m comment --comment "kube-system/kube-dns:dns-tcp" -m tcp -j DNAT --to-destination 10.32.0.5:53
-A KUBE-SEP-AR5TRSQMIM2F553H -s 10.24.10.4/32 -m comment --comment "default/kubernetes:https" -j KUBE-MARK-MASQ
-A KUBE-SEP-AR5TRSQMIM2F553H -p tcp -m comment --comment "default/kubernetes:https" -m recent --set --name KUBE-SEP-AR5TRSQMIM2F553H --mask 255.255.255.255 --rsource -m tcp -j DNAT --to-destination 10.24.10.4:6443
-A KUBE-SEP-WRE3MG3RYMTYWFJZ -s 10.32.0.5/32 -m comment --comment "kube-system/kube-dns:dns" -j KUBE-MARK-MASQ
-A KUBE-SEP-WRE3MG3RYMTYWFJZ -p udp -m comment --comment "kube-system/kube-dns:dns" -m udp -j DNAT --to-destination 10.32.0.5:53
-A KUBE-SERVICES -d 10.96.0.1/32 -p tcp -m comment --comment "default/kubernetes:https cluster IP" -m tcp --dport 443 -j KUBE-SVC-NPX46M4PTMTKRN6Y
-A KUBE-SERVICES -d 10.96.0.10/32 -p udp -m comment --comment "kube-system/kube-dns:dns cluster IP" -m udp --dport 53 -j KUBE-SVC-TCOU7JCQXEZGVUNU
-A KUBE-SERVICES -d 10.96.0.10/32 -p tcp -m comment --comment "kube-system/kube-dns:dns-tcp cluster IP" -m tcp --dport 53 -j KUBE-SVC-ERIFXISQEP7F7OF4
-A KUBE-SERVICES -m comment --comment "kubernetes service nodeports; NOTE: this must be the last rule in this chain" -m addrtype --dst-type LOCAL -j KUBE-NODEPORTS
-A KUBE-SVC-ERIFXISQEP7F7OF4 -m comment --comment "kube-system/kube-dns:dns-tcp" -j KUBE-SEP-5CGXH7DASKB62Q4K
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -m recent --rcheck --seconds 10800 --reap --name KUBE-SEP-AR5TRSQMIM2F553H --mask 255.255.255.255 --rsource -j KUBE-SEP-AR5TRSQMIM2F553H
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -j KUBE-SEP-AR5TRSQMIM2F553H
-A KUBE-SVC-TCOU7JCQXEZGVUNU -m comment --comment "kube-system/kube-dns:dns" -j KUBE-SEP-WRE3MG3RYMTYWFJZ
COMMIT
# Completed on Sat Apr  8 03:39:52 2017
# Generated by iptables-save v1.4.21 on Sat Apr  8 03:39:52 2017
*filter
:INPUT ACCEPT [861:344796]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [856:349424]
:DOCKER - [0:0]
:DOCKER-ISOLATION - [0:0]
:KUBE-FIREWALL - [0:0]
:KUBE-SERVICES - [0:0]
:WEAVE-NPC - [0:0]
:WEAVE-NPC-DEFAULT - [0:0]
:WEAVE-NPC-INGRESS - [0:0]
-A INPUT -j KUBE-FIREWALL
-A FORWARD -j DOCKER-ISOLATION
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A OUTPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A OUTPUT -j KUBE-FIREWALL
-A DOCKER-ISOLATION -j RETURN
-A KUBE-FIREWALL -m comment --comment "kubernetes firewall for dropping marked packets" -m mark --mark 0x8000/0x8000 -j DROP
-A WEAVE-NPC -m state --state RELATED,ESTABLISHED -j ACCEPT
-A WEAVE-NPC -d 224.0.0.0/4 -j ACCEPT
-A WEAVE-NPC -m state --state NEW -j WEAVE-NPC-DEFAULT
-A WEAVE-NPC -m state --state NEW -j WEAVE-NPC-INGRESS
-A WEAVE-NPC-DEFAULT -m set --match-set weave-k?Z;25^M}|1s7P3|H9i;*;MhG dst -j ACCEPT
-A WEAVE-NPC-DEFAULT -m set --match-set weave-iuZcey(5DeXbzgRFs8Szo]<@p dst -j ACCEPT
COMMIT
# Completed on Sat Apr  8 03:39:52 2017
phagunbaya picture phagunbaya on 8 Apr 2017

So the weave bridge has four IP addresses. That shouldn't happen. Any idea what led to that?

I would reboot the host to clear that condition, then repeat the diagnosis.

The other interesting thing is this:

-A KUBE-SEP-WRE3MG3RYMTYWFJZ -s 10.32.0.5/32 -m comment --comment "kube-system/kube-dns:dns" -j KUBE-MARK-MASQ
-A KUBE-SEP-5CGXH7DASKB62Q4K -s 10.32.0.5/32 -m comment --comment "kube-system/kube-dns:dns-tcp" -j KUBE-MARK-MASQ

Can you give the command-line for kube-proxy please; maybe that will help to understand why it's doing that.

bboreham picture bboreham on 8 Apr 2017

May be running kubeadm reset is not deleting all the required things for network plugin. That is the reason for multiple weave IP addresses.

kube-proxy was started by kubeadm

apiVersion: v1
kind: Pod
metadata:
  annotations:
    kubernetes.io/created-by: |
      {"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"DaemonSet","namespace":"kube-system","name":"kube-proxy","uid":"f0a66041-1bed-11e7-9da7-ecb1d7858fe2","apiVersion":"extensions","resourceVersion":"208"}}
    scheduler.alpha.kubernetes.io/affinity: '{"nodeAffinity":{"requiredDuringSchedulingIgnoredDuringExecution":{"nodeSelectorTerms":[{"matchExpressions":[{"key":"beta.kubernetes.io/arch","operator":"In","values":["amd64"]}]}]}}}'
    scheduler.alpha.kubernetes.io/tolerations: '[{"key":"dedicated","value":"master","effect":"NoSchedule"}]'
  creationTimestamp: 2017-04-07T23:57:21Z
  generateName: kube-proxy-
  labels:
    component: kube-proxy
    k8s-app: kube-proxy
    kubernetes.io/cluster-service: "true"
    name: kube-proxy
    tier: node
  name: kube-proxy-7njlr
  namespace: kube-system
  resourceVersion: "263"
  selfLink: /api/v1/namespaces/kube-system/pods/kube-proxy-7njlr
  uid: f0a79cc2-1bed-11e7-9da7-ecb1d7858fe2
spec:
  containers:
  - command:
    - kube-proxy
    - --kubeconfig=/run/kubeconfig
    image: gcr.io/google_containers/kube-proxy-amd64:v1.5.6
    imagePullPolicy: IfNotPresent
    name: kube-proxy
    resources: {}
    securityContext:
      privileged: true
    terminationMessagePath: /dev/termination-log
    volumeMounts:
    - mountPath: /var/run/dbus
      name: dbus
    - mountPath: /run/kubeconfig
      name: kubeconfig
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: default-token-7zr34
      readOnly: true
  dnsPolicy: ClusterFirst
  hostNetwork: true
  nodeName: c4n1
  restartPolicy: Always
  securityContext: {}
  serviceAccount: default
  serviceAccountName: default
  terminationGracePeriodSeconds: 30
  volumes:
  - hostPath:
      path: /etc/kubernetes/kubelet.conf
    name: kubeconfig
  - hostPath:
      path: /var/run/dbus
    name: dbus
  - name: default-token-7zr34
    secret:
      defaultMode: 420
      secretName: default-token-7zr34
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: 2017-04-07T23:57:21Z
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: 2017-04-07T23:57:38Z
    status: "True"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: 2017-04-07T23:57:38Z
    status: "True"
    type: PodScheduled
  containerStatuses:
  - containerID: docker://1b534ac1c2784671c545774eb6cad415a16dfee18882a7323a07c5a2510c1c64
    image: gcr.io/google_containers/kube-proxy-amd64:v1.5.6
    imageID: docker-pullable://gcr.io/google_containers/kube-proxy-amd64@sha256:2ccde09687c6170c2731f295af1f9f6f6d8f48779824baf4af1a6a892f0442db
    lastState: {}
    name: kube-proxy
    ready: true
    restartCount: 0
    state:
      running:
        startedAt: 2017-04-07T23:57:37Z
  hostIP: 10.24.10.4
  phase: Running
  podIP: 10.24.10.4
  startTime: 2017-04-07T23:57:21Z

ip addr after machine reset

[root@c4n1 ~]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether ec:b1:d7:85:8f:e2 brd ff:ff:ff:ff:ff:ff
    inet 10.24.10.4/16 brd 10.24.255.255 scope global eno1
       valid_lft forever preferred_lft forever
    inet6 fe80::d54e:4e88:2b9c:15b4/64 scope link 
       valid_lft forever preferred_lft forever
3: eno1d1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether ec:b1:d7:85:8f:e3 brd ff:ff:ff:ff:ff:ff
4: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN 
    link/ether 02:42:a1:81:54:43 brd ff:ff:ff:ff:ff:ff
    inet 172.17.50.1/24 scope global docker0
       valid_lft forever preferred_lft forever
5: datapath: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1376 qdisc noqueue state UNKNOWN qlen 1000
    link/ether ae:53:cf:27:fe:b5 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::ac53:cfff:fe27:feb5/64 scope link 
       valid_lft forever preferred_lft forever
7: weave: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1376 qdisc noqueue state UP qlen 1000
    link/ether ce:a2:c9:c2:ef:82 brd ff:ff:ff:ff:ff:ff
    inet 10.32.0.1/12 scope global weave
       valid_lft forever preferred_lft forever
    inet6 fe80::cca2:c9ff:fec2:ef82/64 scope link 
       valid_lft forever preferred_lft forever
8: dummy0: <BROADCAST,NOARP> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether 52:23:26:13:e3:16 brd ff:ff:ff:ff:ff:ff
10: vethwe-datapath@vethwe-bridge: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1376 qdisc noqueue master datapath state UP qlen 1000
    link/ether 92:fe:23:6c:8d:65 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::90fe:23ff:fe6c:8d65/64 scope link 
       valid_lft forever preferred_lft forever
11: vethwe-bridge@vethwe-datapath: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1376 qdisc noqueue master weave state UP qlen 1000
    link/ether 2a:c6:cd:28:a4:f7 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::28c6:cdff:fe28:a4f7/64 scope link 
       valid_lft forever preferred_lft forever
13: vxlan-6784: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65470 qdisc noqueue master datapath state UNKNOWN qlen 1000
    link/ether 6a:fe:c7:b2:a8:31 brd ff:ff:ff:ff:ff:ff
15: vethwepl3c06ad1@if14: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1376 qdisc noqueue master weave state UP 
    link/ether 62:e8:1a:99:6b:ac brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet6 fe80::60e8:1aff:fe99:6bac/64 scope link 
       valid_lft forever preferred_lft forever
17: vethwepl5332597@if16: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1376 qdisc noqueue master weave state UP 
    link/ether 8a:b4:c5:5b:64:8a brd ff:ff:ff:ff:ff:ff link-netnsid 1
    inet6 fe80::88b4:c5ff:fe5b:648a/64 scope link 
       valid_lft forever preferred_lft forever

Still cannot connect to google.com from inside pod.

phagunbaya picture phagunbaya on 8 Apr 2017

@bboreham I deleted the cluster by running kubeadm reset and deleted /opt/bni/bin/weave-* then restarted the machine before creating new cluster by kubeadm init. Now it's fine and I can make outbound requests. Thanks for helping me debug this.

phagunbaya picture phagunbaya on 8 Apr 2017
👍2

Thanks for the update @phagunbaya & glad you sorted it out! I will close this GitHub issue then.
Feel free to reopen if it reoccurs and/or you have more information on the root cause.

marccarre picture marccarre on 10 Apr 2017
Was this page helpful?
0 / 5 - 0 ratings

Related issues

cwhenderson20 picture cwhenderson20  路  10Comments
annismckenzie picture annismckenzie  路  6Comments
carldanley picture carldanley  路  7Comments
brb picture brb  路  3Comments
emfrias picture emfrias  路  6Comments