Client-go: portforward.go:331] an error occurred forwarding

[FaKod]

K8s 1.9.3 kubectl on MacOS (to 1.9.3 K8s Cluster on Container Linux)

Doing a simple port-forward (f.e. to tiller) on my mac bash results in:

E0216 15:52:41.029173   40160 portforward.go:331] an error occurred forwarding 44134 -> 44134: error forwarding port 44134 to pod 33eb364bdf5a87418a05acd16e35822ec4146eed7e1fca4d41ab2a8cc43b594b, uid : exit status 1: 2018/02/16 14:52:40 socat[6313] E getaddrinfo("localhost", "NULL", {1,2,1,6}, {}): Temporary failure in name resolution

This exception occurs only after trying to send data.

Any idea what can be the cause of this?

Complete log:

> kubectl -n kube-system port-forward tiller-deploy-5b48764ff7-pm6cd 44134:44134 --v=10
I0216 16:32:23.189909   44953 loader.go:357] Config loaded from file /.../kubeconfig
I0216 16:32:23.210702   44953 round_trippers.go:417] curl -k -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: kubectl/v1.9.3 (darwin/amd64) kubernetes/d283541" https://1.2.3.4/api/v1/namespaces/kube-system/pods/tiller-deploy-5b48764ff7-pm6cd
I0216 16:32:25.047948   44953 round_trippers.go:436] GET https://1.2.3.4/api/v1/namespaces/kube-system/pods/tiller-deploy-5b48764ff7-pm6cd 200 OK in 1837 milliseconds
I0216 16:32:25.048006   44953 round_trippers.go:442] Response Headers:
I0216 16:32:25.048022   44953 round_trippers.go:445]     Content-Type: application/json
I0216 16:32:25.048036   44953 round_trippers.go:445]     Content-Length: 2985
I0216 16:32:25.048048   44953 round_trippers.go:445]     Date: Fri, 16 Feb 2018 15:32:24 GMT
I0216 16:32:25.086489   44953 request.go:873] Response Body: {"kind":"Pod","apiVersion":"v1","metadata":{"name":"tiller-deploy-5b48764ff7-pm6cd","generateName":"tiller-deploy-5b48764ff7-","namespace":"kube-system","selfLink":"/api/v1/namespaces/kube-system/pods/tiller-deploy-5b48764ff7-pm6cd","uid":"6472e10f-12f8-11e8-adc0-0671000007fc","resourceVersion":"2211","creationTimestamp":"2018-02-16T09:04:28Z","labels":{"app":"helm","name":"tiller","pod-template-hash":"1604320993"},"ownerReferences":[{"apiVersion":"extensions/v1beta1","kind":"ReplicaSet","name":"tiller-deploy-5b48764ff7","uid":"64712df6-12f8-11e8-adc0-0671000007fc","controller":true,"blockOwnerDeletion":true}]},"spec":{"volumes":[{"name":"tiller-token-pst7g","secret":{"secretName":"tiller-token-pst7g","defaultMode":420}}],"containers":[{"name":"tiller","image":"gcr.io/kubernetes-helm/tiller:v2.8.1","ports":[{"name":"tiller","containerPort":44134,"protocol":"TCP"},{"name":"http","containerPort":44135,"protocol":"TCP"}],"env":[{"name":"TILLER_NAMESPACE","value":"kube-system"},{"name":"TILLER_HISTORY_MAX","value":"0"}],"resources":{},"volumeMounts":[{"name":"tiller-token-pst7g","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"livenessProbe":{"httpGet":{"path":"/liveness","port":44135,"scheme":"HTTP"},"initialDelaySeconds":1,"timeoutSeconds":1,"periodSeconds":10,"successThreshold":1,"failureThreshold":3},"readinessProbe":{"httpGet":{"path":"/readiness","port":44135,"scheme":"HTTP"},"initialDelaySeconds":1,"timeoutSeconds":1,"periodSeconds":10,"successThreshold":1,"failureThreshold":3},"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent"}],"restartPolicy":"Always","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"tiller","serviceAccount":"tiller","nodeName":"k8s-bc-worker-839b7c4","securityContext":{},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}]},"status":{"phase":"Running","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2018-02-16T09:04:28Z"},{"type":"Ready","status":"True","lastProbeTime":null,"lastTransitionTime":"2018-02-16T09:04:40Z"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2018-02-16T09:04:28Z"}],"hostIP":"1.2.3.4","podIP":"10.244.4.4","startTime":"2018-02-16T09:04:28Z","containerStatuses":[{"name":"tiller","state":{"running":{"startedAt":"2018-02-16T09:04:32Z"}},"lastState":{},"ready":true,"restartCount":0,"image":"gcr.io/kubernetes-helm/tiller:v2.8.1","imageID":"docker-pullable://gcr.io/kubernetes-helm/tiller@sha256:394fb7d5f2fbaca54f6a0dec387cef926f6ae359786c89f7da67db173b97a322","containerID":"docker://9aa5cb229a52d53da9dc53d73588067c668657ee937d034b0713191992b3371d"}],"qosClass":"BestEffort"}}
I0216 16:32:25.130036   44953 round_trippers.go:417] curl -k -v -XPOST  -H "X-Stream-Protocol-Version: portforward.k8s.io" -H "User-Agent: kubectl/v1.9.3 (darwin/amd64) kubernetes/d283541" https://1.2.3.4/api/v1/namespaces/kube-system/pods/tiller-deploy-5b48764ff7-pm6cd/portforward
I0216 16:32:32.044524   44953 round_trippers.go:436] POST https://1.2.3.4/api/v1/namespaces/kube-system/pods/tiller-deploy-5b48764ff7-pm6cd/portforward 101 Switching Protocols in 6914 milliseconds
I0216 16:32:32.044583   44953 round_trippers.go:442] Response Headers:
I0216 16:32:32.044606   44953 round_trippers.go:445]     Connection: Upgrade
I0216 16:32:32.044625   44953 round_trippers.go:445]     Upgrade: SPDY/3.1
I0216 16:32:32.044647   44953 round_trippers.go:445]     X-Stream-Protocol-Version: portforward.k8s.io
I0216 16:32:32.044662   44953 round_trippers.go:445]     Date: Fri, 16 Feb 2018 15:32:29 GMT
Forwarding from 127.0.0.1:44134 -> 44134
Handling connection for 44134
E0216 16:32:36.106965   44953 portforward.go:331] an error occurred forwarding 44134 -> 44134: error forwarding port 44134 to pod 33eb364bdf5a87418a05acd16e35822ec4146eed7e1fca4d41ab2a8cc43b594b, uid : exit status 1: 2018/02/16 15:32:35 socat[13536] E getaddrinfo("localhost", "NULL", {1,2,1,6}, {}): Temporary failure in name resolution

[FaKod]

ok, the reason for this:

• empty /etc/hosts file in kubelet container (rkt on Container Linux)
• because of a missing "--hosts-entry host" in the kubelet-wrapper script