Kind: port-forwarding broken in kindest/node:v1.18.2 but works on kindest/node:v1.18.0

Created on 11 Jul 2020 · 35Comments · Source: kubernetes-sigs/kind

What happened:

port-forwarding to a Kind cluster doesn't appear to work after 1.18.2 (works OK on 1.18.0)

How to reproduce it (as minimally and precisely as possible):

Run:

kind create cluster --image=kindest/node:v1.18.2

Apply this YAML

apiVersion: apps/v1
kind: Deployment
metadata:
  name: example-html
  labels:
    app: example-html
spec:
  selector:
    matchLabels:
      app: example-html
  template:
    metadata:
      labels:
        app: example-html
    spec:
      containers:
      - name: example-html
        image: busybox
        command: ["busybox", "httpd", "-f", "-p", "8000"]
        ports:
        - containerPort: 8000

Wait until the pod is ready
Run

kubectl port-forward deployment/example-html 8000:8000

Visit http://localhost:8000/

Expected result:
a 404 response

Actual result:
Connection hangs indefinitely

In the kubectl port-forward terminal, I see:

E0710 19:56:07.104125 2250228 portforward.go:400] an error occurred forwarding 8000 -> 8000: error forwarding port 8000 to pod 7e6be7fb9a3b95c5c80f206f931b339d955f4fd9423a393f7ee072c477d9f370, uid : failed to execute portforward in network namespace "/var/run/netns/cni-1b6beba4-efb4-40fc-741e-fcce3506a3d3": socat command returns error: exit status 1, stderr: "2020/07/10 23:56:07 socat[1814] E connect(5, AF=2 92.242.140.21:8000, 16): Connection timed out\n"

Anything else we need to know?:

If I start Kind with kind create cluster --image=kindest/node:v1.18.0, port-forwarding works fine

I am so curious what the problem is here, and would love tips on how to debug this further. I've been banging my head against the wall for awhile

Environment:

kind version: (use kind version): kind v0.8.1 go1.14.3 linux/amd64
Kubernetes version: (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.5", GitCommit:"e6503f8d8f769ace2f338794c914a96fc335df0f", GitTreeState:"clean", BuildDate:"2020-07-07T14:04:52Z", GoVersion:"go1.13.12", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.2", GitCommit:"52c56ce7a8272c798dbc29846288d7cd9fbae032", GitTreeState:"clean", BuildDate:"2020-04-30T20:19:45Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"

Docker version: (use docker info):
Server Version: 19.03.11
OS (e.g. from /etc/os-release):
NAME="Ubuntu"
VERSION="20.04 LTS (Focal Fossa)"

kinbug prioritimportant-soon

Source

nicks

Most helpful comment

yes, sorry!! To be clear, port-forwarding is broken with the default image. I was using the --image= flag to try to diagnose why it's broken, didn't mean to imply that I expected this to be a good way to operate kind :joy:

nicks on 11 Jul 2020

❤1 👍1

All 35 comments

fIrst a reminder that using arbitrary images like this is NOT supported. if you're consuming the images we build, please check the release notes for the exact supported images for you release. We do wind up pushing new images later for other releases.

https://github.com/kubernetes-sigs/kind/releases has the images with digests for each kind version.

@aojea this smells like something with gocat https://github.com/containerd/cri/pull/1470