Client-go: HTTP port-forward broke on client terminates

Created on 19 Nov 2017  路  29Comments  路  Source: kubernetes/client-go

Situation

  • Components are connected as Go server -> port-forward -> iOS Client
  • Using chunked transfer encoding to download large movie file from Server to iOS

problem

When iOS client exits with SIGKILL (i.e. restart app from Xcode), the port forward emits an error

E1119 14:59:19.967004    9736 portforward.go:303] error copying from remote stream to local connection: readfrom tcp4 127.0.0.1:8888->127.0.0.1:35390: write tcp4 127.0.0.1:8888->127.0.0.1:35390: write: broken pipe

It is ok, but after this error, all connections to portforward are rejected with timeout.

E1119 15:15:07.167402    9736 portforward.go:271] error creating error stream for port 8888 -> 8888: Timeout occured

Expected behavior

portforward accepts the following requests after broken pipe.

lifecyclrotten

Most helpful comment

Experience same error using Kubernetes in Azure when opening a tunnel to the cluster with az aks browse. Works for a time, then starts to get this error and the tunnel becomes very unstable.

All 29 comments

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

/remove-lifecycle stale
I can confirm that this happens running with linux clients using curl. If the endpoint streams results back to the client, and the client (ie, curl) is terminated via ctrl-c, the port-forward does not respond to future requests until the timeout.

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
/remove-lifecycle stale

/remove-lifecycle rotten

@tomykaira are you still experiencing the same issues? I have the same problem

I definitely still happen. I basically just have to try over and over until it works once

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

/remove-lifecycle stale

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

/remove-lifecycle stale

@yonilerner We had this issue as well on our build servers, though it was not intermittent for this particular port-forward -> curl. Turned out to be an expired cert. Might be worth double-checking.

@s1dequest Thanks for the suggestion, but our certs are fine. port-forward always works, its just not stable

Experience same error using Kubernetes in Azure when opening a tunnel to the cluster with az aks browse. Works for a time, then starts to get this error and the tunnel becomes very unstable.

Can confirm the same on my AKS cluster on azure. The deployment / upgrade goes through and deployment is runned. But the Release pipeline is marked as failed.

4383 portforward.go:363] error copying from remote stream to local connection: readfrom tcp4 127.0.0.1:43163->127.0.0.1:56726: write tcp4 127.0.0.1:43163->127.0.0.1:56726: write: broken pipe

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

/remove-lifecycle stale

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

/remove-lifecycle stale

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

/remove-lifecycle stale

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

/remove-lifecycle stale

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

/remove-lifecycle stale

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Was this page helpful?
0 / 5 - 0 ratings

Related issues

narasago picture narasago  路  3Comments

inetkiller picture inetkiller  路  6Comments

peterwillcn picture peterwillcn  路  5Comments

h0tbird picture h0tbird  路  3Comments

prestonvanloon picture prestonvanloon  路  6Comments