Skaffold: Port forwarding doesn't work anymore since update 0.33.0 -> 0.34.0

[Bastorx]

Sadly no error logs

Expected behavior

Port fowarding should work

Actual behavior

Port fowarding doesn't work

Information

• Skaffold version: 0.34.0
• Operating system: MacOS 10.14.5
• Contents of skaffold.yaml:

apiVersion: skaffold/v1beta13
kind: Config
build:
  tagPolicy:
    sha256: {}
  artifacts:
    - context: api
      image: eu.gcr.io/my-app/api
      sync:
        manual:
          - src: "src/**/*"
            dest: .
    - context: client
      image: eu.gcr.io/my-app/client
      sync:
        manual:
          - src: "src/**/*"
            dest: .
      docker:
        buildArgs:
          NODE_ENV: development
deploy:
  kubectl:
    manifests:
      - ./k8-manifests/client/*
      - ./k8-manifests/api/*
profiles:
  - name: api
    deploy:
      kubectl:
        manifests:
          - ./k8-manifests/api/*
  - name: client
    deploy:
      kubectl:
        manifests:
          - ./k8-manifests/client/*
portForward:
  - resourceType: service
    resourceName: my-app-service
    port: 3000
    localPort: 3000
  - resourceType: service
    resourceName: my-app-client-service
    port: 8080
    localPort: 8080

Steps to reproduce the behavior

1. Create a node/typescript project. Doesn't share sources. Try to forwards port.
2. skaffold dev --port-forwarding

Would like to know if others have this kind of issue since last update

[Bastorx]

It always work one time. After a Ctrl + C and a skaffold delete. It seems doesn't work anymore.

I think Skaffold delete doesn't remove correctly pods and ports fowardings rules.

[prokher]

As for me skaffold dev --port-forward works until I change a source. Skaffold restarts services and that is it, port forwarding does not work anymore. Actually, the issue is here #2408.

[balopat]

This is a race issue: the old pod gets killed and the new pod gets started after kubectl port-forward is already started at the beginning of the new dev loop. The problem is that kubectl port-forward doesn't fail, it just logs things like:

Handling connection for 50053 E0722 16:20:30.827381 155541 portforward.go:400] an error occurred forwarding 50053 -> 50051: error forwarding port 50051 to pod 6b8250b5be8d3e65ed5d9c900cb87966bed006b57cc81617d27b6ba271742815, uid : Error: No such container: 6b8250b5be8d3e65ed5d9c900cb87966bed006b57cc81617d27b6ba271742815 Handling connection for 50053 Handling connection for 50053 Handling connection for 50053 Handling connection for 50053 Handling connection for 50053 Handling connection for 50053 Handling connection for 50053 E0722 16:21:00.929687 155541 portforward.go:340] error creating error stream for port 50053 -> 50051: Timeout occured E0722 16:21:00.969972 155541 portforward.go:362] error creating forwarding stream for port 50053 -> 50051: Timeout occured E0722 16:21:02.989783 155541 portforward.go:362] error creating forwarding stream for port 50053 -> 50051: Timeout occured E0722 16:21:03.998054 155541 portforward.go:362] error creating forwarding stream for port 50053 -> 50051: Timeout occured E0722 16:21:04.598329 155541 portforward.go:340] error creating error stream for port 50053 -> 50051: Timeout occured E0722 16:21:05.577799 155541 portforward.go:362] error creating forwarding stream for port 50053 -> 50051: Timeout occured Handling connection for 50053 E0722 16:21:06.166770 155541 portforward.go:362] error creating forwarding stream for port 50053 -> 50051: Timeout occured E0722 16:21:35.578937 155541 portforward.go:340] error creating error stream for port 50053 -> 50051: Timeout occured Handling connection for 50053 Handling connection for 50053 E0722 16:21:40.688533 155541 portforward.go:400] an error occurred forwarding 50053 -> 50051: error forwarding port 50051 to pod 6b8250b5be8d3e65ed5d9c900cb87966bed006b57cc81617d27b6ba271742815, uid : Error: No such container: 6b8250b5be8d3e65ed5d9c900cb87966bed006b57cc81617d27b6ba271742815 E0722 16:22:10.606373 155541 portforward.go:340] error creating error stream for port 50053 -> 50051: Timeout occured Handling connection for 50053 Handling connection for 50053 E0722 16:22:40.712581 155541 portforward.go:340] error creating error stream for port 50053 -> 50051: Timeout occured E0722 16:22:40.712668 155541 portforward.go:340] error creating error stream for port 50053 -> 50051: Timeout occured

this scenario could happen when the pod gets restarted behind the service due to a failure or manual pod deletion. I think we should restart kubectl port-forward until https://github.com/kubernetes/kubectl/issues/686 gets picked up.

[prokher]

Could there be some workaround for this issue, until the fix released?

[balopat]

The fix is coming in #2543 and probably will be merged today. Then, we'll cut an emergency release.

[Bastorx]

@balopat Seems half resolved... In my example. It works only for only one service. Never both at the same time.
In logs I can see

Port forwarded service/my-app-api-service from remote port 3000 to local port 3000
Port forwarded service/my-app-client-service from remote port 8080 to local port 8080

When I check sudo lsof -iTCP -sTCP:LISTEN -n -P

Only port 3000 is listening. Or sometime. Only 8080...

It's not each time... Sometimes both works... It's hard to specify.

[balopat]

@Bastorx can you try with current bleeding edge? We added some more error handling logic around kubectl portforward.

[balopat]

Closing as these were addressed and now integration tested as well.