Skaffold: Skaffold duplicating port forwarding events

[etanshaul]

See https://github.com/GoogleCloudPlatform/cloud-code-intellij/issues/2572 for context and screenshots.

What I'm seeing:
Using Skaffold with the Cloud Code NodeJs Sample Guestbook Application in debug or continuous dev mode, I'm noticing multiple repeat port forward events (to the same local port):

(skaffold debug)

(skaffold dev)

Notice the many repeated port forward events from the backend service.

What I expected
Only a single port forward event per service/port.

@priyawadhwa perhaps you can help shed some light on why this is happening (I assume it has to do with the initial mongodb connection failure shown in the logs)?

noticed this behavior in Skaffold v 0.29, and 0.30

[priyawadhwa]

Hey @etanshaul, so the port forwarder runs whenever it receives a "pod added" or "pod modified" event. From the logs, it does seem like this this bug is related to the monogodb connection failure, but I'm guessing this doesn't modify the pod spec in any way?

[etanshaul]

afaik, the podspec is not modified in any way. This particular app seems to get stuck in an infinite loop of mongo db failure followed by reforwarding the ports.

@briandealwis in case he encountered this before during testing skaffold debug

[briandealwis]

I haven't looked at the code, but if the container is failing that will cause it's pod status to be updated, which will kick a watcher event. Right?

[etanshaul]

what do you mean by "watcher event" exactly?

Seems like it might be based the podspec's restartPolicy which seems to be defaulted to Always: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#restart-policy

[briandealwis]

Oh! That makes more sense and would explain why the port can be bound to the same local port.

[etanshaul]

Yeah.

I’m not sure if Skaffold can do much here as this can be considered intended behavior, I suppose.

I wonder though - with the move to port forwarding services instead of pods, perhaps the port forward session will survive these container reboots?

[priyawadhwa]

@etanshaul I'll do some testing and get back to you!

[priyawadhwa]

So turns out port forwarding does not survive container reboots, since the connection is broken a new one must be started with kubectl port-forward to connect to the new container (I was hoping Kubernetes would just connect the new container automatically, but looks like it doesn't do that).

[etanshaul]

Thanks for investigating @priyawadhwa

So turns out port forwarding does not survive container reboots

So just to confirm - this is even in the case of port forwarding services, instead of pods, right?

[etanshaul]

And follow up point:

It seems to me that, for now, there is nothing Skaffold can reasonably do here to improve the experience. Do you agree?

That being said, I think the health check effort would be a great place to explore how we can improve this in the future.

[priyawadhwa]

So just to confirm - this is even in the case of port forwarding services, instead of pods, right?

Yah that's the case for services as well, since forwarding a service really just is forwarding to one of the pods attached to the service.

It seems to me that, for now, there is nothing Skaffold can reasonably do here to improve the experience. Do you agree?

Yah, agreed. For now, skaffold will just port-forward pods/services on each dev loop. If the forwarding fails, it won't be restarted until the next loop.

That being said, I think the health check effort would be a great place to explore how we can improve this in the future.

I definitely think integrating the health check effort with port forwarding would be the right way to handle this. Current focus is refactoring port forwarding until it's stable, but this would be the next step after that.

[priyawadhwa]

With the new port forwarding redesign (#2215) I think this should be fixed now -- I'm going to go ahead and close this issue, but please reopen or comment if you still run into this.