Mybinder.org-deploy: tests after deploy can fail due to ingress reloading

[minrk]

If the ingress configuration has not fully loaded before launching the test, there is a chance that the ingress configuration will reload during the build test and drop the connection, resulting in a build failure. If we reliably waited until the ingress configuration was loaded, this wouldn't happen (#181).

[yuvipanda]

It's possible that nginx-ingress is just not something we can fix for this. Part of their design is they don't route to the stable service IP, but watch for individual pod IPs & reload config when any pod churns. While this is a performance boost (you are eliminating one hop on iptables), it also means there's gonna be tons of reloads...

So our options are:

1. Make our clients (JS and tests) more resilient to dropped connections (this we should do anyway)
2. Try moving to a different ingress controller

[minrk]

I agree we should make clients resilient to dropped connections. It's really the server right now that's not resilient, because a connection cannot be resumed - once a request for launch is initiated, a reconnect (which js eventstream will do automatically, we learned) will be treated as a new request for launch. We handle this well in terms of avoiding duplicate builds, but not avoiding multiple launches.

For the tests we have here, we could punt with a longer sleep to wait for the ingress to reload.

[willingc]

Looks like sleep is now set to 10. Probably makes sense in the short term to increase the sleep pending a solution to multiple launches. Do we have a sense of how much longer the sleep should be to catch most cases: 20, 30?

[minrk]

Seems like it takes a while. We could try 30 or 60.

[willingc]

Pick your favorite or split the difference at 45 ;-)

[minrk]

If we have the logs of a failing test on Travis, we might be able to see an example for how long it took for the failing command to be reached.

A travis_retry on the tests would also work, because once this has been interrupted once, it won't happen again until the next deploy.

[willingc]

Looking at the logs of a few failing tests, they seem to be all over the place in terms of time: ~30s, ~30s, ~600s, ~300s

[yuvipanda]

I think https://github.com/jupyterhub/mybinder.org-deploy/pull/258 should also help.

[yuvipanda]

Going to close this as fixed now since we haven't had an issue with this for a while!