Voila: Voila process is leaking threads

[roman-kouzmenko]

Each time a notebook is accessed, voila is adding 3 threads that don't go away even as kernels are culled, eventually depleting all server resources.

Is anyone observing the same / has a fix?

[roman-kouzmenko]

Debugging this a bit, it seems these threads are client channels that might not be disposed of properly, below is the output of lsof|grep [VOILA_PID]|grep TCP:

voila     146700                jovyan    4u     IPv4          753391854       0t0        TCP *:8813 (LISTEN)
voila     146700                jovyan   34u     IPv4          755101912       0t0        TCP localhost:59017->localhost:59017 (ESTABLISHED)
voila     146700                jovyan   47u     IPv4          755260227       0t0        TCP localhost:41223->localhost:41223 (ESTABLISHED)
voila     146700                jovyan   50u     IPv4          754614991       0t0        TCP localhost:49283->localhost:49283 (ESTABLISHED)
ZMQbg/Rea 146700 165411         jovyan    4u     IPv4          753391854       0t0        TCP *:8813 (LISTEN)
ZMQbg/Rea 146700 165411         jovyan   34u     IPv4          755101912       0t0        TCP localhost:59017->localhost:59017 (ESTABLISHED)
ZMQbg/Rea 146700 165411         jovyan   47u     IPv4          755260227       0t0        TCP localhost:41223->localhost:41223 (ESTABLISHED)
ZMQbg/Rea 146700 165411         jovyan   50u     IPv4          754614991       0t0        TCP localhost:49283->localhost:49283 (ESTABLISHED)
ZMQbg/IO/ 146700 165412         jovyan    4u     IPv4          753391854       0t0        TCP *:8813 (LISTEN)
ZMQbg/IO/ 146700 165412         jovyan   34u     IPv4          755101912       0t0        TCP localhost:59017->localhost:59017 (ESTABLISHED)
ZMQbg/IO/ 146700 165412         jovyan   47u     IPv4          755260227       0t0        TCP localhost:41223->localhost:41223 (ESTABLISHED)
ZMQbg/IO/ 146700 165412         jovyan   50u     IPv4          754614991       0t0        TCP localhost:49283->localhost:49283 (ESTABLISHED)
ZMQbg/Rea 146700 165413         jovyan    4u     IPv4          753391854       0t0        TCP *:8813 (LISTEN)
ZMQbg/Rea 146700 165413         jovyan   34u     IPv4          755101912       0t0        TCP localhost:59017->localhost:59017 (ESTABLISHED)
ZMQbg/Rea 146700 165413         jovyan   47u     IPv4          755260227       0t0        TCP localhost:41223->localhost:41223 (ESTABLISHED)
ZMQbg/Rea 146700 165413         jovyan   50u     IPv4          754614991       0t0        TCP localhost:49283->localhost:49283 (ESTABLISHED)
ZMQbg/IO/ 146700 165414         jovyan    4u     IPv4          753391854       0t0        TCP *:8813 (LISTEN)
ZMQbg/IO/ 146700 165414         jovyan   34u     IPv4          755101912       0t0        TCP localhost:59017->localhost:59017 (ESTABLISHED)
ZMQbg/IO/ 146700 165414         jovyan   47u     IPv4          755260227       0t0        TCP localhost:41223->localhost:41223 (ESTABLISHED)
ZMQbg/IO/ 146700 165414         jovyan   50u     IPv4          754614991       0t0        TCP localhost:49283->localhost:49283 (ESTABLISHED)
voila     146700 165415         jovyan    4u     IPv4          753391854       0t0        TCP *:8813 (LISTEN)
voila     146700 165415         jovyan   34u     IPv4          755101912       0t0        TCP localhost:59017->localhost:59017 (ESTABLISHED)
voila     146700 165415         jovyan   47u     IPv4          755260227       0t0        TCP localhost:41223->localhost:41223 (ESTABLISHED)
voila     146700 165415         jovyan   50u     IPv4          754614991       0t0        TCP localhost:49283->localhost:49283 (ESTABLISHED)
ZMQbg/Rea 146700 165417         jovyan    4u     IPv4          753391854       0t0        TCP *:8813 (LISTEN)
ZMQbg/Rea 146700 165417         jovyan   34u     IPv4          755101912       0t0        TCP localhost:59017->localhost:59017 (ESTABLISHED)
ZMQbg/Rea 146700 165417         jovyan   47u     IPv4          755260227       0t0        TCP localhost:41223->localhost:41223 (ESTABLISHED)
ZMQbg/Rea 146700 165417         jovyan   50u     IPv4          754614991       0t0        TCP localhost:49283->localhost:49283 (ESTABLISHED)
ZMQbg/IO/ 146700 165418         jovyan    4u     IPv4          753391854       0t0        TCP *:8813 (LISTEN)
ZMQbg/IO/ 146700 165418         jovyan   34u     IPv4          755101912       0t0        TCP localhost:59017->localhost:59017 (ESTABLISHED)
ZMQbg/IO/ 146700 165418         jovyan   47u     IPv4          755260227       0t0        TCP localhost:41223->localhost:41223 (ESTABLISHED)
ZMQbg/IO/ 146700 165418         jovyan   50u     IPv4          754614991       0t0        TCP localhost:49283->localhost:49283 (ESTABLISHED)
voila     146700 165419         jovyan    4u     IPv4          753391854       0t0        TCP *:8813 (LISTEN)
voila     146700 165419         jovyan   34u     IPv4          755101912       0t0        TCP localhost:59017->localhost:59017 (ESTABLISHED)
voila     146700 165419         jovyan   47u     IPv4          755260227       0t0        TCP localhost:41223->localhost:41223 (ESTABLISHED)
voila     146700 165419         jovyan   50u     IPv4          754614991       0t0        TCP localhost:49283->localhost:49283 (ESTABLISHED)
ZMQbg/Rea 146700 165518         jovyan    4u     IPv4          753391854       0t0        TCP *:8813 (LISTEN)
ZMQbg/Rea 146700 165518         jovyan   34u     IPv4          755101912       0t0        TCP localhost:59017->localhost:59017 (ESTABLISHED)
ZMQbg/Rea 146700 165518         jovyan   47u     IPv4          755260227       0t0        TCP localhost:41223->localhost:41223 (ESTABLISHED)
ZMQbg/Rea 146700 165518         jovyan   50u     IPv4          754614991       0t0        TCP localhost:49283->localhost:49283 (ESTABLISHED)
ZMQbg/IO/ 146700 165519         jovyan    4u     IPv4          753391854       0t0        TCP *:8813 (LISTEN)
ZMQbg/IO/ 146700 165519         jovyan   34u     IPv4          755101912       0t0        TCP localhost:59017->localhost:59017 (ESTABLISHED)
ZMQbg/IO/ 146700 165519         jovyan   47u     IPv4          755260227       0t0        TCP localhost:41223->localhost:41223 (ESTABLISHED)
ZMQbg/IO/ 146700 165519         jovyan   50u     IPv4          754614991       0t0        TCP localhost:49283->localhost:49283 (ESTABLISHED)
voila     146700 165520         jovyan    4u     IPv4          753391854       0t0        TCP *:8813 (LISTEN)
voila     146700 165520         jovyan   34u     IPv4          755101912       0t0        TCP localhost:59017->localhost:59017 (ESTABLISHED)
voila     146700 165520         jovyan   47u     IPv4          755260227       0t0        TCP localhost:41223->localhost:41223 (ESTABLISHED)
voila     146700 165520         jovyan   50u     IPv4          754614991       0t0        TCP localhost:49283->localhost:49283 (ESTABLISHED)

[clydebw]

@roman-kouzmenko I think you're right in that these are client channels not getting cleaned up.

This problem of leaking threads was pointed out to me this morning, and I spent some time today trying to track this down as well. It seems two threads get created here:

https://github.com/voila-dashboards/voila/blob/da39b765a673828e50ffbe8724ae6cc21b5d2d66/voila/handler.py#L166

Although I think these stick around for the app's entire lifecycle, and so are not a problem. Three threads seem to get added each time the notebook is accessed here though, and these don't seem to be cleaned up:

https://github.com/voila-dashboards/voila/blob/da39b765a673828e50ffbe8724ae6cc21b5d2d66/voila/handler.py#L179

I'm going to keep looking, but any help from someone more familiar with the codebase would be much appreciated.

[roman-kouzmenko]

FYI, I mitigated the leak on kubernetes by:
1) creating a liveness probe counting the number of leaked threads and marking the voila container as bad above a certain threshold
2) increasing the grace termination period to a large number to avoid shutting active connections immediately
3) adding a preStop hook to count active kernels and wait on them to finish. If they don't terminate before the grace period in 2), they will get terminated automatically

[dou-du]

We are running Voila app with dokku. The CPUs are keeping eating up by Voila. We have to restart our dokku server every few days. I believe it is caused by this problem. Could anyone fix this issue? Thanks.

[ltalirz]

This is a longstanding issue, see the very related issues
https://github.com/voila-dashboards/voila/issues/209
https://github.com/voila-dashboards/voila/issues/479
https://github.com/voila-dashboards/voila/issues/849

There starts to be some work on it, though, see PRs linked in https://github.com/voila-dashboards/voila/issues/849#issuecomment-801141725