Uwsgi: Max-requests causes all workers to be killed at around the same time

[diwu1989]

If I set the max-requests option, very often, all the workers are restarted at around the same time.

Is it possible to stagger the restart? I still need the max-requests feature, but if you initialize the number of requests handled by each worker by
1/ (N workers) * max_requests
then the restarts will be spread out evenly.

[unbit]

well the max-requests option is a trick for reducing memory usage in case of "unfixable" leaks. So it is a good thing you have an almost fair distribution of requests between workers. Do you have another use case we can think/discuss about ?

[diwu1989]

Yes, I use it as safeguard for that situation too, but if you end up having all your workers being restarted at the same time, the server's latency shoots way up because all the workers are still being re-initialized (especially when they are running in lazy mode).

I think if you spread out the restarts over time, that would still solve the memory issue, but you'd see less of a latency spike. Correct?

[diwu1989]

What I'm suggesting will still give a fair distribution to the workers, it just reduces the probability of all the workers restarting simultaneously.

The initial offset should only be set when the master process FIRST starts and will not reset when the workers restart.

[unbit]

ok, so a new option "--max-request-delta 1000" could change the math to
if (current_requests >= max_request + (wid * 1000)) {
recycle();
}

and should address your issue, right ?

[diwu1989]

Yes that would work.

[diwu1989]

thanks!

[avish]

If I understand correctly, the implemented solution means that worker 0 will always have a lower effective max-requests setting, i.e. it will be recycled more often than worker N. This creates a growing gap between workers on subsequent recycles.

For example: Suppose max-requests is 1000 and delta is 100, and we have 4 workers receiving requests at the same rate. Then the first recycling will happen at 4000, 4100, 4200, 4300 requests; then the second recycling will happen at 5000, 5200, 5400, 5600 requests; the third will happen at 6000, 6300, 6600, 6900 requests; and so on. The gap between different recycles will grow over time, at some point converging on a single point and defeating the purpose of this option which is to avoid recycling multiple workers at the same time. Even if it doesn't converge, the duration of the staggered recycle will grow indefinitely and cannot be predicted.

The original proposal of initializing the requests counter of each worker with 1/N * maxRequests (or a configured delta) only once at the time of first forking, would have created a balanced situation where the first recycling is "unfair" (higher-numbered workers will get their first recycling earlier than lower-numbered workers) but all subsequent recycles are fair in that they happen after the exact same number of requests regardless of worker ID, maintaining the staggering order introduced by the first recycling. In a long-running application which relies on periodic recycling, this balance is crucial. A different way of achieving the same thing is to apply the delta * workerId only at the first recycling, and after that use the regular max-requests setting without any delta.

Another option is to allow a random offset to be added to max-requests per worker, per cycle. This is a weaker guarantee for balance between workers but will still avoid the problem of recycling all workers at the same time, while also not creating a persistent imbalance between workers.

Lastly, these techniques could also apply to the max-worker-lifetime option, except with time delta instead of requests delta.