Locust: OOM error with master/slaves setup (zeromq, windows)

[mparent]

Hi !

Describe the bug

An out of memory error occurs with ZeroMQ trying to allocate a crazy amount of memory in decoded_allocator, sometime up to several petabytes. This might very well be a ZeroMQ bug :
OUT OF MEMORY (bundled\zeromq\src\decoder_allocators.cpp:89)

I added some logs and recompiled pyzmq to check what's going on. Upon further investigation, _max_counters seems to take a crazy value at some point. See zmq_logs.txt
As you can see, allocator instance 0x0000016A9270F700 is constructed with _max_counters=249, but before crash its value has changed to 1557249601288, which causes a malloc of several terabytes.

Steps to reproduce

Sorry, I couldn't find a surefire way to reproduce this one. It seems kind of random. It sometime happens before the test is even started, sometime when the test is stopped. Sometime it doesn't happen at all. It does seem to happen more often when stopping a test in the web UI. Simply run the ps1 attached and do some stuff in the web UI.

Environment

• OS: Windows 10.0.18362.778
• Python version: 3.6
• Locust version: 0.14.6
• Locust files : test_locust.zip

I managed to repro the bug on two computers : my work computer and my personal computer. Both are on Windows 10/Python 3.6 that comes with VS2017, but my personal computer has a pristine python environent, just ran pip install locustio.

Am I doing something I'm not supposed to ?

[heyman]

Interesting. You're not doing anything wrong AFAICT. I suspect this is a windows related issue. Is it possible for you to test if you can reproduce this on a linux or mac machine?

[mparent]

Sure thing ! I'll try on WSL, if that works for you. Worst case, I can set up a Linux VM.

[mparent]

Ok, I tried several times on Ubuntu 18.04 LTS on WSL, running it the exact same way with Powershell Core. I couldn't repro the issue.
I can't say for sure that it won't happen on Linux, considering how inconsistent that bug is, but at the very least I can safely say that it is much less likely to happen. Considering it a Windows-related issue does seem plausible.

[heyman]

Ok, good to know! It might be awhile before I get the chance to try to reproduce this on a Windows machine. Please keep us updated if you test anything else (e.g. another version of ZeroMQ or Python).

[mparent]

Will do !

[anuj-ssharma]

Tested this on my windows machine and I can reproduce this (on v1.0.0 of locust and python 3.8). Used a different locust file to the one attached above.

However, I couldn't really figure out a pattern to the failures but there were a few observations:

1. It always failed for me after all the clients were connected and ready.
2. Chances of failures increased with the increase in the number of workers.
3. Memory consumption of the locust workers seemed to be normal.

Machine Specs:
Windows 10 Version 1903
Intel(R) Core i7-7700HQ CPU @ 2.80 GHz
16GB RAM

[cyberw]

Maybe we should bump the minimum required pyzmq version? Other than that I dont think we can do much without a clear repro case. @anuj-ssharma @mparent can you check your pyzmq versions?

[mparent]

@cyberw 19.0.1 for me.

[cyberw]

Ok, that is the latest, so that shouldnt be an issue...

[cyberw]

I dont have any real ideas on how to solve this, and I hardly use Windows at all these days.

If any of you have the time to do some more digging & finding a fix it would be much appreciated (unfortunately it is unlikely anyone else will fix it for you :-/ )

[mparent]

It's fine, our actual locusts are run on Linux anyway. I'll simply keep working on WSL locally and try to find a fix if I have some time.

[bebeo92]

@cyberw I also faced this issue when run on Window

[cyberw]

@bebeo92 Have you tried updating to latest pyzmq? Can you find any pattern to when it works and when it doesnt?

With no more details there is nothing we can do, sorry... (and with so few of our users running on windows I dont think this issue will get a lot of attention)

perhaps file an issue with pyzmq?

[bebeo92]

@cyberw I think it happens when I click on Stop button. Does it expect behaviour?

[cyberw]

It should work. Sorry, I dont think I can help you...

[bebeo92]

@cyberw I think it still a valid bug, can you contact someone else to verify it?

[cyberw]

I agree, but there is really nobody to contact. this is a project maintained by volunteers.

[cyberw]

like I said, you may have more luck talking to the maintainers of pyzmq itself.