Locust: Web UI does not stop slave servers

[giantryansaul]

Description of issue / feature request

When clicking "Stop" on the web UI, the users on Slave servers will not stop sending requests. The master instance still says the test is in the "running" state, but the stop button has disappeared.

Expected behavior

All slave servers are stopped and the user count goes back to 0.

Actual behavior

All slave servers remain active and the count of users is over 0.

Environment settings (for bug reports)

• OS: Ubuntu 16.04
• Python version: 3.6
• Locust version: 0.9.0

Steps to reproduce (for bug reports)

(can't share my current code, but it is a very simple mix of get and post requests and no processing or validation of data)

• Setup a simple requests task.
• Setup 3 or more slave servers with a master server to distribute task.
• Run in Web UI mode
• Run for over 20 minutes
• Click the Stop button on the Web UI

[Jonnymcc]

Hi @giantryansaul, I have created this PR https://github.com/locustio/locust/pull/927 which I believe will solve the issue you are experiencing. If you have time can you test it out? Thanks!

[giantryansaul]

Thanks @Jonnymcc I don't have any time currently, but I'll see if I can get to setting up a test next week. Thanks!

[andrerivas]

Hey @Jonnymcc, I've tried to use the branch you created but it still does not kill slaves. Not permanently, anyway. I have seen that pressing Stop will briefly stop all requests, but after a few seconds, the slaves start sending requests again.

Is this the correct way to install your branch for testing?

pip install git+https://github.com/Jonnymcc/locust.git@heartbeat#egg=locustio

[Jonnymcc]

That is odd. Looks like you are installing it the right way. In my testing, even today, I cannot replicate the problem you are seeing. Provided the stop signal is sent and the slave receives it and stops, I do not know how the tasks would start up again. Unless the master continued to send hatch jobs.

What do you see in the master logs? Here is a test to see if you are using the right install. I created a master in one shell and a slave in another. Then I closed the other shell (without first stopping the slave). Eventually the master misses the heartbeats and logs the slave as disconnected.

[2019-02-08 08:41:09,617] JonathanMBP.local/INFO/locust.main: Starting web monitor at *:8089
[2019-02-08 08:41:09,618] JonathanMBP.local/INFO/locust.main: Starting Locust 0.9.0
[2019-02-08 08:41:32,529] JonathanMBP.local/INFO/locust.runners: Client 'JonathanMBP.local_13eb0a9bb33744248001d5df851768cd' reported as ready. Currently 1 clients ready to swarm.
[2019-02-08 08:41:57,778] JonathanMBP.local/INFO/locust.runners: Slave JonathanMBP.local_13eb0a9bb33744248001d5df851768cd failed to send heartbeat, setting state to missing.

[LRAbbade]

I'm having the same problem, RPS drops to zero but then it starts to climb again, sometimes creating even more users than originally designated. Only "solution" I've found so far is deleting kube deployment.

[Jonnymcc]

In kubernetes it could be that new pods are coming up and registering as
newly attached slaves. Not sure why the slave pods would exit though. I
thought even when they are not running they do not exit. I may be wrong.

On Mon, May 13, 2019, 11:53 AM Lucas Abbade [e-mail]
wrote:

I'm having the same problem, RPS drops to zero but then it starts to climb
again, sometimes creating even more users than originally designated. Only
"solution" I've found so far is deleting kube deployment.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/locustio/locust/issues/911?email_source=notifications&email_token=AAH6JPQ4RNBWGKVD4PBPEMLPVGFGVA5CNFSM4GCLUU3KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODVIX7EI#issuecomment-491880337,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAH6JPSFLGBFPC3XVNSDXRLPVGFGVANCNFSM4GCLUU3A
.

[LRAbbade]

I think you are right, the problem might be that I'm running from the Python image, and the container probably exits once the application exits, making kubernetes start another instance. I will look into this later today, thanks for the help!

[jayudey-vertex]

I'm still seeing this behavior when running locust 0.11.0 in docker swarm.

UPDATE: I started to look further into the issue this morning. When I installed the current version of master it seems to be working as expected.

[clapero]

I am also seeing this issue with Locust 0.11.0. Same as above, I am deploying the Master and Slaves on different pods on a Kubernetes cluster.
From @Jonnymcc : "Provided the stop signal is sent and the slave receives it and stops" makes me wonder if there are specific ports to open on the Slave pods to actually receive the signals. I currently only have the ports 80, 443 and 8089.

[liuchunming033]

any update?

[tamilhce]

Im also facing this issue, any update on this

[max-rocket-internet]

Also seeing this issue. Quite annoying. Only way to solve it is to delete all locust pods.

[tsykora-verimatrix]

(btw) this is not limited to kubernetes deployments and the issue still persist in locust 0.11.0

[tamilhce]

Any update on this issue ?

[Jonnymcc]

Just tested this out with v0.8.1 on my Mac with two slaves. I let it run for two hours and then stopped it and the slaves stayed stopped without sending more requests. 🤷‍♂

[tamilhce]

@Jonnymcc, the problem here is, if delete /power off the salve machines.In the locust master it still shows the 2 slaves machines in ready state.
Could you delete the slave machines and check in the master if the slaves count changes to zero.

[Jonnymcc]

I stopped a slave abruptly by terminating the shell instead of sending sigterm. This is what I see and is as expected. If I restart the test, only the one ready slave will begin sending requests.

[max-rocket-internet]

Just tested this out with v0.8.1 on my Mac with two slaves

For me it seems to work fine with a very low number of slaves, i.e 2-6. But usually we are running 20-80 and it doesn't work at all.

[yp-photobox]

I'm seeing the same issue running locust 0.11.0 as docker containers on AWS ECS

[sonja455]

I think you are right, the problem might be that I'm running from the Python image, and the container probably exits once the application exits, making kubernetes start another instance. I will look into this later today, thanks for the help!

Did you find a solution for this problem so far? Is there a way to stop kubernetes from starting another instance?

[cgoldberg]

Any update on this issue ?

Updates would be posted here and the issue would be closed.

[finchmeister]

Installing the latest master version in the dockerfile worked for me:

RUN pip install -e git://github.com/locustio/locust.git@master#egg=locustio

Background: The fix was made in #982 and the version was incremented to 0.11.1 however right now, by default pip will give you an older version without the fix.

[jebrage]

This hack worked for me. Also it can be done in the web ui itself.

Hit 'Edit' or 'New test' below status on top right.

Then make number of users=0.

All the users should stop and thus the tests should also stop running.

[jmattiace]

Just confirmed it works for me as well