Locust: autoscaling slaves resets users

[matti]

autoscaling works great, but when a new slave is added, it slows down the RPS curve, because slaves are reset - wouldn't it be better if the slaves would not go all the day down to 0?

[heyman]

I agree. The problem is this line:

https://github.com/locustio/locust/blob/013d64ac6de8e4b6f07fc2b704eeea55601e9858/locust/runners.py#L336

I believe that it might be completely unnecessary, and that we can simply fix the problem by removing that line. Did a quick test that seems to support this, but will look a little deeper into it.

[heyman]

I believe that it might be completely unnecessary, and that we can simply fix the problem by removing that line.

That wasn't really true, but the problem have now been solved in c16dc71

[matti]

works now, thanks

[max-rocket-internet]

@heyman

I'm still seeing this in version 0.13.1.

Logs look OK. Master:

[2019-11-18 16:26:23,697] locust-opa-master-787476b668-mlrz9/INFO/locust.main: Starting web monitor at *:8089
[2019-11-18 16:26:23,698] locust-opa-master-787476b668-mlrz9/INFO/locust.main: Starting Locust 0.13.1
[2019-11-18 16:26:53,182] locust-opa-master-787476b668-mlrz9/INFO/locust.runners: Client 'locust-opa-slave-86cf6df9fb-kmhlw_aebbf1d9b85e4bc4b5d18431d9a4c0d3' reported as ready. Currently 1 clients ready to swarm.
[2019-11-18 16:34:47,209] locust-opa-master-787476b668-mlrz9/INFO/locust.runners: Sending hatch jobs to 1 ready clients
[2019-11-18 16:36:19,065] locust-opa-master-787476b668-mlrz9/INFO/locust.runners: Client 'locust-opa-slave-86cf6df9fb-phgt9_47a162891b3a448ea366205c37607157' reported as ready. Currently 2 clients ready to swarm.
[2019-11-18 16:36:19,065] locust-opa-master-787476b668-mlrz9/INFO/locust.runners: Sending hatch jobs to 2 ready clients
[2019-11-18 16:38:20,539] locust-opa-master-787476b668-mlrz9/INFO/locust.runners: Client 'locust-opa-slave-86cf6df9fb-4kfkf_a2f31c4d7f7d405a95b5ca4d3d8c3d36' reported as ready. Currently 3 clients ready to swarm.
[2019-11-18 16:38:20,539] locust-opa-master-787476b668-mlrz9/INFO/locust.runners: Sending hatch jobs to 3 ready clients
[2019-11-18 16:41:22,522] locust-opa-master-787476b668-mlrz9/INFO/locust.runners: Client 'locust-opa-slave-86cf6df9fb-vvjb4_a0b8d33b274e40b58ff4a8225370dd8b' reported as ready. Currently 4 clients ready to swarm.
[2019-11-18 16:41:22,523] locust-opa-master-787476b668-mlrz9/INFO/locust.runners: Sending hatch jobs to 4 ready clients
[2019-11-18 16:41:22,731] locust-opa-master-787476b668-mlrz9/INFO/locust.runners: Client 'locust-opa-slave-86cf6df9fb-scbd6_065d9980a1ee4629b15b5b79aaed6ff2' reported as ready. Currently 5 clients ready to swarm.
[2019-11-18 16:41:22,731] locust-opa-master-787476b668-mlrz9/INFO/locust.runners: Sending hatch jobs to 5 ready clients
[2019-11-18 16:44:24,490] locust-opa-master-787476b668-mlrz9/INFO/locust.runners: Client 'locust-opa-slave-86cf6df9fb-zfz9d_306a9987e14042efab8aec45720acf28' reported as ready. Currently 6 clients ready to swarm.
[2019-11-18 16:44:24,491] locust-opa-master-787476b668-mlrz9/INFO/locust.runners: Sending hatch jobs to 6 ready clients
[2019-11-18 16:44:24,622] locust-opa-master-787476b668-mlrz9/INFO/locust.runners: Client 'locust-opa-slave-86cf6df9fb-cddmn_06ff9d1f91334e809499cf522bd838cc' reported as ready. Currently 7 clients ready to swarm.
[2019-11-18 16:44:24,623] locust-opa-master-787476b668-mlrz9/INFO/locust.runners: Sending hatch jobs to 7 ready clients
[2019-11-18 16:47:26,348] locust-opa-master-787476b668-mlrz9/INFO/locust.runners: Client 'locust-opa-slave-86cf6df9fb-x95nc_6172fcc3741b4408aa6db4c5b933eee3' reported as ready. Currently 8 clients ready to swarm.
[2019-11-18 16:47:26,348] locust-opa-master-787476b668-mlrz9/INFO/locust.runners: Sending hatch jobs to 8 ready clients
[2019-11-18 16:47:26,567] locust-opa-master-787476b668-mlrz9/INFO/locust.runners: Client 'locust-opa-slave-86cf6df9fb-6fgrl_6c203c03e497472289ed644e2e7ff09f' reported as ready. Currently 9 clients ready to swarm.
[2019-11-18 16:47:26,568] locust-opa-master-787476b668-mlrz9/INFO/locust.runners: Sending hatch jobs to 9 ready clients

Slaves look like this:

[2019-11-18 16:26:36,988] locust-opa-slave-86cf6df9fb-kmhlw/INFO/locust.main: Starting Locust 0.13.1
[2019-11-18 16:34:47,210] locust-opa-slave-86cf6df9fb-kmhlw/INFO/locust.runners: Hatching and swarming 1000 clients at the rate 0.5 clients/s...
[2019-11-18 16:36:19,066] locust-opa-slave-86cf6df9fb-kmhlw/INFO/locust.runners: Killing 500 locusts
[2019-11-18 16:38:20,540] locust-opa-slave-86cf6df9fb-kmhlw/INFO/locust.runners: Killing 167 locusts
[2019-11-18 16:41:22,525] locust-opa-slave-86cf6df9fb-kmhlw/INFO/locust.runners: Killing 83 locusts
[2019-11-18 16:41:22,732] locust-opa-slave-86cf6df9fb-kmhlw/INFO/locust.runners: Killing 50 locusts
[2019-11-18 16:44:24,492] locust-opa-slave-86cf6df9fb-kmhlw/INFO/locust.runners: Killing 33 locusts
[2019-11-18 16:44:24,626] locust-opa-slave-86cf6df9fb-kmhlw/INFO/locust.runners: Killing 24 locusts
[2019-11-18 16:47:26,352] locust-opa-slave-86cf6df9fb-kmhlw/INFO/locust.runners: Killing 18 locusts
[2019-11-18 16:47:26,572] locust-opa-slave-86cf6df9fb-kmhlw/INFO/locust.runners: Killing 14 locusts
[2019-11-18 16:50:28,707] locust-opa-slave-86cf6df9fb-kmhlw/INFO/locust.runners: Killing 11 locusts
[2019-11-18 16:50:28,766] locust-opa-slave-86cf6df9fb-kmhlw/INFO/locust.runners: Killing 9 locusts

[max-rocket-internet]

@heyman can we reopen this issue?

[heyman]

I'm still seeing this in version 0.13.1

Ah, yes, I managed to reproduce this with the --reset-stats parameter specified (I assume you use that?).

It's actually a different cause than the original issue (though a similar symptom). I've opened a new issue: #1168