Locust: network_timeout is not respected

[rokcarl]

I'm using network_timeout = 1.0 on a FastHttpLocust subclass, but the slow pages still succeed. I would expect that after 1.0s, the request would stop and another one would start. Additionally, I would expect the page load to count as an error. What I'm seeing though is that Locust is waiting for the page to load and counting it as a success.

I know there's a test in the codebase for this, but it only checks if there was an error in the slow page, but this could be due to other things.

Code for my Flask test web server app.py

import random
import time
from flask import Flask
app = Flask(__name__)

@app.route('/')
def hello_world():
    time.sleep(random.random() * 0.9)
    return 'Hello, World!'


@app.route('/slow')
def slow():
    sleep = 3 + random.random()
    time.sleep(sleep)
    return f'Took me {sleep} s.'



@app.route('/400')
def bad_request():
    return 'Bad Request for your data', 400

My locust.py file:

from locust import TaskSet
from locust.contrib.fasthttp import FastHttpLocust
from locust.wait_time import constant_pacing


def index(l):
    l.client.get("/slow")


class UserBehavior(TaskSet):
    tasks = {index: 1}


class WebsiteUser(FastHttpLocust):
    task_set = UserBehavior
    wait_time = constant_pacing(1.0)
    network_timeout = 1.0
    connection_timeout = 1.0

I run the web server with flask run --reload (need pip install flask) and run the locust without the GUI using locust -f locust.py --no-web -c 10 -r 10 -t 10s --host http://localhost:5000.

Here are the results

locust -f locust.py --no-web -c 10 -r 10 -t 10s --host http://localhost:5000
[2020-04-03 12:07:19,301] Roks-MacBook-Pro-2.local/INFO/locust.runners: Hatching and swarming 10 users at the rate 10 users/s (0 users already running)...
[2020-04-03 12:07:20,229] Roks-MacBook-Pro-2.local/INFO/locust.runners: All locusts hatched: WebsiteUser: 10 (0 already running)
[2020-04-03 12:07:20,306] Roks-MacBook-Pro-2.local/INFO/locust.main: Run time limit set to 10 seconds
[2020-04-03 12:07:20,306] Roks-MacBook-Pro-2.local/INFO/locust.main: Starting Locust 0.14.5
 Name                                                          # reqs      # fails     Avg     Min     Max  |  Median   req/s failures/s
--------------------------------------------------------------------------------------------------------------------------------------------
--------------------------------------------------------------------------------------------------------------------------------------------
 Aggregated                                                         0     0(0.00%)       0       0       0  |       0    0.00    0.00

 Name                                                          # reqs      # fails     Avg     Min     Max  |  Median   req/s failures/s
--------------------------------------------------------------------------------------------------------------------------------------------
--------------------------------------------------------------------------------------------------------------------------------------------
 Aggregated                                                         0     0(0.00%)       0       0       0  |       0    0.00    0.00

 Name                                                          # reqs      # fails     Avg     Min     Max  |  Median   req/s failures/s
--------------------------------------------------------------------------------------------------------------------------------------------
 GET /slow                                                         10     0(0.00%)    3574    3100    4001  |    3400    0.00    0.00
--------------------------------------------------------------------------------------------------------------------------------------------
 Aggregated                                                        10     0(0.00%)    3574    3100    4001  |    3400    0.00    0.00

 Name                                                          # reqs      # fails     Avg     Min     Max  |  Median   req/s failures/s
--------------------------------------------------------------------------------------------------------------------------------------------
 GET /slow                                                         10     0(0.00%)    3574    3100    4001  |    3400    0.00    0.00
--------------------------------------------------------------------------------------------------------------------------------------------
 Aggregated                                                        10     0(0.00%)    3574    3100    4001  |    3400    0.00    0.00

 Name                                                          # reqs      # fails     Avg     Min     Max  |  Median   req/s failures/s
--------------------------------------------------------------------------------------------------------------------------------------------
 GET /slow                                                         20     0(0.00%)    3516    3095    4001  |    3400    1.43    0.00
--------------------------------------------------------------------------------------------------------------------------------------------
 Aggregated                                                        20     0(0.00%)    3516    3095    4001  |    3400    1.43    0.00

[2020-04-03 12:07:30,307] Roks-MacBook-Pro-2.local/INFO/locust.main: Time limit reached. Stopping Locust.
[2020-04-03 12:07:30,309] Roks-MacBook-Pro-2.local/INFO/locust.main: Shutting down (exit code 0), bye.
[2020-04-03 12:07:30,309] Roks-MacBook-Pro-2.local/INFO/locust.main: Cleaning up runner...
[2020-04-03 12:07:30,309] Roks-MacBook-Pro-2.local/INFO/locust.main: Running teardowns...
 Name                                                          # reqs      # fails     Avg     Min     Max  |  Median   req/s failures/s
--------------------------------------------------------------------------------------------------------------------------------------------
 GET /slow                                                         21     0(0.00%)    3502    3095    4001  |    3400    1.91    0.00
--------------------------------------------------------------------------------------------------------------------------------------------
 Aggregated                                                        21     0(0.00%)    3502    3095    4001  |    3400    1.91    0.00

Percentage of the requests completed within given times
 Type                 Name                                                           # reqs    50%    66%    75%    80%    90%    95%    98%    99%  99.9% 99.99%   100%
------------------------------------------------------------------------------------------------------------------------------------------------------
 GET                  /slow                                                              21   3400   3600   3700   3800   4000   4000   4000   4000   4000   4000   4000
------------------------------------------------------------------------------------------------------------------------------------------------------
 None                 Aggregated                                                         21   3400   3600   3700   3800   4000   4000   4000   4000   4000   4000   4000

You can see that all the requests were successful, but all of them should have failed and there should have been more requests, as each one should fail after 1.0 s.

Environment:

• OS: Mac OS
• Python version: Python 3.7.1
• Locust version: 0.14.5

[heyman]

I've added some additional checks to the test but that did not make it fail: 12c4d33b5a37cf71d22c7fc3d78b700f4778e974

[cyberw]

@rokcarl Can you try pulling latest master and see if it works? Since https://github.com/locustio/locust/pull/1304 , Locust classes must explicitly be declared abstract, so you may have ended up running the base FastHttpLocust class instead of WebsiteUser. I fixed FastHttpLocust in 461eafaf5355f2263d87bb4d1d36c5b808de2971

[heyman]

Can you try pulling latest master and see if it works? Since #1304 , Locust classes must explicitly be declared abstract, so you may have ended up running the base FastHttpLocust class instead of WebsiteUser.

Good catch!

[rokcarl]

I tried it from master, but now I get the following error:

[2020-04-06 17:46:46,736] docker-desktop/ERROR/locust.core: Cannot choose from an empty sequence
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/locust/core.py", line 260, in run
    self.schedule_task(self.get_next_task())
  File "/usr/local/lib/python3.7/site-packages/locust/core.py", line 419, in get_next_task
    return random.choice(self.locust.tasks)
  File "/usr/local/lib/python3.7/random.py", line 261, in choice
    raise IndexError('Cannot choose from an empty sequence') from None
IndexError: Cannot choose from an empty sequence

Which is weird because there's a test in locust to make sure that I get a different error if the tasks are empty. Have the same code as I had above. I also tried tasks = [], the same. I tried extending HttpLocust instead of FastHttpLocust, but it was the same again.

pip freeze

certifi==2020.4.5.1
chardet==3.0.4
click==7.1.1
ConfigArgParse==1.2
Flask==1.1.2
gevent==1.5a3
geventhttpclient-wheels==1.3.1.dev3
greenlet==0.4.15
idna==2.9
itsdangerous==1.1.0
Jinja2==2.11.1
locustio==0.14.5
MarkupSafe==1.1.1
msgpack==1.0.0
psutil==5.7.0
pyzmq==19.0.0
requests==2.23.0
six==1.14.0
urllib3==1.25.8
Werkzeug==1.0.1

I got this ☝️ by basically running pip install https://github.com/locustio/locust/archive/master.zip.

[cyberw]

Hmm. Not sure what is the problem (it could definitely be related to 1.0 changes), but if you use this syntax it should work:

from locust import task
class WebsiteUser(FastHttpLocust):
    @task 
    def index(self):
        self.client.get("/slow")
    wait_time = constant_pacing(1.0)
    network_timeout = 1.0
    connection_timeout = 1.0

[rokcarl]

Yes, this works and the timeout works as well, I ran it against my mock website with a timeout of 0.5 + random.random() (so between 0.5 and 1.5) and half of the requests were successful and half were a failure.

Thank you!