Locust: Using FastHttpLocust gives a 'dict object

Created on 21 Dec 2019  路  7Comments  路  Source: locustio/locust

Describe the bug

Expected behavior

Actual behavior

Steps to reproduce

Environment settings

  • OS:
  • Python version:
  • Locust version:
bug non-critical
Source
picture krishnakeshan

Most helpful comment

Hi @feizalasmoro, @cyberw, I had the same problem last week. Since this issue does not have any details in it, I was going to open a new one, but maybe its better to reopen this one instead.

Here are the details of what happened to me and how I solved the problem in my test:

After using transformer to generate a script and running a successful test with the default settings, I wanted to see the difference in performance when using FastHttpLocust. Unfortunately, a post request in my test was not working (for two different reasons, I'll detail them below), with this traceback:

[2020-04-17 18:54:04,093] localhost/ERROR/stderr: File "/home/iadmin/stresstests/locust/test2.py", line 19, in on_start
    self.client.post(path='/my-app/j_spring_security_check', name='/my-app/j_spring_security_check', timeout=30, allow_redirects=False, headers={'User-Agent': 'Locust-Request', 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8', 'Accept-Language': 'en-US,en;q=0.5', 'Accept-Encoding': 'gzip, deflate', 'Content-Type': 'application/x-www-form-urlencoded', 'Content-Length': '73', 'Origin': '', 'DNT': '1', 'Connection': 'keep-alive', 'Upgrade-Insecure-Requests': '1'}, data={"j_username":username, "j_password":password, "desired_dashboard":"", "login_mode_embed":"false"} )
[2020-04-17 18:54:04,093] localhost/ERROR/stderr: 
[2020-04-17 18:54:04,093] localhost/ERROR/stderr: File "/home/iadmin/anaconda3/lib/python3.7/site-packages/locust/contrib/fasthttp.py", line 228, in post
    return self.request("POST", path, data=data, **kwargs)
[2020-04-17 18:54:04,093] localhost/ERROR/stderr: 
[2020-04-17 18:54:04,094] localhost/ERROR/stderr: File "/home/iadmin/anaconda3/lib/python3.7/site-packages/locust/contrib/fasthttp.py", line 170, in request
    response = self._send_request_safe_mode(method, url, payload=data, headers=headers, **kwargs)
[2020-04-17 18:54:04,094] localhost/ERROR/stderr: 
[2020-04-17 18:54:04,094] localhost/ERROR/stderr: File "/home/iadmin/anaconda3/lib/python3.7/site-packages/locust/contrib/fasthttp.py", line 116, in _send_request_safe_mode
    return self.client.urlopen(url, method=method, **kwargs)
[2020-04-17 18:54:04,094] localhost/ERROR/stderr: 
[2020-04-17 18:54:04,094] localhost/ERROR/stderr: File "/home/iadmin/anaconda3/lib/python3.7/site-packages/geventhttpclient/useragent.py", line 352, in urlopen
    last_error = self._handle_error(e, url=req.url)
[2020-04-17 18:54:04,094] localhost/ERROR/stderr: 
[2020-04-17 18:54:04,094] localhost/ERROR/stderr: File "/home/iadmin/anaconda3/lib/python3.7/site-packages/geventhttpclient/useragent.py", line 319, in _handle_error
    raise reraise(type(e), e, sys.exc_info()[2])
[2020-04-17 18:54:04,094] localhost/ERROR/stderr: 
[2020-04-17 18:54:04,094] localhost/ERROR/stderr: File "/home/iadmin/anaconda3/lib/python3.7/site-packages/six.py", line 693, in reraise
    raise value
[2020-04-17 18:54:04,094] localhost/ERROR/stderr: 
[2020-04-17 18:54:04,094] localhost/ERROR/stderr: File "/home/iadmin/anaconda3/lib/python3.7/site-packages/geventhttpclient/useragent.py", line 347, in urlopen
    resp = self._urlopen(req)
[2020-04-17 18:54:04,094] localhost/ERROR/stderr: 
[2020-04-17 18:54:04,094] localhost/ERROR/stderr: File "/home/iadmin/anaconda3/lib/python3.7/site-packages/locust/contrib/fasthttp.py", line 305, in _urlopen
    body=request.payload, headers=request.headers)
[2020-04-17 18:54:04,094] localhost/ERROR/stderr: 
[2020-04-17 18:54:04,094] localhost/ERROR/stderr: File "/home/iadmin/anaconda3/lib/python3.7/site-packages/geventhttpclient/client.py", line 236, in request
    sock.sendfile(body)
[2020-04-17 18:54:04,094] localhost/ERROR/stderr: 
[2020-04-17 18:54:04,094] localhost/ERROR/stderr: File "/home/iadmin/anaconda3/lib/python3.7/site-packages/gevent/_socket3.py", line 671, in sendfile
    return self._sendfile_use_send(file, offset, count)
[2020-04-17 18:54:04,094] localhost/ERROR/stderr: 
[2020-04-17 18:54:04,094] localhost/ERROR/stderr: File "/home/iadmin/anaconda3/lib/python3.7/site-packages/gevent/_socket3.py", line 609, in _sendfile_use_send
    file_read = file.read
[2020-04-17 18:54:04,094] localhost/ERROR/stderr: 
[2020-04-17 18:54:04,095] localhost/ERROR/stderr: AttributeError: 'dict' object has no attribute 'read'
[2020-04-17 18:54:04,095] localhost/ERROR/stderr: 
[2020-04-17 18:54:04,095] localhost/ERROR/stderr: 2020-04-17T21:54:04Z
[2020-04-17 18:54:04,095] localhost/ERROR/stderr: 
[2020-04-17 18:54:04,095] localhost/ERROR/stderr: <Greenlet at 0x7f1555b744d0: start_locust(<test2.WebsiteUser object at 0x7f1555b0d390>)> failed with AttributeError

What was really strange to me is that this issue only happened when I defined the header 'Content-Type': 'application/x-www-form-urlencoded'. Looking through the client code, I think this is the cause:

        if payload:
            # Adjust headers depending on payload content
            content_type = req_headers.get('content-type', None)
            if not content_type and isinstance(payload, dict):
                req_headers['content-type'] = "application/x-www-form-urlencoded; charset=utf-8"
                payload = urlencode(payload)

As we can see, if you do not define the content-type but pass a dict as payload (parameter data), the client urlencodes it. However, if you do set it beforehand, we get the AttributeError exception.

After encoding the body, however, I ran into the second cause of this issue (this affects GET requests as well): leaving the parameters timeout and allow_redirects from my original test in the requests. Looking through the code again, if you leave unrecognized named parameters in the request they are added as the request body:

    def urlopen(self, url, method='GET', response_codes=valid_response_codes,
                headers=None, payload=None, to_string=False, debug_stream=None, **kwargs):
        """ Open an URL, do retries and redirects and verify the status code
        """
        # POST or GET parameters can be passed in **kwargs
        if kwargs:
            if not payload:
                payload = kwargs
            elif isinstance(payload, dict):
                payload.update(kwargs)

After removing both parameters, my test worked as expected.

I'm not sure what is the best solution here, but I'd like to suggest two possible options:

  • Update the documentation, to be more precise about when you must urlencode the data parameter
  • urlencode the data parameter before calling self._send_request_safe_mode in fasthttp.py

Environment

  • OS: Ubuntu 18.04.4 LTS
  • Python version: 3.7.4
  • Locust version: 0.14.5
  • Locust command line that you ran: locust -f test2.py
  • Locust file contents (anonymized if necessary):
picture rphsantos on 22 Apr 2020
👍3

All 7 comments

I guess this worked itself out?

cyberw picture cyberw on 22 Dec 2019

Hi @krishnakeshan I got the same issue. Do you mind to share the solution? Thanks

feizalasmoro picture feizalasmoro on 20 Apr 2020

Hi @krishnakeshan I got the same issue. Do you mind to share the solution? Thanks

Could you share some specifics like your test plan and maybe what the dict contains?

cyberw picture cyberw on 20 Apr 2020

Hi @feizalasmoro, @cyberw, I had the same problem last week. Since this issue does not have any details in it, I was going to open a new one, but maybe its better to reopen this one instead.

Here are the details of what happened to me and how I solved the problem in my test:

After using transformer to generate a script and running a successful test with the default settings, I wanted to see the difference in performance when using FastHttpLocust. Unfortunately, a post request in my test was not working (for two different reasons, I'll detail them below), with this traceback:

[2020-04-17 18:54:04,093] localhost/ERROR/stderr: File "/home/iadmin/stresstests/locust/test2.py", line 19, in on_start
    self.client.post(path='/my-app/j_spring_security_check', name='/my-app/j_spring_security_check', timeout=30, allow_redirects=False, headers={'User-Agent': 'Locust-Request', 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8', 'Accept-Language': 'en-US,en;q=0.5', 'Accept-Encoding': 'gzip, deflate', 'Content-Type': 'application/x-www-form-urlencoded', 'Content-Length': '73', 'Origin': '', 'DNT': '1', 'Connection': 'keep-alive', 'Upgrade-Insecure-Requests': '1'}, data={"j_username":username, "j_password":password, "desired_dashboard":"", "login_mode_embed":"false"} )
[2020-04-17 18:54:04,093] localhost/ERROR/stderr: 
[2020-04-17 18:54:04,093] localhost/ERROR/stderr: File "/home/iadmin/anaconda3/lib/python3.7/site-packages/locust/contrib/fasthttp.py", line 228, in post
    return self.request("POST", path, data=data, **kwargs)
[2020-04-17 18:54:04,093] localhost/ERROR/stderr: 
[2020-04-17 18:54:04,094] localhost/ERROR/stderr: File "/home/iadmin/anaconda3/lib/python3.7/site-packages/locust/contrib/fasthttp.py", line 170, in request
    response = self._send_request_safe_mode(method, url, payload=data, headers=headers, **kwargs)
[2020-04-17 18:54:04,094] localhost/ERROR/stderr: 
[2020-04-17 18:54:04,094] localhost/ERROR/stderr: File "/home/iadmin/anaconda3/lib/python3.7/site-packages/locust/contrib/fasthttp.py", line 116, in _send_request_safe_mode
    return self.client.urlopen(url, method=method, **kwargs)
[2020-04-17 18:54:04,094] localhost/ERROR/stderr: 
[2020-04-17 18:54:04,094] localhost/ERROR/stderr: File "/home/iadmin/anaconda3/lib/python3.7/site-packages/geventhttpclient/useragent.py", line 352, in urlopen
    last_error = self._handle_error(e, url=req.url)
[2020-04-17 18:54:04,094] localhost/ERROR/stderr: 
[2020-04-17 18:54:04,094] localhost/ERROR/stderr: File "/home/iadmin/anaconda3/lib/python3.7/site-packages/geventhttpclient/useragent.py", line 319, in _handle_error
    raise reraise(type(e), e, sys.exc_info()[2])
[2020-04-17 18:54:04,094] localhost/ERROR/stderr: 
[2020-04-17 18:54:04,094] localhost/ERROR/stderr: File "/home/iadmin/anaconda3/lib/python3.7/site-packages/six.py", line 693, in reraise
    raise value
[2020-04-17 18:54:04,094] localhost/ERROR/stderr: 
[2020-04-17 18:54:04,094] localhost/ERROR/stderr: File "/home/iadmin/anaconda3/lib/python3.7/site-packages/geventhttpclient/useragent.py", line 347, in urlopen
    resp = self._urlopen(req)
[2020-04-17 18:54:04,094] localhost/ERROR/stderr: 
[2020-04-17 18:54:04,094] localhost/ERROR/stderr: File "/home/iadmin/anaconda3/lib/python3.7/site-packages/locust/contrib/fasthttp.py", line 305, in _urlopen
    body=request.payload, headers=request.headers)
[2020-04-17 18:54:04,094] localhost/ERROR/stderr: 
[2020-04-17 18:54:04,094] localhost/ERROR/stderr: File "/home/iadmin/anaconda3/lib/python3.7/site-packages/geventhttpclient/client.py", line 236, in request
    sock.sendfile(body)
[2020-04-17 18:54:04,094] localhost/ERROR/stderr: 
[2020-04-17 18:54:04,094] localhost/ERROR/stderr: File "/home/iadmin/anaconda3/lib/python3.7/site-packages/gevent/_socket3.py", line 671, in sendfile
    return self._sendfile_use_send(file, offset, count)
[2020-04-17 18:54:04,094] localhost/ERROR/stderr: 
[2020-04-17 18:54:04,094] localhost/ERROR/stderr: File "/home/iadmin/anaconda3/lib/python3.7/site-packages/gevent/_socket3.py", line 609, in _sendfile_use_send
    file_read = file.read
[2020-04-17 18:54:04,094] localhost/ERROR/stderr: 
[2020-04-17 18:54:04,095] localhost/ERROR/stderr: AttributeError: 'dict' object has no attribute 'read'
[2020-04-17 18:54:04,095] localhost/ERROR/stderr: 
[2020-04-17 18:54:04,095] localhost/ERROR/stderr: 2020-04-17T21:54:04Z
[2020-04-17 18:54:04,095] localhost/ERROR/stderr: 
[2020-04-17 18:54:04,095] localhost/ERROR/stderr: <Greenlet at 0x7f1555b744d0: start_locust(<test2.WebsiteUser object at 0x7f1555b0d390>)> failed with AttributeError

What was really strange to me is that this issue only happened when I defined the header 'Content-Type': 'application/x-www-form-urlencoded'. Looking through the client code, I think this is the cause:

        if payload:
            # Adjust headers depending on payload content
            content_type = req_headers.get('content-type', None)
            if not content_type and isinstance(payload, dict):
                req_headers['content-type'] = "application/x-www-form-urlencoded; charset=utf-8"
                payload = urlencode(payload)

As we can see, if you do not define the content-type but pass a dict as payload (parameter data), the client urlencodes it. However, if you do set it beforehand, we get the AttributeError exception.

After encoding the body, however, I ran into the second cause of this issue (this affects GET requests as well): leaving the parameters timeout and allow_redirects from my original test in the requests. Looking through the code again, if you leave unrecognized named parameters in the request they are added as the request body:

    def urlopen(self, url, method='GET', response_codes=valid_response_codes,
                headers=None, payload=None, to_string=False, debug_stream=None, **kwargs):
        """ Open an URL, do retries and redirects and verify the status code
        """
        # POST or GET parameters can be passed in **kwargs
        if kwargs:
            if not payload:
                payload = kwargs
            elif isinstance(payload, dict):
                payload.update(kwargs)

After removing both parameters, my test worked as expected.

I'm not sure what is the best solution here, but I'd like to suggest two possible options:

  • Update the documentation, to be more precise about when you must urlencode the data parameter
  • urlencode the data parameter before calling self._send_request_safe_mode in fasthttp.py

Environment

  • OS: Ubuntu 18.04.4 LTS
  • Python version: 3.7.4
  • Locust version: 0.14.5
  • Locust command line that you ran: locust -f test2.py
  • Locust file contents (anonymized if necessary):
rphsantos picture rphsantos on 22 Apr 2020
👍3

Yes. That was exactly what happen in my case. Sorry I haven't got the time to add all details.

feizalasmoro picture feizalasmoro on 24 Apr 2020

IMO we should wrap geventhttpclient so that it gets the same behaviour as requests. If someone creates a PR to fix this, I am +1.

At the same time we might change it so that it automatically converts a dict type data parameter to json (which requests does). I was thinking about doing that when I added the json parameter (9a090163fcb748d577f9aa780c320fdcceb91246), but decided against it because I was lazy and didnt want to break anything :)

cyberw picture cyberw on 24 Apr 2020

I seem to experience the same problem. In order to solve some other problem I followed these instructions, but I got an error

Traceback (most recent call last):
  File "src/gevent/greenlet.py", line 854, in gevent._gevent_cgreenlet.Greenlet.run
  File "/Users/adietz/Work/10_NSE/1.19_SSCX_Portal/dev/salpeter/locustfile.py", line 51, in <lambda>
    group.spawn(lambda: self.client.get(url, headers=headers, verify=False))
  File "/Users/adietz/Envs/mac_1.19_salpeter/lib/python3.8/site-packages/locust/contrib/fasthttp.py", line 245, in get
    return self.request("GET", path, **kwargs)
  File "/Users/adietz/Envs/mac_1.19_salpeter/lib/python3.8/site-packages/locust/contrib/fasthttp.py", line 189, in request
    response = self._send_request_safe_mode(method, url, payload=data, headers=headers, **kwargs)
  File "/Users/adietz/Envs/mac_1.19_salpeter/lib/python3.8/site-packages/locust/contrib/fasthttp.py", line 109, in _send_request_safe_mode
    return self.client.urlopen(url, method=method, **kwargs)
  File "/Users/adietz/Envs/mac_1.19_salpeter/lib/python3.8/site-packages/geventhttpclient/useragent.py", line 359, in urlopen
    last_error = self._handle_error(e, url=req.url)
  File "/Users/adietz/Envs/mac_1.19_salpeter/lib/python3.8/site-packages/geventhttpclient/useragent.py", line 326, in _handle_error
    raise reraise(type(e), e, sys.exc_info()[2])
  File "/Users/adietz/Envs/mac_1.19_salpeter/lib/python3.8/site-packages/six.py", line 703, in reraise
    raise value
  File "/Users/adietz/Envs/mac_1.19_salpeter/lib/python3.8/site-packages/geventhttpclient/useragent.py", line 354, in urlopen
    resp = self._urlopen(req)
  File "/Users/adietz/Envs/mac_1.19_salpeter/lib/python3.8/site-packages/locust/contrib/fasthttp.py", line 402, in _urlopen
    resp = client.request(
  File "/Users/adietz/Envs/mac_1.19_salpeter/lib/python3.8/site-packages/geventhttpclient/client.py", line 236, in request
    sock.sendfile(body)
  File "/Users/adietz/Envs/mac_1.19_salpeter/lib/python3.8/site-packages/gevent/_socket3.py", line 684, in sendfile
    return self._sendfile_use_send(file, offset, count)
  File "/Users/adietz/Envs/mac_1.19_salpeter/lib/python3.8/site-packages/gevent/_socket3.py", line 622, in _sendfile_use_send
    file_read = file.read
AttributeError: 'dict' object has no attribute 'read'
2020-12-04T06:55:32Z <Greenlet at 0x107ce76a0: <lambda>> failed with AttributeError
  • OS: MacOS 10.14.6
  • Python version: 3.8.6
  • Locust version: 1.4.1
alex4200 picture alex4200 on 4 Dec 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

kowalcj0 picture kowalcj0  路  3Comments
ghost picture ghost  路  3Comments
dolohow picture dolohow  路  3Comments
radiantone picture radiantone  路  3Comments
max-rocket-internet picture max-rocket-internet  路  3Comments