Locust: FastHttpLocust is very slow at returning the response text

[cyberw]

Getting resp.text when using FastHttpLocust does a string decode (fasthttp.py:257) and it is crazy slow for large responses (~12s for a 1.5MB payload).

    with self.client.request(method, f"https://{host}{url}", name, catch_response=True) as resp:
        t = resp.text: # this takes ages
        ....
        return resp

[cyberw]

Oh, and also it doesnt cache the value, so every call to .text will need to decode the string again.

I'm looking in to it...

[cyberw]

Oh, the actual problem is the character encoding detection in apparent_encoding()

[cyberw]

I can solve this by using cchardet instead of chardet. Encoding detection time drops from 12 seconds to 0.03s for me.

@heyman ok to add this as a dependency?

[cyberw]

Or we could just read the Content-Type: charset header?

[heyman]

Or we could just read the Content-Type: charset header?

Yeah, maybe that's best. And assume ISO-8859-1 if none is specified. I guess it could cause problems when the server is setting incorrect chaset headers, though it shouldn't be that big of a problem since users of a load testing tool usually have control over both client and server side.

[cyberw]

I'll have a go at it