Locust: How to fix "Connection pool is full, discarding connection" warning?

[alex4200]

Describe the bug

When I make more than 15 requests on locust in parallel, I get the warning

WARNING/urllib3.connectionpool: Connection pool is full, discarding connection:

which, according to a comment HERE, is caused by "logging"(?), and could be the reason of some underlying issue.

I tried to use the line logging.basicConfig(level=logging.ERROR) in the locustfile.py, but this does not solve the issue.

Expected behavior

The expected behavior is that locust is able to perform, like, 50 requests in parallel for 50 users. That is what I would expect from a performance testing tool.

Actual behavior

I get the warning

WARNING/urllib3.connectionpool: Connection pool is full, discarding connection:

and I am not sure locust is doing the correct thing (i.e. making like 20 requests in parallel).

Steps to reproduce

Here is the part of the code that is (hopefully) doing a bunch of requests in parallel (just like a browser would do is a user opens a web page):

    def make_parallel_requests(self, requests):
        group = Group()
        uid = str(uuid.uuid4())[:8]
        print(f"{uid} Starting {len(requests)} requests in parallel.")
        for request in requests:
            if request["method"] == "GET":
                url = request["url"]
                headers = request["headers"]
                headers["Authorization"] = "bearer " + ACCESS_TOKEN
                group.spawn(lambda: self.client.get(url, headers=headers, verify=False))
        group.join()
        print(f"{uid} Requests done")

It looks like that if the number of requests is smaller than 15 there is no such warning. But there will be printed that warning for every additional request.

Maybe I can ignore the warning and locust is, in fact, performing the requests in parallel?

Environment

• OS: MacOS 10.14.6
• Python version: 3.8.6
• Locust version: 1.4.1
• Locust command line that you ran: locust --headless -u 1 -r 1 -t 100s --csv-full-history --csv test --only-summary
• Locust file contents (anonymized if necessary): see above.

[cyberw]

The "normal" way of doing this in locust would be to run multiple users instead of having one user do parallell requests. But if you need to do it this way, maybe you can increase urllib3:s pool size? Check the documentation for requests & urllib as this isnt really a locust thing.

Another option would be to spawn your own clients (locust.clients.HttpSession) and use them.

[alex4200]

I have seen this documentation on the pool size. So I just put these two line into the code locustfile.py?

import urllib3
http = urllib3.PoolManager(num_pools=50)

[alex4200]

No, this does not seem to work. What am I doing wrong, @cyberw?

[cyberw]

Did you work it out? Sorry for not responding. This is not really a locust issue and requires more in depth knowledge of requests/urllib3 than I have...

[alex4200]

@cyberw No, I have not found a solution yet. Maybe I need to dive in to requests/urllib myself a bit more...

[aek]

Hi @alex4200
Try it like this example(copy the __init__ method to your Locust User)

from requests.adapters import HTTPAdapter

class CustomUser(HttpUser):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.client.mount('https://', HTTPAdapter(pool_maxsize=50))
        self.client.mount('http://', HTTPAdapter(pool_maxsize=50))

    def make_parallel_requests(self, requests):
        group = Group()
        uid = str(uuid.uuid4())[:8]
        print(f"{uid} Starting {len(requests)} requests in parallel.")
        for request in requests:
            if request["method"] == "GET":
                url = request["url"]
                headers = request["headers"]
                headers["Authorization"] = "bearer " + ACCESS_TOKEN
                group.spawn(lambda: self.client.get(url, headers=headers, verify=False))
        group.join()
        print(f"{uid} Requests done")

[alex4200]

@aek Thanks, that seems to work!

And if someone else is interested in the solution, here the import of HTTPAdapter:

from requests.adapters import HTTPAdapter

[max-rocket-internet]

It would be cool we could set this on HttpUser instead of using super though, right?

[cyberw]

It would be cool we could set this on HttpUser instead of using super though, right?

Sure! PR is welcome :)

[aek]

I think that it will be better to allow users to provide an instance of their own HTTPAdapter instance. That way you could do any customization on your side like I have done in the __init__ method. I strongly believe that the FastHttpUser class could be easily converted into a requests HTTPAdapter and reuse all the stuffs already done in that matter.
Using it like this you could use your own batteries to power on the same engine

[max-rocket-internet]

@aek can't wait to see your PR 🚀

[aek]

@max-rocket-internet I'm planning to test that idea by implementing a requests HTTPAdapter using pycurl.
That will be a proof of concept of this idea that could be backported to the rework of FastHTTPUser with the geventhttpclient converting it into a requests HTTPAdapter