Locust: Infinite recursion error when testing https sites

[jobi]

Description of issue / feature request

When opening a connection to an HTTPs site, I'm triggering an infinite recursion error:

[2017-09-14 22:19:39,370] BOS-SC-JBil2/ERROR/stderr: File "/Users/jbilien/src/scout-load/virtualenv/lib/python3.6/site-packages/requests/sessions.py", line 521, in get
    return self.request('GET', url, **kwargs)
[2017-09-14 22:19:39,386] BOS-SC-JBil2/ERROR/stderr:
[2017-09-14 22:19:39,386] BOS-SC-JBil2/ERROR/stderr: File "/Users/jbilien/src/scout-load/virtualenv/lib/python3.6/site-packages/locust/clients.py", line 114, in request
    response = self._send_request_safe_mode(method, url, **kwargs)
[2017-09-14 22:19:39,387] BOS-SC-JBil2/ERROR/stderr:
[2017-09-14 22:19:39,387] BOS-SC-JBil2/ERROR/stderr: File "/Users/jbilien/src/scout-load/virtualenv/lib/python3.6/site-packages/locust/clients.py", line 158, in _send_request_safe_mode
    return requests.Session.request(self, method, url, **kwargs)
[2017-09-14 22:19:39,387] BOS-SC-JBil2/ERROR/stderr:
[2017-09-14 22:19:39,387] BOS-SC-JBil2/ERROR/stderr: File "/Users/jbilien/src/scout-load/virtualenv/lib/python3.6/site-packages/requests/sessions.py", line 508, in request
    resp = self.send(prep, **send_kwargs)
[2017-09-14 22:19:39,387] BOS-SC-JBil2/ERROR/stderr:
[2017-09-14 22:19:39,387] BOS-SC-JBil2/ERROR/stderr: File "/Users/jbilien/src/scout-load/virtualenv/lib/python3.6/site-packages/requests/sessions.py", line 618, in send
    r = adapter.send(request, **kwargs)
[2017-09-14 22:19:39,387] BOS-SC-JBil2/ERROR/stderr:
[2017-09-14 22:19:39,387] BOS-SC-JBil2/ERROR/stderr: File "/Users/jbilien/src/scout-load/virtualenv/lib/python3.6/site-packages/requests/adapters.py", line 440, in send
    timeout=timeout
[2017-09-14 22:19:39,387] BOS-SC-JBil2/ERROR/stderr:
[2017-09-14 22:19:39,387] BOS-SC-JBil2/ERROR/stderr: File "/Users/jbilien/src/scout-load/virtualenv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 601, in urlopen
    chunked=chunked)
[2017-09-14 22:19:39,387] BOS-SC-JBil2/ERROR/stderr:
[2017-09-14 22:19:39,388] BOS-SC-JBil2/ERROR/stderr: File "/Users/jbilien/src/scout-load/virtualenv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 346, in _make_request
    self._validate_conn(conn)
[2017-09-14 22:19:39,388] BOS-SC-JBil2/ERROR/stderr:
[2017-09-14 22:19:39,388] BOS-SC-JBil2/ERROR/stderr: File "/Users/jbilien/src/scout-load/virtualenv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 850, in _validate_conn
    conn.connect()
[2017-09-14 22:19:39,388] BOS-SC-JBil2/ERROR/stderr:
[2017-09-14 22:19:39,388] BOS-SC-JBil2/ERROR/stderr: File "/Users/jbilien/src/scout-load/virtualenv/lib/python3.6/site-packages/urllib3/connection.py", line 314, in connect
    cert_reqs=resolve_cert_reqs(self.cert_reqs),
[2017-09-14 22:19:39,388] BOS-SC-JBil2/ERROR/stderr:
[2017-09-14 22:19:39,388] BOS-SC-JBil2/ERROR/stderr: File "/Users/jbilien/src/scout-load/virtualenv/lib/python3.6/site-packages/urllib3/util/ssl_.py", line 269, in create_urllib3_context
    context.options |= options
[2017-09-14 22:19:39,388] BOS-SC-JBil2/ERROR/stderr:
[2017-09-14 22:19:39,388] BOS-SC-JBil2/ERROR/stderr: File "/usr/local/Cellar/python3/3.6.2/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 459, in options
    super(SSLContext, SSLContext).options.__set__(self, value)
[2017-09-14 22:19:39,388] BOS-SC-JBil2/ERROR/stderr:
[2017-09-14 22:19:39,388] BOS-SC-JBil2/ERROR/stderr: File "/usr/local/Cellar/python3/3.6.2/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 459, in options
    super(SSLContext, SSLContext).options.__set__(self, value)
[2017-09-14 22:19:39,388] BOS-SC-JBil2/ERROR/stderr:
[2017-09-14 22:19:39,389] BOS-SC-JBil2/ERROR/stderr: File "/usr/local/Cellar/python3/3.6.2/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 459, in options
    super(SSLContext, SSLContext).options.__set__(self, value)
[2017-09-14 22:19:39,389] BOS-SC-JBil2/ERROR/stderr:
[2017-09-14 22:19:39,389] BOS-SC-JBil2/ERROR/stderr: [Previous line repeated 316 more times]

A similar issue is being discussed at https://github.com/requests/requests/issues/3752 and https://github.com/gevent/gevent/issues/903. Suggestions to update to gevent 1.2 haven't helped.

Environment settings (for bug reports)

• OS: macOS 10.12.6
• Python version: 3.6.2
• Locust version: 8c4eeae4d1001f6b01cd5caf01273a0f2393405e

[hamann]

Getting the same error for https site

• Alpine 3.6
• Python version: 3.6.1
• Locust version: 0.8a3

[heyman]

Could you provide a minimal locustfile that produces this error? If possible also specify some public host that the test can be run against. I was unable to reproduce it on Python 3.5 against https://google.com.

[heyman]

Oh, just realised that the two issues you linked were related to Python 3.6. Will try to reproduce with that.

[hamann]

$ cat locustfile.py
from locust import HttpLocust, TaskSet, task

class HttpsTestBehaviour(TaskSet):

    @task(1)
    def get(self):
      self.client.get("/")

class HttpsTest(HttpLocust):
    task_set = HttpsTestBehaviour
    min_wait = 2
    max_wait = 5

$ cat Dockerfile
FROM alpine:3.6
RUN apk -U add ca-certificates python3 python3-dev py3-pip build-base && \
    pip3 install locustio==0.8a3 && \
    pip3 install pyzmq && \
    apk del python3-dev && \
    rm -r /var/cache/apk/*
ENTRYPOINT [ "/usr/bin/locust" ]
$ docker build -t local/locust:0.8a3 .
$ docker run --rm --net=host -v $PWD/locustfile.py:/locustfile.py local/locust:0.8a3 -f /locustfile.py --host=https://google.com

# open browser on http://localhost:8089 and watch errors in console

[heyman]

Found the issue! #651 had changed so that we applied the gevent monkey patching after we import requests which caused the issue. This is fixed in master now, and I'm about to push out a new version on PyPI.

[heyman]

New version released to PyPI.