Google-cloud-python: httplib2.Http is not thread-safe

[pcostell]

This is from a StackOverflow post. I've done some debugging from the Datastore side but I don't think these requests ever make it into the Datastore part of the stack. I'd appreciate if someone look at this from the gcloud-python team. Note that this issue was originally reported via [email protected] In July.

I have a Python Django application running on a Google Compute instance. It is using gcloudoem to interface from Django to Google Datastore. gcloudoem uses the same underlying code to communicate with Datastore as gcloud-python 0.5.x

At what seems to be completely random times, I will get SSL errors happening when trying to talk to Datastore. There is no pattern in where in my application code these happen. It's just during a random call to Datastore. Here are the two flavours of errors:

ERROR:django.request:Internal Server Error: /complete/google-oauth2/
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/django/core/handlers/base.py", line 111, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/django/views/decorators/cache.py", line 52, in _wrapped_view_func
    response = view_func(request, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/django/views/decorators/csrf.py", line 57, in wrapped_view
    return view_func(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/social/apps/django_app/utils.py", line 51, in wrapper
    return func(request, backend, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/social/apps/django_app/views.py", line 28, in complete
    redirect_name=REDIRECT_FIELD_NAME, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/social/actions.py", line 43, in do_complete
    user = backend.complete(user=user, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/social/backends/base.py", line 41, in complete
    return self.auth_complete(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/social/utils.py", line 229, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/social/backends/oauth.py", line 387, in auth_complete
    *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/social/utils.py", line 229, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/social/backends/oauth.py", line 396, in do_auth
    return self.strategy.authenticate(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/social/strategies/django_strategy.py", line 96, in authenticate
    return authenticate(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/django/contrib/auth/__init__.py", line 60, in authenticate
    user = backend.authenticate(**credentials)
  File "/usr/local/lib/python2.7/dist-packages/social/backends/base.py", line 82, in authenticate
    return self.pipeline(pipeline, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/social/backends/base.py", line 85, in pipeline
    out = self.run_pipeline(pipeline, pipeline_index, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/social/backends/base.py", line 112, in run_pipeline
    result = func(*args, **out) or {}
  File "/usr/local/lib/python2.7/dist-packages/social/pipeline/social_auth.py", line 20, in social_user
    social = backend.strategy.storage.user.get_social_auth(provider, uid)
  File "./social_gc/storage.py", line 105, in get_social_auth
    return cls.objects.get(provider=provider, uid=uid)
  File "/usr/local/lib/python2.7/dist-packages/gcloudoem/queryset/__init__.py", line 162, in get
    num = len(clone)
  File "/usr/local/lib/python2.7/dist-packages/gcloudoem/queryset/__init__.py", line 126, in __len__
    self._fetch_all()
  File "/usr/local/lib/python2.7/dist-packages/gcloudoem/queryset/__init__.py", line 370, in _fetch_all
    self._result_cache = list(self.iterator())
  File "/usr/local/lib/python2.7/dist-packages/gcloudoem/datastore/query.py", line 480, in __iter__
    self.next_page()
  File "/usr/local/lib/python2.7/dist-packages/gcloudoem/datastore/query.py", line 452, in next_page
    transaction_id=transaction and transaction.id,
  File "/usr/local/lib/python2.7/dist-packages/gcloudoem/datastore/connection.py", line 249, in run_query
    response = self._rpc('runQuery', request, datastore_pb.RunQueryResponse)
  File "/usr/local/lib/python2.7/dist-packages/gcloudoem/datastore/connection.py", line 159, in _rpc
    data=request_pb.SerializeToString()
  File "/usr/local/lib/python2.7/dist-packages/gcloudoem/datastore/connection.py", line 134, in _request
    body=data
  File "/usr/local/lib/python2.7/dist-packages/oauth2client/client.py", line 589, in new_request
    redirections, connection_type)
  File "/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py", line 1609, in request
    (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
  File "/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py", line 1351, in _request
    (response, content) = self._conn_request(conn, request_uri, method, body, headers)
  File "/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py", line 1307, in _conn_request
    response = conn.getresponse()
  File "/usr/lib/python2.7/httplib.py", line 1127, in getresponse
    response.begin()
  File "/usr/lib/python2.7/httplib.py", line 453, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python2.7/httplib.py", line 409, in _read_status
    line = self.fp.readline(_MAXLINE + 1)
  File "/usr/lib/python2.7/socket.py", line 480, in readline
    data = self._sock.recv(self._rbufsize)
  File "/usr/lib/python2.7/ssl.py", line 734, in recv
    return self.read(buflen)
  File "/usr/lib/python2.7/ssl.py", line 621, in read
    v = self._sslobj.read(len or 1024)
SSLError: [SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1752)

Unfortunately, for the second, I don't have a full stacktrace handy:

[SSL: DECRYPTION_FAILED_OR_BAD_RECORD_MAC] decryption failed or bad record mac (_ssl.c:1752)

These errors don't happen when I am using the GCD tool. Does anyone have any idea what is happening here? Is this some sort of networking problem?

[pcostell]

Also -- this is using gcloudoem, which most (all?) of the datastore code was taken from gcloud-python. If you do think this is only an issue with gcloudoem and not gcloud-python, let's close this issue and reopen it for the gcloudoem owner.

[jgeewax]

Unfortunately, even if we did find and fix a bug in gcloud-python, gcloud-datastore-oem / gcloudoem copied, pasted and modified the code, so our changes won't fix anything for this particular situation.

I think it's worth a quick shot at trying to reproduce on our side, but we still need to open a bug at gcloud-datastore-oem.

[pcostell]

True -- but based on the type of error I'd expect it to also be an error in gcloud-python (unless we've already fixed it). If we are fairly confident it isn't, we can just open a bug against gcloud-datastore-oem and forget about it(may be worthwhile to do this anyways! -- the bug, not the forgetting)

[dhermes]

Can you provide code to repro? Like, loop 5,000 times and then it happens probably?

[rstuart85]

I originally reported this problem and I can provide a little more context.

The problem happens quite regularly when I am developing code from my local machine using the production datastore. However, if I use the gcd tool locally instead of actual datastore, it never happens. It happens less regularly in production from a compute instance. It might also be worth noting that the code is run in a docker container.

I've tried installing pyOpenSSL to see if the problem is rectified but it appears to have no affect.

I will write some code using only gcloud-python to see if I can reproduce the problem. I will set it running locally first because I'm slightly concerned about the potential cost it could accumulate on gcloud. I'll report back with results.

[rstuart85]

Here is the code I am running. I will report back when something goes wrong.

[dhermes]

Thanks @rstuart85! It looks like a very simple script :+1:

[rstuart85]

The script has been running for a couple of days with no errors. So there appears to be something related to my setup that is causing the SSL errors. I'll keep investigating.

[rstuart85]

Another update. It isn't just Datastore that gets the problem. Here is a stacktrace from trying to access a bucket using gcloud-python.

ERROR:django.request:Internal Server Error: /shared_image/ded1e2fa794e425b866ffdcf5ffbccda/
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/django/core/handlers/base.py", line 111, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/django/views/decorators/http.py", line 41, in inner
    return func(request, *args, **kwargs)
  File "./research_portal/views.py", line 623, in shared
    blob = gcloud_utils.get_object(shared_id, bucket=storage.get_bucket(settings.SHARED_BUCKET))
  File "./research_portal/gcloud_utils.py", line 31, in get_object
    return bucket.get_blob(name)
  File "/usr/local/lib/python2.7/dist-packages/gcloud/storage/bucket.py", line 190, in get_blob
    path=blob.path)
  File "/usr/local/lib/python2.7/dist-packages/gcloud/storage/connection.py", line 229, in api_request
    method=method, url=url, data=data, content_type=content_type)
  File "/usr/local/lib/python2.7/dist-packages/gcloud/storage/connection.py", line 141, in _make_request
    return self._do_request(method, url, headers, data)
  File "/usr/local/lib/python2.7/dist-packages/gcloud/storage/connection.py", line 166, in _do_request
    body=data)
  File "/usr/local/lib/python2.7/dist-packages/oauth2client/client.py", line 589, in new_request
    redirections, connection_type)
  File "/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py", line 1609, in request
    (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
  File "/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py", line 1351, in _request
    (response, content) = self._conn_request(conn, request_uri, method, body, headers)
  File "/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py", line 1307, in _conn_request
    response = conn.getresponse()
  File "/usr/lib/python2.7/httplib.py", line 1127, in getresponse
    response.begin()
  File "/usr/lib/python2.7/httplib.py", line 453, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python2.7/httplib.py", line 409, in _read_status
    line = self.fp.readline(_MAXLINE + 1)
  File "/usr/lib/python2.7/socket.py", line 480, in readline
    data = self._sock.recv(self._rbufsize)
  File "/usr/lib/python2.7/ssl.py", line 734, in recv
    return self.read(buflen)
  File "/usr/lib/python2.7/ssl.py", line 621, in read
    v = self._sslobj.read(len or 1024)
SSLError: [SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1752)

And here is the code:

def get_object(name, bucket=None):
    retries = 0
    while True:
        try:
            if not bucket:
                bucket = storage.get_bucket(settings.UPLOADS_BUCKET)
            return bucket.get_blob(name)
        except InternalServerError as e:  # Retry when Google has an error
            logger.exception("Got a Google Storage error trying to upload a file.")
            if retries < 3:
                sleep = random.random() * 2
                time.sleep(sleep)
                retries += 1
                continue
            raise e

As you can see, I've tried to sidestep the errors using retires.

[rstuart85]

Just incase it is worth while, here is the Dockerfile I use to build the environment this runs from.

FROM ubuntu:15.04
MAINTAINER Ryan Stuart <[email protected]>

# Install packages
RUN apt-get update && apt-get install -y \
                                build-essential \ 
                                git \
                                libffi-dev \
                                libjpeg-dev \
                                libsqlite3-dev \
                                libssl-dev \
                                libyaml-dev \
                                libxml2-dev \
                                libxslt1-dev \
                                python \
                                python-dev \
                                python-setuptools \
                                sqlite3 \
                                zlib1g-dev

# Setup ssh acces to private repos
RUN mkdir /root/.ssh/
ADD id_rsa /root/.ssh/id_rsa
RUN chmod 500 /root/.ssh/id_rsa && echo "    IdentityFile ~/.ssh/id_rsa" >> /etc/ssh/ssh_config
RUN touch /root/.ssh/known_hosts && ssh-keyscan github.com >> /root/.ssh/known_hosts

# Setup the application
RUN easy_install -U pip
RUN pip install apsw==3.8.7.3.post1
RUN mkdir /research_portal
ADD . /research_portal
WORKDIR /research_portal
RUN pip install -r requirements.txt
RUN python -m nltk.downloader punkt
ENV PYTHONUNBUFFERED=1 \
    STATIC_URL='https://storage.googleapis.com/static.kapiche.com/research/b0fa/'

And the requirements.txt:

caterpillar>=1.0.0.dev7
-e git+ssh://[email protected]/Kapiche/caterpillar-influence.git@master#egg=caterpillar-influence
-e git+ssh://[email protected]/Kapiche/caterpillar-topics.git@master#egg=caterpillar-topics
celery~=3.1.17
Django~=1.7.9
django-debug-toolbar~=1.3.2
django-gravatar2
gcloud~=0.4.2
gcloudoem>=0.1.0rc10
goose-extractor~=1.0.25
markdown~=2.6.2
opbeat
pdfminer==20140328
pycrypto~=2.6.1
pyopenssl~=0.15.1
python-social-auth~=0.2.7
requests~=2.6.0
stripe~=1.22.2
wsgiref==0.1.2
xlrd==0.9.3
uwsgi

[dhermes]

It's worrisome. I'm curious if the error is

• In the ssl module from the stdlib
• In your docker environ
• In some resource leak in httplib2

Maybe we should call in @jcgregorio to see if he has encountered this anywhere else with httplib2

[dhermes]

I'm curious if turning off stdin buffering (PYTHONUNBUFFERED=1) makes a difference. I highly doubt it, but it's interesting to see.

Have you tried catching the ssl.SSLError exception and entering a debugger (I prefer ipdb over pdb but either will work).

[rstuart85]

No, but I'll give that a try. It happens both inside and outside of docker (on the production server which uses docker and on my local laptop which doesn't), so I doubt it is docker related. My local machine is OSX and the server is created using docker-machine.

[dhermes]

Very good to know! Local laptop is running which OS?

[rstuart85]

OSX

[dhermes]

Nice. And the docker instance is running on OS X or on a linux VM somewhere?

[rstuart85]

Linux VM on Google Compute created using docker-machine.

[dhermes]

Good. This (almost) certainly means the issue is in the libraries.

[rstuart85]

I've seen issues related to SSL with httplib2, but they are almost always related to httplib2 using its own bundled certificate store. I can't find anything related to these issues.

[dhermes]

"these issues" meaning the "bundled certificate store" or the "SSL: WRONG_VERSION_NUMBER"

[rstuart85]

The two exceptions I get which are "SSL: WRONG_VERSION_NUMBER" and "SSL: DECRYPTION_FAILED_OR_BAD_RECORD_MAC"

[dhermes]

Did you ever find a stracktrace for DECRYPTION_FAILED_OR_BAD_RECORD_MAC?

Some places where the error occurs (they don't really illuminate):
https://github.com/python/cpython/blob/2.7/Lib/ssl.py#L734
https://github.com/python/cpython/blob/2.7/Lib/ssl.py#L621
https://github.com/python/cpython/blob/2.7/Modules/_ssl.c#L1752
https://hg.python.org/cpython/file/v2.7.10/Lib/ssl.py#l621
https://hg.python.org/cpython/file/v2.7.10/Modules/_ssl.c

What versions of oauth2client, httplib2 and gcloud-python do you have? From what I can tell, you have version 0.4.3 of our library, which is very old.

---

Alternate theory: this error occurs because you are finding a Google server which doesn't speak the right SSL protocol.

From an openssl mailing list:

Versions in client/server SSL records do not agree.
Probably your client sends SSL2 ... and server is configured only for SSL3/TLS1.
In this situation server does not accept SSL2 ... is being manifested by "wrong version number" error.
To resolve this error you may disable SSL2 on client or enable SSL2 handshake on server.
tcpdump output from wrong session handshake may be helpful too.

UPDATE: Some sections of _ssl.c may be relevant here.
UPDATE 2: It looks like other people have experienced this error using a Google API. (SSLv3 is deprecated).

[dhermes]

You could use Wireshark to capture the traffic and make a Capture Filter host www.googleapis.com and then see if SSL3 is used. I did it for some light traffic on my Ubuntu 14.04 laptop and only saw TLSv1.2.

---

It seems that Python ssl.py has disabled SSLv3:

    # SSLv3 has problematic security and is only required for really old
    # clients such as IE6 on Windows XP
    context.options |= OP_NO_SSLv3

Double-check that file on your systems to make sure SSLv3 isn't the problem?

[rstuart85]

I haven't had another stacktrace yet for DECRYPTION_FAILED_OR_BAD_RECORD_MAC (all the recent ones have been WRONG_VERSION_NUMBER) but I know they come from _ssl.c (1752).

The post from the mailing list is interesting but the only people that could elaborate on that would be Google Engineers I assume? Do all your servers speak SSL2?

[rstuart85]

I will check SSLv3 support.

[dhermes]

After POODLE, SSLv1, SSLv2 and SSLv3 are deprecated (SSLv1 isn't even implemented). SSLv23 is actually the default in ssl.py. TLSv1, TLSv1_1 and TLSv1_2 are the other supported protocol variants. The line _ssl.c (1752) is just parroting back an error from the system SSL libraries so it doesn't tell us much.

I checked on my machine with Python 2.7.6 (default on Ubuntu 14.04) and SSLv3 is still enabled.

[dhermes]

@rstuart85 Were you able to check your python -V and the contents of ssl.py? Also, were you able to reproduce the error with a debugger (pdb or ipdb)?

I may also try to reproduce on my machine and monitor with Wireshark to find out what was happening.

[rstuart85]

@krisrogers is getting this problem regularly from his development machine so he is going to debug it and report back here.

[dhermes]

Great thanks. I've been running an infinite loop with just getting an object from a bucket and having no error (not even a 500) after

Iteration: 21463, 3368.84s

UPDATE: I killed the script after no failures in

Iteration: 34801, 5414.76s

This was on my local machine and it was connected to the web via my apartment WiFi.

[rstuart85]

I'm pretty sure it is a client side issue because it happens a lot more in development rather than production. Either way, @krisrogers should be able to give some more info shortly.

[dhermes]

Good to know. Thanks.

[rstuart85]

We can now no longer reproduce this on our development environment. But this is a screenshot from our monitoring tool. All bar 1 error is related to Datastore/Storage. Not quite sure what to do next.

[rstuart85]

Here are some of the other types of errors.

[theacodes]

Could this be a threading issue? When I used a connection across threads
I'd get random SSL errors.

On Wed, Dec 9, 2015, 6:06 PM Ryan Stuart [email protected] wrote:

Here are some of the other types of errors.
[image: screen shot 2015-12-10 at 11 54 26 am]
https://cloud.githubusercontent.com/assets/2389518/11704767/72f2dcf6-9f36-11e5-8e74-53a628b56fd5.png

—
Reply to this email directly or view it on GitHub
https://github.com/GoogleCloudPlatform/gcloud-python/issues/1214#issuecomment-163463428
.

[dhermes]

Ooohh goody I love a new lead! @jonparrott you mean a gcloud-python Connection?

@rstuart85 is that the case?

[theacodes]

Yes, although I can't say for sure if it was the same kind of errors. But
if the app this is occurring in is multithreaded (or is hosted by a
multithreaded wsgi server) and shares a client/connection that can
certainly cause inexplicable errors.

(I'm currently at a team offsite, so I can't really test it out)

On Wed, Dec 9, 2015, 9:18 PM Danny Hermes [email protected] wrote:

Ooohh goody I love a new lead! @jonparrott https://github.com/jonparrott
you mean a gcloud-python Connection?

@rstuart85 https://github.com/rstuart85 is that the case?

—
Reply to this email directly or view it on GitHub
https://github.com/GoogleCloudPlatform/gcloud-python/issues/1214#issuecomment-163489567
.

[rstuart85]

Yes, it looks to be threading related. We can now reproduce reliably using our application from multiple browser tabs. @krisrogers is gathering information and posting it here shortly.

[rstuart85]

Also, we host the application using uwsgi in multithreaded mode. So @jonparrott is on the money there. Glad we are getting somewhere with this!

[theacodes]

Wow. Total shot in the dark there. You could try installing from #1274 and seeing if that fixes the issue? (though that PR is not reviewed or approved yet).

[rstuart85]

I've re-implemented my test script that shows the issue. The issue appears every time for me.

[rstuart85]

Using the basis of the solution used by @jonparrott worked great for gcloudoem.

[theacodes]

Awesome, @rstuart85. Glad to know that we both fixed your issue and confirmed that the patch works.

[dhermes]

I love that we Jon found the cause. Since I found a ton of other bugs on the web that experienced this with Google APIs, I still want to create a very simple repro-script (likely won't use gcloud-python or at least won't use much of it).

[dhermes]

I am playing around and not able to force the error with 15 threads using the same httplib2.Http object.

@rstuart85 Can you give a sense of how many threads were active?

(FWIW, I am using a different origin for all 15 threads and that may be mitigating the issue. I am going to try to make a datastore example as suggested by @rstuart85 and then extract the pieces that are not tied to gcloud-python or oauth2client)

[rstuart85]

@dhermes as I mentioned, it happens every time I run the revised version of my script linked to above. All you need is two active HTTP connections from the same origin talking to the same service at the same time. Hence why I used sleeps in the code. My code used 4 threads.

[dhermes]

Thanks. Sorry for losing the thread of the conversation.

[dhermes]

Yay! I got my first repro!

Exception in thread Thread-2:
...
SSLError: [Errno 1] _ssl.c:1429: error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number

and

Exception in thread Thread-1:
...
SSLError: [Errno 1] _ssl.c:1429: error:1408F081:SSL routines:SSL3_GET_RECORD:block cipher pad is wrong

---

I'm working on throwing away as much of the extra (e.g. all of gcloud-python) as possible so that we can dive deep on the failure (or more likely, someone from httplib2 can).

[rstuart85]

Sounds like a plan. I'm pretty gob smacked that more people aren't reporting this problem. I couldn't find much about it out there and I've personally being trying to convince people for a few months that it is an issue.

[dhermes]

The lack of thread-safety in httplib2 is a well-known thing:
https://developers.google.com/api-client-library/python/guide/thread_safety

Though we don't really publicly advertise that we use httplib2 underneath.

I have a "broken down" isolated example with httplib2 only: https://gist.github.com/dhermes/aa7b6186ede2a9d0b471

I'm sure someone could use it to find out how the bits are going to the wrong thread and maybe even fix it.

[dhermes]

@rstuart85 RE: "more people reporting" it seems there is chatter:

It looks like other people have experienced this error using a Google API. (SSLv3 is deprecated).

[bendemaree]

I'm pretty gob smacked that more people aren't reporting this problem.

We discovered this several months ago, and as advised in #926, we ended up mimicking httplib2's interface in requests which let us replace the HTTP implementation entirely, solving our threading issues.

EDIT: specifically #908.

Our library is now in production and I'm working on getting it open-sourced.

I'm sure someone could use it to find out how the bits are going to the wrong thread and maybe even fix it.

IIRC the primary threading issue is related to httplib2's global shared cache, which is not thread-safe. I think we observed SSL contexts ending up shared between threads.

[bendemaree]

Okay. We've released gcloud_requests as a way to use requests rather than httplib2. Relevant to #926 and #908 as well.

There are basic system smoke tests for the other (not Cloud Datastore) connection wrappers, but I am waiting for an empty Cloud Platform project to be set up to isolate our environment.

[dhermes]

Massive :+1:. Mind if we link to this in our README.md and auth.rst documents?

[bendemaree]

'course not, but it might be wise to do so with a caveat (for the time being). :smile: We'll have better system tests soon, and might even get adventurous and try to incorporate the system tests here. It looks like I'd need to hijack the CLIENT that's bootstrapped in most of the tests, though.

[dhermes]

Neat! We can try to make it more modularized? Or you could just monkey-patch the CLIENT on the module object(s) before actually running the test cases.

[bendemaree]

I think monkey-patching will be the least friction and should work just fine.

[dhermes]

I'm still tracking this for fun. The core issue comes from the fact that a connection is re-used for the same protocol / host combination and it is stored in a dictionary (Http.connections = {}) across threads.

From here, I'm still curious how the connection itself causes the issue.

[dhermes]

The source of httplib.py (from the 2.7 branch) describes the flow an object goes through.

In particular, if the HTTPConnection object is not idle, it won't allow starting a connection:

        if self.__state == _CS_IDLE:
            self.__state = _CS_REQ_STARTED
        else:
            raise CannotSendRequest()

But this error inherits (indirectly) httplib.HTTPException. This is problematic in httplib2 because it catches ANY httplib.HTTPException after a failed connect()/request() and tries again one more time. (This may be a "feature" rather than a bug for most subclasses of httplib.HTTPException)

It still doesn't showcase how the streams get crossed, still digging there.

[sadovnychyi]

Not really related to this issue, but I found the same problem while using oauth2client to authorize my requests by calling credentials.authorize(Http()).

[sadovnychyi]

As a fast workaround I wrote a custom wrapper around requests to pretend to be httplib2.

class CustomHttp(object):
  def __init__(self, timeout=None):
    self.timeout = timeout

  def request(self, uri, method='GET', body=None, headers=None,
              redirections=None, connection_type=None):
    if connection_type is not None:
      uri = '%s://%s' % (connection_type, uri)
    resp = requests.request(method=method, url=uri, data=body, headers=headers,
                            timeout=self.timeout)
    resp.status = resp.status_code
    return resp, resp.content

credentials.authorize(CustomHttp())

Works great now. Maybe useful to somebody.

[bendemaree]

@sadovnychyi If you're looking for something tested (and already in the wild) we did something similar.

[dhermes]

Thanks for chiming in @bendemaree! @jonparrott is simultaneously working on httplib2shim and removing it as a dependency within oauth2client.

[theacodes]

httplib2shim should be a drop-in replacement. I encourage anyone looking
for threadsafety in the near term to use and please report any bugs. It'll
help me abstract httplib2 from oauth2client.

On Tue, Feb 23, 2016 at 12:27 PM Danny Hermes [email protected]
wrote:

Thanks for chiming in @bendemaree https://github.com/bendemaree!
@jonparrott https://github.com/jonparrott is simultaneously working on
httplib2shim https://github.com/GoogleCloudPlatform/httplib2shim and
removing it as a dependency within oauth2client.

—
Reply to this email directly or view it on GitHub
https://github.com/GoogleCloudPlatform/gcloud-python/issues/1214#issuecomment-187888560
.

[dhermes]

@jonparrott Would you like us to advertise it in our docs?

[theacodes]

Sure, but definitely advertise with caution.

On Tue, Feb 23, 2016 at 12:46 PM Danny Hermes [email protected]
wrote:

@jonparrott https://github.com/jonparrott Would you like us to
advertise it in our docs?

—
Reply to this email directly or view it on GitHub
https://github.com/GoogleCloudPlatform/gcloud-python/issues/1214#issuecomment-187894927
.

[BobDohnal]

I am not sure if this is related, but this httplib2 issue can easily be replicated on AppEngine as soon as sockets are enabled for httplib using this app.yaml configuration and give it a bit load through TaskQueue using googleapiclient and oauth2client:

env_variables:
   GAE_USE_SOCKETS_HTTPLIB : 'anyvalue'

I jus hit a similiar issue with standard httplib which explodes with this exception, which I think leads to a similar issue with incorrectly sharing the same object:

File "/base/data/home/runtimes/python27/python27_dist/lib/python2.7/python_std_lib/httplib.py", line 1033, in getresponse
    raise ResponseNotReady()

[dhermes]

Totally related. Thanks for adding more context.

[eric-optimizely]

Is there any progress to report on this issue? I use this library in my flexible environment instances (which are required to be configured as threadsafe) and see a lot of possibly related errors in the logs. I've tried some of the workarounds listed here, but none of them resolved the issues.

[theacodes]

@eric-optimizely I would suggest you use the library in a thread-safe manner, e.g, construct a new client instance per-request.

[tseaver]

@eric-optimizely, @jonparrott holding a client instance per thread, assuming your application can be set up like that, would work as well.

[eric-optimizely]

@jonparrott @tseaver Using the client parameter for each method call definitely helps to clear up some issues, but I still see the following on occasion. Is it related to this issue?

...Redacted this part of the trace...
  File "/env/local/lib/python2.7/site-packages/gcloud/bigquery/table.py", line 711, in insert_data
    data=data)
  File "/env/local/lib/python2.7/site-packages/gcloud/connection.py", line 339, in api_request
    target_object=_target_object)
  File "/env/local/lib/python2.7/site-packages/gcloud/connection.py", line 240, in _make_request
    return self._do_request(method, url, headers, data, target_object)
  File "/env/local/lib/python2.7/site-packages/gcloud/connection.py", line 269, in _do_request
    body=data)
  File "/env/local/lib/python2.7/site-packages/oauth2client/client.py", line 597, in new_request
    self._refresh(request_orig)
  File "/env/local/lib/python2.7/site-packages/oauth2client/client.py", line 863, in _refresh
    self._do_refresh_request(http_request)
  File "/env/local/lib/python2.7/site-packages/oauth2client/client.py", line 895, in _do_refresh_request
    self.token_uri, method='POST', body=body, headers=headers)
  File "/env/local/lib/python2.7/site-packages/httplib2/__init__.py", line 1609, in request
    (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
  File "/env/local/lib/python2.7/site-packages/httplib2/__init__.py", line 1351, in _request
    (response, content) = self._conn_request(conn, request_uri, method, body, headers)
  File "/env/local/lib/python2.7/site-packages/httplib2/__init__.py", line 1272, in _conn_request
    conn.connect()
  File "/usr/lib/python2.7/httplib.py", line 1212, in connect
    server_hostname=server_hostname)
  File "/usr/lib/python2.7/ssl.py", line 350, in wrap_socket
    _context=self)
  File "/usr/lib/python2.7/ssl.py", line 566, in __init__
    self.do_handshake()
  File "/usr/lib/python2.7/ssl.py", line 788, in do_handshake
    self._sslobj.do_handshake()
SSLEOFError: EOF occurred in violation of protocol (_ssl.c:581)" 

[dhermes]

@eric-optimizely That error occurs when multiple threads read bytes from a payload at once, which gives invalid crypto bits. This likely means multiple threads have access to the same httplib2.Http object.

[eric-optimizely]

@dhermes Ok, thanks for confirming. I've used the client attribute wherever possible and am still seeing this error occasionally. I've seen at least one case where a gcloud function is reusing the provided client, so I suspect there could be others.

[dhermes]

Yeah, your current best bet is to create a thread-local Client.

[rstuart85]

@eric-optimizely Just in case it's useful: I fixed this for my stuff a while ago by making the connection object threadsafe. You can see the changes here. Hope it's useful.

[eric-optimizely]

@rstuart85 Cool, thanks! This looks like it's based on #1274 which should probably just be merged by @dhermes and @jonparrott , unless it's still incomplete or there's some other blocker. I'm not particularly interested in maintaining a fork of this library -- I just want it to work correctly on Google's own cloud platform.

[rstuart85]

@eric-optimizely I think the outcome of the discussion here was that it should be fixed upstream. Maybe @dhermes can confirm or deny that.

[dhermes]

Yes I am taking this on right now, the goal is to get a divorce from httplib2 and allow BYO transport with some nice defaults / examples.

[speedplane]

I just re-deployed my app and I went from never seeing this issue, to getting thousands of them. Did anything recently change? They come in a number of flavors:

• [SSL: DECRYPTION_FAILED_OR_BAD_RECORD_MAC] decryption failed or bad record mac (_ssl.c:1750)
• [SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1750)
• Content purported to be compressed with gzip but failed to decompress.
• [SSL: BLOCK_CIPHER_PAD_IS_WRONG] block cipher pad is wrong (_ssl.c:1750)

I was previously sharing http connections across threads. I'll make them theadsafe and see if that helps.

[speedplane]

Making my use of httplib thread-safe fixed the issue. Not sure what I did to cause this error to pop up, but I was tinkering with some related code, so I may have caused it.

If it helps anyone else, below is the code that I use to get the authentication token. It relies on a dictionary PRIVATE_KEY to lookup the service account email and the PEM file (which should probably just be inputs into the function).

PRIVATE_KEY = {'my-project' : {'email':'[email protected]', 'key':'my_key.pem'}}
auth_cache = threading.local()
auth_cache_lock = threading.Lock()
def get_auth(project, force = False, scopes = None):
    global auth_cache, auth_cache_lock
    # At some point oauth2client had breaking changes. There are different 
    # copies of this library floating around, so handle whichever default imports.  
    try:
        from oauth2client.client import SignedJwtAssertionCredentials
    except ImportError as e:
        SignedJwtAssertionCredentials = None
        from oauth2client import crypt as oauth_crypt
        from oauth2client.service_account import ServiceAccountCredentials
    if project not in PRIVATE_KEY:
        raise Exception("Cannot connect to %s"%project)
    with auth_cache_lock:
        if not hasattr(auth_cache, 'auths'):
            auth_cache.auths = {}

        if not force and project in auth_cache.auths:
            return auth_cache.auths[project]
        f = None
        key = PRIVATE_KEY[project]['key']
        for path in ('updater/', '../updater/', ''):
            pfile = path + key
            try:
                f = file(pfile, 'rb')
            except IOError:
                continue
            else:
                break
        if not f:
            raise Exception("Could not find key file.")
        pem_contents = f.read()
        f.close()
        if scopes == None:
            scopes = DEFAULT_SCOPES
        # The first parameter, service_account_name, is the Email address created 
        # for the Service account. It must be the email address associated with 
        # the key that was created.
        if SignedJwtAssertionCredentials:
            # The old way of doing it.
            credential = SignedJwtAssertionCredentials(
              PRIVATE_KEY[project]['email'], pem_contents, scope = scopes)
        else:
            # The new way (improvement?)
            signer = oauth_crypt.Signer.from_string(pem_contents)
            credential = ServiceAccountCredentials(
                PRIVATE_KEY[project]['email'], signer, scopes = scopes)
            credential._private_key_pkcs8_pem = pem_contents

        http = httplib2.Http()
        http = credential.authorize(http)
        auth_cache.auths[project] = http
        return http

[theacodes]

Superseded by the discussion in #1346

[dhermes]

For those still following this issue, #3674 is putting the final nail in our usage of httplib2

[Ark-kun]

I have the issues with WRONG_VERSION_NUMBER even when I have a single client per thread (just doing two requests one after another in the same function).

The workaround for me was to clear the connections before making each request: http.connection = {}.