Google-api-python-client: SSLEOFError: EOF occurred in violation of protocol (_ssl.c:590)

[carlopires]

After installing this package on python 2.7.10 and using Google Admin API I get tons of these errors in an intermittent fashion:

  File "/usr/local/lib/python2.7/site-packages/oauth2client/util.py", line 135, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/googleapiclient/http.py", line 827, in execute
    method=str(self.method), body=self.body, headers=self.headers)
  File "/usr/local/lib/python2.7/site-packages/googleapiclient/http.py", line 167, in _retry_request
    raise exception
SSLEOFError: EOF occurred in violation of protocol (_ssl.c:590)

I'm using Ubuntu 14.04:

$ openssl version
OpenSSL 1.0.1f 6 Jan 2014


ubuntu@tests:~$ python
Python 2.7.10 (default, Sep  7 2015, 14:45:00) 
[GCC 4.8.4] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import ssl
>>> ssl.OPENSSL_VERSION
'OpenSSL 1.0.1f 6 Jan 2014'

Any ideas on how to fix this?

[daspecster]

If this wasn't installed on a fresh machine/virtualenv you might try updating oauth2client and google-api-python-client.

[carlopires]

@daspecster I'm using latest versions from pypi of these packages. I even installed google-api-python-client from github but had no success.

[daspecster]

@dhermes will have some more ideas. It almost looks like connectivity issues?

Which admin API's are you calling?

[carlopires]

I'm using Directory API calls, more precisely:
https://developers.google.com/admin-sdk/directory/v1/reference/members/list

I noticed it happens more when I issue several API calls in parallel. Does it make sense?

This is happening with Drive API as well (files and permissions).

[theacodes]

This is likely happening because you're using a single client across multiple threads. httplib2 is not threadsafe and therefore neither is google-api-python-client. I recommend that you construct a separate client for each thread.

[theacodes]

(closing for now, feel free to continue to comment. we can re-open if needed)

[daspecster]

Sorry @jonparrott, I clicked this from my email thinking it was in gcloud-python.

[carlopires]

@jonparrott what make you think I'm running multiples threads? I'm running eventlet to do concurrent access.

[theacodes]

@daspecster I'll never complain about you accidentally helping me out here. :)

@carlopires this is usually the most likely culprit when it comes to exceptions deep in the stack like this. If it's not, then the next most likely culprit is that your openssl version is out of date. I would recommend updating openssl as a first step.

[carlopires]

@jonparrott did you see I'm using latest openssl from Ubuntu 14.04 + Python 2.7.10?

So, Is Python 2.7.10 + upgraded Ubuntu 14.04 incompatible with google-api-python-client?

[theacodes]

latest openssl from Ubuntu 14.04 + Python 2.7.10?

As it stands that version of openssl is actually 2 years old (it was released in 2014).

So, Is Python 2.7.10 + upgraded Ubuntu 14.04 incompatible with google-api-python-client?

No. But it's still likely that your usage of eventlet is what is causing issues here. Httplib2 can still have issues with non-thread-based parallelism. I'd recommend trying to construct one httplib2 instance & client per green thread and see if that resolves it.

[dhermes]

This particular line is the problem

if conn_key in self.connections:
    conn = self.connections[conn_key]

it is not safe for concurrent access. The EOF error you see is because multiple actors are reading from the same stream of bytes concurrently and so one of the actors gets to EOF before it expected to be done reading the crypto payload (since one of the other actors took part of the payload).

[carlopires]

@jonparrott I tried make my own adapter using python requests + pyOpenSSL + certifi as well. Same error.

I'm going to compile Python + latest OpenSSL manually to see if Google servers accept my requests.

FYI, although openssl in Ubuntu is versioned that old, they backports most of security patches:
http://changelogs.ubuntu.com/changelogs/pool/main/o/openssl/openssl_1.0.1f-1ubuntu2.11/changelog

@dhermes thanks for the hint. I'll check this.

[dhermes]

The fact that you got the same error without using httplib2 is a bit concerning. Though maybe google-api-python-client used httplib2 under the covers without you noticing.

Have you seen: https://developers.google.com/api-client-library/python/guide/thread_safety

[carlopires]

@dhermes I checked this.. the error happens with requests generated by python requests[security] as well.

What make this hard to debug is because it is intermittent. Sometimes happens, some not. I'll try to confirm if it is related to concurrency use.

[theacodes]

It's probably very likely that this library is sneakily using httplib2
without you noticing. You can try using httplib2shim and see if that fixes
things, but the simplest thing to do is just construct a new client for
every thread/greenlet/etc.

On Thu, Jul 21, 2016, 9:53 AM Carlo Pires [email protected] wrote:

@dhermes https://github.com/dhermes I checked this.. the error happens
with requests generated by python requests[security] as well.

What make this hard to debug is because it is intermittent. Sometimes
happens, some not. I'll try to confirm if it is related to concurrency use.

—
You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub
https://github.com/google/google-api-python-client/issues/253#issuecomment-234315093,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAPUc57_wLlIa_Co7ojt-JeVCdDSyZhLks5qX6QTgaJpZM4JRXBd
.

[carlopires]

@jonparrott I did it for httplib2: one HTTP client generated per request. And the error happens anyway.

[dhermes]

Can you share some code?

[carlopires]

@dhermes I'm preparing a test case for this.

[dhermes]

Thanks

[carlopires]

Just updating... I compiled Python 2.7.12 + Latest OpenSSL (v1.0.2h) from sources. And the error continues. Same behavior with httplib2 or python requests.

Python 2.7.12 (default, Jul 21 2016, 19:11:09) 
[GCC 4.8.4] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import ssl
>>> ssl.OPENSSL_VERSION
'OpenSSL 1.0.2h  3 May 2016'
>>>

[theacodes]

If you're creating the http client in every thread and still seeing this,
then this is an issue with the lower level http libraries and not with this
library.

On Thu, Jul 21, 2016, 1:25 PM Carlo Pires [email protected] wrote:

Just updating... I compile Python 2.7.12 + Latest OpenSSL (v1.0.2h) from
sources. And the error continues. Same behavior with httplib2 or python
requests.

Python 2.7.12 (default, Jul 21 2016, 19:11:09)
[GCC 4.8.4] on linux2
Type "help", "copyright", "credits" or "license" for more information.

import ssl
ssl.OPENSSL_VERSION
'OpenSSL 1.0.2h 3 May 2016'

—
You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub
https://github.com/google/google-api-python-client/issues/253#issuecomment-234372915,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAPUc2OklAMsaAqIq9MivBXIeA1k_OeUks5qX9WzgaJpZM4JRXBd
.

[carlopires]

@jonparrott What do you mean by lower level http libraries?

[theacodes]

As in httplib2 and the underlying urlopen and SSL libraries.

On Thu, Jul 21, 2016, 1:29 PM Carlo Pires [email protected] wrote:

@jonparrott https://github.com/jonparrott What do you mean by lower
level http libraries?

—
You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub
https://github.com/google/google-api-python-client/issues/253#issuecomment-234374135,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAPUc1Wj6hpfv4HC9bsAwLRvP_ZnoF-Oks5qX9atgaJpZM4JRXBd
.

[dhermes]

@carlopires It'll be easiest to diagnose with actual code for us to run.

[carlopires]

@dhermes I'm doing this right now... I just wanted to make sure it is not related to Ubuntu OpenSSL version.

[dhermes]

I'm 99.9% sure this is caused by the non-threadsafe dictionary I linked to above. I am running the same version of Ubuntu as you are and have never had a problem.

[carlopires]

I tested this issue and found out that it is reproducible with the following configurations (using Ubuntu 14.04):

• default python + OpenSSL from Ubuntu 14.04
• Python 2.7.12 + OpenSSL 1.0.2h

The script used to reproduce the error is this:
https://gist.github.com/carlopires/efbf607f746c5acb05b452571e784bbe

@dhermes As you can see in the script, I'm instantiating the HTTP client inside the eventlet task, so I'm not susceptible to the problem you mentioned.

The error is _intermittent_, so it is not easily reproducible in a dev environment. It is more prone to occur in a high number of concurrent requests to Google servers.

The way I'm dealing with it is by reducing the concurrency and retrying the requests.

I think it is a problem in Google servers throttling system.

[theacodes]

@carlopires does this same issue occur if you use multiprocessing for concurrency?

[carlopires]

@jonparrott It always happen if you increase the number of connections originating from the same machine. It doesn't matter if it is from a single process or from multi processes (I have both scenarios). The problem is related to the number of concurrent requests going to Google.

[theacodes]

Thanks @carlopires for thoroughly investigating this.

Do you think there's anything this library can do to alleviate this?

[carlopires]

@jonparrott As the problem arrives from the use of this library it would be good to have some way to inform the developers that this error can be erroneously reported by Google servers in high concurrency scenarios.

[theacodes]

So documentation? Do you have any recommendations on the best way to surface this?

[dhermes]

Have you seen https://github.com/eventlet/eventlet/issues/145