Google-api-python-client: socket.timeout with will cause api client to become unuseable

Created on 23 Aug 2018 · 16Comments · Source: googleapis/google-api-python-client

We recently ran into a socket.timeout issue using the google api client, and noticed that then all our requests would fail in the same way. I later find that connections are cached here: https://github.com/httplib2/httplib2/blob/master/python3/httplib2/__init__.py#L1759

however there is no code to remove cached connections that are bad. This is a major oversight.

Here's the callstack we hit:

    batch_entry.batch.execute(http=http, num_retries=num_retries)
File "/pyenv/lib/python3.6/site-packages/googleapiclient/_helpers.py", line 130, in positional_wrapper
    return wrapped(*args, **kwargs)
File "/pyenv/lib/python3.6/site-packages/googleapiclient/http.py", line 1465, in execute
    self._execute(http, self._order, self._requests, num_retries)
File "/pyenv/lib/python3.6/site-packages/googleapiclient/http.py", line 1394, in _execute
    body=body, headers=headers)
File "/pyenv/lib/python3.6/site-packages/googleapiclient/http.py", line 185, in _retry_request
    raise exception
File "/pyenv/lib/python3.6/site-packages/googleapiclient/http.py", line 165, in _retry_request
    resp, content = http.request(uri, method, *args, **kwargs)
File "/pyenv/lib/python3.6/site-packages/oauth2client/transport.py", line 175, in new_request
    redirections, connection_type)
File "/pyenv/lib/python3.6/site-packages/oauth2client/transport.py", line 282, in request
    connection_type=connection_type)
File "/pyenv/lib/python3.6/site-packages/httplib2/__init__.py", line 1514, in request
    (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
File "/pyenv/lib/python3.6/site-packages/httplib2/__init__.py", line 1264, in _request
    (response, content) = self._conn_request(conn, request_uri, method, body, headers)
File "/pyenv/lib/python3.6/site-packages/httplib2/__init__.py", line 1217, in _conn_request
    response = conn.getresponse()
File "/usr/local/lib/python3.6/http/client.py", line 1331, in getresponse
    response.begin()
File "/usr/local/lib/python3.6/http/client.py", line 297, in begin
    version, status, reason = self._read_status()
File "/usr/local/lib/python3.6/http/client.py", line 258, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/usr/local/lib/python3.6/socket.py", line 586, in readinto
    return self._sock.recv_into(b)
File "/usr/local/lib/python3.6/ssl.py", line 1009, in recv_into
    return self.read(nbytes, buffer)
File "/usr/local/lib/python3.6/ssl.py", line 871, in read
    return self._sslobj.read(len, buffer)
File "/usr/local/lib/python3.6/ssl.py", line 631, in read
    v = self._sslobj.read(len, buffer)
socket.timeout: The read operation timed out

p2 will not fix bug

Source

thehesiod

All 16 comments

looks like this should do something similar to what's being done in the cloud-sdk retry code: http://testcompany.info/google-cloud-sdk/lib/third_party/apitools/base/py/http_wrapper.py

thehesiod on 23 Aug 2018

note: a separate callstack happens via discovery.build

thehesiod on 23 Aug 2018

I think a better solution would be for httplib2 to handle most of these cases, like socket.*. Higher level errors can be handled by the google client

thehesiod on 23 Aug 2018

see: https://github.com/httplib2/httplib2/issues/18

thehesiod on 23 Aug 2018

This has been resolved as of httplib2 v0.12.0.

busunkim96 on 6 Dec 2018

actually no, it's post 0.12.0 (806de52a2939d95800e43b1ab96be45a33b882d1 vs 806de52a2939d95800e43b1ab96be45a33b882d1 for 0.12.0), supposedly 12.1. This should be re-opened

thehesiod on 6 Dec 2018

👍1

Can confirm.


Traceback (most recent call last):
  File "./query.py", line 88, in <module>
    if __name__ == '__main__': get_front_page(1,1)
  File "./query.py", line 67, in get_front_page
    service = get_authenticated_service()
  File "./query.py", line 54, in get_authenticated_service
    service = build(API_SERVICE_NAME, API_VERSION, credentials=credentials)
  File "/home/nexor/.pyenv/versions/3.6.0/lib/python3.6/site-packages/googleapiclient/_helpers.py", line 130, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/home/nexor/.pyenv/versions/3.6.0/lib/python3.6/site-packages/googleapiclient/discovery.py", line 222, in build
    requested_url, discovery_http, cache_discovery, cache, developerKey)
  File "/home/nexor/.pyenv/versions/3.6.0/lib/python3.6/site-packages/googleapiclient/discovery.py", line 272, in _retrieve_discovery_doc
    resp, content = http.request(actual_url)
  File "/home/nexor/.local/lib/python3.6/site-packages/httplib2/__init__.py", line 1924, in request
    cachekey,
  File "/home/nexor/.local/lib/python3.6/site-packages/httplib2/__init__.py", line 1595, in _request
    conn, request_uri, method, body, headers
  File "/home/nexor/.local/lib/python3.6/site-packages/httplib2/__init__.py", line 1501, in _conn_request
    conn.connect()
  File "/home/nexor/.local/lib/python3.6/site-packages/httplib2/__init__.py", line 1289, in connect
    sock.connect((self.host, self.port))
socket.timeout: timed out

Checked to see if credentials were expired but it's fine. Not sure why this is happening

Requirement already up-to-date: httplib2 in /home/nexor/.local/lib/python3.6/site-packages (0.12.0)

mvrozanti on 6 Dec 2018

Anyone got any quickfix in the meantime? I don't seem to be able to get past authentication, ever. But the same script was working a couple days ago. quickstart.py sample also stopped working.

mvrozanti on 7 Dec 2018

Magically started working again

mvrozanti on 11 Dec 2018

v0.12.1 of httplib2 has been released and contains the update to clear dead connections.

busunkim96 on 10 Jun 2019

🎉1

https://github.com/googleapis/google-api-python-client/issues/480

echan00 on 7 Oct 2019

httplib2 v.0.14.0 error still happen

topoleov on 7 Dec 2019

@soogly If you're experiencing this issue please open a new issue with more details on what's happening. Repository maintainers have a harder time keeping track of responses to closed issues.

Thanks!

busunkim96 on 9 Dec 2019

Hello, this error is happening multiple times a day.

The read operation timed out
Traceback (most recent call last):
File "", line 197, in
File "", line 34, in get_results
File "/usr/local/lib/python3.8/site-packages/googleapiclient/_helpers.py", line 130, in positional_wrapper
return wrapped(args, *kwargs)
File "/usr/local/lib/python3.8/site-packages/googleapiclient/http.py", line 849, in execute
resp, content = _retry_request(
File "/usr/local/lib/python3.8/site-packages/googleapiclient/http.py", line 184, in _retry_request
raise exception
File "/usr/local/lib/python3.8/site-packages/googleapiclient/http.py", line 165, in _retry_request
resp, content = http.request(uri, method, args, *kwargs)
File "/usr/local/lib/python3.8/site-packages/oauth2client/transport.py", line 173, in new_request
resp, content = request(orig_request_method, uri, method, body,
File "/usr/local/lib/python3.8/site-packages/oauth2client/transport.py", line 280, in request
return http_callable(uri, method=method, body=body, headers=headers,
File "/usr/local/lib/python3.8/site-packages/httplib2/__init__.py", line 1982, in request
(response, content) = self._request(
File "/usr/local/lib/python3.8/site-packages/httplib2/__init__.py", line 1650, in _request
(response, content) = self._conn_request(
File "/usr/local/lib/python3.8/site-packages/httplib2/__init__.py", line 1589, in _conn_request
response = conn.getresponse()
File "/usr/local/lib/python3.8/http/client.py", line 1322, in getresponse
response.begin()
File "/usr/local/lib/python3.8/http/client.py", line 303, in begin
version, status, reason = self._read_status()
File "/usr/local/lib/python3.8/http/client.py", line 264, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/usr/local/lib/python3.8/socket.py", line 669, in readinto
return self._sock.recv_into(b)
File "/usr/local/lib/python3.8/ssl.py", line 1241, in recv_into
return self.read(nbytes, buffer)
File "/usr/local/lib/python3.8/ssl.py", line 1099, in read
return self._sslobj.read(len, buffer)
socket.timeout: The read operation timed out
Script failed.

Any suggestions how to get rid of it

Thanks

Vmarkelov on 25 Feb 2020

Please reference #709 to track socket timeout issues.

busunkim96 on 25 Feb 2020

Hello, this error is happening multiple times a day.

The read operation timed out
Traceback (most recent call last):
File "", line 197, in
File "", line 34, in get_results
File "/usr/local/lib/python3.8/site-packages/googleapiclient/_helpers.py", line 130, in positional_wrapper
return wrapped(args, kwargs)
File "/usr/local/lib/python3.8/site-packages/googleapiclient/http.py", line 849, in execute
resp, content = _retry_request(
File "/usr/local/lib/python3.8/site-packages/googleapiclient/http.py", line 184, in _retry_request
raise exception
File "/usr/local/lib/python3.8/site-packages/googleapiclient/http.py", line 165, in _retry_request
resp, content = http.request(uri, method, *args, *kwargs)
File "/usr/local/lib/python3.8/site-packages/oauth2client/transport.py", line 173, in new_request
resp, content = request(orig_request_method, uri, method, body,
File "/usr/local/lib/python3.8/site-packages/oauth2client/transport.py", line 280, in request
return http_callable(uri, method=method, body=body, headers=headers,
File "/usr/local/lib/python3.8/site-packages/httplib2/init.py", line 1982, in request
(response, content) = self._request(
File "/usr/local/lib/python3.8/site-packages/httplib2/init.py", line 1650, in _request
(response, content) = self._conn_request(
File "/usr/local/lib/python3.8/site-packages/httplib2/init**.py", line 1589, in _conn_request
response = conn.getresponse()
File "/usr/local/lib/python3.8/http/client.py", line 1322, in getresponse
response.begin()
File "/usr/local/lib/python3.8/http/client.py", line 303, in begin
version, status, reason = self._read_status()
File "/usr/local/lib/python3.8/http/client.py", line 264, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/usr/local/lib/python3.8/socket.py", line 669, in readinto
return self._sock.recv_into(b)
File "/usr/local/lib/python3.8/ssl.py", line 1241, in recv_into
return self.read(nbytes, buffer)
File "/usr/local/lib/python3.8/ssl.py", line 1099, in read
return self._sslobj.read(len, buffer)
socket.timeout: The read operation timed out
Script failed.

Any suggestions how to get rid of it

Thanks

After hours of hacking api code, found the solution.

import socket
socket.setdefaulttimeout(4000) 


import argparse
import googleapiclient.discovery
from google.oauth2 import service_account
import mimetypes
import json
import sys
import os

Make note the order - socket.setdefaulttimeout(timeoutInSec) needs to be before any import of googleapiclient.
No magic happens instead ;)