Google-api-python-client: `socket.timeout: timed out` when using full IPv6 stack.

[n8felton]

Environment details

• OS: macOS 10.14.5 (18F132) Mojave
• Python version: 3.7.3
• pip version: 19.1.1
• google-api-python-client version: 1.7.9

Steps to reproduce

Pre-steps:

• Have a full IPv6 stack with an IPv6 address. In my case, I have the following from Spectrum, through a Linksys WRT3200ACM router.

inet6 fe80::4ab:197d:b439:8d97%en0 prefixlen 64 secured scopeid 0x9 
inet6 2604:6000:1013:a0eb:cca:87bb:d7ed:91b6 prefixlen 64 autoconf secured 
inet6 2604:6000:1013:a0eb:91e1:1128:7728:8e1 prefixlen 64 autoconf temporary 

Error:

1. Follow the steps for the Python Quickstart (https://developers.google.com/docs/api/quickstart/python)
2. run python quickstart.py
3. python exits with socket.timeout: timed out error

If I open System Preferences > Network > Wi-FI > Advanced > TCP/IP and change the IPv6 configuration from Automatically to Link-local only, the quickstart.py properly runs.

[busunkim96]

A few folks have filed issues about socket timeouts recently - thank you for the additional info!

[vladignatyev]

Socket problem is still here, in 2020.

Environment details

• OS: Alpine (docker image https://hub.docker.com/_/python python:3.8.0-alpine)

Container runs on AWS EC2 instance.

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/rq/worker.py", line 886, in perform_job
    rv = job.perform()
  File "/usr/local/lib/python3.8/site-packages/rq/job.py", line 664, in perform
    self._result = self._execute()
  File "/usr/local/lib/python3.8/site-packages/rq/job.py", line 670, in _execute
    return self.func(*self.args, **self.kwargs)
  File "/usr/src/app/searchconsole/data.py", line 90, in load_data_for_property
    save_data_batch(property, batch)
  File "/usr/src/app/searchconsole/data.py", line 103, in save_data_batch
    for data_batch in data_batches:
  File "/usr/src/app/searchconsole/google/reports.py", line 103, in generate_lazy
    response = request.execute()
  File "/usr/local/lib/python3.8/site-packages/googleapiclient/_helpers.py", line 130, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/googleapiclient/http.py", line 849, in execute
    resp, content = _retry_request(
  File "/usr/local/lib/python3.8/site-packages/googleapiclient/http.py", line 184, in _retry_request
    raise exception
  File "/usr/local/lib/python3.8/site-packages/googleapiclient/http.py", line 165, in _retry_request
    resp, content = http.request(uri, method, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/google_auth_httplib2.py", line 197, in request
    response, content = self.http.request(
  File "/usr/local/lib/python3.8/site-packages/httplib2/__init__.py", line 1982, in request
    (response, content) = self._request(
  File "/usr/local/lib/python3.8/site-packages/httplib2/__init__.py", line 1650, in _request
    (response, content) = self._conn_request(
  File "/usr/local/lib/python3.8/site-packages/httplib2/__init__.py", line 1589, in _conn_request
    response = conn.getresponse()
  File "/usr/local/lib/python3.8/http/client.py", line 1322, in getresponse
    response.begin()
  File "/usr/local/lib/python3.8/http/client.py", line 303, in begin
    version, status, reason = self._read_status()
  File "/usr/local/lib/python3.8/http/client.py", line 264, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/local/lib/python3.8/socket.py", line 669, in readinto
    return self._sock.recv_into(b)
  File "/usr/local/lib/python3.8/ssl.py", line 1241, in recv_into
    return self.read(nbytes, buffer)
  File "/usr/local/lib/python3.8/ssl.py", line 1099, in read
    return self._sslobj.read(len, buffer)
socket.timeout: The read operation timed out

[gselva]

@vladignatyev - I had this error and was able to get around it by updating /etc/gai.conf to prefer IPV4 over IPV6 (uncomment the line precedence ::ffff:0:0/96 100 ).

[cheptsov]

I have this issue too

[tuliocgon]

I'm experiencing this right now, the whole weekend, a code that was previoudly fine now i can't get pass authentication:

.../python3.8/site-packages/httplib2/__init__.py, line 1324, in connect
sock.connect((self.host.self.port__
socket.timetou: time out

Any advices how to solve this ?

[vlad-oles]

I also started experiencing this issue just a day ago, with no modification to the code:

    service = build('sheets', 'v4', credentials=get_creds())
  File "/home/me/anaconda3/lib/python3.7/site-packages/googleapiclient/_helpers.py", line 134, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/home/me/anaconda3/lib/python3.7/site-packages/googleapiclient/discovery.py", line 225, in build
    requested_url, discovery_http, cache_discovery, cache, developerKey
  File "/home/me/anaconda3/lib/python3.7/site-packages/googleapiclient/discovery.py", line 282, in _retrieve_discovery_doc
    resp, content = http.request(actual_url)
  File "/home/me/anaconda3/lib/python3.7/site-packages/httplib2/__init__.py", line 1991, in request
    cachekey,
  File "/home/me/anaconda3/lib/python3.7/site-packages/httplib2/__init__.py", line 1651, in _request
    conn, request_uri, method, body, headers
  File "/home/me/anaconda3/lib/python3.7/site-packages/httplib2/__init__.py", line 1557, in _conn_request
    conn.connect()
  File "/home/me/anaconda3/lib/python3.7/site-packages/httplib2/__init__.py", line 1324, in connect
    sock.connect((self.host, self.port))

• OS: Ubuntu 18.04.5 LTS
• Python: 3.7.6
• pip: 20.0.2
• googleapiclient: 1.8.0

Uncommenting the line precedence ::ffff:0:0/96 100 in /etc/gai.conf fixed the problem.

[SurferJeffAtGoogle]

My first attempt to reproduce on a Windows machine did not manifest the bug.
Trying Linux....

[SurferJeffAtGoogle]

Tried on Linux, haven't manifested the bug yet. Still working on it...

[SurferJeffAtGoogle]

How often do you see the socket timeout? With every request? What fraction of your requests?

[SurferJeffAtGoogle]

Has your code called socket.setdefaulttimeout(timeout) before calling this function?

What value is returned by socket.getdefaulttimeout() immediately before calling the API?

[SurferJeffAtGoogle]

After studying the code in the client library, I don't see any red flags in the client library itself. An n8felton's stack trace demonstrates, the google client calls httplib2, which calls socket functions, where the timeout happens.

[SurferJeffAtGoogle]

I'm trying to reproduce this issue on Alpine Linux running on an EC2 instance. When I run the quickstart (https://developers.google.com/docs/api/quickstart/python), it asks me to visit a localhost website to authenticate, which I can't do, since I'm connected via ssh with no graphical user interface.

Does someone have a simpler way to reproduce the misbehavior?

[cfiner]

I'm having the same issue. Code was running fine a month ago, tried today and I get a timeout error.

[---------------------------------------------------------------------------
timeout                                   Traceback (most recent call last)
<ipython-input-44-75c2f3716daf> in <module>()
----> 1 service = build('calendar','v3', credentials=credentials)

/Users/me/opt/anaconda3/lib/python3.7/site-packages/googleapiclient/_helpers.py in positional_wrapper(*args, **kwargs)
    132                 elif positional_parameters_enforcement == POSITIONAL_WARNING:
    133                     logger.warning(message)
--> 134             return wrapped(*args, **kwargs)
    135 
    136         return positional_wrapper

/Users/me/opt/anaconda3/lib/python3.7/site-packages/googleapiclient/discovery.py in build(serviceName, version, http, discoveryServiceUrl, developerKey, model, requestBuilder, credentials, cache_discovery, cache, client_options)
    223       options to set user options on the client.
    224       (1) The API endpoint should be set through client_options. If API endpoint
--> 225       is not set, `GOOGLE_API_USE_MTLS_ENDPOINT` environment variable can be used
    226       to control which endpoint to use.
    227       (2) client_cert_source is not supported, client cert should be provided using

/Users/me/opt/anaconda3/lib/python3.7/site-packages/googleapiclient/discovery.py in _retrieve_discovery_doc(url, http, cache_discovery, cache, developerKey)
    280                 base=discovery_url,
    281                 http=http,
--> 282                 developerKey=developerKey,
    283                 model=model,
    284                 requestBuilder=requestBuilder,

/Users/me/opt/anaconda3/lib/python3.7/site-packages/httplib2/__init__.py in request(self, uri, method, body, headers, redirections, connection_type)
   1974                         headers,
   1975                         redirections,
-> 1976                         cachekey,
   1977                     )
   1978         except Exception as e:

/Users/me/opt/anaconda3/lib/python3.7/site-packages/httplib2/__init__.py in _request(self, conn, host, absolute_uri, request_uri, method, body, headers, redirections, cachekey)
   1638 
   1639         (response, content) = self._conn_request(
-> 1640             conn, request_uri, method, body, headers
   1641         )
   1642 

/Users/me/opt/anaconda3/lib/python3.7/site-packages/httplib2/__init__.py in _conn_request(self, conn, request_uri, method, body, headers)
   1544             try:
   1545                 if conn.sock is None:
-> 1546                     conn.connect()
   1547                 conn.request(method, request_uri, body, headers)
   1548             except socket.timeout:

/Users/me/opt/anaconda3/lib/python3.7/site-packages/httplib2/__init__.py in connect(self)
   1315                 if has_timeout(self.timeout):
   1316                     sock.settimeout(self.timeout)
-> 1317                 sock.connect((self.host, self.port))
   1318 
   1319                 self.sock = self._context.wrap_socket(sock, server_hostname=self.host)

timeout: timed out](url)

[max-sixty]

If another case is helpful, I'm getting the same error when attempting to instantiate a storagetransfer client, running from GKE. From my laptop, using the same docker image, I can't repro.

   File "/usr/local/lib/python3.8/site-packages/{...}/load.py", line 137, in _storage_transfer_client
     return googleapiclient.discovery.build("storagetransfer", "v1", cache=MemoryCache())
   File "/usr/local/lib/python3.8/site-packages/googleapiclient/_helpers.py", line 134, in positional_wrapper
     return wrapped(*args, **kwargs)
   File "/usr/local/lib/python3.8/site-packages/googleapiclient/discovery.py", line 245, in build
     content = _retrieve_discovery_doc(
   File "/usr/local/lib/python3.8/site-packages/googleapiclient/discovery.py", line 305, in _retrieve_discovery_doc
     resp, content = http.request(actual_url)
   File "/usr/local/lib/python3.8/site-packages/httplib2/__init__.py", line 1985, in request
     (response, content) = self._request(
   File "/usr/local/lib/python3.8/site-packages/httplib2/__init__.py", line 1650, in _request
     (response, content) = self._conn_request(
   File "/usr/local/lib/python3.8/site-packages/httplib2/__init__.py", line 1589, in _conn_request
     response = conn.getresponse()
   File "/usr/local/lib/python3.8/http/client.py", line 1347, in getresponse
     response.begin()
   File "/usr/local/lib/python3.8/http/client.py", line 307, in begin
     version, status, reason = self._read_status()
   File "/usr/local/lib/python3.8/http/client.py", line 268, in _read_status
     line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
   File "/usr/local/lib/python3.8/socket.py", line 669, in readinto
     return self._sock.recv_into(b)
   File "/usr/local/lib/python3.8/ssl.py", line 1241, in recv_into
     return self.read(nbytes, buffer)
   File "/usr/local/lib/python3.8/ssl.py", line 1099, in read
     return self._sslobj.read(len, buffer)
 socket.timeout: The read operation timed out

[max-sixty]

My problem was solved by duplicating my GKE cluster, and it working on the new GKE cluster. So not sure what the actual problem was — calling build with any valid service python -c 'import googleapiclient.discovery; googleapiclient.discovery.build("compute", "v1")' would hang before timing out.

[SurferJeffAtGoogle]

Thank you @max-sixty, that's a really useful data point. I will follow up with GKE engineers. Is the old, malfunctioning cluster still running?

[SurferJeffAtGoogle]

I filed internal tracking bug 171068451.

[max-sixty]

Hi @SurferJeffAtGoogle — this ended up being a GKE issue, and the "it works on another cluster" was because the problem hadn't spread to another zone at that point; nothing to do with the cluster itself (it later spread to all zones).

I can't see it in the Incident History, but it was listed under https://console.cloud.google.com/support/cases;tab=knownissues for a while. Our Google support case number was 25392673. Given it spanned half a dozen regions, I would have expected to see it in the Incident History.

Let me know any questions! This was fairly painful for us — seemingly more so than for others, I still haven't fully resolved why.

[hkdevandla]

@max-sixty , can you tell us if you are still seeing the timeout issue? There was an issue related to API discovery service between 2020-10-06 to 2020-10-08 from GCE VMs which may have caused this timeout issue as well and the changes were rolled back to mitigate the issue. So want to check if it is still happening. Please advice.

[max-sixty]

This was fixed with the rollback and is no longer an issue for us, I should have made that clearer above.

There was an issue related to API discovery service between 2020-10-06 to 2020-10-08 from GCE VMs which may have caused this timeout issue as well and the changes were rolled back to mitigate the issue.

I can't see this on the incident history, I'm guessing you see this internally

[SurferJeffAtGoogle]

There have been recent changes to our network configuration. Would someone who has experienced this issue please confirm it's still happening?

[meeranagpal]

I just started getting this issue 2 days ago. I'm using the Google Custom Search API. Does anyone know how I can solve this?

Traceback (most recent call last): File "serp_googlecs.py", line 73, in <module> format_search_results('75_dataset.csv') File "serp_googlecs.py", line 56, in format_search_results queries, names = combined_serp_cs(data) File "serp_googlecs.py", line 50, in combined_serp_cs res = google_api_search(query, my_api_key, my_cse_id) File "serp_googlecs.py", line 32, in google_api_search res = service.cse().list(q=search_term, cx=cse_id, **kwargs).execute() File "/Users/h35528/opt/anaconda3/envs/py37/lib/python3.7/site-packages/googleapiclient/_helpers.py", line 134, in positional_wrapper return wrapped(*args, **kwargs) File "/Users/h35528/opt/anaconda3/envs/py37/lib/python3.7/site-packages/googleapiclient/http.py", line 901, in execute headers=self.headers, File "/Users/h35528/opt/anaconda3/envs/py37/lib/python3.7/site-packages/googleapiclient/http.py", line 204, in _retry_request raise exception File "/Users/h35528/opt/anaconda3/envs/py37/lib/python3.7/site-packages/googleapiclient/http.py", line 177, in _retry_request resp, content = http.request(uri, method, *args, **kwargs) File "/Users/h35528/opt/anaconda3/envs/py37/lib/python3.7/site-packages/httplib2/__init__.py", line 1994, in request cachekey, File "/Users/h35528/opt/anaconda3/envs/py37/lib/python3.7/site-packages/httplib2/__init__.py", line 1651, in _request conn, request_uri, method, body, headers File "/Users/h35528/opt/anaconda3/envs/py37/lib/python3.7/site-packages/httplib2/__init__.py", line 1557, in _conn_request conn.connect() File "/Users/h35528/opt/anaconda3/envs/py37/lib/python3.7/site-packages/httplib2/__init__.py", line 1324, in connect sock.connect((self.host, self.port)) socket.timeout: timed out

[DanielAviv]

I just got it, previously working code now does not pass the auth.

File "lib/python3.8/site-packages/httplib2/__init__.py", line 1324, in connect sock.connect((self.host, self.port)) socket.timeout: timed out

Any fix for this?

Edit: Uncommenting the line worked for me on Ubuntu 20

I also started experiencing this issue just a day ago, with no modification to the code:

    service = build('sheets', 'v4', credentials=get_creds())
  File "/home/me/anaconda3/lib/python3.7/site-packages/googleapiclient/_helpers.py", line 134, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/home/me/anaconda3/lib/python3.7/site-packages/googleapiclient/discovery.py", line 225, in build
    requested_url, discovery_http, cache_discovery, cache, developerKey
  File "/home/me/anaconda3/lib/python3.7/site-packages/googleapiclient/discovery.py", line 282, in _retrieve_discovery_doc
    resp, content = http.request(actual_url)
  File "/home/me/anaconda3/lib/python3.7/site-packages/httplib2/__init__.py", line 1991, in request
    cachekey,
  File "/home/me/anaconda3/lib/python3.7/site-packages/httplib2/__init__.py", line 1651, in _request
    conn, request_uri, method, body, headers
  File "/home/me/anaconda3/lib/python3.7/site-packages/httplib2/__init__.py", line 1557, in _conn_request
    conn.connect()
  File "/home/me/anaconda3/lib/python3.7/site-packages/httplib2/__init__.py", line 1324, in connect
    sock.connect((self.host, self.port))

• OS: Ubuntu 18.04.5 LTS
• Python: 3.7.6
• pip: 20.0.2
• googleapiclient: 1.8.0

Uncommenting the line precedence ::ffff:0:0/96 100 in /etc/gai.conf fixed the problem.

[SurferJeffAtGoogle]

@parthea If you're seeking a very challenging bug to investigate, try this one!

[parthea]

@nulld mentioned a workaround in #563 where he had to move the socket.setdefaulttimeout() function before import googleapiclient. If it solves the problem, then credit goes to @nulld ! Either way, we can use this data point to better understand whether there are multiple issues going on. Please can someone who is experiencing the issue try the workaround and report back?

[busunkim96]

Reassigning to @hkdevandla. Googlers see status in 171349628

[parthea]

Remove priority:p2 label based on internal status b/171349628