Google-api-python-client: Google Drive Batch Request Error: socket.timeout: The read operation timed out

[hsyyid]

Possibly related to #563

Occasionally when executing a batch request using the Python client library callingdrive.files().copy the operation fails with the error socket.timeout: The read operation timed out

Environment details

Running Python 3.7 on Google Cloud Functions environment

Relevant Code

def copy_files(access_token, file_ids, folder_ids, labels):
    def copy_callback(request_id, response, exception):
        if exception:
            print(exception)
        else:
            print(response)

    creds = AccessTokenCredentials(access_token, 'my-user-agent/1.0')
    drive_service = build('drive', 'v3', credentials=creds)

    move_files = drive_service.new_batch_http_request(callback=copy_callback)

    for label, id in zip(labels, file_ids):
        move_files.add(
            drive_service.files().copy(fileId=id, body={"parents": [folder_ids[label]]})
        )

    move_files.execute()

Error

Traceback (most recent call last):
  File "/env/lib/python3.7/site-packages/google/cloud/functions/worker.py", line 313, in run_http_function
    result = _function_handler.invoke_user_function(flask.request)
  File "/env/lib/python3.7/site-packages/google/cloud/functions/worker.py", line 215, in invoke_user_function
    return call_user_function(request_or_event)
  File "/env/lib/python3.7/site-packages/google/cloud/functions/worker.py", line 208, in call_user_function
    return self._user_function(request_or_event)
  File "/user_code/main.py", line 19, in sort
    request_json['labels'])
  File "/user_code/drive.py", line 51, in copy_files
    move_files.execute()
  File "/env/lib/python3.7/site-packages/google_api_python_client-1.7.8-py3.7.egg/googleapiclient/_helpers.py", line 130, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/env/lib/python3.7/site-packages/google_api_python_client-1.7.8-py3.7.egg/googleapiclient/http.py", line 1450, in execute
    self._execute(http, self._order, self._requests)
  File "/env/lib/python3.7/site-packages/google_api_python_client-1.7.8-py3.7.egg/googleapiclient/http.py", line 1382, in _execute
    headers=headers)
  File "/env/lib/python3.7/site-packages/oauth2client/transport.py", line 175, in new_request
    redirections, connection_type)
  File "/env/lib/python3.7/site-packages/oauth2client/transport.py", line 282, in request
    connection_type=connection_type)
  File "/env/lib/python3.7/site-packages/httplib2/__init__.py", line 1926, in request
    cachekey,
  File "/env/lib/python3.7/site-packages/httplib2/__init__.py", line 1595, in _request
    conn, request_uri, method, body, headers
  File "/env/lib/python3.7/site-packages/httplib2/__init__.py", line 1533, in _conn_request
    response = conn.getresponse()
  File "/opt/python3.7/lib/python3.7/http/client.py", line 1321, in getresponse
    response.begin()
  File "/opt/python3.7/lib/python3.7/http/client.py", line 296, in begin
    version, status, reason = self._read_status()
  File "/opt/python3.7/lib/python3.7/http/client.py", line 257, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/opt/python3.7/lib/python3.7/socket.py", line 589, in readinto
    return self._sock.recv_into(b)
  File "/opt/python3.7/lib/python3.7/ssl.py", line 1052, in recv_into
    return self.read(nbytes, buffer)
  File "/opt/python3.7/lib/python3.7/ssl.py", line 911, in read
    return self._sslobj.read(len, buffer)
socket.timeout: The read operation timed out

Thanks!

[busunkim96]

Hi! Thanks for the report. I've seen issues in other client libraries saying that timeouts can be a little weird in the Cloud Functions environment. Would you mind showing us what's in your requirements.txt?

[hsyyid]

@busunkim96

# Function dependencies, for example:
# package>=version
sklearn>=0.0
nltk>=3.4
oauth2client>=0.0
pandas>=0.23.0

[ktbernoulli]

Hiya, I have been seeing this error when I run reports for Google Analytics, a lot during 11am to 3pm PDT yesterday & just today again so far at 3am PDT. Is there a solution to this so far? I also updated to the 1.7.9 version on all forms.

[busunkim96]

I haven't had the chance to dig into this further, unfortunately.

How long are your functions running for, on average? Are you close to the 9 minute max, or does this happen earlier?

If each request invocation takes a while, it may be worth moving the client instantiation out into the global scope. There are other networking best practices for cloud functions here.

drive_service = build('drive', 'v3', credentials=creds)

def copy_files(access_token, file_ids, folder_ids, labels):
    def copy_callback(request_id, response, exception):
        if exception:
            print(exception)
        else:
            print(response)

    creds = AccessTokenCredentials(access_token, 'my-user-agent/1.0')

    move_files = drive_service.new_batch_http_request(callback=copy_callback)

    for label, id in zip(labels, file_ids):
        move_files.add(
            drive_service.files().copy(fileId=id, body={"parents": [folder_ids[label]]})
        )

    move_files.execute()

[busunkim96]

There are a few other support channels that might be worth reaching out to:

https://cloud.google.com/functions/docs/getting-support

[ktbernoulli]

Whoop, I apologize; I am seeing this error not on Google's Cloud platform, but running locally.
Intermittently works, but my function does not run for that long, a minute at most.
Do you recommend increasing the timeout for now?

[busunkim96]

@ktbernoulli Are you in Cloud Functions or in another environment?

[ktbernoulli]

I am in my own local environment; Python 3.6, package version 1.7.9 ):

[ktbernoulli]

Hello, after some more investigation, it might have been a latency issue with the GA account I am creating reports from; I am not sure why it would cause a read operation issue though ):
Thank you for your responses so far ~

[ilpersi]

I am getting the very same error when using Batch Request containing requests to the drive_service.files().copy() endpoint.

After different tests, I've noticed that this depends on the size of the batch: if I run small batches (e.g., five calls at time), I don't get the issue.

For the moment, my work-around is to have a wrapper class around the BatchHttpRequest class that takes care of running the requests in small chunks.

[hsyyid]

@ilpersi doesn't that defeat the point of a batch request? If the size of the request is related it should be handled on Google's side, or they should list any limitations to the size of batch requests in the docs

[ilpersi]

@hsyyid I agree with you and in the I came to the same conclusion: my current program is not using batch requests anymore.

The size hint is a nice one as it makes it easier to reproduce this issue.

[busunkim96]

@ilpersi Are you also experiencing this in the Cloud Functions environment?

[ilpersi]

@busunkim96 No, I am running the code locally on my Windows laptop.

I saw that this very same issue is also happening, much less frequently, with normal calls (not batch) to the drive.files.copy end point.

[amoussoubaruch]

Hi everyone!

Have you have any solution for this issue??

Thanks in advanced

[busunkim96]

Thank you @AlJohri for linking that issue! :)

The default timeout 60 seconds, which is probably too short for many batch requests. It makes sense that the behavior would go away when batches were made smaller (smaller batches -> faster requests).

https://github.com/googleapis/google-api-python-client/blob/f0b025b95b2fb5fd0369d7233614e35b9c86af30/googleapiclient/http.py#L1777-L1792

import socket
from googleapiclient import discovery

socket.setdefaulttimeout(600)  # set timeout to 10 minutes
client = discovery.build("drive", "v3")

I am closing this issue now, but please do open a new issue if you are still experiencing problems.