Google-api-python-client: Broken Pipe exceptions raised after idle periods

[tbonfort]

I'm using long-lived client objects like so:

client = discovery.build(....)
task = client.get_something(....).execute(num_retries=5)
some_long_running_process(task)
client.do_another_thing().execute(num_retries=5)

And running into exceptions on the calls to do_another_thing():

Traceback (most recent call last):
  File "/opt/app/gcp_helper.py", line 168, in get_size
    instanceGroupManager=instance_group).execute(num_retries=5)
  File "/opt/venv/local/lib/python2.7/site-packages/oauth2client/util.py", line 137, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/opt/venv/local/lib/python2.7/site-packages/googleapiclient/http.py", line 755, in execute
    method=str(self.method), body=self.body, headers=self.headers)
  File "/opt/venv/local/lib/python2.7/site-packages/googleapiclient/http.py", line 93, in _retry_request
    resp, content = http.request(uri, method, *args, **kwargs)
  File "/opt/venv/local/lib/python2.7/site-packages/oauth2client/client.py", line 633, in new_request
    self._refresh(request_orig)
  File "/opt/venv/local/lib/python2.7/site-packages/oauth2client/contrib/gce.py", line 129, in _refresh
    META, headers={'Metadata-Flavor': 'Google'})
  File "/opt/venv/local/lib/python2.7/site-packages/httplib2/__init__.py", line 1609, in request
    (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
  File "/opt/venv/local/lib/python2.7/site-packages/httplib2/__init__.py", line 1351, in _request
    (response, content) = self._conn_request(conn, request_uri, method, body, headers)
  File "/opt/venv/local/lib/python2.7/site-packages/httplib2/__init__.py", line 1273, in _conn_request
    conn.request(method, request_uri, body, headers)
  File "/usr/lib/python2.7/httplib.py", line 1001, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib/python2.7/httplib.py", line 1035, in _send_request
    self.endheaders(body)
  File "/usr/lib/python2.7/httplib.py", line 997, in endheaders
    self._send_output(message_body)
  File "/usr/lib/python2.7/httplib.py", line 850, in _send_output
    self.send(msg)
  File "/usr/lib/python2.7/httplib.py", line 826, in send
    self.sock.sendall(data)
  File "/usr/lib/python2.7/socket.py", line 224, in meth
    return getattr(self._sock,name)(*args)
error: [Errno 32] Broken pipe"

I would expect the num_retries argument to handle these exceptions in https://github.com/google/google-api-python-client/blob/master/googleapiclient/http.py#L145, but for the time being I have to implement my own retry logic around all API calls to overcome this.

[theacodes]

It's likely that c6425a00e3c fixed this, but we haven't cut a new release. Can you try the HEAD version and see if you still see this issue?

[tbonfort]

Still happening on master:

Traceback (most recent call last):
  File "/opt/app/gcp_helper.py", line 168, in get_size
    instanceGroupManager=instance_group).execute(num_retries=5)
  File "/opt/venv/local/lib/python2.7/site-packages/oauth2client/util.py", line 135, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/opt/venv/local/lib/python2.7/site-packages/googleapiclient/http.py", line 820, in execute
    method=str(self.method), body=self.body, headers=self.headers)
  File "/opt/venv/local/lib/python2.7/site-packages/googleapiclient/http.py", line 147, in _retry_request
    resp, content = http.request(uri, method, *args, **kwargs)
  File "/opt/venv/local/lib/python2.7/site-packages/oauth2client/client.py", line 632, in new_request
    self._refresh(request_orig)
  File "/opt/venv/local/lib/python2.7/site-packages/oauth2client/contrib/gce.py", line 129, in _refresh
    META, headers={'Metadata-Flavor': 'Google'})
  File "/opt/venv/local/lib/python2.7/site-packages/httplib2/__init__.py", line 1609, in request
    (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
  File "/opt/venv/local/lib/python2.7/site-packages/httplib2/__init__.py", line 1351, in _request
    (response, content) = self._conn_request(conn, request_uri, method, body, headers)
  File "/opt/venv/local/lib/python2.7/site-packages/httplib2/__init__.py", line 1273, in _conn_request
    conn.request(method, request_uri, body, headers)
  File "/usr/lib/python2.7/httplib.py", line 1001, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib/python2.7/httplib.py", line 1035, in _send_request
    self.endheaders(body)
  File "/usr/lib/python2.7/httplib.py", line 997, in endheaders
    self._send_output(message_body)
  File "/usr/lib/python2.7/httplib.py", line 850, in _send_output
    self.send(msg)
  File "/usr/lib/python2.7/httplib.py", line 826, in send
    self.sock.sendall(data)
  File "/usr/lib/python2.7/socket.py", line 224, in meth
    return getattr(self._sock,name)(*args)
error: [Errno 32] Broken pipe"

[theacodes]

Is that a socket.error or an IO/OSError? Seems to be an IOError - we may just have to add some additional logic here. If you want to make a PR, I'll be happy to review it.

[nathanielmanistaatgoogle]

Fixed in 713b1de1078008c0353e00820e1.

[kapi90]

Hi, I'm using the Cloud-Vision API in python with similar conditions (a long-lived object and long idle sessions between requests)

I get the same error message a lot, therefore I wonder if this could be the same problem mentioned above, so pulling the latest commit should fix it.

I copy the related part from traceback here:

in detect_text responses = request.execute(num_retries=num_retries) File "/opt/python/run/venv/lib/python2.7/site-packages/oauth2client/util.py", line 135, in positional_wrapper return wrapped(_args, *_kwargs) File "/opt/python/run/venv/lib/python2.7/site-packages/googleapiclient/http.py", line 755, in execute method=str(self.method), body=self.body, headers=self.headers) File "/opt/python/run/venv/lib/python2.7/site-packages/googleapiclient/http.py", line 93, in _retry_request resp, content = http.request(uri, method, _args, *_kwargs) File "/opt/python/run/venv/lib/python2.7/site-packages/oauth2client/client.py", line 621, in new_request redirections, connection_type) File "/opt/python/run/venv/lib/python2.7/site-packages/httplib2/__init__.py", line 1609, in request (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey) File "/opt/python/run/venv/lib/python2.7/site-packages/httplib2/__init__.py", line 1351, in _request (response, content) = self._conn_request(conn, request_uri, method, body, headers) File "/opt/python/run/venv/lib/python2.7/site-packages/httplib2/__init__.py", line 1273, in _conn_request conn.request(method, request_uri, body, headers) File "/usr/lib64/python2.7/httplib.py", line 1001, in request self._send_request(method, url, body, headers) File "/usr/lib64/python2.7/httplib.py", line 1035, in _send_request self.endheaders(body) File "/usr/lib64/python2.7/httplib.py", line 997, in endheaders self._send_output(message_body) File "/usr/lib64/python2.7/httplib.py", line 850, in _send_output self.send(msg) File "/usr/lib64/python2.7/httplib.py", line 826, in send self.sock.sendall(data) File "/usr/lib64/python2.7/ssl.py", line 701, in sendall v = self.send(data[count:]) File "/usr/lib64/python2.7/ssl.py", line 667, in send v = self._sslobj.write(data) error: [Errno 32] Broken pipe

[theacodes]

Let us know if it fixes you.

@nathanielmanistaatgoogle should we cut another minor release?

[nathanielmanistaatgoogle]

@jonparrott Probably! Have you any time in the next few days?

[theacodes]

Maybe Friday. I'll make a bug for it.

[klday]

I'm having this issue using the google bigquery API. I'm pulling data from Azure SQL Server in batches and sending to bigquery. I have to use batches with some of the tables because of their size. Because of filtering, ordering and batching of some of the tables, the pulls from Azure take longer than pulls from other tables. I'm getting the broken pipe error on tables with longer pulls.

I've tried several different batch sizes with the same result.

Any advice would be greatly appreciated.

[theacodes]

@klday are you experiencing this with the latest release?

[klday]

Sorry, I left out some important information:

I'm using pandas to_gbq() to import the data to BigQuery.

Here is my terminal output for the traceback:
Streaming Insert is 100.0% Complete
Traceback (most recent call last):
File "datascience/app_sql_to_bq/main.py", line 146, in
logging_level = args.logging_level)
File "datascience/app_sql_to_bq/main.py", line 83, in main
table_time = bq.sql_data_to_bq_table(sql_resource,bq_resource,sql_table_name,bq_table_id,sql_schema_columns,sql_query,sql_columns,sql_filtered_size_query,step_size=step_size)
File "./datascience/utils/bigqueryUtils.py", line 630, in sql_data_to_bq_table
clean_df.to_gbq(bq_dataset_table_id, bq_project_id, if_exists='append', private_key=key)
File "/opt/conda/lib/python3.5/site-packages/pandas/core/frame.py", line 899, in to_gbq
if_exists=if_exists, private_key=private_key)
File "/opt/conda/lib/python3.5/site-packages/pandas/io/gbq.py", line 724, in to_gbq
connector.load_data(dataframe, dataset_id, table_id, chunksize)
File "/opt/conda/lib/python3.5/site-packages/pandas/io/gbq.py", line 453, in load_data
body=body).execute()
File "/opt/conda/lib/python3.5/site-packages/oauth2client/util.py", line 137, in positional_wrapper
return wrapped(_args, *_kwargs)
File "/opt/conda/lib/python3.5/site-packages/googleapiclient/http.py", line 833, in execute
method=str(self.method), body=self.body, headers=self.headers)
File "/opt/conda/lib/python3.5/site-packages/googleapiclient/http.py", line 173, in _retry_request
raise exception
File "/opt/conda/lib/python3.5/site-packages/googleapiclient/http.py", line 160, in _retry_request
resp, content = http.request(uri, method, _args, *_kwargs)
File "/opt/conda/lib/python3.5/site-packages/oauth2client/transport.py", line 169, in new_request
redirections, connection_type)
File "/opt/conda/lib/python3.5/site-packages/httplib2/__init__.py", line 1314, in request
(response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
File "/opt/conda/lib/python3.5/site-packages/httplib2/__init__.py", line 1064, in _request
(response, content) = self._conn_request(conn, request_uri, method, body, headers)
File "/opt/conda/lib/python3.5/site-packages/httplib2/__init__.py", line 988, in _conn_request
conn.request(method, request_uri, body, headers)
File "/opt/conda/lib/python3.5/http/client.py", line 1106, in request
self._send_request(method, url, body, headers)
File "/opt/conda/lib/python3.5/http/client.py", line 1151, in _send_request
self.endheaders(body)
File "/opt/conda/lib/python3.5/http/client.py", line 1102, in endheaders
self._send_output(message_body)
File "/opt/conda/lib/python3.5/http/client.py", line 936, in _send_output
self.send(message_body)
File "/opt/conda/lib/python3.5/http/client.py", line 908, in send
self.sock.sendall(data)
File "/opt/conda/lib/python3.5/ssl.py", line 891, in sendall
v = self.send(data[count:])
File "/opt/conda/lib/python3.5/ssl.py", line 861, in send
return self._sslobj.write(data)
File "/opt/conda/lib/python3.5/ssl.py", line 586, in write
return self._sslobj.write(data)
BrokenPipeError: [Errno 32] Broken pipe

[klday]

The latest release of ...?
I just installed the api tools so I believe I am using the latest release of those.

[theacodes]

This library, 1.5.2?

[klday]

Honestly, I have no idea. I just downloaded the google api tools for the cloud platform. Those are the most recent tools. And, I installed those using the command line. I'm not sure what library you're referring to, but I'm also relatively new to programming.

[theacodes]

@klday no worries. Can you open up python and run the following and give me the output?

>>> import googleapiclient
>>> googleapiclient.__version__
'1.5.1'
>>> import oauth2client
>>> print oauth2client.__version__
2.2.0

[klday]

Yes, thank you. The output is:

import googleapiclient
googleapiclient.version
'1.5.2'
import oauth2client
Traceback (most recent call last):
File "", line 1, in
ImportError: No module named 'oauthclient'

I am working in a docker container. I have been creating a google cloud resource to work with the bigquery api tools using a key (pretty sure I'm explaining that correctly).

[theacodes]

well, in that case I don't know. It could be an underlying issue with httplib2.

In which case you can try using httplib2shim to see if it fixes it.

[klday]

Ok, thank you. I'll give that a shot.

[LordAro]

FWIW, I just got this with 1.5.3 (From FreeBSD 10.1, python 3.5.1 if that matters). I guess there's no progress with it?

Have updated to 1.5.4 (w/ oauth2client 4.0.0) now, will see what happens

[theacodes]

@LordAro this should definitely be occurring less often. If you can provide a full stacktrace that will be helpful.

[LordAro]

Sure

[2016/10/23 18:31:15] (I:googleapiclient.discovery) URL being requested: GET https://www.googleapis.com/youtube/v3/videos?id=5_LxyhCJp
sM&part=snippet%2CcontentDetails%2Cstatistics&alt=json&key=<snip>
[2016/10/23 18:31:15] (E:asyncio) Unhandled exception in event task
future: <Task finished coro=<LinkInfo.scan_privmsg() done, defined at /home/urybot/.pyenv/versions/3.5.1/lib/python3.5/asyncio/corouti
nes.py:204> exception=BrokenPipeError(32, 'Broken pipe')>
Traceback (most recent call last):
  File "/home/urybot/.pyenv/versions/3.5.1/lib/python3.5/asyncio/tasks.py", line 239, in _step
    result = coro.send(None)
  File "/home/urybot/.pyenv/versions/3.5.1/lib/python3.5/asyncio/coroutines.py", line 206, in coro
    res = func(*args, **kw)
  File "/home/urybot/csbot/csbot/plugins/linkinfo.py", line 155, in scan_privmsg
    result = self.get_link_info(part)
  File "/home/urybot/csbot/csbot/plugins/linkinfo.py", line 189, in get_link_info
    result = h.handler(url, match)
  File "/home/urybot/csbot/csbot/plugins/youtube.py", line 153, in page_handler
    response = self._yt(url)
  File "/home/urybot/csbot/csbot/plugins/youtube.py", line 86, in _yt
    json = self.get_video_json(vid_id)
  File "/home/urybot/csbot/csbot/plugins/youtube.py", line 73, in get_video_json
    response = self.client.videos().list(id=id, part='snippet,contentDetails,statistics').execute(http=self.http)
  File "/home/urybot/.pyenv/versions/3.5.1/lib/python3.5/site-packages/oauth2client/util.py", line 137, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/home/urybot/.pyenv/versions/3.5.1/lib/python3.5/site-packages/googleapiclient/http.py", line 833, in execute
    method=str(self.method), body=self.body, headers=self.headers)
  File "/home/urybot/.pyenv/versions/3.5.1/lib/python3.5/site-packages/googleapiclient/http.py", line 173, in _retry_request
    raise exception
  File "/home/urybot/.pyenv/versions/3.5.1/lib/python3.5/site-packages/googleapiclient/http.py", line 160, in _retry_request
    resp, content = http.request(uri, method, *args, **kwargs)
  File "/home/urybot/.pyenv/versions/3.5.1/lib/python3.5/site-packages/httplib2/__init__.py", line 1314, in request
    (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
  File "/home/urybot/.pyenv/versions/3.5.1/lib/python3.5/site-packages/httplib2/__init__.py", line 1064, in _request
    (response, content) = self._conn_request(conn, request_uri, method, body, headers)
  File "/home/urybot/.pyenv/versions/3.5.1/lib/python3.5/site-packages/httplib2/__init__.py", line 988, in _conn_request
    conn.request(method, request_uri, body, headers)
  File "/home/urybot/.pyenv/versions/3.5.1/lib/python3.5/http/client.py", line 1083, in request
    self._send_request(method, url, body, headers)
  File "/home/urybot/.pyenv/versions/3.5.1/lib/python3.5/http/client.py", line 1128, in _send_request
    self.endheaders(body)
  File "/home/urybot/.pyenv/versions/3.5.1/lib/python3.5/http/client.py", line 1079, in endheaders
    self._send_output(message_body)
  File "/home/urybot/.pyenv/versions/3.5.1/lib/python3.5/http/client.py", line 911, in _send_output
    self.send(msg)
  File "/home/urybot/.pyenv/versions/3.5.1/lib/python3.5/http/client.py", line 885, in send
    self.sock.sendall(data)
  File "/home/urybot/.pyenv/versions/3.5.1/lib/python3.5/ssl.py", line 886, in sendall
    v = self.send(data[count:])
  File "/home/urybot/.pyenv/versions/3.5.1/lib/python3.5/ssl.py", line 856, in send
    return self._sslobj.write(data)
  File "/home/urybot/.pyenv/versions/3.5.1/lib/python3.5/ssl.py", line 581, in write
    return self._sslobj.write(data)
BrokenPipeError: [Errno 32] Broken pipe

[Bobspadger]

I'm also getting the broken pipe on BatchHttpRequest()

I can get through thousands of calls to the shopping api, but if we have a quiet period, then I get the broken pipe.

[theacodes]

This issue is most likely due to httplib2's connection pooling logic. I'd suggest trying httplib2shim.

[Bobspadger]

@jonparrott currently testing this and it seems to have fixed the problem

Is it worth making this the required method for connecting ?

[theacodes]

@Bobspadger probably not, but we're soon replacing oauth2client in this library with google-auth, which will allow us to use urllib3 and requests without needing httplib2shim.

[Bobspadger]

@jonparrott excellent, requests makes life so much easier!

[dhyon]

@jonparrott Any timeline on this replacement endeavor? It seems like a great idea and this bug is truly annoying.

[theacodes]

@jntkflow early next year at the earliest. I recommend using httplib2shim in the meantime.

[nharpalani]

Hey @jonparrott, any update on timeline here? It looks like the new google-auth python library now exists?

[theacodes]

@nharpalani it's looking unlikely that we'll be able to replace httplib2 in this library without massive breaking changes. We're working on a similar effort over on the google-cloud-python package and there's a small chance that work might be transferrable here, but no promises.

Closing for now.

[nharpalani]

Thanks for the update, Jon!

[ryantuck]

Is there any workaround for this? I'm guessing redefining client right before each new API call might do it?

[janneh]

I'm experiencing this issue as well...

Using:

Python 3.6.7

>>> import googleapiclient
>>> googleapiclient.__version__
'1.7.8'

Stack trace:

File "/var/task/pubsub_publisher.py", line 42, in handler
  pubsub.projects().topics().publish(topic=_GOOGLE_PUBSUB_TOPIC_NAME, body=body).execute() 
File "/var/task/googleapiclient/_helpers.py", line 130, in positional_wrapper
  return wrapped(*args, **kwargs)
File "/var/task/googleapiclient/http.py", line 846, in execute
  method=str(self.method), body=self.body, headers=self.headers)
File "/var/task/googleapiclient/http.py", line 183,in _retry_request
  raise exception
File "/var/task/googleapiclient/http.py", line 164, in _retry_request
  resp, content = http.request(uri, method, *args, **kwargs)
File "/var/task/google_auth_httplib2.py", line 198, in request
  uri, method, body=body, headers=request_headers, **kwargs)
File "/var/task/httplib2/__init__.py", line 1924, in request
  cachekey,
File "/var/task/httplib2/__init__.py", line 1595, in _request
  conn, request_uri, method, body, headers
File "/var/task/httplib2/__init__.py", line 1502, in _conn_request
  conn.request(method, request_uri, body, headers)
File "/var/lang/lib/python3.6/http/client.py", line 1239, in request
  self._send_request(method, url, body, headers, encode_chunked)
File "/var/lang/lib/python3.6/http/client.py", line 1285, in _send_request
  self.endheaders(body, encode_chunked=encode_chunked)
File "/var/lang/lib/python3.6/http/client.py", line 1234, in endheaders
  self._send_output(message_body, encode_chunked=encode_chunked)
File "/var/lang/lib/python3.6/http/client.py", line 1065, in _send_output
  self.send(chunk)
File "/var/lang/lib/python3.6/http/client.py", line 986, in send
  self.sock.sendall(data)
File "/var/lang/lib/python3.6/ssl.py", line 975, in sendall
  v = self.send(byte_view[count:])
File "/var/lang/lib/python3.6/ssl.py", line 944, in send
  return self._sslobj.write(data)
File "/var/lang/lib/python3.6/ssl.py", line 642, in write
  return self._sslobj.write(data)
BrokenPipeError: [Errno 32] Broken pipe

It's something that happens now and then, not super common. Constantly recurring though.

As an example in a sample of 7,741,334 publishes I have 6 of those errors.

[SkYNewZ]

Always the issue

[Bobspadger]

This broken pipe issue has occurred again for me now I'm moving to googles v2.1 content API and i'm using a service account (no longer httplib2shim as I've followed the instructions from your example and using all the google libraries.

Is the answer to go back to using httplib2shim.

I too have a long life client - does this need rebuilding before every run ?

[SkYNewZ]

?

[ianseyer]

This is now happening to me via a Cloud Run execution that happens every 30 minutes. What's the status of this fix?

Why was the issue closed when there is no solution or fix in place?

[HaririAli]

Hi,

I'm having the same issue via Cloud Run also. I randomly get "Broken Pipe" or "connection reset by peer" errors.
I am using the latest version of the Google API client so it seems this has not been fixed yet.

[HaririAli]

OK this is just a FYI in case someone is having the same problem. I am getting the Broken Pipe error while using Firebase messaging via Cloud Run.
As @theacodes mentioned, this has been fixed in google-cloud-python, which includes APIs for many cloud services, but it doesn't include an API for Firebase messaging.
It seems that this issue will not be fixed soon for Firebase messaging, so try PyFCM. It is based on requests module, so I assume it should not have this issue.