Google-api-python-client: discovery.build( ) raises ResourceWarning: unclosed <ssl.SSLSocket fd=13,

[ramonmedeiros]

Environment details

• OS:
  Linux hu3br 4.18.16-300.fc29.x86_64 #1 SMP Sat Oct 20 23:24:08 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
• Python version:
  Python 3.6.7

• pip version:
  pip 19.0.1 from /home/ramonm/git/bigtable-backups/.venv/lib/python3.6/site-packages/pip (python 3.6)

• google-api-python-client version:

Name: google-api-python-client
Version: 1.7.8
Summary: Google API Client Library for Python
Home-page: http://github.com/google/google-api-python-client/
Author: Google Inc.
Author-email: None
License: Apache 2.0
Location: /home/ramonm/git/bigtable-backups/.venv/lib/python3.6/site-packages
Requires: google-auth-httplib2, six, uritemplate, google-auth, httplib2
Required-by:

Steps to reproduce

Instantiate a dataflow, v1b3 API with googleapis.discovery

> /home/ramonm/git/bigtable-backups/.venv/lib/python3.6/site-packages/bigtable_backups/dataflow.py(117)__init__()
-> self.api_service = build(API_COMPONENT, API_VERSION, cache_discovery=False).projects().locations()
(Pdb) n
/home/ramonm/git/bigtable-backups/.venv/lib/python3.6/site-packages/google/auth/_default.py:66: UserWarning: Your application has authenticated using end user credentials from Google Cloud SDK. We recommend that most server applications use service accounts instead. If your application continues to use end user credentials from Cloud SDK, you might receive a "quota exceeded" or "API not enabled" error. For more information about service accounts, see https://cloud.google.com/docs/authentication/.
  warnings.warn(_CLOUD_SDK_CREDENTIALS_WARNING)
/home/ramonm/git/bigtable-backups/.venv/lib/python3.6/site-packages/googleapiclient/_helpers.py:130: ResourceWarning: unclosed <ssl.SSLSocket fd=13, family=AddressFamily.AF_INET, type=2049, proto=6, laddr=('10.43.33.41', 48878), raddr=('74.125.133.95', 443)>
  return wrapped(*args, **kwargs)
> /home/ramonm/git/bigtable-backups/.venv/lib/python3.6/site-packages/bigtable_backups/dataflow.py(120)__init__()

Making sure to follow these steps will guarantee the quickest resolution possible.

Thanks!

[berendo]

We're also seeing this while using other API endpoints (other than dataflow) when -Wall is in effect for python.

[busunkim96]

@berendo Could you elaborate on what -Wall is?

[berendo]

Could you elaborate on what -Wall is?

Sure, it’s running Python with python -Wall. See the -W option here.

[busunkim96]

@berendo Thank you!

This is straightforward to reproduce:

from googleapiclient import discovery

client = discovery.build("dataflow", "v1b3")

$ python3 -Wall make-client.py 
~/.local/lib/python3.6/site-packages/googleapiclient/_helpers.py:130: ResourceWarning: unclosed <ssl.SSLSocket fd=3, family=AddressFamily.AF_INET6, type=2049, proto=6, laddr=('2620:15c:10f:202:e603:43de:a156:cc2', 33710, 0, 0), raddr=('2607:f8b0:400a:809::200a', 443, 0, 0)>

[busunkim96]

The request to fetch the discovery doc results in the ResourceWarning.

from googleapiclient import discovery
import requests
import httplib2

# Doesn't raise socket resource warning
dataflow_discovery_doc = requests.get("https://dataflow.googleapis.com/$discovery/rest?version=v1b3").json()
client = discovery.build_from_document(dataflow_discovery_doc)

# Raises socket resource warning
client = discovery.build("dataflow", "v1b3")

# Function to retrieve public discovery doc
doc = discovery._retrieve_discovery_doc("https://www.googleapis.com/discovery/v1/apis", httplib2.Http(), cache_discovery=False)

httplib2 seems to leave sockets open when requests are successful.

import httplib2

http = httplib2.Http()
http.request("http://example.com")

I'm not sure if there's a good way to ask that the connection be closed in httplib2. I filed https://github.com/httplib2/httplib2/issues/148 to ask the maintainers.

From similar discussions on requests (https://github.com/psf/requests/issues/3912) it may be okay to let this be.

[wintermelons]

Since https://github.com/httplib2/httplib2/issues/148 is now closed, any plans to add .close() to discovery?

[busunkim96]

@wintermelons Thank you for the ping! It looks like httplib2 hasn't yet made a release with the change.

[wintermelons]

FYI: httplib2 has made a new release :slightly_smiling_face:

[imjuanleonard]

Bumping this up for the code sanity, any updates?

[mike9005]

This is causing a lot of disruption for us - it causes containers running Airflow tasks against GCP to consistently run out of available sockets.

[busunkim96]

Hello folks,

Apologies for leaving this unattended for so long.

f this is actively causing pain you can manually call the httplib2.http.close() method when you are done with the client.

from googleapiclient import discovery

client = discovery.build("drive", "v3")

client._http.http.close()

Ideally the library should clean up httplib2.http it creates. httplib2 instances could be shared across clients so I think it will require some care to get the logic correct.

[haizaar]

@ramonmedeiros Can you please elaborate why did you close this issue?

[haizaar]

@busunkim96 Thank you so much for fixing this! Looking forward for the next release!