Google-api-python-client: Call to `googleapiclient.discovery.build` hangs for several minutes

[nolanmar511]

We've (the Cloud Profiler team) been seeing some prober and E2E test failures for our python agent (repo at https://github.com/GoogleCloudPlatform/cloud-profiler-python) where calls to googleapiclient.discovery.build hangs for until our benchmark application finishes (this is 10 minutes), and are wondering if this is expected.

Environment details

• OS type and version: Linux (Using GCP image project ubuntu-os-cloud and image family ubuntu-1804-lts)
• Python version: Python 3.5
• pip version: pip --version
• google-api-python-client version: 1.12.3 (will confirm this)

Steps to reproduce

This is consistently happening in our E2E tests and probers which run on GCE VMs in us-west2-a (we also run probers in asia-east1-c and europe-west1-b, which aren't experiencing this) when we call:

 googleapiclient.discovery.build(
        'cloudprofiler',
        'v2',
        http=http,
        cache_discovery=False,
        requestBuilder=ProfilerHttpRequest,
        discoveryServiceUrl= googleapiclient.discovery.DISCOVERY_URI)

[humbletyler]

Hello; we are experiencing a similar hang on googleapiclient.discovery.build suddenly; it started yesterday around 7PM pacific time.

We've had some luck getting around the error by upgrading our google-api-python-client and oauth2client packages, but are still running into issues overall...

[humbletyler]

It turns out the luck reinstalling the google-api-python-client and oauth2client packages were totally random, and the Zone issue you pointed out was the case (all of our workers were in us-west2-a)

[busunkim96]

I would recommend downloading the discovery json and using build_from_document. https://github.com/googleapis/google-api-python-client/blob/30eff9d8276919b8c4e50df2d3b1982594423692/googleapiclient/discovery.py#L394 That way you won't be dependent on the discovery doc being up.

This quarter we'll be restructuring the library to have a copy of all the discovery docs locally so the workaround will be temporary.

[busunkim96]

If you do have a support contract, please make sure to file a case. https://cloud.google.com/support/

Some other folks have reached out with similar reports. The issue seems to consistently be us-west.

---

Here's more detail on how to use build_from_json:

Find your API in https://www.googleapis.com/discovery/v1/apis. and open the discoveryRestUrl. That will be the current discovery document for the API.

Abusive Experience Report v1 is https://abusiveexperiencereport.googleapis.com/$discovery/rest?version=v1 for example. Download that JSON locally.

{
  "kind": "discovery#directoryList",
  "discoveryVersion": "v1",
  "items": [
    {
      "kind": "discovery#directoryItem",
      "id": "abusiveexperiencereport:v1",
      "name": "abusiveexperiencereport",
      "version": "v1",
      "title": "Abusive Experience Report API",
      "description": "Views Abusive Experience Report data, and gets a list of sites that have a significant number of abusive experiences.",
      "discoveryRestUrl": "https://abusiveexperiencereport.googleapis.com/$discovery/rest?version=v1",
      "icons": {
        "x16": "https://www.gstatic.com/images/branding/product/1x/googleg_16dp.png",
        "x32": "https://www.gstatic.com/images/branding/product/1x/googleg_32dp.png"
      },
      "documentationLink": "https://developers.google.com/abusive-experience-report/",
      "preferred": true
    },

Read the file and pass it to build_from_document.

from googleapiclient import discovery

with open("abusiveexperienncereport_v1.json") as f:
    document = f.read()
service = build_from_document(document)

NOTE: Discovery documents do change periodically so you should re-fetch the doc at least once a week. You can check if it has changed by referencing the revision field.

[rvandegrift]

We're affected by this issue and have opened a support case. After digging into the code, we discovered that there's a discovery doc cache. But it depends on specific old versions of oauth2client - see [1] for the details. Installing
oauth2client==3.0.0 greatly reduced the impact in my test cases.

I will open another issue for the caching depending on an old version of oauth2client.

Edited to add:

• I can reproduce in us-central1, so it's not limited to us-west.
• I've only seen the issue on the compute engine api.

[1] - https://github.com/googleapis/google-api-python-client/blob/30eff9d8276919b8c4e50df2d3b1982594423692/googleapiclient/discovery_cache/file_cache.py#L32

[tswast]

Does this error happen if you use discoveryServiceUrl=googleapiclient.discovery.V2_DISCOVERY_URI?

The www.googleapis.com endpoint doesn't have the same SLA as the actual oneplatform service, so if supported by the service, the V2_DISCOVERY_URI will be more reliable.

[rvandegrift]

@tswast interesting! I asked support if the compute engine api had a rate limit adjusted last night, but haven't received a response.

I tried that option, but it doesn't work for me:

>>> from googleapiclient import discovery
>>> discovery.build('compute', 'v1')
<googleapiclient.discovery.Resource object at 0x7ffb6cfa74a8>
>>> discovery.build('compute', 'v1', discoveryServiceUrl=discovery.V2_DISCOVERY_URI)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/ross/.local/lib/python3.7/site-packages/googleapiclient/_helpers.py", line 134, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/home/ross/.local/lib/python3.7/site-packages/googleapiclient/discovery.py", line 303, in build
    raise UnknownApiNameOrVersion("name: %s  version: %s" % (serviceName, version))
googleapiclient.errors.UnknownApiNameOrVersion: name: compute  version: v1

[busunkim96]

It appears this is related to a config rollout in the Discovery Service.

Googlers, see incident 23306.

A rollout is in progress and is expected to complete within the next 7 hours.

[busunkim96]

Incident is now resolved.