Google-cloud-python: Core: add retry on 503 errors.

[bmenasha]

The google-api-python-client and gcloud CLI both retry on 503 SERVICE UNAVAILABLE errors yet the current version of google-cloud-python doesn't.

https://github.com/google/google-api-python-client/blob/master/googleapiclient/http.py#L94

We seem to have coded a retry on 500 and 429 , but not 503.

if_transient_error = if_exception_type((
    exceptions.InternalServerError,
    exceptions.TooManyRequests))

Please consider adding 503 as retryable.

I'm running 2.7.11 but I expect same behavior with 3

503 The service is currently unavailable.
  File "./create_policies.py", line 48, in <module>
    <class 'google.api_core.exceptions.ServiceUnavailable'>
print('Created ' +  future.result().name)
  File "/home/bmenasha/.pyenv/versions/2.7.11-clutered/lib/python2.7/site-packages/concurrent/futures/_base.py", line 462, in result
    return self.__get_result()
  File "/home/bmenasha/.pyenv/versions/2.7.11-clutered/lib/python2.7/site-packages/concurrent/futures/thread.py", line 63, in run
    result = self.fn(*self.args, **self.kwargs)
  File "./create_policies.py", line 11, in create_policy
    policy = alert_client.create_alert_policy(project, policy)
  File "/home/bmenasha/.pyenv/versions/2.7.11-clutered/lib/python2.7/site-packages/google/cloud/monitoring_v3/gapic/alert_policy_service_client.py", line 399, in create_alert_policy
    request, retry=retry, timeout=timeout, metadata=metadata)
  File "/home/bmenasha/.pyenv/versions/2.7.11-clutered/lib/python2.7/site-packages/google/api_core/gapic_v1/method.py", line 139, in __call__
    return wrapped_func(*args, **kwargs)
  File "/home/bmenasha/.pyenv/versions/2.7.11-clutered/lib/python2.7/site-packages/google/api_core/retry.py", line 263, in retry_wrapped_func
    on_error=on_error,
  File "/home/bmenasha/.pyenv/versions/2.7.11-clutered/lib/python2.7/site-packages/google/api_core/retry.py", line 178, in retry_target
    return target()
  File "/home/bmenasha/.pyenv/versions/2.7.11-clutered/lib/python2.7/site-packages/google/api_core/timeout.py", line 206, in func_with_timeout
    return func(*args, **kwargs)
  File "/home/bmenasha/.pyenv/versions/2.7.11-clutered/lib/python2.7/site-packages/google/api_core/grpc_helpers.py", line 56, in error_remapped_callable
    six.raise_from(exceptions.from_grpc_error(exc), exc)
  File "/home/bmenasha/.pyenv/versions/2.7.11-clutered/lib/python2.7/site-packages/six.py", line 737, in raise_from
    raise value
google.api_core.exceptions.ServiceUnavailable: 503 The service is currently unavailable.

Metadata-Version: 2.0
Name: google-cloud
Version: 0.22.0
Summary: API Client library for Google Cloud
Home-page: https://github.com/GoogleCloudPlatform/google-cloud-python
Author: Google Cloud Platform
Author-email: [email protected]
License: Apache 2.0
Location: /home/bmenasha/.pyenv/versions/2.7.11-clutered/lib/python2.7/site-packages
Requires: google-cloud-resource-manager, google-cloud-monitoring, google-cloud-happybase, google-cloud-language, google-cloud-storage, google-cloud-translate, google-cloud-error-reporting, google-cloud-vision, google-cloud-runtimeconfig, google-cloud-core, google-cloud-bigtable, google-cloud-bigquery, google-cloud-pubsub, google-cloud-datastore, google-cloud-logging, google-cloud-dns

[tseaver]

@bmenasha Thanks for the report! @theacodes retrying 503 responses seems reasonable to me.

One quick note: the google-cloud umbrella package is deprecated, and will stop including any of the API-specific packages as dependencies in its next release (about three weeks from now). After that point, you will need to install the API-specific packages directly.

[xmedeko]

A storage copy operation fenced by Retry just failed for me by 503, too. Also, the polling.RETRY_PREDICATE contains BadGateway HTTP 502 exception.

Please, fix this simple issue, add ServiceUnavailable and BadGateway to if_transient_error and uniry it with polling.RETRY_PREDICATE.

Ugly hack until the issue is solved:

orig_if_transient_error = retry.if_transient_error
retry.if_transient_error = lambda ex: orig_if_transient_error(ex) \
                                      or isinstance(ex, (exceptions.ServiceUnavailable, exceptions.BadGateway))
retry.Retry.__init__ = functools.partialmethod(retry.Retry.__init__, predicate = retry.if_transient_error)

[xmedeko]

@tseaver Thanks for the fix. What about BadGateway HTTP 502 exception? It's contained in the polling.RETRY_PREDICATE too.