Google-cloud-python: PubSub Client 0.29.1: list_subscriptions has transient failures

[anorth2]

1. Specify the API at the beginning of the title (for example, "BigQuery: ...")
   General, Core, and Other are also allowed as types
2. OS type and version
   Mac High Sierra and Alpine based Docker Image

3. Python version and virtual environment information python --version
   Python 3.6.3

4. google-cloud-python version pip show google-cloud, pip show google-<service> or pip freeze
   google-cloud-pubsub==0.29.1

5. Stacktrace if available

INFO:2017-12-01 09:59:45,134:Retrieving existing topics and subscriptions for project oem-services-dev
Traceback (most recent call last):
  File "/usr/local/Cellar/python3/3.6.3/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/google/api_core/grpc_helpers.py", line 51, in error_remapped_callable
    return callable_(*args, **kwargs)
  File "/Users/alexander.north/Library/Python/3.6/lib/python/site-packages/grpc/_channel.py", line 492, in __call__
    return _end_unary_response_blocking(state, call, False, deadline)
  File "/Users/alexander.north/Library/Python/3.6/lib/python/site-packages/grpc/_channel.py", line 440, in _end_unary_response_blocking
    raise _Rendezvous(state, None, None, deadline)
grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with (StatusCode.INVALID_ARGUMENT, Invalid resource name given (name=projects/oem-services-dev/topics/us.gcr.io%2Foem-services-dev). Refer to https://cloud.google.com/pubsub/overview#names for more information.)>






Traceback (most recent call last):
  File "pubsub_check.py", line 183, in <module>
    create(args.project, topic=args.topic, subscription=args.subscription, mapping=args.mapping)
  File "pubsub_check.py", line 131, in create
    existing_topics, existing_subscriptions = get_existing(project)
  File "pubsub_check.py", line 74, in get_existing
    existing_subscriptions.extend([subscription.name for subscription in list_subscriptions_in_topic(e_topic)])
  File "pubsub_check.py", line 74, in <listcomp>
    existing_subscriptions.extend([subscription.name for subscription in list_subscriptions_in_topic(e_topic)])
  File "/usr/local/Cellar/python3/3.6.3/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/google/api_core/page_iterator.py", line 186, in _items_iter
    for page in self._page_iter(increment=False):
  File "/usr/local/Cellar/python3/3.6.3/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/google/api_core/page_iterator.py", line 217, in _page_iter
    page = self._next_page()
  File "/usr/local/Cellar/python3/3.6.3/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/google/api_core/page_iterator.py", line 499, in _next_page
    response = self._method(self._request)
  File "/usr/local/Cellar/python3/3.6.3/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/google/api_core/gapic_v1/method.py", line 134, in __call__
    return wrapped_func(*args, **kwargs)
  File "/usr/local/Cellar/python3/3.6.3/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/google/api_core/retry.py", line 260, in retry_wrapped_func
    on_error=on_error,
  File "/usr/local/Cellar/python3/3.6.3/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/google/api_core/retry.py", line 177, in retry_target
    return target()
  File "/usr/local/Cellar/python3/3.6.3/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/google/api_core/timeout.py", line 206, in func_with_timeout
    return func(*args, **kwargs)
  File "/usr/local/Cellar/python3/3.6.3/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/google/api_core/grpc_helpers.py", line 53, in error_remapped_callable
    six.raise_from(exceptions.from_grpc_error(exc), exc)
  File "<string>", line 3, in raise_from
google.api_core.exceptions.InvalidArgument: 400 Invalid resource name given (name=projects/oem-services-dev/topics/us.gcr.io%2Foem-services-dev). Refer to https://cloud.google.com/pubsub/overview#names for more information.

This topic appeared to be autocreated by something in gcloud, but we also see the exception for other topics which do for sure follow pubsub naming requirements.
i.e.

google.api_core.exceptions.InvalidArgument: 400 Invalid resource name given (name=projects/oem-services-dev/topics/ctp-test-topic). Refer to https://cloud.google.com/pubsub/overview#names for more information.

1. Steps to reproduce
   Execute the provided code and hope it fails. Running the exact same command twice resulted in one failing and one passing.

Also note this is still affected by: https://github.com/GoogleCloudPlatform/google-cloud-python/issues/4426

Even when this code works, it still returns all subscriptions rather than just for that topic.

1. Code example

def list_topics(project):
    """Lists all Pub/Sub topics in the given project."""
    logging.debug(f"Retrieving topics for project: {project}")
    project_path = publisher.project_path(project)
    return publisher.list_topics(project_path)

def list_subscriptions_in_topic(topic_path):
    """Lists all subscriptions for a given topic."""
    logging.debug(f"Retrieving subscriptions for topic: {topic_path}")
    return subscriber.list_subscriptions(topic_path)


def get_existing(project):
    # retrieve existing topics and subscriptions to compare against
    # this will return the path for each
    # this list comprehension extracts just the topic name from the list of topic objects
    logging.info(f"Retrieving existing topics and subscriptions for project {project}")
    existing_topics = [topic.name for topic in list_topics(project)]
    existing_subscriptions = []
    for e_topic in existing_topics:
        # this list comprehension extracts just the subscription name from the list of subscription objects
        existing_subscriptions.extend([subscription.name for subscription in list_subscriptions_in_topic(e_topic)])
    # currently the library lists all subscriptions not just for that topic
    # so I call set on the the end list to remove duplicates.
    # This should be forward compatible once Google fixes the library
    existing_subscriptions = set(existing_subscriptions)
    return existing_topics, existing_subscriptions

Using GitHub flavored markdown can help make your request clearer.
See: https://guides.github.com/features/mastering-markdown/

[dhermes]

First, this response is coming directly from the backend, so there isn't much in the client we can do to fix the backend. One thing that could be better would to wrap that scary looking _Rendezvous in one of our api_core exceptions. I'm actually surprised it isn't wrapped. UPDATE: I see you've updated the stacktrace to show the wrapped exception is in fact google.api_core.exceptions.InvalidArgument.

As I try to reproduce, it looks like your failure was caused by get_existing, yes?

---

From the docs referenced in the error messgae

The resource-name must start with a letter, and contain only letters ([A-Za-z]), numbers ([0-9]), dashes (-), underscores (_), periods (.), tildes (~), plus (+) or percent signs (%). It must be between 3 and 255 characters in length, and cannot begin with the string goog.

As you mentioned, the resource names ctp-test-topic and us.gcr.io%2Foem-services-dev both seem valid according to this.

>>> import re
>>> MOSTLY = re.compile(r'^[A-Za-z][A-Za-z0-9-_.~+%]{2,254}$')
>>> def verify(resource_name):
...     if resource_name.lower().startswith('goog'):
...         return False
...     return MOSTLY.match(resource_name) is not None
... 
>>> verify('ctp-test-topic')
True
>>> verify('us.gcr.io%2Foem-services-dev')
True
>>> verify('goog')
False
>>> verify('ab')
False
>>> verify('abc')
True

[anorth2]

Yes the error is raising in get_existing, though it appears to be an underlying Subscriber issue since the iterator returned by list_subscriptions is what's failing.

[dhermes]

@lukesneeringer @jonparrott Anybody on the backend team that could look at this?

[dhermes]

>>> import google.auth
>>> import google.cloud.pubsub_v1
>>> _, project = google.auth.default()
>>> publisher = google.cloud.pubsub_v1.PublisherClient()
>>> path1 = publisher.topic_path(project, 'ctp-test-topic')
>>> path2 = publisher.topic_path(project, 'us.gcr.io%2Foem-services-dev')
>>> path3 = publisher.topic_path(project, 'invalid!')
>>> path4 = publisher.topic_path(project, 'goog-bad')
>>> publisher.create_topic(path1)
name: "projects/${PROJECT}/topics/ctp-test-topic"

>>> publisher.create_topic(path2)
name: "projects/${PROJECT}/topics/us.gcr.io%2Foem-services-dev"

>>> publisher.create_topic(path3)
Traceback (most recent call last):
  File "${SITE_PACKAGES}/google/api_core/grpc_helpers.py", line 51, in error_remapped_callable
    return callable_(*args, **kwargs)
  File "${SITE_PACKAGES}/grpc/_channel.py", line 492, in __call__
    return _end_unary_response_blocking(state, call, False, deadline)
  File "${SITE_PACKAGES}/grpc/_channel.py", line 440, in _end_unary_response_blocking
    raise _Rendezvous(state, None, None, deadline)
grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with (StatusCode.INVALID_ARGUMENT, Invalid resource name given (name=projects/${PROJECT}/topics/invalid!). Refer to https://cloud.google.com/pubsub/overview#names for more information.)>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "${SITE_PACKAGES}/pubsub/google/cloud/pubsub_v1/_gapic.py", line 42, in <lambda>
    fx = lambda self, *a, **kw: wrapped_fx(self.api, *a, **kw)
  File "${SITE_PACKAGES}/pubsub/google/cloud/pubsub_v1/gapic/publisher_client.py", line 225, in create_topic
    return self._create_topic(request, retry=retry, timeout=timeout)
  File "${SITE_PACKAGES}/google/api_core/gapic_v1/method.py", line 135, in __call__
    return wrapped_func(*args, **kwargs)
  File "${SITE_PACKAGES}/google/api_core/retry.py", line 260, in retry_wrapped_func
    on_error=on_error,
  File "${SITE_PACKAGES}/google/api_core/retry.py", line 177, in retry_target
    return target()
  File "${SITE_PACKAGES}/google/api_core/timeout.py", line 206, in func_with_timeout
    return func(*args, **kwargs)
  File "${SITE_PACKAGES}/google/api_core/grpc_helpers.py", line 53, in error_remapped_callable
    six.raise_from(exceptions.from_grpc_error(exc), exc)
  File "<string>", line 3, in raise_from
google.api_core.exceptions.InvalidArgument: 400 Invalid resource name given (name=projects/${PROJECT}/topics/invalid!). Refer to https://cloud.google.com/pubsub/overview#names for more information.
>>>
>>> publisher.create_topic(path4)
Traceback (most recent call last):
  File "${SITE_PACKAGES}/google/api_core/grpc_helpers.py", line 51, in error_remapped_callable
    return callable_(*args, **kwargs)
  File "${SITE_PACKAGES}/grpc/_channel.py", line 492, in __call__
    return _end_unary_response_blocking(state, call, False, deadline)
  File "${SITE_PACKAGES}/grpc/_channel.py", line 440, in _end_unary_response_blocking
    raise _Rendezvous(state, None, None, deadline)
grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with (StatusCode.INVALID_ARGUMENT, Invalid resource name given (name=projects/${PROJECT}/topics/goog-bad). Refer to https://cloud.google.com/pubsub/overview#names for more information.)>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "${SITE_PACKAGES}/pubsub/google/cloud/pubsub_v1/_gapic.py", line 42, in <lambda>
    fx = lambda self, *a, **kw: wrapped_fx(self.api, *a, **kw)
  File "${SITE_PACKAGES}/pubsub/google/cloud/pubsub_v1/gapic/publisher_client.py", line 225, in create_topic
    return self._create_topic(request, retry=retry, timeout=timeout)
  File "${SITE_PACKAGES}/google/api_core/gapic_v1/method.py", line 135, in __call__
    return wrapped_func(*args, **kwargs)
  File "${SITE_PACKAGES}/google/api_core/retry.py", line 260, in retry_wrapped_func
    on_error=on_error,
  File "${SITE_PACKAGES}/google/api_core/retry.py", line 177, in retry_target
    return target()
  File "${SITE_PACKAGES}/google/api_core/timeout.py", line 206, in func_with_timeout
    return func(*args, **kwargs)
  File "${SITE_PACKAGES}/google/api_core/grpc_helpers.py", line 53, in error_remapped_callable
    six.raise_from(exceptions.from_grpc_error(exc), exc)
  File "<string>", line 3, in raise_from
google.api_core.exceptions.InvalidArgument: 400 Invalid resource name given (name=projects/${PROJECT}/topics/goog-bad). Refer to https://cloud.google.com/pubsub/overview#names for more information.
>>>
>>> publisher.delete_topic(path1)
>>> publisher.delete_topic(path2)
>>>

[anorth2]

Hey can we get this prioritized. It's failing the majority of the time and currently blocks our CD pipeline (we use dynamic pubsub creation per branch as part of our deployment process)

[dhermes]

@anorth2 I'll do my best to upstream it but it's a server issue, not a library issue, so there is nothing (actionable) to prioritize in google-cloud-pubsub.

[dhermes]

@anorth2 Did you get this resolved?

/cc @kir-titievsky

[dhermes]

@anorth2 I'm doing a "bug sweep" for Pub / Sub so am going to close this on the assumption it got resolved.

I'm inclined to keep it closed since it's a backend issue, not an issue with our client (AFAICT).

It may make sense to file an issue with the API team directly:

• https://cloud.google.com/support/docs/issue-trackers
• https://issuetracker.google.com/savedsearches/559741
• https://issuetracker.google.com/issues/new?component=187173&template=0

IIUC many users also have support contracts with Google Cloud Platform.

[anorth2]

We had this show up again, so I opened the issue with the backend team. Thanks for your help.