Google-cloud-python: Stackdriver: list descending entries fails after a short moment

[iddan]

1. OS type and version: macOS 10.13.4
2. Python version and virtual environment information: Python 3.6.5, venv using pipenv
3. google-cloud-python version: 0.33.1
4. Stacktrace if available:
   
   
   

   Stacktrace

Traceback (most recent call last):
  File "/Users/iddan/.local/share/virtualenvs/stackdriver-cli-7guWFJlM/lib/python3.6/site-packages/google/gax/retry.py", line 121, in inner
    return to_call(*args)
  File "/Users/iddan/.local/share/virtualenvs/stackdriver-cli-7guWFJlM/lib/python3.6/site-packages/google/gax/retry.py", line 68, in inner
    return a_func(*updated_args, **kwargs)
  File "/Users/iddan/.local/share/virtualenvs/stackdriver-cli-7guWFJlM/lib/python3.6/site-packages/grpc/_channel.py", line 500, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/Users/iddan/.local/share/virtualenvs/stackdriver-cli-7guWFJlM/lib/python3.6/site-packages/grpc/_channel.py", line 434, in _end_unary_response_blocking
    raise _Rendezvous(state, None, None, deadline)
grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with (StatusCode.RESOURCE_EXHAUSTED, Insufficient tokens for quota 'logging.googleapis.com/read_requests' and limit 'ReadRequestsPerMinutePerProject' of service 'logging.googleapis.com' for consumer 'project_number:307713667560'.)>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/Users/iddan/stackdriver-cli/stackdriver/cli.py", line 11, in 
    for entry in client.list_entries(order_by=DESCENDING):
  File "/Users/iddan/.local/share/virtualenvs/stackdriver-cli-7guWFJlM/lib/python3.6/site-packages/google/api_core/page_iterator.py", line 186, in _items_iter
    for page in self._page_iter(increment=False):
  File "/Users/iddan/.local/share/virtualenvs/stackdriver-cli-7guWFJlM/lib/python3.6/site-packages/google/api_core/page_iterator.py", line 223, in _page_iter
    page = self._next_page()
  File "/Users/iddan/.local/share/virtualenvs/stackdriver-cli-7guWFJlM/lib/python3.6/site-packages/google/api_core/page_iterator.py", line 430, in _next_page
    items = six.next(self._gax_page_iter)
  File "/Users/iddan/.local/share/virtualenvs/stackdriver-cli-7guWFJlM/lib/python3.6/site-packages/google/gax/__init__.py", line 465, in __next__
    response = self._func(self._request, **self._kwargs)
  File "/Users/iddan/.local/share/virtualenvs/stackdriver-cli-7guWFJlM/lib/python3.6/site-packages/google/gax/api_callable.py", line 376, in inner
    return a_func(*args, **kwargs)
  File "/Users/iddan/.local/share/virtualenvs/stackdriver-cli-7guWFJlM/lib/python3.6/site-packages/google/gax/retry.py", line 127, in inner
    ' classified as transient', exception)
google.gax.errors.RetryError: RetryError(Exception occurred in retry method that was not classified as transient, caused by <_Rendezvous of RPC that terminated with (StatusCode.RESOURCE_EXHAUSTED, Insufficient tokens for quota 'logging.googleapis.com/read_requests' and limit 'ReadRequestsPerMinutePerProject' of service 'logging.googleapis.com' for consumer 'project_number:307713667560'.)>)

6. Steps to reproduce - Install google cloud python - Write script importing stackdriver list_entries() with order_by=DESCENDING - Run for a short moment until it crashes 7. Code example

from google.cloud import logging
from google.cloud.logging import DESCENDING

client = logging.Client.from_service_account_json("credentials.json")

for entry in client.list_entries(order_by=DESCENDING):
    print(entry)

[tseaver]

@iddan Thanks for the report! It looks like you are hitting quota exhaustion, likely due to a very large number of log entries for your project. Best practice would be to filter the log entries.

[iddan]

I'm trying to view logs after a very strict filter in my local console. I don't see why when using order_by DESCENDING while using an iterator would hit the quota. Am I doing something wrong?
Worth mentioning this is a recommended usage of the API

[tseaver]

@iddan Your example doesn't show any filtering at all.

@liyanhui1228 Does anything about order_by DESCENDING trigger extra quota usage?

[liyanhui1228]

Read requests per minute per user is 60 by default. It's possible that it still hit the quota after you apply your filter. The order_by field doesn't increase the quota usage.

[iddan]

What I'm really looking for is tailing logs. I've managed to properly tail logs but I got really surprised I'm allowed to call list_entries() once a second. Is there a different method? How does the Logs Viewer stream logs?

[tseaver]

@iddan We see this error occasionally in our system tests (#5303). I've just committed a fix there which adds ResourceExhausted to the set of manual retries for tests which call logger.list_entries (PR #5486).

Are you using gRPC (it is on by default unless you set the GOOGLE_CLOUD_DISABLE_GRPC environment variable)? If so, we should probably update the gRPC configuration to ensure that ResrouceExhausted is considered one of the idempotent response codes for automatic retry when running under gRPC.

[iddan]

I use gRPC. My problem is I tried to use this API for constant tailing (similar to what the Stackdriver UI does) and I couldn't find a better suiting API to use.

[tseaver]

If you need to call the API more than once a second, you could wrap it using google.api_core.retry.Retry, which adds retry with exponential backoff for transient errors. E.g.:

from google.api_core.retry import Retry

retry_list_entries = Retry()(client.list_entries)

for entry in retry_list_entries(order_by=DESCENDING):
    print(entry)

Note that this will be expensive for any project which accumulates a large number of log entries over time: a real-world example would likely filter the log entries, and maybe only list the N most recent on each call.

[iddan]

and maybe only list the N most recent on each call.

How can I do that?

[tseaver]

List the N most recent log entries:

from google.api_core.retry import Retry

def list_recent_entries(client, num_entries=10):
    retry_list_entries = Retry()(client.list_entries)
    count = 0
    for entry in retry_list_entries(order_by=DESCENDING):
        print(entry)
        count += 1
        if count > num_entries:
            break

[yiga2]

@tseaver not sure Retry will do so if 403 rateLimitExceeded is hit.

That error is currently not listed as transient API error as per https://google-cloud-python.readthedocs.io/en/latest/core/retry.html?highlight=retry#google.api_core.retry.if_transient_error

Would be good to add but in the meantime, I guess you can pass the error as custom predicate. However the error type is not defined in the library or in http_client (at best it matches 403 FORBIDDEN).

We went down the road of a custom linear retry (sleep 2 sec)

[iddan]

Eventually we migrated to another logs product with a better UX instead of extending Stackdriver

[tseaver]

@iddan I'm sorry we weren't able to work out how to address your usecase better with GCP logging.