Cloud-custodian: gcp - allow bypass if API has not been enabled or used in project

[mikejgray]

Is your feature request related to a problem? Please describe.
Whenever a policy attempts to hit an API that has been disabled or has not been used yet in a project, a 403 error is returned and halts Custodian execution.

Describe the solution you'd like
I'd like to either bypass this failure and continue or have a flag I can use to bypass it.

Describe alternatives you've considered
I can't think of an alternative that might work across numerous projects with a variable number of APIs that are disabled. The only one that makes sense is turning on all APIs in all projects but there are potential security concerns there.

Additional context

2020-08-20 15:26:46,367: custodian.policy:DEBUG Running policy:gcp-function-public resource:gcp.function region:default c7n:0.9.4
2020-08-20 15:26:47,249: googleapiclient.http:WARNING Invalid JSON content from response: b'{\n  "error": {\n    "code": 403,\n    "message": "Cloud Functions API has not been used in project 123456789123 before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/cloudfunctions.googleapis.com/overview?project=123456789123 then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry.",\n    "status": "PERMISSION_DENIED",\n    "details": [\n      {\n        "@type": "type.googleapis.com/google.rpc.Help",\n        "links": [\n          {\n            "description": "Google developers console API activation",\n            "url": "https://console.developers.google.com/apis/api/cloudfunctions.googleapis.com/overview?project=123456789123"\n          }\n        ]\n      }\n    ]\n  }\n}\n'
2020-08-20 15:26:47,250: custodian.output:DEBUG metric:PolicyException Count:1 policy:gcp-function-public restype:gcp.function
2020-08-20 15:26:47,251: custodian.output:ERROR Error while executing policy
Traceback (most recent call last):
  File "/Users/mgray/.pyenv/versions/3.7.5/lib/python3.7/site-packages/c7n/policy.py", line 291, in run
    resources = self.policy.resource_manager.resources()
  File "/Users/mgray/.pyenv/versions/3.7.5/lib/python3.7/site-packages/c7n_gcp/query.py", line 203, in resources
    resources = self._fetch_resources(q)
  File "/Users/mgray/.pyenv/versions/3.7.5/lib/python3.7/site-packages/c7n_gcp/query.py", line 223, in _fetch_resources
    return self.augment(self.source.get_resources(query)) or []
  File "/Users/mgray/.pyenv/versions/3.7.5/lib/python3.7/site-packages/c7n_gcp/query.py", line 86, in get_resources
    return self.query.filter(self.manager, **query)
  File "/Users/mgray/.pyenv/versions/3.7.5/lib/python3.7/site-packages/c7n_gcp/query.py", line 61, in filter
    client, enum_op, params, path)
  File "/Users/mgray/.pyenv/versions/3.7.5/lib/python3.7/site-packages/c7n_gcp/query.py", line 66, in _invoke_client_enum
    for page in client.execute_paged_query(enum_op, params):
  File "/Users/mgray/.pyenv/versions/3.7.5/lib/python3.7/site-packages/c7n_gcp/client.py", line 412, in execute_paged_query
    response = self._execute(request)
  File "/Users/mgray/.pyenv/versions/3.7.5/lib/python3.7/site-packages/retrying.py", line 49, in wrapped_f
    return Retrying(*dargs, **dkw).call(f, *args, **kw)
  File "/Users/mgray/.pyenv/versions/3.7.5/lib/python3.7/site-packages/retrying.py", line 206, in call
    return attempt.get(self._wrap_exception)
  File "/Users/mgray/.pyenv/versions/3.7.5/lib/python3.7/site-packages/retrying.py", line 247, in get
    six.reraise(self.value[0], self.value[1], self.value[2])
  File "/Users/mgray/.pyenv/versions/3.7.5/lib/python3.7/site-packages/six.py", line 703, in reraise
    raise value
  File "/Users/mgray/.pyenv/versions/3.7.5/lib/python3.7/site-packages/retrying.py", line 200, in call
    attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
  File "/Users/mgray/.pyenv/versions/3.7.5/lib/python3.7/site-packages/c7n_gcp/client.py", line 480, in _execute
    num_retries=self._num_retries)
  File "/Users/mgray/.pyenv/versions/3.7.5/lib/python3.7/site-packages/googleapiclient/_helpers.py", line 134, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/Users/mgray/.pyenv/versions/3.7.5/lib/python3.7/site-packages/googleapiclient/http.py", line 907, in execute
    raise HttpError(resp, content, uri=self.uri)
googleapiclient.errors.HttpError: <HttpError 403 when requesting https://cloudfunctions.googleapis.com/v1/projects/project/locations/-/functions?alt=json returned "Cloud Functions API has not been used in project 123456789123 before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/cloudfunctions.googleapis.com/overview?project=123456789123 then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry.". Details: "[{'@type': 'type.googleapis.com/google.rpc.Help', 'links': [{'description': 'Google developers console API activation', 'url': 'https://console.developers.google.com/apis/api/cloudfunctions.googleapis.com/overview?project=123456789123'}]}]">
Traceback (most recent call last):
  File "/Users/mgray/.pyenv/versions/3.7.5/lib/python3.7/site-packages/c7n/cli.py", line 362, in main
    command(config)
  File "/Users/mgray/.pyenv/versions/3.7.5/lib/python3.7/site-packages/c7n/commands.py", line 136, in _load_policies
    return f(options, list(policies))
  File "/Users/mgray/.pyenv/versions/3.7.5/lib/python3.7/site-packages/c7n/commands.py", line 282, in run
    policy()
  File "/Users/mgray/.pyenv/versions/3.7.5/lib/python3.7/site-packages/c7n/policy.py", line 1170, in __call__
    resources = PullMode(self).run()
  File "/Users/mgray/.pyenv/versions/3.7.5/lib/python3.7/site-packages/c7n/policy.py", line 291, in run
    resources = self.policy.resource_manager.resources()
  File "/Users/mgray/.pyenv/versions/3.7.5/lib/python3.7/site-packages/c7n_gcp/query.py", line 203, in resources
    resources = self._fetch_resources(q)
  File "/Users/mgray/.pyenv/versions/3.7.5/lib/python3.7/site-packages/c7n_gcp/query.py", line 223, in _fetch_resources
    return self.augment(self.source.get_resources(query)) or []
  File "/Users/mgray/.pyenv/versions/3.7.5/lib/python3.7/site-packages/c7n_gcp/query.py", line 86, in get_resources
    return self.query.filter(self.manager, **query)
  File "/Users/mgray/.pyenv/versions/3.7.5/lib/python3.7/site-packages/c7n_gcp/query.py", line 61, in filter
    client, enum_op, params, path)
  File "/Users/mgray/.pyenv/versions/3.7.5/lib/python3.7/site-packages/c7n_gcp/query.py", line 66, in _invoke_client_enum
    for page in client.execute_paged_query(enum_op, params):
  File "/Users/mgray/.pyenv/versions/3.7.5/lib/python3.7/site-packages/c7n_gcp/client.py", line 412, in execute_paged_query
    response = self._execute(request)
  File "/Users/mgray/.pyenv/versions/3.7.5/lib/python3.7/site-packages/retrying.py", line 49, in wrapped_f
    return Retrying(*dargs, **dkw).call(f, *args, **kw)
  File "/Users/mgray/.pyenv/versions/3.7.5/lib/python3.7/site-packages/retrying.py", line 206, in call
    return attempt.get(self._wrap_exception)
  File "/Users/mgray/.pyenv/versions/3.7.5/lib/python3.7/site-packages/retrying.py", line 247, in get
    six.reraise(self.value[0], self.value[1], self.value[2])
  File "/Users/mgray/.pyenv/versions/3.7.5/lib/python3.7/site-packages/six.py", line 703, in reraise
    raise value
  File "/Users/mgray/.pyenv/versions/3.7.5/lib/python3.7/site-packages/retrying.py", line 200, in call
    attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
  File "/Users/mgray/.pyenv/versions/3.7.5/lib/python3.7/site-packages/c7n_gcp/client.py", line 480, in _execute
    num_retries=self._num_retries)
  File "/Users/mgray/.pyenv/versions/3.7.5/lib/python3.7/site-packages/googleapiclient/_helpers.py", line 134, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/Users/mgray/.pyenv/versions/3.7.5/lib/python3.7/site-packages/googleapiclient/http.py", line 907, in execute
    raise HttpError(resp, content, uri=self.uri)
googleapiclient.errors.HttpError: <HttpError 403 when requesting https://cloudfunctions.googleapis.com/v1/projects/project/locations/-/functions?alt=json returned "Cloud Functions API has not been used in project 123456789123 before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/cloudfunctions.googleapis.com/overview?project=123456789123 then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry.". Details: "[{'@type': 'type.googleapis.com/google.rpc.Help', 'links': [{'description': 'Google developers console API activation', 'url': 'https://console.developers.google.com/apis/api/cloudfunctions.googleapis.com/overview?project=123456789123'}]}]">
> /Users/mgray/.pyenv/versions/3.7.5/lib/python3.7/site-packages/googleapiclient/http.py(907)execute()
-> raise HttpError(resp, content, uri=self.uri)
(Pdb)

[kapilt]

yeah.. we should treat it like we do aws services that are not in region and just skip.. its a little bit harder to do then that since we can't pre-compute that and skip execution, its a runtime characteristic during execution, but catching and moving on is the right thing.

[kapilt]

fixed in #6208, thanks @kkessels