Google-cloud-python: BigQuery DataTransfer: create_transfer_config not using default application credentials

[JohnPaton]

Environment details

1. Specify the API at the beginning of the title (for example, "BigQuery: ...")
   General, Core, and Other are also allowed as types ✔️
2. OS type and version: macOS Mojave 10.14
3. Python version and virtual environment information: Python 3.7.1
4. google-cloud- version:

$ pip freeze | grep google
google-api-core==1.9.0
google-auth==1.6.3
google-cloud-bigquery==1.11.2
google-cloud-bigquery-datatransfer==0.3.0
google-cloud-core==0.29.1
google-resumable-media==0.3.2
googleapis-common-protos==1.5.9

Steps to reproduce

Just gcloud auth login to a project with BQ DataTransfer API enabled

Code example

from google.cloud import bigquery_datatransfer
from google.protobuf.struct_pb2 import Struct
client_dt = bigquery_datatransfer.DataTransferServiceClient()

# https://github.com/googleapis/google-cloud-python/issues/6449
parent = "{}/locations/{}".format(client_dt.project_path("project_name"), "europe")

# test that we have access 
# this works fine -> credentials are being used
configs = list(client_dt.list_transfer_configs(parent))

# create minimal TransferConfig
# TransferConfig.params can't be a dict for some reason
params = Struct()
params.update({
    "write_disposition": "WRITE_TRUNCATE",
    "destination_table_name_template": "table_name",
    "query": "SELECT 'test' AS test"
})
tc_dict = {
    "display_name": "jp_test_scheduled_query",
    "destination_dataset_id": "writable",
    "data_source_id": "scheduled_query",
    "schedule": "every monday 09:00",
    "params": params
}
tc = bigquery_datatransfer.types.TransferConfig(**tc_dict)

# the big moment
client_dt.create_transfer_config(parent, tc)

Stack trace

---------------------------------------------------------------------------
_Rendezvous                               Traceback (most recent call last)
~/.virtualenvs/gcloud/lib/python3.7/site-packages/google/api_core/grpc_helpers.py in error_remapped_callable(*args, **kwargs)
     56         try:
---> 57             return callable_(*args, **kwargs)
     58         except grpc.RpcError as exc:

~/.virtualenvs/gcloud/lib/python3.7/site-packages/grpc/_channel.py in __call__(self, request, timeout, metadata, credentials, wait_for_ready)
    548                                       wait_for_ready)
--> 549         return _end_unary_response_blocking(state, call, False, None)
    550 

~/.virtualenvs/gcloud/lib/python3.7/site-packages/grpc/_channel.py in _end_unary_response_blocking(state, call, with_call, deadline)
    465     else:
--> 466         raise _Rendezvous(state, None, None, deadline)
    467 

_Rendezvous: <_Rendezvous of RPC that terminated with:
    status = StatusCode.INVALID_ARGUMENT
    details = "Failed to find a valid credential. The request to create a transfer config is supposed to contain an authorization code."
    debug_error_string = "{"created":"@1554993819.642496000","description":"Error received from peer","file":"src/core/lib/surface/call.cc","file_line":1039,"grpc_message":"Failed to find a valid credential. The request to create a transfer config is supposed to contain an authorization code.","grpc_status":3}"
>

The above exception was the direct cause of the following exception:

InvalidArgument                           Traceback (most recent call last)
<ipython-input-42-4e6364d93fab> in <module>
----> 1 test = client_dt.create_transfer_config(parent, tc)

~/.virtualenvs/gcloud/lib/python3.7/site-packages/google/cloud/bigquery_datatransfer_v1/gapic/data_transfer_service_client.py in create_transfer_config(self, parent, transfer_config, authorization_code, retry, timeout, metadata)
    495 
    496         return self._inner_api_calls["create_transfer_config"](
--> 497             request, retry=retry, timeout=timeout, metadata=metadata
    498         )
    499 

~/.virtualenvs/gcloud/lib/python3.7/site-packages/google/api_core/gapic_v1/method.py in __call__(self, *args, **kwargs)
    141             kwargs["metadata"] = metadata
    142 
--> 143         return wrapped_func(*args, **kwargs)
    144 
    145 

~/.virtualenvs/gcloud/lib/python3.7/site-packages/google/api_core/retry.py in retry_wrapped_func(*args, **kwargs)
    268                 sleep_generator,
    269                 self._deadline,
--> 270                 on_error=on_error,
    271             )
    272 

~/.virtualenvs/gcloud/lib/python3.7/site-packages/google/api_core/retry.py in retry_target(target, predicate, sleep_generator, deadline, on_error)
    177     for sleep in sleep_generator:
    178         try:
--> 179             return target()
    180 
    181         # pylint: disable=broad-except

~/.virtualenvs/gcloud/lib/python3.7/site-packages/google/api_core/timeout.py in func_with_timeout(*args, **kwargs)
    212             """Wrapped function that adds timeout."""
    213             kwargs["timeout"] = next(timeouts)
--> 214             return func(*args, **kwargs)
    215 
    216         return func_with_timeout

~/.virtualenvs/gcloud/lib/python3.7/site-packages/google/api_core/grpc_helpers.py in error_remapped_callable(*args, **kwargs)
     57             return callable_(*args, **kwargs)
     58         except grpc.RpcError as exc:
---> 59             six.raise_from(exceptions.from_grpc_error(exc), exc)
     60 
     61     return error_remapped_callable

~/.virtualenvs/gcloud/lib/python3.7/site-packages/six.py in raise_from(value, from_value)

InvalidArgument: 400 Failed to find a valid credential. The request to create a transfer config is supposed to contain an authorization code.

It seems to expect an explicit credential, even though list_transfer_configs works just fine using the defaults.

I have also tried to use the token from gcloud auth application-default print-access-token, but to no avail, it seems to be invalid:

client_dt.create_transfer_config(parent, tc, "<my_token>")

---------------------------------------------------------------------------
_Rendezvous                               Traceback (most recent call last)
~/.virtualenvs/gcloud/lib/python3.7/site-packages/google/api_core/grpc_helpers.py in error_remapped_callable(*args, **kwargs)
     56         try:
---> 57             return callable_(*args, **kwargs)
     58         except grpc.RpcError as exc:

~/.virtualenvs/gcloud/lib/python3.7/site-packages/grpc/_channel.py in __call__(self, request, timeout, metadata, credentials, wait_for_ready)
    548                                       wait_for_ready)
--> 549         return _end_unary_response_blocking(state, call, False, None)
    550 

~/.virtualenvs/gcloud/lib/python3.7/site-packages/grpc/_channel.py in _end_unary_response_blocking(state, call, with_call, deadline)
    465     else:
--> 466         raise _Rendezvous(state, None, None, deadline)
    467 

_Rendezvous: <_Rendezvous of RPC that terminated with:
    status = StatusCode.INVALID_ARGUMENT
    details = "Request contains an invalid argument."
    debug_error_string = "{"created":"@1554994044.809585000","description":"Error received from peer","file":"src/core/lib/surface/call.cc","file_line":1039,"grpc_message":"Request contains an invalid argument.","grpc_status":3}"
>

The above exception was the direct cause of the following exception:

InvalidArgument                           Traceback (most recent call last)
<ipython-input-46-8cfb05f8c77c> in <module>
----> 1 test = client_dt.create_transfer_config(parent, tc, token)

~/.virtualenvs/gcloud/lib/python3.7/site-packages/google/cloud/bigquery_datatransfer_v1/gapic/data_transfer_service_client.py in create_transfer_config(self, parent, transfer_config, authorization_code, retry, timeout, metadata)
    495 
    496         return self._inner_api_calls["create_transfer_config"](
--> 497             request, retry=retry, timeout=timeout, metadata=metadata
    498         )
    499 

~/.virtualenvs/gcloud/lib/python3.7/site-packages/google/api_core/gapic_v1/method.py in __call__(self, *args, **kwargs)
    141             kwargs["metadata"] = metadata
    142 
--> 143         return wrapped_func(*args, **kwargs)
    144 
    145 

~/.virtualenvs/gcloud/lib/python3.7/site-packages/google/api_core/retry.py in retry_wrapped_func(*args, **kwargs)
    268                 sleep_generator,
    269                 self._deadline,
--> 270                 on_error=on_error,
    271             )
    272 

~/.virtualenvs/gcloud/lib/python3.7/site-packages/google/api_core/retry.py in retry_target(target, predicate, sleep_generator, deadline, on_error)
    177     for sleep in sleep_generator:
    178         try:
--> 179             return target()
    180 
    181         # pylint: disable=broad-except

~/.virtualenvs/gcloud/lib/python3.7/site-packages/google/api_core/timeout.py in func_with_timeout(*args, **kwargs)
    212             """Wrapped function that adds timeout."""
    213             kwargs["timeout"] = next(timeouts)
--> 214             return func(*args, **kwargs)
    215 
    216         return func_with_timeout

~/.virtualenvs/gcloud/lib/python3.7/site-packages/google/api_core/grpc_helpers.py in error_remapped_callable(*args, **kwargs)
     57             return callable_(*args, **kwargs)
     58         except grpc.RpcError as exc:
---> 59             six.raise_from(exceptions.from_grpc_error(exc), exc)
     60 
     61     return error_remapped_callable

~/.virtualenvs/gcloud/lib/python3.7/site-packages/six.py in raise_from(value, from_value)

InvalidArgument: 400 Request contains an invalid argument.

Any ideas?

[JohnPaton]

Dumb mistake on my part - first creating a scheduled query through the UI triggered the OAuth consent screen, and after that my default creds worked fine. Sorry for the spam!