Google-cloud-python: Core: GOOGLE_CLOUD_PROJECT env var is not supported unless credentials are also provided in the environment

[vimota]

Environment details

• Python 3.6.6, Anaconda

• Name: google-cloud-bigquery, Version: 1.11.3

  Steps to reproduce

  1. Set GOOGLE_CLOUD_PROJECT env var to the desired project.
  2. run bigquery.client(credentials=FooCredentials())

Code example

client = bigquery.Client(credentials=KaggleKernelCredentials())

Stack trace

Traceback (most recent call last):
  File "/input/tests/test_bigquery.py", line 112, in test_project_with_env_var_project_default_credentials
    client = bigquery.Client()
  File "/tmp/patches/lib/python3.6/site-packages/sitecustomize.py", line 42, in <lambda>
    bq_client, *args, **kwargs)
  File "/tmp/patches/lib/python3.6/site-packages/sitecustomize.py", line 35, in monkeypatch_bq
    return bq_client(*args, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/google/cloud/bigquery/client.py", line 150, in __init__
    project=project, credentials=credentials, _http=_http
  File "/opt/conda/lib/python3.6/site-packages/google/cloud/client.py", line 223, in __init__
    _ClientProjectMixin.__init__(self, project=project)
  File "/opt/conda/lib/python3.6/site-packages/google/cloud/client.py", line 175, in __init__
    project = self._determine_default(project)
  File "/opt/conda/lib/python3.6/site-packages/google/cloud/client.py", line 190, in _determine_default
    return _determine_default_project(project)
  File "/opt/conda/lib/python3.6/site-packages/google/cloud/_helpers.py", line 186, in _determine_default_project
    _, project = google.auth.default()
  File "/opt/conda/lib/python3.6/site-packages/google/auth/_default.py", line 317, in default
    raise exceptions.DefaultCredentialsError(_HELP_MESSAGE)
google.auth.exceptions.DefaultCredentialsError: Could not automatically determine credentials. Please set GOOGLE_APPLICATION_CREDENTIALS or explicitly create credentials and re-run the application. For more information, please see https:/
/cloud.google.com/docs/authentication/getting-started

I looked into it for a while, the issue here is that the way the core library tries to determine the default project is by calling google.auth.default() and ignoring the credentials returned (https://github.com/googleapis/google-cloud-python/blob/master/core/google/cloud/_helpers.py#L186). But that method is mostly used to determine the credentials, so it throws an error if it doesn't find one in the environment (https://github.com/googleapis/google-auth-library-python/blob/master/google/auth/_default.py#L317). Maybe the fix is for google.auth to have a default_project() method that only looks for that? Or maybe more useful: a flag parameter in default() that is raise_missing_credentials?

[tseaver]

Thanks for the report.

ISTM that the right API surface is for google.auth.default to take a flag (e.g. project_only?), and return None, explicit_project_id if no credentials are found and that flag is passed.

@theacodes Does that seem right to you?

[theacodes]

This is by design. Credentials and project information must come from the
same place (principle of least surprise).

On Thu, May 23, 2019 at 10:59 AM Tres Seaver [e-mail]
wrote:

Thanks for the report.

ISTM that the right API surface is for google.auth.default to take a flag
(e.g. project_only?), and return None, explicit_project_id if no
credentials are found and that flag is passed.

@theacodes https://github.com/theacodes Does that seem right to you?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/googleapis/google-cloud-python/issues/8108?email_source=notifications&email_token=AAB5I42IYZHMLG3VJCHMU6DPW3LRVA5CNFSM4HOZTRL2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODWDALMA#issuecomment-495322544,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAB5I42Y27HVPAAIRO6IGWDPW3LRVANCNFSM4HOZTRLQ
.

[vimota]

This is super surprising - there's no such documentation (https://github.com/googleapis/google-auth-library-python/blob/master/google/auth/environment_vars.py#L19), and plenty of use cases where you would want to set a default project but not be able to set a default credential in the environment.

[theacodes]

Explicitly setting the project in the constructor is the best way to
express that.

On Thu, May 23, 2019 at 11:17 AM Victor Mota [e-mail]
wrote:

This is super surprising - there's no such documentation (
https://github.com/googleapis/google-auth-library-python/blob/master/google/auth/environment_vars.py#L19),
and plenty of use cases where you would want to set a default project but
not be able to set a default credential in the environment.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/googleapis/google-cloud-python/issues/8108?email_source=notifications&email_token=AAB5I4YIKSQDRZMLUYPA6GDPW3NTNA5CNFSM4HOZTRL2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODWDBZ7Y#issuecomment-495328511,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAB5I43W2TH2JHYDCHTOIODPW3NTNANCNFSM4HOZTRLQ
.

[vimota]

Thanks for getting back :)
If credentials and project information must come from the
same place, wouldn't you also expect an error thrown if the project id isn't specified in the environment but the credentials is?
Or rather how come you're expected to set credentials at the environment level and allowed to only set the project in the constructor but not the other way around? That doesn't seem consistent.

[theacodes]

You can set the credentials in the environment using the
GOOGLE_APPLICATION_CREDENTIALS env var.

On Thu, May 23, 2019, 12:00 PM Victor Mota [e-mail] wrote:

Thanks for getting back :)
If credentials and project information must come from the
same place, wouldn't you also expect an error thrown if the project id
isn't specified in the environment but the credentials is?
Or rather how come you're expected to set credentials at the environment
level and allowed to only set the project in the constructor but not the
other way around? That doesn't seem consistent.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/googleapis/google-cloud-python/issues/8108?email_source=notifications&email_token=AAB5I4ZYACEYZFHLY7RJNKLPW3STZA5CNFSM4HOZTRL2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODWDFRDI#issuecomment-495343757,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAB5I424WX6BNUO7YRGGPOLPW3STZANCNFSM4HOZTRLQ
.

[vimota]

Right, I got that :) I mean, why do you have to set GOOGLE_APPLICATION_CREDENTIALS to be able to set GOOGLE_CLOUD_PROJECT, but you don't have to set GOOGLE_CLOUD_PROJECT to set GOOGLE_APPLICATION_CREDENTIALS? The library only throws an error in one direction, which leads to confusion.

[theacodes]

The short answer is "historical reasons". The long answer is complicated.
This is why we really encourage people to explicitly pass credentials and
project to the client constructor and not depend on the often surprising
behavior of application default credentials.

It was a good idea at the time but wow did it have some unexpected
consequences.

On Thu, May 23, 2019, 12:40 PM Victor Mota [e-mail] wrote:

Right, I got that :) I mean, why do you have to set
GOOGLE_APPLICATION_CREDENTIALS to be able to set GOOGLE_CLOUD_PROJECT, but
you don't have to set GOOGLE_CLOUD_PROJECT to set
GOOGLE_APPLICATION_CREDENTIALS? The library only throws an error in one
direction, which leads to confusion.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/googleapis/google-cloud-python/issues/8108?email_source=notifications&email_token=AAB5I4526MPLAKMVYNH3SYLPW3XLTA5CNFSM4HOZTRL2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODWDI5JI#issuecomment-495357605,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAB5I4ZS53WZQ6FOPYKSMT3PW3XLTANCNFSM4HOZTRLQ
.

[tswast]

As a workaround, the BigQuery library does accept a default project in the Client constructor:

import os
client = bigquery.Client(
    credentials=KaggleKernelCredentials(),
    project=os.getenv("GOOGLE_CLOUD_PROJECT"),
)

[vimota]

Yep, that's what we ended up doing (https://github.com/Kaggle/docker-python/blob/master/patches/sitecustomize.py#L27) - thanks :)

[tswast]

Closing as "Working as Intended".

To reduce possible sources of confusion, the discovered project and credentials must come from the same source. Use an explicit project ID in the constructor if different behavior is required.