Google-cloud-python: DefaultCredentialsError is raised when getting client using JSON credentials on Docker

[panuta]

I'm trying to connect to Datastore from inside Docker container running on my local machine when I got this DefaultCredentialsError.

Python 2.7.13 (default, Apr 18 2017, 22:03:41)
[GCC 6.3.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from google.cloud import datastore
D1004 04:45:28.018781655     107 env_linux.c:66]             Warning: insecure environment read function 'getenv' used
>>> client = datastore.Client()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.7/site-packages/google/cloud/datastore/client.py", line 204, in __init__
    project=project, credentials=credentials, _http=_http)
  File "/usr/lib/python2.7/site-packages/google/cloud/client.py", line 211, in __init__
    _ClientProjectMixin.__init__(self, project=project)
  File "/usr/lib/python2.7/site-packages/google/cloud/client.py", line 165, in __init__
    project = self._determine_default(project)
  File "/usr/lib/python2.7/site-packages/google/cloud/datastore/client.py", line 221, in _determine_default
    return _determine_default_project(project)
  File "/usr/lib/python2.7/site-packages/google/cloud/datastore/client.py", line 77, in _determine_default_project
    project = _base_default_project(project=project)
  File "/usr/lib/python2.7/site-packages/google/cloud/_helpers.py", line 179, in _determine_default_project
    _, project = google.auth.default()
  File "/usr/lib/python2.7/site-packages/google/auth/_default.py", line 281, in default
    credentials, project_id = checker()
  File "/usr/lib/python2.7/site-packages/google/auth/_default.py", line 142, in _get_explicit_environ_credentials
    os.environ[environment_vars.CREDENTIALS])
  File "/usr/lib/python2.7/site-packages/google/auth/_default.py", line 65, in _load_credentials_from_file
    'File {} was not found.'.format(filename))
google.auth.exceptions.DefaultCredentialsError: File "/root/.google/##################.json" was not found.

My Docker container is from jfloff/alpine-python:2.7 image. I have set Docker volume to let this container access credentials file from the host and I have already make sure that Python can see this file. What I don't understand is that if I try to call function _load_credentials_from_file directly, it can read credentials file without problem. Only got DefaultCredentialsError when calling through client = datastore.Client()

>>> from google.auth._default import _load_credentials_from_file
>>> _load_credentials_from_file('/root/.google/##################.json')
(<google.oauth2.service_account.Credentials object at 0x7f0b47ef3f90>, u'############')

Another thing that I noticed is that when I import datastore using from google.cloud import datastore I got the following warning (You can also see the warning at the stacktrace above)

Warning: insecure environment read function 'getenv' used

So I googled around this warning and see that it might have something to do with grpc library.

• https://github.com/grpc/grpc/issues/8104
• https://github.com/nodejs/docker-node/issues/436

Now I have no idea how to fix issue, please help me

Thank you

[dhermes]

@panuta Thanks for filing. Have you seen our auth guide? Also, is the filename in the error:

google.auth.exceptions.DefaultCredentialsError: File "/root/.google/##################.json" was not found.

the same as the filename you are manually loading?

```python

_load_credentials_from_file('/root/.google/##################.json')
````

[panuta]

@dhermes Yes, I did read and follow the guide. I download json key and set its path to GOOGLE_APPLICATION_CREDENTIALS environment variable according to the guide.

Also filename is is the same. I copy&paste it :)

To give you more info, I have uploaded my application to Google Compute Engine and run it from there using the same code that I run on my local machine. It runs perfectly fine. Data was saved to Datastore correctly. So this issue only happened inside Docker on my local machine.

Here is my Dockerfile and requirements that I'm using

FROM python:3.6-alpine3.6
ENV PYTHONUNBUFFERED 1

# System dependencies
RUN apk add --no-cache \
        tzdata \
        build-base \
        bash \
        libffi-dev \
        openssl-dev \
        libxml2-dev \
        libxslt-dev

# Set Timezone
RUN cp /usr/share/zoneinfo/Asia/Bangkok /etc/localtime
RUN echo "Asia/Bangkok" > /etc/timezone

# Python environments
COPY ./requirements/scrapyd.txt /tmp/requirements.txt
RUN pip install -r /tmp/requirements.txt

# scrapyd
RUN mkdir -p /scrapyd/data
COPY ./compose/scrapyd/scrapyd.conf /scrapyd/scrapyd.conf

VOLUME /scrapyd/data
EXPOSE 6800

WORKDIR /scrapyd
CMD ["scrapyd"]

scrapyd

slack_log_handler

google-cloud-datastore

NOTE that since I asked this question, I have changed Docker image from jfloff/alpine-python:2.7 to python:3.6-alpine3.6 but the issue is still persisted.

[dhermes]

@jonparrott Can we move this over to https://github.com/GoogleCloudPlatform/google-auth-library-python/?

@panuta Thanks for the details. It's very surprising that _load_credentials_from_file fails to find the file when called by google.auth.default() but not when called directly.

How are you running Docker on your local machine? The file in /root/.google won't appear there unless you do something like --volume /some/local/path:/root/.google when you call docker run.

[panuta]

@dhermes I solved it. It's my own fault. I have set env variable with double quote around the string like

GOOGLE_APPLICATION_CREDENTIALS="/root/.google/############.json"

Took me a while to see this obvious error. Such an embarrassing mistake. Thank you for trying to help me figuring it out.

[dhermes]

Good deal! 👍

[MajdiRabia]

This saved my day ! Thanks panuta

[kennyaires]

I made exactly the same mistake, thanks to share the case.

[lewisMachilika]

you gys how are you getting it right i have also the same error

[maxnelson]

@lewisMachilika when you set the environment variable, make sure to use single quote instead of double-quotes surrounding your path, i.e.
export GOOGLE_APPLICATION_CREDENTIALS=‘/path/to/your_credentials.json’

The reason so many people are having this issue is probably because they use double-quotes in the documentation:
https://cloud.google.com/docs/authentication/getting-started#auth-cloud-implicit-python

Someone should fix this in order to avoid further confusion amongst developers.

[faltajan]

Hello guys.
I have still this issue, and after trying all steps written above, I am still getting same error - file not found.

Strange is - script is looking for file in GOOGLE_APPLICATION_CREDENTIALS - it mean, it can read it from ENV, but complains about not finding it.

Any other sugestion how to solve this?
Thanks.

ERROR
google.auth.exceptions.DefaultCredentialsError: File /Users/janfalta//Users/janfalta/key.json was not found

this is my statement in BASH_PROFILE
export GOOGLE_APPLICATION_CREDENTIALS='/Users/janfalta//Users/janfalta/key.json '

PRINTENV
printenv TERM_PROGRAM=Apple_Terminal SHELL=/bin/bash TERM=xterm-256color TMPDIR=/var/folders/k1/vxmdz_591lqf_k1p3czxb4_m0000gn/T/ GOOGLE_APPLICATION_CREDENTIALS=/Users/janfalta//Users/janfalta/key.json Apple_PubSub_Socket_Render=/private/tmp/com.apple.launchd.7VuVFJmiK4/Render TERM_PROGRAM_VERSION=421.2 OLDPWD=/Users/janfalta TERM_SESSION_ID=7C78DC52-1147-46F5-A419-55BD12BEE6C4 USER=janfalta SSH_AUTH_SOCK=/private/tmp/com.apple.launchd.2Bof01U0Gw/Listeners PATH=/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin LANG=cs_CZ.UTF-8 XPC_FLAGS=0x0 PS1=$ \u@\e[1;36m [\w]:\e[m \n XPC_SERVICE_NAME=0 SHLVL=1 HOME=/Users/janfalta LOGNAME=janfalta _=/usr/bin/printenv

[jkyl]

@faltajan in your case it may be that this path:

GOOGLE_APPLICATION_CREDENTIALS=/Users/janfalta//Users/janfalta/key.json

is incorrect (/Users/janfalta/ is duplicated).