Google-cloud-python: `from_service_account_json` throws error

[crestenstclair]

Currently, I'm getting the following error while using gcloud in a python 3.4.3 virtualenv:

Traceback (most recent call last):
  File "/usr/lib/python3.4/configparser.py", line 762, in get
    value = d[option]
  File "/var/lib/jenkins/workspace/d-files-with-client-library-J2HHEFIKRSP5J5SH4BYKZ7MKYZEJFGDHIM4IBDJ7MCBN24XYA53A/venv/lib/python3.4/collections/__init__.py", line 814, in __getitem__
    return self.__missing__(key)            # support subclasses that define __missing__
  File "/var/lib/jenkins/workspace/d-files-with-client-library-J2HHEFIKRSP5J5SH4BYKZ7MKYZEJFGDHIM4IBDJ7MCBN24XYA53A/venv/lib/python3.4/collections/__init__.py", line 806, in __missing__
    raise KeyError(key)
KeyError: 'project'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "deploy/deploy.py", line 128, in <module>
    cli()
  File "/var/lib/jenkins/workspace/d-files-with-client-library-J2HHEFIKRSP5J5SH4BYKZ7MKYZEJFGDHIM4IBDJ7MCBN24XYA53A/venv/lib/python3.4/site-packages/click/core.py", line 716, in __call__
    return self.main(*args, **kwargs)
  File "/var/lib/jenkins/workspace/d-files-with-client-library-J2HHEFIKRSP5J5SH4BYKZ7MKYZEJFGDHIM4IBDJ7MCBN24XYA53A/venv/lib/python3.4/site-packages/click/core.py", line 696, in main
    rv = self.invoke(ctx)
  File "/var/lib/jenkins/workspace/d-files-with-client-library-J2HHEFIKRSP5J5SH4BYKZ7MKYZEJFGDHIM4IBDJ7MCBN24XYA53A/venv/lib/python3.4/site-packages/click/core.py", line 1060, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/var/lib/jenkins/workspace/d-files-with-client-library-J2HHEFIKRSP5J5SH4BYKZ7MKYZEJFGDHIM4IBDJ7MCBN24XYA53A/venv/lib/python3.4/site-packages/click/core.py", line 889, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/var/lib/jenkins/workspace/d-files-with-client-library-J2HHEFIKRSP5J5SH4BYKZ7MKYZEJFGDHIM4IBDJ7MCBN24XYA53A/venv/lib/python3.4/site-packages/click/core.py", line 534, in invoke
    return callback(*args, **kwargs)
  File "deploy/deploy.py", line 26, in upload_code_to_CDN_command
    upload_code_to_CDN(branch)
  File "deploy/deploy.py", line 41, in upload_code_to_CDN
    '/path/to/keyfile) <---- This is not a typo, it has been redacted
  File "/var/lib/jenkins/workspace/d-files-with-client-library-J2HHEFIKRSP5J5SH4BYKZ7MKYZEJFGDHIM4IBDJ7MCBN24XYA53A/venv/lib/python3.4/site-packages/google/cloud/client.py", line 61, in from_service_account_json
    return cls(*args, **kwargs)
  File "/var/lib/jenkins/workspace/d-files-with-client-library-J2HHEFIKRSP5J5SH4BYKZ7MKYZEJFGDHIM4IBDJ7MCBN24XYA53A/venv/lib/python3.4/site-packages/google/cloud/storage/client.py", line 53, in __init__
    http=http)
  File "/var/lib/jenkins/workspace/d-files-with-client-library-J2HHEFIKRSP5J5SH4BYKZ7MKYZEJFGDHIM4IBDJ7MCBN24XYA53A/venv/lib/python3.4/site-packages/google/cloud/client.py", line 185, in __init__
    _ClientProjectMixin.__init__(self, project=project)
  File "/var/lib/jenkins/workspace/d-files-with-client-library-J2HHEFIKRSP5J5SH4BYKZ7MKYZEJFGDHIM4IBDJ7MCBN24XYA53A/venv/lib/python3.4/site-packages/google/cloud/client.py", line 141, in __init__
    project = self._determine_default(project)
  File "/var/lib/jenkins/workspace/d-files-with-client-library-J2HHEFIKRSP5J5SH4BYKZ7MKYZEJFGDHIM4IBDJ7MCBN24XYA53A/venv/lib/python3.4/site-packages/google/cloud/client.py", line 154, in _determine_default
    return _determine_default_project(project)
  File "/var/lib/jenkins/workspace/d-files-with-client-library-J2HHEFIKRSP5J5SH4BYKZ7MKYZEJFGDHIM4IBDJ7MCBN24XYA53A/venv/lib/python3.4/site-packages/google/cloud/_helpers.py", line 312, in _determine_default_project
    project = _default_service_project_id()
  File "/var/lib/jenkins/workspace/d-files-with-client-library-J2HHEFIKRSP5J5SH4BYKZ7MKYZEJFGDHIM4IBDJ7MCBN24XYA53A/venv/lib/python3.4/site-packages/google/cloud/_helpers.py", line 245, in _default_service_project_id
    return config.get(_GCLOUD_CONFIG_SECTION, _GCLOUD_CONFIG_KEY)
  File "/usr/lib/python3.4/configparser.py", line 765, in get
    raise NoOptionError(option, section)

[crestenstclair]

The .json file DOES exist, and the user does have access to it.

[daspecster]

Thanks for reporting!

@csaintc, it looks like the key file is missing the project ID somehow?
If you look at the file, can you confirm that the project ID is in your .json file?
The actual key in the JSON file is project_id.

[crestenstclair]

yeah, I have the project_id in my keyfile. It's fresh from gcloud

[theacodes]

@daspecster @csaintc older key files may be missing the project id, so it's possible. If it is, you can just generate a new key file for the same service account from the API / Cloud Console.

[crestenstclair]

I'll go a head and re-generate my keyfile for a sanity check

[crestenstclair]

Went ahead and did so, same error.

[daspecster]

@csaintc, is this on google app engine?

[crestenstclair]

@daspecster No, it's on a Google Compute Engine instance running Jenkins. This is a service account keyfile.json that I upload to the box in order to authenticate with gcloud, so i can upload my compiled front-end code to a CDN.

[daspecster]

@csaintc, have you exported the GOOGLE_APPLICATION_CREDENTIALS environment variable?

$ export GOOGLE_APPLICATION_CREDENTIALS=/path/to/credentials.json

It appears to not have found that, and then is continuing on to use the gcloud cli configuration project which also doesn't exist. I think we should probably throw a better error for this.

[crestenstclair]

@daspecster Setting the environment variable worked!

I was under the impression that from_service_account_json would use the path I provided. (or maybe that's the bug )

[daspecster]

Great!

Yeah it should, I believe, get the project with from_service_account_json.
I'll see if I can reproduce that.

[theacodes]

@daspecster don't sink too much time into it, google-auth will handle project ID detection when it's ready.

[daspecster]

@jonparrott, ah good call.

Incase this is needed in the future, I think the issue is around here somewhere.

@csaintc, I'm going to close this for now. I'm glad you're up and running!