Google-cloud-python: (IAM) Service account key created with python can't be used with the client

[adamdavis40208]

1. OSx 10.14.4
2. Python version 3.7.3
3. google-cloud-iam version: 0.1.0

I'm trying to attempt to create a service account, give it permissions, and then use it, all at once.

The issue that I'm running into is that generating the service account json, and then loading that json seem to be two differing formats.

There are two formats to the service account file.

Generating one with the python client, and then trying to re-use that with the SDK produces this error:

ValueError: Service account info was not in the expected format, missing fields token_uri, client_email.

The auth documentation page says:

Because the formatting differs between each method, it's easiest to generate a key using the same method you plan to use when making future API calls. For example, if you're using gcloud, also generate your key using gcloud.

But that doesn't seem to be true.

Is there a way you can generate a service account json in the python SDK and then use that in the python SDK?

Sample code:

    from google.oauth2 import service_account
    iam_service = googleapiclient.discovery.build('iam', 'v1')
    """Creates a key for a service account."""
    key = iam_service.projects().serviceAccounts().keys().create(
        name='projects/-/serviceAccounts/' + service_account_email, body={}
    ).execute()

    logger.debug('Created key: ' + key['name'])
    with open("service_account.json", 'w') as key_file:
        key_file.write(json.dumps(key)) 

    new_svc_accounts_creds = service_account.Credentials.from_service_account_file(
            filename="service_account.json",
            scopes=['https://www.googleapis.com/auth/devstorage.full_control'])

[tseaver]

First, we don't (yet) have support for the full IAM API here, which is why your example has to use the older googleapis wrapper to invoke projects.serviceAccounts.keys.create: the issue is therefore off-topic for this repository.

Looking at those docs: the payload returned by create is a ServiceAccountKey resource, which is not the same JSON repr as a normal service account file.

To get the "JSON service account file" format, you need to pass the privateKeyType field in the body of your request, e.g. (untested):

    iam_service = googleapiclient.discovery.build('iam', 'v1')
    request_body = {
        'privateKeyType': 'TYPE_GOOGLE_CREDENTIALS_FILE', # might need to be 3?
    }
    key = iam_service.projects().serviceAccounts().keys().create(
        name='projects/-/serviceAccounts/' + service_account_email,
        body=request_body
    ).execute()

And then use the privateKeyData key of the returned ServiceAccountKey resource, e.g.:

    with open("service_account.json", 'w') as key_file:
        key_file.write(json.dumps(key['privateKeyData'])) 

If that doesn't work for you, please use one of the resources on the IAM API support page to get better support for your usecase.

[adamdavis40208]

TYPE_GOOGLE_CREDENTIALS_FILE seems to be the default according to https://cloud.google.com/iam/reference/rest/v1/projects.serviceAccounts.keys#ServiceAccountPrivateKeyType

I'll put in a support request. thanks for the help!

[rsimac]

I have exact same issue where freshly created service account key is failing to get loaded into credentials with ValueError: Service account info was not in the expected format, missing fields token_uri, client_email. failure. All code and APIs python, latest.

@adamdavis40208 kindly pls if you submitted this issue 'elsewhere' do post a reference here or PM me. Thanks!

Evidence:

request=iam.projects().serviceAccounts().keys().create(name=ret['name'], 
   body={'privateKeyType':'TYPE_GOOGLE_CREDENTIALS_FILE'})

key=request.execute()


>>> print json.dumps(key, indent=4) #just to verify what we got
{
    "keyOrigin": "GOOGLE_PROVIDED", 
    "name": "goodandvalidname", 
    "validBeforeTime": "2029-06-28T15:09:59Z", 
    "privateKeyData": "datadata", 
    "privateKeyType": "TYPE_GOOGLE_CREDENTIALS_FILE", 
    "keyAlgorithm": "KEY_ALG_RSA_2048", 
    "validAfterTime": "2019-07-01T15:09:59Z"
}

>>> credentials = google.oauth2.service_account.Credentials.from_service_account_info(key)
Traceback (most recent call last):
  File "/home/user/.p2/pool/plugins/org.python.pydev.core_7.2.1.201904261721/pysrc/_pydevd_bundle/pydevd_exec.py", line 3, in Exec
    exec exp in global_vars, local_vars
  File "<console>", line 1, in <module>
  File "/home/user/.local/lib/python2.7/site-packages/google/oauth2/service_account.py", line 193, in from_service_account_info
    info, require=['client_email', 'token_uri'])
  File "/home/user/.local/lib/python2.7/site-packages/google/auth/_service_account_info.py", line 51, in from_dict
    'fields {}.'.format(', '.join(missing)))
ValueError: Service account info was not in the expected format, missing fields token_uri, client_email.

[rsimac]

Answering my own issue and probably helping others...

The 'key' we get from python APIs is NOT the 'json key' as obtained from gcloud. The dict we get from iam.projects().serviceAccounts().keys().create() contains the field privateKeyData which itself contains ENTIRE 'json key' one needs to authenticate to google cloud.

The data in this field is base64 encoded and needs decoding, and subsequently dumping to json. Below is the snippet from functional code, demonstrating the credentials are loaded back from such key:

request=iam.projects().serviceAccounts().keys().create(name=ret['name'], 
   body={'privateKeyType':'TYPE_GOOGLE_CREDENTIALS_FILE'})

key=request.execute()

key=base64.decodestring(key['privateKeyData'])
key=json.loads(key)

credentials = google.oauth2.service_account.Credentials.from_service_account_info(key)

I figured this out by stepping thru gcloud service account key creating, line by line, using python debugger. Hope this helps others.

[adamdavis40208]

I went with a gcloud subprocess. I'm not sure about the status of the request, I'll reach out to my gcp TAM