Google-cloud-python: Document how to use Service Accounts with dev_appserver

[dhermes]

It can be done via

$ dev_appserver.py \
> --appidentity_email_address ${EMAIL_ADDRESS} \
> --appidentity_private_key_path ${PATH_TO_KEY} \
> ${APPLICATION_DIR}

This will make implicit credentials "just work" on the dev appserver for GAE.

[dhermes]

From dev_appserver.py --help

Application Identity:
  --appidentity_email_address APPIDENTITY_EMAIL_ADDRESS
                        email address associated with a service account that
                        has a downloadable key. May be None for no local
                        application identity. (default: None)
  --appidentity_private_key_path APPIDENTITY_PRIVATE_KEY_PATH
                        path to private key file associated with service
                        account (.pem format). Must be set if
                        appidentity_email_address is set. (default: None)

ISTM that .pem is not necessary, but if it is, that is quite alright

[dhermes]

Played around a bit with this and the dev_appserver support is pretty bad. It doesn't support JSON keyfiles or P12 keyfiles, so a user has manually take one of them and extract the PEM contents of the key. Even still, after doing that, it only supports PEM PKCS#1 keys (BEGIN RSA PRIVATE KEY) while the keys in JSON keyfiles are PKCS#8 (BEGIN PRIVATE KEY).

[dhermes]

OK I was able to get it working with a JSON keyfile (H/T S/O), but I don't know if it's worth documenting since it is so broken difficult to use:

$ python
>>> import os
>>> JSON_CREDS = os.getenv('GOOGLE_APPLICATION_CREDENTIALS')
>>> JSON_CREDS
'/path/to/keyfile.json'
>>> import json
>>> with open(JSON_CREDS, 'rb') as fh:
...     key_json = json.load(fh)
...
>>> pkcs8_pem = key_json['private_key']
>>> key_json['client_email']
'[email protected]'
>>> name, ext = os.path.splitext(JSON_CREDS)
>>> PEM_FILE = name + '-PKCS8.pem'
>>> with open(PEM_FILE, 'wb') as fh:
...     fh.write(pkcs8_pem)
...
>>>
$ # Then convert to PKCS1
$ openssl rsa -in /path/to/keyfile-PKCS8.pem -out /path/to/keyfile-PKCS1.pem

and after doing this, the PKCS#1 PEM-file can be used:

$ dev_appserver.py \
> [email protected] \
> --appidentity_private_key_path=/path/to/keyfile-PKCS1.pem \
> ${APPLICATION_DIR}

[dhermes]

@jgeewax Do you think this is worth adding to our auth doc or should I just give up on it and close out this issue?

[voscausa]

Yes, because you do not have to store the key in your project. You can also use this setup to create signed urls to access Google Cloud storage in the development server using app_identity.sign blob() and the app default service account, without modifying your application. This default service account cannot be used as an ouath2client service account, because it does not have a client_id.

[dhermes]

@jgeewax Can you weigh in here?

[dhermes]

@jgeewax Bump

[dhermes]

@jgeewax Bump

[jgeewax]

@dhermes , Can you help me understand why auth is coming up in the local-development sense?

Is it that the app server needs to make authenticated requests elsewhere? Or is this just to simulate that auth is working correctly in a local environment? If the latter, it seems weird the server accepts the private key (wouldn't the server hang onto the public key, and validate the client with that?)

Either way, I think we need to document the "traditional ways" to manipulate private keys into different formats simply because people don't know how to do this (even those of us that are familiar still have to look it up...). This gets particularly nasty with Windows users who I suspect have issues with this...

Then it definitely doesn't hurt to explain this use case - the question is more about where it'd go. I'd expect the first piece to live under the auth article in gcloud-common, and the other one to be in some "local development" article.

[dhermes]

• People run App Engine apps in production and dev_appserver
• Using gcloud when auth "just works" on App Engine, the App Identity Service is used (by oauth2client)
• In dev_appserver, most GAE services are mocked (e.g. save datastore entities to a local DB)
• By default, the App Identity Service has no mock
• A user can provide their own keys to emulate the App Identity service in the dev_appserver (via the flags --appidentity_email_address and --appidentity_private_key_path) for features like signing a blob

---

RE: Manipulate private keys, this should never be necessary. The JSON key has a very standard format (PKCS#8 keys as PEM).

Key manipulation is only necessary because key support is implemented so poorly in dev_appserver. It assumes someone took a P12 key that was converted to PKCS#1 as PEM. (P12 is a format essentially DERP-recated by Google since it is harder to work with.)

(The dev_appserver code is essentially what's in https://github.com/google/oauth2client/pull/1 and nothing else.) I'd be happy to submit a patch to dev_appserver to get legitimate support as is implemented in _pure_python_crypt (though P12 keys can only be supported by pyOpenSSL).

[lukesneeringer]

Hello,
One of the challenges of maintaining a large open source project is that sometimes, you can bite off more than you can chew. As the lead maintainer of google-cloud-python, I can definitely say that I have let the issues here pile up.

As part of trying to get things under control (as well as to empower us to provide better customer service in the future), I am declaring a "bankruptcy" of sorts on many of the old issues, especially those likely to have been addressed or made obsolete by more recent updates.

My goal is to close stale issues whose relevance or solution is no longer immediately evident, and which appear to be of lower importance. I believe in good faith that this is one of those issues, but I am scanning quickly and may occasionally be wrong. If this is an issue of high importance, please comment here and we will reconsider. If this is an issue whose solution is trivial, please consider providing a pull request.

Thank you!