Gsutil: Add support to service account json key authentication

[ilyavaiser]

By analogy with the gcloud utility, it's very lacking

[houglum]

Gsutil offers service account authorization. gsutil config -e will prompt for the private key file containing your service account credentials. Were you inquiring about some other form of service account auth?

[ilyavaiser]

This is unsuitable for use in scripts. When I have a file with a key. And can not use promt. The gcloud utility allows to use non-interactive authorization via a key file

[houglum]

A couple of potential workarounds:

• If you're able to use a gcloud installation of gsutil, it should take care of the service account auth for you (creating a boto file on your behalf, behind the scenes, that points to this keyfile).
• If you're using a standalone installation of gsutil, you can create your boto file beforehand. If you're running the scripts on remote VMs, you can copy both the boto file and the key file to the VM. (This is the approach I generally use.)

[ilyavaiser]

I do not need workarounds. I want gsutil to work as well as gcloud with service account authorization. This will greatly facilitate the tasks of automation.

[houglum]

We purposely make the gsutil config process interactive because of all the edge cases, the main one being that different information is necessary to create the config file depending on whether you specify a JSON or P12 keyfile. Once you have a valid config file, you can alter it and package it (alongside the keyfile you're already packaging) for use with automated scripts. And if you want to use gcloud's credential management system with gsutil, that's easily achievable by installing and configuring gcloud.

I'd argue that it's just as easy to run a gsutil config command as it is to construct a short boto file like this:

[Credentials]
gs_service_key_file = /path/to/keyfile.json

[GSUtil]
default_project_id = id-for-my-project

or to do what gcloud does under the hood for most config options -- passing them as arguments to gsutil on the command line; this has the benefit of not fiddling with creating files:

gsutil -o "Credentials:gs_service_key_file=/path/to/keyfile.json" \
    -o "default_project_id=id-for-my-project" \
    <rest of command>

[ilyavaiser]

This is a great feature, I know about it. But this is inconvenient. Why do I need another file when I want to unify with the gcloud utility?

Just add the ability to specify the key.json file as a parameter, without additional files. This will make life easier for many.

[ilyavaiser]

And it would be nice to point this case to the documentation with a concrete example. Thank you!

[mloughran]

A concrete example where this causes issues: I am using the gs_service_key_file workaround mentioned above, used within some wrapper scripts. However if a user has also configured a default user account via gcloud auth application-default login, the command will fail with this rather unhelpful error:

$ gsutil -o "Credentials:gs_service_key_file=/path/to/keyfile.json" rsync ...
Caught non-retryable exception while listing gs://<snip>/: CommandException: \
You have multiple types of configured credentials ([u'Oauth 2.0 User Account', u'OAuth 2.0 \
Service Account']), which is not supported. One common way this happens is if you run gsutil \
config to create credentials and later run gcloud auth, and create a second set of credentials. Your \
boto config path is: ['<snip>']. For more help, see "gsutil help creds".
CommandException: Caught non-retryable exception - aborting rsync

I would also argue that an --auth command line flag would be much simpler. Thanks.

[NikkyAI]

the mentioned workaround does not seem to work anymore
without any error message or debug log, it just prints the help message and stops

[AndrewJDR]

Just to bring up another reason it would be handy to have a well-supported way to do this in scripts:
I have a script that fires off many separate gsutil command executions, and each time it may use one of many different service accounts. Simply being able to specify a different service account json to the gsutil command is really the easiest way to deal with this. I'm not even sure how to do it with a boto file, since there's frequent swapping between many service accounts on these different gsutil executions, as mentioned.

The Credentials:gs_service_key_file workaround is working for now, but I'm on an older version of gsutil. It's concerning that it may stop working when I upgrade, as @NikkyAI mentioned above it no longer works :(

[SabyasachiNayak]

@NikkyAI @AndrewJDR I got the same error but after adding GSUtil to default_project_id it worked for me.
Here is the command:-

gsutil -o "Credentials:gs_service_key_file=/path/to/keyfile.json" \
-o "GSUtil:default_project_id=id-for-my-project" \