Gsutil: Have gsutil verify google access key and secret provided to `gsutil config -a`

[tedsta]

TLDR it'd be nice if gsutil config -a would verify the google access key and secret before writing them to the .boto file.

gsutil config gave me the following error message even though I had already authenticated using gcloud auth login. (somewhat unrelated, perhaps I should open a separate issue?)

CommandException: OAuth2 is the preferred authentication mechanism with the Cloud SDK. Run "gcloud auth login" to configure authentication, unless you want to authenticate with an HMAC access key and secret, in which case run "gsutil config -a".

Anyway, I was able to work around it with gsutil config -a. I copied and pasted my credentials from the browser and proceeded to configure my .boto file. Then I tried to run some code that made use of gsutil and got some cryptic error messages about authentication. It took me a while to figure it out, but it turns out I accidentally copied a little more than just the google secret key from the browser so there was some garbage at the end of gs_secret_access_key.

[thobrla]

The original message is working as intended; gcloud based installs configure credentials via gcloud auth login and you should only run gsutil config if you want to configure HMAC credentials (-a flag) or generate a generic .boto configuration file (-n flag).

The challenge here is that gsutil config -a doesn't have any information about the resources you intend to access with this key, nor does it have your instruction to access a specific resource. Even if we added a mechanism to do this , presumably you would get a similar error to the one that you saw when you actually tried to use the invalid token.

So perhaps we could improve the error message that you saw? What about it did you find confusing?

[tedsta]

Oh alright. I guess we'd better update our README then! (PerfKitBenchmarker)

The error messages could be improved on our end - you're right. (Something like "Please check gs_access_key_id and gs_secret_access_key in your .boto file."). And something like "~/.boto file not found. Did you run gsutil config -a?" rather than a traceback when an exception gets raised because the .boto file is not found. I guess the confusing part was that I assumed I copied over my credentials properly since the gsutil command completed successfully.

[thobrla]

I was referring to improving gsutil's error messages - were those confusing?

Ensuring you copied credentials properly is challenging for the reasons I previously described. That being said, I'm open to suggestions there if you have any.

[thobrla]

Closing this out - please reopen if you have suggestions as to how to improve gsutil's error message.

[ceefour]

GS Python Library Documentation here https://cloud.google.com/storage/docs/xml-api/gspythonlibrary#credentials says :

Configure the .boto file with the service account. You can do this with gsutil:
gsutil config -e
The command will prompt you for the service account email address and the location of the service account private key (.p12). Be sure to have the private key on the computer where you are running the gsutil command.

But following that results in error:

>gsutil config -e
CommandException: OAuth2 is the preferred authentication mechanism with the Cloud SDK. Run "gcloud auth login" to configure authentication, unless you want to authenticate with an HMAC access key and secret, in which case run "gsutil config -a".

Either the documentation is wrong or I'm wrong. :( Just how many authentication options are there, I'm really confused. Googling the problem only leads to this page, which I'm not sure whether it's correct or not.

I've tried gsutil config -n as in the above comments, but running my app still:

> python .\gs2cl.py
Traceback (most recent call last):
  File ".\gs2cl.py", line 40, in <module>
    list_gs()
  File ".\gs2cl.py", line 30, in list_gs
    for obj in uri.get_bucket():
  File "C:\Program Files\Anaconda2\lib\site-packages\boto\storage_uri.py", line 181, in get_bucket
    conn = self.connect()
  File "C:\Program Files\Anaconda2\lib\site-packages\boto\storage_uri.py", line 140, in connect
    **connection_args)
  File "C:\Program Files\Anaconda2\lib\site-packages\boto\gs\connection.py", line 47, in __init__
    suppress_consec_slashes=suppress_consec_slashes)
  File "C:\Program Files\Anaconda2\lib\site-packages\boto\s3\connection.py", line 191, in __init__
    validate_certs=validate_certs, profile_name=profile_name)
  File "C:\Program Files\Anaconda2\lib\site-packages\boto\connection.py", line 569, in __init__
    host, config, self.provider, self._required_auth_capability())
  File "C:\Program Files\Anaconda2\lib\site-packages\boto\auth.py", line 991, in get_auth_handler
    'Check your credentials' % (len(names), str(names)))
boto.exception.NoAuthHandlerFound: No handler was ready to authenticate. 3 handlers were checked. ['OAuth2Auth', 'OAuth2ServiceAccountAuth', 'HmacAuthV1Handler'] Check your credentials

When I open the (generic) .boto file, it says:

# Google OAuth2 credentials are managed by the Cloud SDK and
# do not need to be present in this file.

So one documentation says you have to configure .boto file, while the .boto file itself says credentials do not need to be present in this file. Who to trust? And what does it mean by "managed by the Cloud SDK"? I just want my simple Python script to access GS, this authentication is way to complex... The "access key + secret access key" of the old days is way easier.

I'm not the only one confused : https://stackoverflow.com/questions/38398066/gsutil-config-a-which-key-to-use

[ceefour]

At last, it works. Thanks to: https://stackoverflow.com/a/38399516/122441 , if someone gets here because same fate as me, here's how:

1. Go to https://console.cloud.google.com/storage/settings > Interoperability and create your HMAC Key. it's called HMAC, the magic word.
2. Run gsutil config -a and input thing you get from step 1. All done! :)

Why these two simple steps are not available in the docs is beyond me ...

[mhoug89]

gsutil help config shows that the -n option generates a .boto file without any auth configured, so it's consistent with the error you were seeing.

Also, it looks like you're using a gsutil installation that came packaged with gcloud. I also assume you're not using the latest version (4.27), as the error message improved in the last release (see e43b040). To generate a boto file that uses p12 credentials, you can run gcloud config set pass_credentials_to_gsutil false, then run gsutil config -e, followed by gcloud config set pass_credentials_to_gsutil true to make gcloud's auth magic work normally again.

On that note, you may want to create your boto file somewhere other than the default location (~/.boto), as gsutil will try to pull in your auth methods from this file in addition to the OAuth2 configuration gcloud sends. If you already have gsutil set up to use OAuth2 user credentials via running gcloud auth login (this is the norm), then you configure service account keyfile auth (p12 or json) in your newly created boto file, you'll likely get an error like this next time you run gsutil:

CommandException: You have multiple types of configured credentials (['Oauth 2.0 User Account', 'OAuth 2.0 Service Account']), which is not supported. One common way this happens is if you run gsutil config to create credentials and later run gcloud auth, and create a second set of credentials. Your boto config path is: ['/home/USER/.boto', '/home/USER/.config/gcloud/legacy_credentials/[email protected]/.boto']. For more help, see "gsutil help creds".

This can be avoided by just using HMAC credentials as you mentioned above, OR:
Creating your boto file in another path via gsutil config -e -o /path/to/new/botofile, and telling boto it should look there when running your application by setting the BOTO_CONFIG environment variable to refer to the file's location -- just make sure you unset the environment variable if you're going to run gsutil afterward, so that it doesn't pick up your additional auth configs. In Bash, you could set the environment variable temporarily for one command like BOTO_CONFIG=/path/to/new/botofile ./gs2cl.py. But according to this thread, Windows makes it a bit tougher -- you have to explicitly set the var, run your command, and unset the var to accomplish the same thing (e.g. set BOTO_CONFIG=/path/to/new/botofile && ./gs2cl.py && set BOTO_CONFIG=).

...also, point taken - the docs should be updated to point all this stuff out.

[ceefour]

Thanks for the explanation. Honestly it feels like a crash lesson in cryptography.

IMHO the beginner quickstart should just let the developer use HMAC in 2 simple steps (without even downloading gsutil!), but also put "If you want more advanced authentication mechanism, read on..."