Dvc: Google Cloud Storate ― Response header must contain location

[IamGianluca]

Hi guys,

Once again, thanks for the great tool!

I've tried to upload into Google Cloud Storage my data, so that everyone in my team can download it to reproduce locally our pipeline. Unfortunately, the request failed due to a missing location header in the response.

$ dvc remote add -d upstream gs://dvc-bucket/my-path

Setting 'upstream' as a default remote.        
$ cat .dvc/config 
['remote "upstream"']
url = gs://dvc-bucket/my-path
[core]
remote = upstream             
$ dvc push                      
Preparing to upload data to 'gs://dvc-bucket/my-path'
Preparing to collect status from gs://dvc-bucket/my-path
[##############################] 100% Collecting information
[##############################] 100% Analysing status.
Warning: Failed to upload '/home/gianluca/my-project/.dvc/cache/3d/1660bb393e019d9caf935878bdb13c' to 'dvc-bucket/my-path/3d/1660bb393e019d9caf935878bdb13c': 403 POST https://www.googleapis.com/upload/storage/v1/b/dvc-bucket/o?uploadType=resumable: ('Response headers must contain header', 'location')
Warning: Failed to upload '/home/gianluca/my-project/.dvc/cache/b1/7308416bdc6ab08fb87b81856dbeb0' to 'dvc-bucket/my-path/b1/7308416bdc6ab08fb87b81856dbeb0': 403 POST https://www.googleapis.com/upload/storage/v1/b/dvc-bucket/o?uploadType=multipart: ('Request failed with status code', 403, 'Expected one of', <HTTPStatus.OK: 200>)
Warning: Failed to upload '/home/gianluca/my-project/.dvc/cache/32/9a425a7ed07e6a170d2755626ed085' to 'dvc-bucket/my-path/32/9a425a7ed07e6a170d2755626ed085': 403 POST https://www.googleapis.com/upload/storage/v1/b/dvc-bucket/o?uploadType=resumable: ('Response headers must contain header', 'location')
Warning: Failed to upload '/home/gianluca/my-project/.dvc/cache/1d/1b75fd5ca8d3efdc25177d92c66eff' to 'dvc-bucket/my-path/1d/1b75fd5ca8d3efdc25177d92c66eff': 403 POST https://www.googleapis.com/upload/storage/v1/b/dvc-bucket/o?uploadType=resumable: ('Response headers must contain header', 'location')

I have the latest version of dvc installed.

$ dvc --version
0.23.2

Please let me know if I can help debugging the issue.

[efiop]

Hm, really strange. We've had a similar error reported today in the discord, which turned out to be creds-related. Running gcloud beta auth application-default login helped in that case. It might be related to google-cloud-storage version even, we are forcing 1.13.0, but there is 1.13.2 already released. If you are using pip to install dvc, could you please try upgrading your google-cloud-storage and see if the problem still occurs? I.e. pip install --upgrade google-cloud-storage.

[IamGianluca]

Hi @efiop

I'm trying to run the command from within a Google Compute Engine virtual machine. I'm not sure if this could help with debugging the issue.

I was running google-cloud-storage==1.13.0. Upgrading to 1.13.2 with pip install --upgrade google-cloud-storage, unfortunately, didn't help. I can still replicate the issue.

I didn't run the first command because:

$ gcloud beta auth application-default login

You are running on a Google Compute Engine virtual machine.
The service credentials associated with this virtual machine
will automatically be used by Application Default
Credentials, so it is not necessary to use this command.

If you decide to proceed anyway, your user credentials may be visible
to others with access to this virtual machine. Are you sure you want
to authenticate with your personal account?

Do you want to continue (Y/n)?  n

ERROR: (gcloud.beta.auth.application-default.login) Aborted by user.

[efiop]

@IamGianluca The guy that reported a similar issue today was also running gcp vm and that command helped him. Not quite sure what is going on precisely. Investigating...

[IamGianluca]

@efiop I run the command and managed to upload the data in the Cloud Storage bucket.

$ dvc push
 Preparing to upload data to 'gs://dvc-bucket/my-path'
 Preparing to collect status from gs://dvc-bucket/my-path
 [#########                     ] 30% Collecting information/home/gianluca/.pyenv/versions/3.6.7/envs/bid36/lib/python3.6/site-packages/google/auth/_default.py:66: UserWarning: Your application has authenticated using end user credentials from Google Cloud SDK. We recomme
 nd that most server applications use service accounts instead. If your application continues to use end user credentials from Cloud SDK, you might receive a "quota exceeded" or "API not enabled" error. For more information about service accounts, see https://cloud.google.com$
 docs/authentication/
   warnings.warn(_CLOUD_SDK_CREDENTIALS_WARNING)
 /home/gianluca/.pyenv/versions/3.6.7/envs/bid36/lib/python3.6/site-packages/google/auth/_default.py:66: UserWarning: Your application has authenticated using end user credentials from Google Cloud SDK. We recommend that most server applications use service accounts inste
 ad. If your application continues to use end user credentials from Cloud SDK, you might receive a "quota exceeded" or "API not enabled" error. For more information about service accounts, see https://cloud.google.com/docs/authentication/
   warnings.warn(_CLOUD_SDK_CREDENTIALS_WARNING)
 [##############################] 100% Collecting information
 [##############################] 100% Analysing status.
 /home/gianluca/.pyenv/versions/3.6.7/envs/bid36/lib/python3.6/site-packages/google/auth/_default.py:66: UserWarning: Your application has authenticated using end user credentials from Google Cloud SDK. We recommend that most server applications use service accounts inste
 ad. If your application continues to use end user credentials from Cloud SDK, you might receive a "quota exceeded" or "API not enabled" error. For more information about service accounts, see https://cloud.google.com/docs/authentication/
   warnings.warn(_CLOUD_SDK_CREDENTIALS_WARNING)
 /home/gianluca/.pyenv/versions/3.6.7/envs/bid36/lib/python3.6/site-packages/google/auth/_default.py:66: UserWarning: Your application has authenticated using end user credentials from Google Cloud SDK. We recommend that most server applications use service accounts inste
 ad. If your application continues to use end user credentials from Cloud SDK, you might receive a "quota exceeded" or "API not enabled" error. For more information about service accounts, see https://cloud.google.com/docs/authentication/
   warnings.warn(_CLOUD_SDK_CREDENTIALS_WARNING)
 /home/gianluca/.pyenv/versions/3.6.7/envs/bid36/lib/python3.6/site-packages/google/auth/_default.py:66: UserWarning: Your application has authenticated using end user credentials from Google Cloud SDK. We recommend that most server applications use service accounts inste
 ad. If your application continues to use end user credentials from Cloud SDK, you might receive a "quota exceeded" or "API not enabled" error. For more information about service accounts, see https://cloud.google.com/docs/authentication/
   warnings.warn(_CLOUD_SDK_CREDENTIALS_WARNING)
 /home/gianluca/.pyenv/versions/3.6.7/envs/bid36/lib/python3.6/site-packages/google/auth/_default.py:66: UserWarning: Your application has authenticated using end user credentials from Google Cloud SDK. We recommend that most server applications use service accounts inste
 ad. If your application continues to use end user credentials from Cloud SDK, you might receive a "quota exceeded" or "API not enabled" error. For more information about service accounts, see https://cloud.google.com/docs/authentication/
   warnings.warn(_CLOUD_SDK_CREDENTIALS_WARNING)
 /home/gianluca/.pyenv/versions/3.6.7/envs/bid36/lib/python3.6/site-packages/google/auth/_default.py:66: UserWarning: Your application has authenticated using end user credentials from Google Cloud SDK. We recommend that most server applications use service accounts inste
 ad. If your application continues to use end user credentials from Cloud SDK, you might receive a "quota exceeded" or "API not enabled" error. For more information about service accounts, see https://cloud.google.com/docs/authentication/
   warnings.warn(_CLOUD_SDK_CREDENTIALS_WARNING)
 /home/gianluca/.pyenv/versions/3.6.7/envs/bid36/lib/python3.6/site-packages/google/auth/_default.py:66: UserWarning: Your application has authenticated using end user credentials from Google Cloud SDK. We recommend that most server applications use service accounts inste
 ad. If your application continues to use end user credentials from Cloud SDK, you might receive a "quota exceeded" or "API not enabled" error. For more information about service accounts, see https://cloud.google.com/docs/authentication/
   warnings.warn(_CLOUD_SDK_CREDENTIALS_WARNING)
 /home/gianluca/.pyenv/versions/3.6.7/envs/bid36/lib/python3.6/site-packages/google/auth/_default.py:66: UserWarning: Your application has authenticated using end user credentials from Google Cloud SDK. We recommend that most server applications use service accounts inste
 ad. If your application continues to use end user credentials from Cloud SDK, you might receive a "quota exceeded" or "API not enabled" error. For more information about service accounts, see https://cloud.google.com/docs/authentication/
   warnings.warn(_CLOUD_SDK_CREDENTIALS_WARNING)
 (1/4): [##############################] 100% models/stage1.joblib
 (2/4): [##############################] 100% data/test.parquet
 (3/4): [##############################] 100% data/train.parquet
 (4/4): [##############################] 100% data/performance.parquet

How the warnings ahead it seems Google doesn't like that I'm using my personal user to upload to the Cloud Storage bucket, as opposed to use a service account.

I've also tried to download that data using dvc pull from another machine, but it fails. Once again, it seems to be an issue with not using a service account.

$ dvc pull
Preparing to download data from 'gs://dvc-bucket/my-path'
Preparing to collect status from gs://dvc-bucket/my-path
[#########                     ] 30% Collecting information/home/gianluca/.pyenv/versions/3.6.8/envs/my-path/lib/python3.6/site-packages/google/auth/_default.py:66: UserWarning: Your application has authenticated using end user credentials from Google Cloud SDK. We recommend that most server applications use service accounts instead. If your application continues to use end user credentials from Cloud SDK, you might receive a "quota exceeded" or "API not enabled" error. For more information about service accounts, see https://cloud.google.com/docs/authentication/
  warnings.warn(_CLOUD_SDK_CREDENTIALS_WARNING)
Error: Failed to pull data from the cloud: Project was not passed and could not be determined from the environment.

Having any troubles? Hit us up at dvc.org/support, we are always happy to help!

[efiop]

Hm, that is interesting. It used to work correctly, right? Looks like something changed on gcp auth side.

[IamGianluca]

I personally haven't tried it before using a Cloud Storage bucket, so I cannot confirm it :-)

[IamGianluca]

I can confirm that even after updating DVC to version 0.24.2 the issue is still ongoing. I still see the following error message:

Error: failed to pull data from the cloud - Project was not passed and could not be determined from the environment.

[efiop]

Hi @IamGianluca !

Sorry for the delay. Have you tried configuring projectname for your gs remote?

[efiop]

@IamGianluca Do you still experience the same issue?

[efiop]

Closing as stale. Please feel free to reopen if you are still experiencing this issue.

[luxedo]

I'm having the same issue

dvc --version           
0.35.7

Running dvc push -v I get the following output:
https://gist.github.com/luxedo/ad000abdbb4e5a717e7da15e1e0b65d3

I think that it has something to do with GCP:
` ERROR: failed to upload '/home/john/imagem-mariposas/.dvc/cache/08/4536cb9f552b642099a71351310ae2.dir' to 'tarvos-datasets/08/4536cb9f552b642099a71351310ae2.dir' - 403 POST https://www.googleapis.com/upload/storage/v1/b/tarvos-datasets/o?uploadType=multipart: ('Request failed with status code', 403, 'Expected one of', <HTTPStatus.OK: 200>)

[efiop]

Related discord discussion https://discordapp.com/channels/485586884165107732/485596304961962003/567428611460694046

[luxedo]

@efiop suggested the following workaround and it worked:

gcloud beta auth application-default login