Google-api-python-client: httplib2 v0.16.0 breaks the library

[lzzy12]

Recently updated httplib2 library (v0.16.0) have a breaking change causing this Exception:

httplib2.RedirectMissingLocation: Redirected but response is missing a location: header

Steps to reproduce:

1. pip install --upgrade httplib2
2. Now use the google drive library to upload a file (This is where I encountered the bug, it may break while using other stuffs a well)

It should either force to use 0.15.0 version of httplib2 or adapt the breaking changes introduced in the new version.

A temporary fix for those using the client library would be to add httplib2==0.15.0 in requirements,txt of the project

[busunkim96]

@lzzy12 Thank you for the report! Would you be able to provide a bit of code demonstrating the issue?

[HarlanH]

A set of old Airflow tasks failed last night due to this error, it looks like. Should be able to confirm tomorrow that pinning the library is a workaround.

Stack trace looks like this:

[busunkim96]

It looks like there were some changes to how redirects are handled in 0.16.0. See https://github.com/httplib2/httplib2/commit/45441b2f0ea6180853b087f77169f44ce726a508#diff-c6669c781a2dee1b2d2671cab4e21c66. I'll have to go through more carefully to figure out what's causing the error.

For those of you who are experiencing issues, are you all using oauth2client (the deprecated auth client)? Has anyone experienced this using google-auth and google-auth-oauthlib?

[lzzy12]

Has anyone experienced this using google-auth and google-auth-oauthlib?

Yes me!
I am not using the deprecated oauth2client

[busunkim96]

Sorry, one more question. It looks like @HarlanH is using python2. @lzzy12 are you also using python 2?

Edit: I see https://github.com/httplib2/httplib2/issues/156 which may be relevant.

Edit 2: It looks like the maintainers of httplib2 have come up with a workaround (allowing users to choose which redirects to follow) which will come out in the next release. It might be worth pinging the API team to see if they can add the missing header.

For now, continue to use httplib2==0.15.0.

Edit 3: The Google Drive API Team uses the 308 response as (Resume Incomplete), rather than Permanent Redirect per the RFC.

[lzzy12]

Sorry, one more question. It looks like @HarlanH is using python2. @lzzy12 are you also using python 2?

No. Python 3.8

[temoto]

httplib2 v0.17.0 is just released with API to handle this. Sorry, I couldn't quickly find where to apply the following patch in this repo, aside from samples (which should probably be modified too).

 http = httplib2.Http()
+http.redirect_codes = set(http.redirect_codes) - {308}

[temoto]

Side note: what a wonderful ironic mystery this issue number is 308 backwards.

[SaschaHeyer]

I am also experiencing the same behavior, yesterday logic was still working today not anymore.
Is there already a solution for it?

[temoto]

@SaschaHeyer yes it needs tests https://github.com/googleapis/google-api-python-client/pull/813

[mcampos-quinn]

Experienced this behavior uploading a large file that (presumably?) required a resumable upload, but with a file under 5MB the behavior does not happen.

[lzzy12]

I am also experiencing the same behavior, yesterday logic was still working today not anymore.
Is there already a solution for it?

As mentioned in OP, you can pin httplib2 to v0.15.0 as a temporary fix

[rsmbyk]

You just need to execute the .Upload() again until it is uploaded. I think it is due to incomplete upload of a large file just as @mcampos-quinn said.

[temoto]

Merge #813 is proper solution, they rightfully asked for tests so it doesn't happen again. Someone please add test to that PR. It just needs to run existing upload test with particular version of dependency.

Pin httplib2==0.15.0 is temporary workaround.

@rsmbyk Upload() again is misusing API at best, I don't think it will work, please don't do that.

[rsmbyk]

@rsmbyk Upload() again is misusing API at best, I don't think it will work, please don't do that.

I had this error and tried to ignore the exception. But when I access the id of the file it said the file haven't been uploaded. I ran the .Upload() again and successfully uploaded the file. That's why I thought it was okay to do that as the file had not been uploaded to the drive yet.

[busunkim96]

Sorry about the delay folks, will get a test added to that PR and merged either today or early next week. Please continue to use 0.15.0 in the meantime.

[mengyyy]

python 3.6.9
Ubuntu 18.04 x64

---- (full traceback above) ----
File /root/gdrive.py, line 389, in upload
    result = self.files_api.create(body=file_metadata, media_body=media, fields="*", supportsTeamDrives=True).execute()
File /usr/local/lib/python3.6/dist-packages/googleapiclient/_helpers.py, line 134, in positional_wrapper
    return wrapped(*args, **kwargs)
File /usr/local/lib/python3.6/dist-packages/googleapiclient/http.py, line 862, in execute
    _, body = self.next_chunk(http=http, num_retries=num_retries)
File /usr/local/lib/python3.6/dist-packages/googleapiclient/_helpers.py, line 134, in positional_wrapper
    return wrapped(*args, **kwargs)
File /usr/local/lib/python3.6/dist-packages/googleapiclient/http.py, line 1037, in next_chunk
    self.resumable_uri, method="PUT", body=data, headers=headers
File /usr/local/lib/python3.6/dist-packages/oauth2client/transport.py, line 175, in new_request
    redirections, connection_type)
File /usr/local/lib/python3.6/dist-packages/oauth2client/transport.py, line 282, in request
    connection_type=connection_type)
File /usr/local/lib/python3.6/dist-packages/httplib2/__init__.py, line 1991, in request
    cachekey,
File /usr/local/lib/python3.6/dist-packages/httplib2/__init__.py, line 1690, in _request
    content,

RedirectMissingLocation: Redirected but the response is missing a Location: header.

[temoto]

@mengyyy pip3 install -U google-api-python-client

[mengyyy]

@mengyyy pip3 install -U google-api-python-client

root@it:~# pip3 show google-api-python-client
Name: google-api-python-client
Version: 1.8.0
Summary: Google API Client Library for Python
Home-page: http://github.com/google/google-api-python-client/
Author: Google Inc.
Author-email: None
License: Apache 2.0
Location: /usr/local/lib/python3.6/dist-packages
Requires: google-auth-httplib2, uritemplate, google-auth, httplib2, google-api-core, six

[mengyyy]

@mengyyy pip3 install -U google-api-python-client

And a new Error

---- (full traceback above) ----
File <ipython-input-1-6a45cf6e5d3b>, line 87, in get_credentials
    self.new_credentials()
File <ipython-input-1-6a45cf6e5d3b>, line 71, in new_credentials
    self.credentials = flow.step2_exchange(auth_code)
File /usr/local/lib/python3.6/dist-packages/oauth2client/_helpers.py, line 133, in positional_wrapper
    return wrapped(*args, **kwargs)
File /usr/local/lib/python3.6/dist-packages/oauth2client/client.py, line 2089, in step2_exchange
    raise FlowExchangeError(error_msg)

FlowExchangeError: invalid_requestMissing required parameter: code

[temoto]

@mengyyy likely you have old httplib2 on that system. pip3 install -U google-api-python-client httplib2

And invalid_requestMissing is just another error, it's not related here.

[mengyyy]

@mengyyy likely you have old httplib2 on that system. pip3 install -U google-api-python-client httplib2

And invalid_requestMissing is just another error, it's not related here.

I try it in another server and upte these two package but get the same result.

root@px:~# pip3 show httplib2 google-api-python-client
Name: httplib2
Version: 0.17.0
Summary: A comprehensive HTTP client library.
Home-page: https://github.com/httplib2/httplib2
Author: Joe Gregorio
Author-email: [email protected]
License: MIT
Location: /usr/local/lib/python3.6/dist-packages
Requires:
Required-by: oauth2client, google-auth-httplib2, google-api-python-client
---
Name: google-api-python-client
Version: 1.8.0
Summary: Google API Client Library for Python
Home-page: http://github.com/google/google-api-python-client/
Author: Google Inc.
Author-email: None
License: Apache 2.0
Location: /usr/local/lib/python3.6/dist-packages
Requires: uritemplate, google-api-core, six, google-auth-httplib2, google-auth, httplib2
Required-by:

[temoto]

@mengyyy sorry my last try at workaround is google-api-python-client==1.7.12 it's known to work. Otherwise you should open new issue in this repo.

[ivanistheone]

I'm facing the same issue running this sample code.

I pinned httplib2 to 0.15.0 (using pip install -U httplib2==0.15.0) and upload worked.

Output of pip freeze in the working state is

cachetools==4.1.0
certifi==2020.4.5.1
chardet==3.0.4
dill==0.2.9
future==0.16.0
gapic-google-cloud-speech-v1==0.15.3
google-api-python-client==1.7.12
google-auth==1.14.1
google-auth-httplib2==0.0.3
google-cloud-core==0.24.1
google-cloud-language==0.24.1
google-cloud-speech==0.25.1
google-cloud-storage==1.1.1
google-gax==0.15.16
google-resumable-media==0.0.2
googleapis-common-protos==1.51.0
grpcio==1.28.1
httplib2==0.15.0
idna==2.9
oauth2client==3.0.0
ply==3.8
proto-google-cloud-speech-v1==0.15.3
protobuf==3.11.3
pyasn1==0.4.8
pyasn1-modules==0.2.8
pydub==0.16.5
requests==2.23.0
rsa==4.0
six==1.10.0
uritemplate==3.0.1
urllib3==1.25.9

the doesn't work state would have been the same as above but with httplib2==0.17.3

[smrgit]

just in case this issue comes back to life -- I just ran into this same ( _RedirectMissingLocation_ ) error when using bq load

$ gcloud --version
Google Cloud SDK 297.0.0
bq 2.0.58
core 2020.06.12
gsutil 4.51

I then updated my gcloud components and the problem seemed to go away.

UPDATE: it seems that updating the SDK was not the solution -- rather copying the file that I wanted to upload into BigQuery first to a GCS bucket and then loading from there seems to have been the solution to the problem.

[RajeshwariS]

Seeing this issue on httplib2 version 0.18.1 as well (Python 3.6.8)

Pip freeze:

cachetools==4.1.1
certifi==2020.6.20
chardet==3.0.4
google-api-core==1.21.0
google-api-python-client==1.9.3
google-auth==1.18.0
google-auth-httplib2==0.0.3
googleapis-common-protos==1.52.0
httplib2==0.18.1
idna==2.10
oauth2client==4.1.3
protobuf==3.12.2
pyasn1==0.4.8
pyasn1-modules==0.2.8
PyDrive==1.3.1
pytz==2020.1
PyYAML==5.3.1
requests==2.24.0
rsa==4.6
six==1.15.0
uritemplate==3.0.1
urllib3==1.25.9

I had to downgrade to 0.15.0 seeing the above suggestions.
I see the above commit - Add compatibility with httplib2 0.16.0+ , but not sure why 0.18 (0.16.0+) is still breaking.

[pkenjora]

Looks like the error is back... I've upgraded ALL libraries to most recent:

google-api-python-client
google-auth
google-auth-oauthlib
google-auth-httplib2

[busunkim96]

@pkenjora Could you please open a new issue referencing this one? Replies to closed issues are more likely to be missed by repo maintainers.

It would also be helpful if you could show the full pip freeze (with versions).

Thanks!

[shadow-light]

New issue at https://github.com/googleapis/google-api-python-client/issues/1006

[pkenjora]

FIX: The error is caught at the wrong layer. This is actually an application layer error due to incorrect schema.

Double check your schema, for me I had an incorrect time zone format "Los_Angeles/America" instead of "America/Los_Angeles", which triggered this error.

Check your request schema!