Google-api-python-client: Not possible to provide `http` parameter in `next_chunk()` with resumable upload? HttpError 400

[yohplala]

Hi,

This ticket is probably pointless, but maybe it hilights the need of a better error message?

The following code works fine with chunk size larger than 1024*1024, but using a chunksize of 1024 ends in a HttpError 400. Why is that so?

from google.oauth2 import service_account
import googleapiclient.discovery as gad
import googleapiclient.http as gah

# Run once at start.
creds = service_account.Credentials.from_service_account_file(creds).with_scopes(['https://www.googleapis.com/auth/drive'])
drive = gad.build('drive', 'v3', credentials=creds)

# Run in a loop / implement both thread-safety and caching in http object.
filename = 'my_file.txt'
cache_path='.cache'
# Using 1024 ends up in an error.
# Using 1024*1024 works fine
# Using default parameter also works fine, but is actually not a resumable upload.
media = gah.MediaFileUpload(file_name, chunksize=1024, resumable=True) # Here
file_metadata = {'name': file_name, 'parents': [folder_id]}
request = self.drive.files().create(body=file_metadata, media_body=media, fields='id')
auth_http = _auth.authorized_http(creds)
auth_http.cache = httplib2.FileCache(cache_path)
response = None
while response is None:
        status, response = request.next_chunk(num_retries=4, http=auth_http)

Here is the error message I get when chunksize=1024.

HttpError: <HttpError 400 when requesting https://www.googleapis.com/upload/drive/v3/files?fields=id&alt=json&uploadType=resumable returned "Bad Request">

Please, why is that so?

I thank you for your support.
Have a good day,
Bests,

[busunkim96]

The docs mention that the chunk size needs to be a multiple of 256KB. I think that's what's happening here? https://github.com/googleapis/google-api-python-client/tree/master/docs/media.md#resumable-media-chunked-upload

Note: Chunk size restriction: There are some chunk size restrictions based on the size of the file you are uploading. Files larger than 256 KB (256 * 1024 B) must have chunk sizes that are multiples of 256 KB. For files smaller than 256 KB, there are no restrictions. In either case, the final chunk has no limitations; you can simply transfer the remaining bytes.

This restriction doesn't seem to be mentioned in the code anywhere, which is a bit odd.

[yohplala]

Hi @busunkim96, I think you are right. I did 2 more tests:

• one with chunksize=256*1024 : works just fine
• one with chunksize=128*1024 : same error messagee: HttpError 400

Should I leave the ticket open so that maybe you implement a check directly on chunksize with a more understandable error message in case it is not a multiple of 256KB?

I let you decide.
Have a good day, and thanks a lot for your support! You're great!

[parthea]

@yohplala I'm sorry that it took a while to address this issue. Thanks for providing the sample code. I was able to run it and see that the library was not processing the error message details in the HTTP 400 response because google-api-python-client was expecting json format. I modified google-api-python-client locally to populate the non-json error details and I can see the error details is populated. I'm going to submit a PR.

googleapiclient.errors.HttpError: <HttpError 400 when requesting https://www.googleapis.com/upload/drive/v3/files?fields=id&alt=json&uploadType=resumable returned "Bad Request". Details: "Invalid request.  The number of bytes uploaded is required to be equal or greater than 262144, except for the final request (it's recommended to be the exact multiple of 262144).  The received request contained 131072 bytes, which does not meet this requirement.">