Google-cloud-python: Unable to upload empty file?

[danqing]

macOS 10.12.6, Python 3.6.2, google-cloud-storage 1.2.0.

It seems that if a file is empty, I cannot upload it:

blob = storage.Blob(dest_filename, bucket, chunk_size=262144)
blob.upload_from_filename(src_filename, content_type=content_type)

Error:

File "storage.py", line 88, in upload_from_file
    blob.upload_from_filename(src_filename, content_type=content_type)
File "app/env/lib/python3.6/site-packages/google/cloud/storage/blob.py", line 934, in upload_from_filename
    size=total_bytes)
File "app/env/lib/python3.6/site-packages/google/cloud/storage/blob.py", line 891, in upload_from_file
    client, file_obj, content_type, size, num_retries)
File "app/env/lib/python3.6/site-packages/google/cloud/storage/blob.py", line 818, in _do_upload
    client, stream, content_type, size, num_retries)
File "app/env/lib/python3.6/site-packages/google/cloud/storage/blob.py", line 771, in _do_resumable_upload
    response = upload.transmit_next_chunk(transport)
File "app/env/lib/python3.6/site-packages/google/resumable_media/requests/upload.py", line 392, in transmit_next_chunk
    method, url, payload, headers = self._prepare_request()
File "app/env/lib/python3.6/site-packages/google/resumable_media/_upload.py", line 531, in _prepare_request
    self._stream, self._chunk_size, self._total_bytes)
File "app/env/lib/python3.6/site-packages/google/resumable_media/_upload.py", line 805, in get_next_chunk
    u'Stream is already exhausted. There is no content remaining.')
ValueError: Stream is already exhausted. There is no content remaining.

When uploading a folder, it'd be ideal if I don't need to check whether a file is empty before uploading. Or am I doing something wrong?

Thanks!

[tseaver]

@danqing Thanks for the report! I believe this is a bug in https://github.com/GoogleCloudPlatform/google-resumable-media-python/, maybe due to the problematic behavior of read (returning an empty string at EOF).

@dhermes Can you clarify?

[danqing]

Friendly ping @dhermes :)

[lukesneeringer]

@dhermes I filed GoogleCloudPlatform/google-resumable-media-python#25 to fix this, but need guidance on whether it is correct.

[dhermes]

I believe GoogleCloudPlatform/google-resumable-media-python#25 is ready to merge.

@danqing The fix may not lead to immedate releases of google-resumable-media==0.2.3 (and possibly google-cloud-storage==1.3.1), but in the meanwhile, if you don't set the chunk size, then a MultipartUpload will be used, which has no issue sending an empty file.

[danqing]

I see. Thanks @dhermes ! I've already implemented a workaround - will wait until 0.2.3 to get rid of it :)

[dhermes]

Closing this based on https://github.com/GoogleCloudPlatform/google-resumable-media-python/releases/tag/0.2.3

[dhermes]

Blerg, the release build https://circleci.com/gh/GoogleCloudPlatform/google-resumable-media-python/189 collided with a different build https://circleci.com/gh/GoogleCloudPlatform/google-resumable-media-python/188 and made some flaky failures.

Hopefully https://circleci.com/gh/GoogleCloudPlatform/google-resumable-media-python/190 will actually succeed and push the release to PyPI

[dhermes]

https://pypi.org/project/google-cloud-storage/1.3.1/ exists now (in addition to https://pypi.org/project/google-resumable-media/0.2.3/)