Gsutil: cp command hangs when uploading large files

Created on 23 May 2017 · 4Comments · Source: GoogleCloudPlatform/gsutil

/usr/local/bin/gsutil -o GSUtil:parallel_composite_upload_threshold=10M -D cp gs://google-bucket/myfile.gz s3://s3-bucket/s3dir/

$ gsutil version -l
gsutil version: 4.26
checksum: PACKAGED_GSUTIL_INSTALLS_DO_NOT_HAVE_CHECKSUMS (!= 7dec1285a94a82cf663408e343b2dd66)
boto version: 2.46.1
python version: 2.7.12
OS: Linux xxx.amzn1.x86_64
multiprocessing available: True
using cloud sdk: False
config path(s): /home/myhome/.boto
gsutil path: /usr/local/bin/gsutil
compiled crcmod: True
installed via package manager: True
editable install: False

Source

tispratik

Most helpful comment

Hello,
I'm trying to rsync from gcs to s3 buckets but some files are bigger than 5 GiB.

I receive this error: "exceeds the maximum gsutil-supported size for an S3 upload. S3 objects greater than 5 GiB in size require multipart uploads, which gsutil does not support."

As you can see here https://docs.aws.amazon.com/AmazonS3/latest/dev/qfacts.html there is a limitation in S3 where you have to use multipart upload for files bigger than 5 GiB.

Please help me, how can I fix that?

Any help will be really appreciated!

Best Regards
Fabio Rigato

fabio-rigato on 20 Feb 2020

👍3

All 4 comments

The file in quesion myfile.gz is about 600MB, the parameter does seem to split the bytes while fetching from google cloud storage but does not split when uploading to amazon s3.

This works perfectly fine for tiny files.

tispratik on 23 May 2017

Seems like we will need to make use of S3s multi-part API for uploads. Hadoop does this as well using the parameter fs.s3n.multipart.uploads.enabled.

http://docs.aws.amazon.com/AmazonS3/latest/dev/uploadobjusingmpu.html

tispratik on 23 May 2017

I am experiencing the same problem with infinite rsync hanging while copying files around 50GB

hagen1778 on 7 Aug 2017