Aws-sdk-java: TransferManager MultiPart Upload From InputStream

[joeyvmason]

If I perform a multipart upload using TransferManager with InputStream and ContentLength provided, only 1 thread is used. Is this expected behavior? It seems that if we know the content length, we should be able to upload in parallel. This forces users to write InputStreams to disk if they wish to upload to S3 as multipart upload.

    public void store(String bucket, String key, InputStream inputStream, long contentLength) {
        ObjectMetadata objectMetadata = new ObjectMetadata();
        objectMetadata.setContentLength(contentLength);
        PutObjectRequest putObjectRequest = new PutObjectRequest(bucket, key, inputStream, objectMetadata);
        transferManager.upload(putObjectRequest); // Only 1 thread from executor is used
    }

public void store(String bucket, String key, File file) {
        ObjectMetadata objectMetadata = new ObjectMetadata();
        objectMetadata.setContentLength(contentLength);
        PutObjectRequest putObjectRequest = new PutObjectRequest(bucket, key, file);
        transferManager.upload(putObjectRequest); // Multiple threads are used
    }

[zhangzhx]

Yeah, it is an intended behavior. TransferManager will perform a multiple threads upload when the underlying file size is greater than the threshold (16MB by default) and it is a file instead of a stream.

https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-s3/src/main/java/com/amazonaws/services/s3/transfer/internal/TransferManagerUtils.java#L71

[zhangzhx]

Hi, do you still have questions for this? I'm closing this issue for now, and please feel free to reopen if you have further questions.

[jwsohn]

Is there any workaround to enable multiple thread upload for inputstream? (i.e. using PutObjectRequest(bucket, key, inputStream, objectMetadata)) The length of the entire inputstream is known and stored in objectMetadata.

[dagnir]

@jwsohn Sorry, this isn't possible for InputStream. Parallel upload would need to be able to read from different segments in the stream concurrently, i.e. we would need random access to the data.

[joeyvmason]

@dagnir Wouldn't this be possible if a buffer were used?

[jwsohn]

Thanks a lot, dgnir.

[Sukesh-Alluri]

Ran into this issue, only to find that library doesn't process parallel streams as the document says unless it's from a file :(

It will be nice incorporate some streaming ideas with already implemented back pressure techniques for this so that we can parallelise uploads/downloads from InputStream.

[sivagiri]

How to upload a larger file greater than 10 GB using InputStream (only PutObject is possible ) in java

[qualityking]

I have the same issue this is almost 2 years old thread, I have the very same requirement, where we are hosting a service layer internally and reading chunked streams form object publishers at our server layer.
now at our server, we do not want to hold the entire stream buffered in memory also wants to avoid DiskIO, so we want to push this stream directly to S3
but this operation is very very slow compared with direct file upload instead of stream.

we have also noticed, it is uploading only 12k in one go while waiting for the whole stream (content length is known) 50mb file is take more than a minute

[sivagiri]

Yes but i wonder how others do for example git hub is there any standard way u find to upload a large file if so could u help me am much waited for this

[gabhishekprdxn]

I have the same issue this is almost 2 years old thread, I have the very same requirement, where we are hosting a service layer internally and reading chunked streams form object publishers at our server layer.
now at our server, we do not want to hold the entire stream buffered in memory also wants to avoid DiskIO, so we want to push this stream directly to S3
but this operation is very very slow compared with direct file upload instead of stream.

we have also noticed, it is uploading only 12k in one go while waiting for the whole stream (content length is known) 50mb file is take more than a minute

Is there any solution for this?

[igrishaev]

Got running into the same trouble; still actual.