Aws-sdk-cpp: Process large files on S3 in chunks or in stream

[stefanmoro]

Hi all!

I'm currently working on an application which will serve binary content to clients based on large files stored on S3. These files can be up to several GiB in size. The application will perform processing of these files on-the-fly as a client request comes in.

In this scenario, time to first byte for the client is really important, more important than total throughput. The way I'd like to implement this is to fetch data from S3 and process it as it comes in, either by relatively small chunks or via a std:istream style interface.

Is there a recommended way to accomplish this using the aws c++ sdk?

Here are a couple of the things I've tried so far:

I've played around with both the S3Client::GetObject functionality and the TransferManager but I can't really figure out how to do what I want to do.

Here's a snippet where I use S3Client::GetObject

Aws::SDKOptions options;
Aws::InitAPI(options);

Aws::S3::S3Client s3_client;
Aws::S3::Model::GetObjectRequest objectRequest;
objectRequest.WithBucket(S3_BKT).WithKey(S3_KEY);
auto getObjectOutcome = s3_client.GetObject(objectRequest);

if (getObjectOutcome.IsSuccess()) {
  processAndSendToClient(getObjectOutcome.GetResult().GetBody())
}

Aws::ShutdownAPI(options);

The problem with this approach is that s3_client.GetObject(objectRequest); is blocking until entire file is downloaded which means that processing (and sending response to client) can't start until after entire file is downloaded.

In the TransferManager I played around with using the DownloadProgressCallback, but the TransferHandle received in the callback doesn't seem to allow me to access the actual bytes transferred, only the number of bytes transferred. I've also looked at the TransferManager::DownloadFile function which takes a CreateDownloadStreamCallback for writing the data to. I guess this function could potentially be used, given that I write my own iostream implementation that would support being written to by the download thread and read from by the processing (main) thread but it feels like there should be a simpler solution.

Thanks
Stefan

[jt70471]

Take a look at the Range request header referenced here:

http://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectGET.html

Basically, you can get the object in any size chunks you wish.

[stefanmoro]

Hi Jeff and thanks for the feedback.

I forgot to mention that but I had a look at, and played around with, byte range requests. However, I was hoping to solve this without using byte ranges since each byte range request also involves the overhead of initializing the transfer. For small chunk this overhead will be non-negligible.

I made a small test where I first fetched 1 KiB, then 2 KiB and then 3 KiB using ranges. The test showed that there was very little time difference between the three requests (~140ms each on my local computer). This means that if I would want to process the data in small chunks, I would spend a vast majority of the time initializing each byte range transfer.

Any other ideas?

Regards Stefan

[jt70471]

Stefan, with such small byte ranges, I'm not surprised that the connection overhead is making the end-to-end response times so similar.

The only other thing I would suggest you look at is Aws::Transfer::TransferManager::DownloadFile. There's a version that writes the output to a stream. I have not used it, so I can't comment whether or not the bytes are available when downloaded or whether the entire download has to complete, etc.

Good luck, would be interested to see if that works out and whether you can control the size of the "chunks" that are streamed, etc.

[stefanmoro]

Hi Jeff,

Yes I have also looked into the TransferManager::DownloadFile whcih writes to an output stream. After looking at the TransferManager::DownloadFile code it seems like the transfer manger chunks the transfer into parts, using S3Client GetObject with range for each part and then writes those parts to the output stream. So, that wouldn't really accomplish what I want either.
I guess it seems like I'll have to consider using bigger chunks instead so that the connection overhead does not eat up such a high percentage of the total transfer time.

Thanks!

[iceboundflame]

+1 on this issue; also looking for a simple, high-performance way to create an istream from an S3 file, such that the contents would be read or written as available. Calling GetObject with small ranges is much less efficient, as we have to wait for each chunk to be fully read before processing them, and a new request is created for each chunk.

[stefanmoro]

Hi,

For those interested, I did not come up with a neat solution for what I wanted to achieve using the AWS C++ SDK. Instead I built a small S3 client using poco http library which allows streaming data over http.

If any new features are available in the AWS C++ SDK that would allow me to achieve this, it'd be great to get input on that!

Thanks

[nikt12]

@stefanmoro @jt70471 Hello, folks.
I met same problem and I'm interested is there some progress in aws-sdk-cpp api?

[saatvikshah]

Same problem here too - This is a really useful feature to have.

I think one way to get around it for the time being might be to write a boost iostreams compatible source(like here https://www.boost.org/doc/libs/1_58_0/libs/iostreams/doc/tutorial/container_source.html) which can then be treated like a stream within boost::iostreams. I still need to try it out though.

[webbnh]

I too would appreciate a solution for this (we're also in the business of processing potentially large, binary files, and we would prefer not having to wait for the entire thing to arrive before starting in on processing it).

In my case, I took the bull by the horns and implemented my own derivations of std::streambuf and std::iostream and passed the result to the TransferManager. Unfortunately, the file is downloaded in chunks which arrive out of order, so the reader frequently has to block until the desired chunk is ready, but it does work. However, the details are really gnarly, and we ended up storing everything in memory anyway (the buffer management was more than I felt like tangling with), but, with a big file, we manage to get about halfway through processing it by the time the download completes, so that's a win.

[rdettai]

It seems to me that it is a bug that s3_client.GetObject(objectRequest); is blocking until the entire request is downloaded. Usually, if the body is returned as an stream (here Aws::IOStream) it's precisely to be streamable ;-) I believe this issue should be re-qualified as a bug.

[colin1227]

Take a look at the Range request header referenced here:

http://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectGET.html

Basically, you can get the object in any size chunks you wish.

This makes total sense, I had made the chunk sizes but they mean nothing if you don't use them, smh.

For future nerds trying to figure this out in node, here's the documentation and the code I used:

    const AWS = require("aws-sdk");

    const s3 = new AWS.S3({
      accessKeyId: process.env.YOUR_AWS_KEY_ID,
      secretAccessKey: process.env.YOUR_AWS_SECRET_ACCESS_ID,
      region: proccess.env.YOUR_AWS_REGION
    });

    const range = req.headers.range;
    const videoSize = 4389391;   // arbitrary amount of bytes but what I've been testing with

    // see NOTES[0]
    const CHUNK_SIZE = 10 ** 6;
    const start = Number(range.replace(/\D/g, ""));
    const end = Math.min(start + CHUNK_SIZE, videoSize - 1);
    const contentLength = end - start + 1;

    // see NOTES[1]
    const contentRange = `${0}-${videoSize - 1}/${videoSize}`;

    const params = {
      Bucket: process.env.AWS_BUCKET_NAME,
      Key: name,
      Range: `bytes=${contentRange}`
    };

    const memeObj = await s3.getObject(params);

NOTES:
0.) Took these vars from this article about 206.
1.) I also use this variable when sending headers to the client, but I don't show that here.