Docs.rs: Uploading large files to S3 can use an excessive amount of RAM

[pietroalbini]

While investigating #656 we discovered that our current upload code uses a lot of RAM when uploading large files to S3. Unscientific tests show that the RAM usage is 10x the size of the file we're trying to upload. We should change the implementation to reduce the amount of RAM usage, or to limit how much data we upload at once.

[jyn514]

I think this can be closed now, after profiling the new implementation (unscientifically) it uses 120 MB to upload a 100 MB file, which seems reasonable. Possibly using mmap would reduce that even further if you're interested in that.

[pietroalbini]

Hmm, another possibility to reduce the amount of used RAM is to pass a streaming body to the request, somehow converting a stream of the file content to it.

[jyn514]

I think the only way to do that without first reading the file into memory (and copying it into user space and back) is with mmap. I haven't tried to implement this yet though.

[Mark-Simulacrum]

I would not try to do mmap. Ultimately that's not even necessarily sound to use in Rust, and regardless, our files are usually on the order of dozens of megabytes at most.

If we do want to optimize, we should avoid using the standard allocator for buffering in memory, instead reading into something like a bump allocator so that we can drop the file contents without potentially increasing fragmentation etc in the primary allocator.

But I personally don't think we should do anything here, files are tiny.

[jyn514]

I'm onboard with this, the largest file we've served in the last week is only 45 MB. I think it's more important to limit the memory usage in a batch: if we have 1000 files at 10 MB each, that's way more likely to blow out memory than uploading one enormous file.

[Mark-Simulacrum]

Right, yes, if we have problems around memory usage I would personally decrease parallelism rather than try to be smarter about how each file is handled.

Ultimately 99% of the files we upload are probably less than one megabyte.