S3fs-fuse: Improve random writes and appends

[CAFxX]

In the README.md the performance of random writes and appends is listed as a limitation:

Limitations

Generally S3 cannot offer the same performance or semantics as a local file system. More specifically:

• random writes or appends to files require rewriting the entire file

If this is still the case, was it ever considered to use multipart upload, and specifically both Upload Part and Upload Part - Copy, to speed it up?

Specifically, an append operation would become (simplifying):

• initiate multipart upload
• upload part - copy (old object)
• upload part (data to append)
• complete multipart upload
• delete old object

A random write operation (e.g. "helloworld" -> "hello World") would become:

• initiate multipart upload
• upload part - copy (old object, off 0 len 5 "hello")
• upload part (changed data, " W")
• upload part - copy (old object, off 6 len 4 "orld")
• complete multipart upload
• delete old object

This approach would probably be beneficial when the existing object is big and the amount of data to append or changed is small, so some heuristics may be needed. In addition, these heuristics should also consider that (as noted below by @sqlbot) the upload part API has some restrictions in terms of minimum/maximum allowable part size ("Part size: 5 MB to 5 GB, last part can be < 5 MB")

[sqlbot]

S3 itself would not allow such an operation at small scale. The Multipart Upload API, whether the part data is uploaded or copied from another object, requires that each part (except the last part) have a minimum size of 5 MB.

http://docs.aws.amazon.com/AmazonS3/latest/dev/qfacts.html

[CAFxX]

@sqlbot so the appending scenario would work without any problem (assuming the _old_ object is at least 5mb in size, something that in an append scenario seems fairly common).

For random writes it could be still useful in certain limited scenarios (e.g. VM images) where it would still provide substantial benefits.

This was kinda implied by the last phrase in my first post ("This approach would probably be beneficial when the existing object is big and the amount of data to append or changed is small."), I will now update it to clarify it.

[ggtakec]

@CAFxX @sqlbot I'm sorry for very late reply.
I implemented this feature in #1027 .
If you can, please try to use master branch latest code after merging it.

[ggtakec]

This issue is closed because #1098 has been merged.