Google-drive-ocamlfuse: Caching before upload is very slow

[fjen]

File are written to the cache before the real upload starts. The problem is that this caching reaches only ~2 MB/s and therefor takes a very long time.
After turning on the debug mode, it appears that the local database is queried and updated after every 128k.

This repeats in the logfile for every 128k:

[28.453607] TID=88: write /gocryptfs [131072 bytes] 5111808 0
[28.453629] TID=88: BEGIN: Getting metadata from context
[28.453640] TID=88: END: Getting metadata: Valid
[28.453651] TID=88: BEGIN: Loading resource /gocryptfs (trashed=false) from db
[28.454181] TID=88: END: Loading resource /gocryptfs (trashed=false) from db: Found (id=2, state=ToUpl
oad)
[28.454724] TID=88: BEGIN: Writing local file (path=/gocryptfs, trashed=false)
[28.454965] TID=88: END: Writing local file (path=/gocryptfs, trashed=false)
[28.454980] TID=88: BEGIN: Updating resource in db (id=2, state=ToUpload)
[28.516588] TID=88: END: Updating resource in db (id=2)
[28.516612] TID=88: BEGIN: Updating cache size (delta=131072) in db
[28.577785] TID=88: END: Updating cache size (new size=5242880) in db
[28.578019] TID=89: getxattr /gocryptfs security.capability
[28.578041] TID=89: BEGIN: Getting metadata from context
[28.578053] TID=89: END: Getting metadata: Valid
[28.578064] TID=89: BEGIN: Loading resource /gocryptfs (trashed=false) from db
[28.578612] TID=89: END: Loading resource /gocryptfs (trashed=false) from db: Found (id=2, state=ToUpl
oad)

Is there a way to speedup this process?

[astrada]

No sorry, you should patch your kernel and libfuse (see issue #128). Or maybe you can try another client, like rclone that AFAIK caches everything in RAM so it should be faster.

[fjen]

The question is whether the database updates are really necessary. I think they contribute in a significant way to this slow behaviour.
Rclone unfortunately doesn't support other fuse filesystems (enfs, gocryptfs) above it.

[vi]

eatmydata google-drive-ocamlfuse makes upload fast.

[astrada]

In version 0.7.1, I added a new config option (write_buffers). If you turn it to true, it caches files in memory before uploading. It should speed up uploads. It's still experimental, so turn it on at your risk. :wink:

[vi]

Does it stop doing fsync for each uploaded file (or part of a large file)?

[astrada]

It writes to the filesystem only when there are no more free buffers (max_memory_cache_size is the total cache size and memory_buffer_size is the size of single buffers). Since version 0.7.0, the writes to the sqlite3 db are also cached in memory and flushed every metadata_memory_cache_saving_interval seconds.

[astrada]

And in version 0.7.10, I've added async_upload_queue config option to start async uploads.