Google-drive-ocamlfuse: Google Drive ocamlfuse is very slow

[wallnas]

Good morning,
i have install ocmlfuse on ubuntu 14.04 64 bit and is very slow for download and upload.

my config is:
max_cache_size_mb=512
new_revision=true
sqlite3_busy_timeout=500
read_only=false
conflict_resolution=server
verification_code=
drawing_format=png
document_format=odt
form_format=ods
docs_file_extension=true
keep_duplicates=false
presentation_format=pdf
download_docs=true
client_secret=
metadata_cache_time=60
client_id=
spreadsheet_format=ods
debug=false
umask=0o002

There is a method to speed ?

Tanks for help.

[idevin]

Same thing

[astrada]

You can try to increase metadata_cache_time. This way there are less server requests e the cache will not be invalidated too frequently.

[wallnas]

I have modify metadata_cache_time = 180 and 360 : There is very slow.

[lightsurge]

I'm also experiencing this slow-down even with a high metadata_cache_time. In my case I gzip a large amount of data into the google drive folder... so possibly because it gradually writes the compressed file into the google drive folder this is causing excessive syncing. I'm going to try moving pre-prepared files rather than performing heavy operations on files inside the sync folder.

[ardhipoetra]

Second this.

I know this is old issue, I got only 100-150 kB/s with FUSE. In the other hand, I can upload 2-3 MB/s with browser. Tried changing the metadata_cache_time produces the same result.

My concern (for now) only for uploading though.

[astrada]

You can try setting async_upload=true, or you can try the new beta version (0.6.0). that you can install from a new PPA, if you are on Ubuntu.

[ardhipoetra]

Still, the speed very slow. Now it has around 400kB/s. Moreover, I uploaded from SSD, so I don't think the bottleneck is from the disk.

As for comparison, I can get good performance with https://github.com/odeke-em/drive

[astrada]

I think the problem is that FUSE uses blocks of 128KB (at maximum), so you have to wait for a (relatively slow) copy to end before the upload begins. If your use case requires frequent uploads of very big files, you should go with https://github.com/odeke-em/drive. You can then use my app to stream the uploaded files (if they contain videos, for example).

[ardhipoetra]

rclone also implements FUSE to mount cloud storage, one of them is google drive. I tried that before and mostly can get higher speed although it still in experimental stage. I will look into the details later.

[astrada]

OK, but from the source code you linked:

Limitations

This can only write files seqentially, it can only seek when reading.
Rclone mount inherits rclone's directory handling. In rclone's world
directories don't really exist. This means that empty directories
will have a tendency to disappear once they fall out of the directory
cache.

I think that rclone has a simple in-memory model, so it's optimized for uploading but has those limitations.

[hbakhtiyor]

i'm experiencing the same issue with speed of uploading

[vi]

Maybe the slowness is due to not using the database properly: https://github.com/astrada/google-drive-ocamlfuse/issues/308#issue-230220653. It fully syncs the database state to disk every 128 kilobytes.

Using libeatmydata hack forces unsafe, unsynced database mode and makes uploads fast.

[wifiuk]

i also get crippling slow folder loading and file transfers

[henricook]

@vi You mention a libeatmydata hack, do you have details? I'm also only getting ~100k/s with ocamlfuse on Ubuntu Disco (i'm doing a ~20,000 photo and video file upload)

[vi]

@henricook In my case the bottleneck was (SQLite?) database that google-drive-ocamlfuse uses (or used) for internal bookkeeping. Every now and then (a file uploaded or a chunk of larger file uploaded) it commits transaction to that database and waits until it is fully persisted.

This wait is slow, especially with HDD. You typically want it to happen if something important is being save to database, e.g. a bookmark in your browser.

eatmydata on Linux turns off that "wait for database transaction is fully persisted" aspect, making SQLite belive that it persisted while in reality it is still in cache, allowing the application to continue earlier. This trades resilience against sudden system poweroffs for speed.

SQLite has a built-in "unsafe, unflushed" mode that skips it, achieving the same effect without using external modifier libraries. I expect google-drive-ocamlfuse to have an option for it, if not default for it.

[henricook]

Thanks for that explanation @vi - not what i expected. Since my goals were fast one-off upload of a few large folders i've switched to rclone and had a really pleasant time

[astrada]

Since version 0.7.0, I introduced config option metadata_memory_cache=true to cache metadata in memory and flush to sqlite periodically. And since 0.7.1, I introduced write_buffers=true to cache writes in memory (it's off by default because it's still experimental).