Google-drive-ocamlfuse: Slow Navigation

[Mensen]

Thank you for actually making some software that can make use of my google drive!

However, whenever I explore the drive, either using a GUI or directly in the terminal it takes a long time to even enter a directory. When I try to move files in the directory or rename files/folders it can take up to 20-30s.

This does not happen with any other directory except the gdrive folder.
I've set the cache size to 1GB in the hopes it would basically just download my whole drive locally for fast access but the issue does not change.

Is this just an inherent problem, or is there something I can do?

[Mensen]

google-drive-ocamlfuse, version 0.5.3
Copyright (C) 2012-2013 Alessandro Strada
License MIT

[astrada]

Do you have a lot of Google Docs, Sheets and Slides? Those documents cannot be cached efficiently for a number of reasons. You can try to turn off documents download, editing the config file, setting the download_docs option to false:

download_docs=false

This should make things faster.

[Mensen]

Thanks for the quick reply... I don't use google docs though.

The files are just 'regular' random files, usually of very small size in the order of just a few kb... however the most recent copying of one whole directory of 1.8 Mb was timed at a fairly consistent 1.1 kb/s

[astrada]

OK. In this case, if you plan to have lots of client side operations, you can try to raise the cache timeout of metadata, e.g.:

metadata_cache_time=600

(or even higher if you usually don't do server side changes). This will avoid excessive polling in that scenario.

[Vladx71]

Hi,

Thanks for your work, because at this moment this is the only way to access google drive from Linux (above the crappy web interface). However it is very very slow... not just the folder browsing. but opening any file is extremely slow. Currently I'm using the following options to mount my drives

/root/.opam/system/bin/google-drive-ocamlfuse -m -o big_writes -label $1 $*

is there any further fine-tune option to speed-up access to files stored on gdrive or this is cannot be faster due to google/technology?

thank you
L:

[astrada]

Is it slow just the first time you open a file or even when you open the same file more than once?
Did you try putting a big value in metadata_cache_time?

[Vladx71]

it seems it is very very slow when I open directory first, when I open
directory second time it is fast, so it is looks like related to caching.
I've increased metadata_cache_time to 600 but does not help a lot. for
example opening a directory with a ~30 files takes several minutes.

L:

On 2 July 2014 19:25, Alessandro Strada [email protected] wrote:

Is it slow just the first time you open a file or even when you open the
same file more than once?
Did you try putting a big value in metadata_cache_time?

—
Reply to this email directly or view it on GitHub
https://github.com/astrada/google-drive-ocamlfuse/issues/58#issuecomment-47808318
.

[astrada]

Do you have many Google Docs (office-like files)? Those files can't be efficiently cached, so they slow down a lot the resource fetching. If you don't need them on your desktop, you can turn them completely off, putting download_docs=false in the config file. Otherwise, you can turn your docs into links, putting:

document_format=desktop
presentation_format=desktop
spreadsheet_format=desktop
drawing_format=desktop
form_format=desktop

in the config file, and then using -cc command line option to clear the cache.

[Vladx71]

Not at all. I have just a few documents like this and navigation is very
slow in directories where I have no google docs at all

L:

On 3 July 2014 08:48, Alessandro Strada [email protected] wrote:

Do you have many Google Docs (office-like files)? Those files can't be
efficiently cached, so they slow down a lot the resource fetching. If you
don't need them on your desktop, you can turn them completely off, putting
download_docs=false in the config file. Otherwise, you can turn your docs
into links, putting:

document_format=desktop
presentation_format=desktop
spreadsheet_format=desktop
drawing_format=desktop
form_format=desktop

in the config file, and then using -cc command line option to clear the
cache.

—
Reply to this email directly or view it on GitHub
https://github.com/astrada/google-drive-ocamlfuse/issues/58#issuecomment-47872443
.

[astrada]

Then I don't think you can gain more speed. The only thing you can do is try to measure the server side latency: you can enable debug output (with -debug) and every operation will be tracked with timestamps. This way you can see if it's slow because of the network.

[jcward]

I have the same problem with very slow access, from ls to nautilus, because I do have a lot of Google Docs in my drive, and this did work for me:

you can turn your docs into links...

document_format=desktop
presentation_format=desktop
spreadsheet_format=desktop
drawing_format=desktop
form_format=desktop

Then google-drive-ocamlfuse -cc to clear cache.

Thanks, @astrada!

One question: While links are a good workaround, is it possible to allow fast directory listing (e.g. ls and find), and only perform the slow document fetch/conversion later when the file is opened? Maybe report a 1-byte file (or some other special flag, like a symlink or named pipe) until the file is actually opened?

[astrada]

One question: While links are a good workaround, is it possible to allow fast directory listing
(e.g. ls and find), and only perform the slow document fetch/conversion later when the file is
opened? Maybe report a 1-byte file (or some other special flag, like a symlink or named pipe)
until the file is actually opened?

Unfortunately it's not easy because it requires opening files with O_DIRECT option, and the FUSE binding I'm using doesn't allow that. I'm keeping this issue open as a reminder.

[jcward]

Ok, thanks! The links feature is nice and will work for now.

[aplund]

I'm not sure if this is the right place for this comment, but it seems that doing lots of metadata requests (e.g. lstat()) is also extremely slow if the metadata cache expiry has occurred, say when you start up at the beginning of the day. From my connection, one lstat() request takes anywhere from 500ms-1000ms. If you are getting the sizes of 1000 files, there's 10-20minutes in just checking metadata.

This is particularly a problem when doing an rsync update. Is there any way to get some kind of bulk metadata update out of the google drive api, or perhaps a more efficient way of asking rsync to check file sizes?

[astrada]

The easiest thing that I can think of is running ls -lR before rsync to warm the cache. I don't know if it's quicker than running just rsync.

[Torsten85]

@astrada couldn't you fetch the whole google drive (drive.files.list should do this without many requests and also delivers all the stats needed) and instead of flushing the cache after metadata_cache_time simply poll drive.changes.get and flush only the parts of the cache that really have changes? This would dramatically speed navigation up.

[astrada]

Well, not really. drive.files.list without filters can be quite slow if you have many files. You can check it enabling lost+found and querying that folder. The changes are already polled the way you are suggesting. Global cache invalidation occurs only if there are more than 50 changes between 2 calls to drive.changes.get.

[Torsten85]

@astrada ... and a call to drive.changes.get happens every metadata_cache_time seconds right?

Currently the first attempt to navigation after mounting is painfully slow but every following navigation attempt is quite fast... but after some time (maybe an hour or so) navigation is slow again.

[astrada]

... and a call to drive.changes.get happens every metadata_cache_time seconds right?

Yes, right.

Currently the first attempt to navigation after mounting is painfully slow but every following navigation attempt is quite fast... but after some time (maybe an hour or so) navigation is slow again.

It can be slow if you have lots of Google Docs. As they are exported, I've have to download them fully to get their size (otherwise the will be shown as zero-bytes). If you don't care about exporting documents, you could switch formats to desktop, or turning them off setting download_docs=false in the config.

[OndraZizka]

Where is the config file? --> ~/.gdfuse/default/config

[OndraZizka]

Hi, here's my take on this: to my mountGdrive script, I added tree /gdrive/ & which caches all the files in the background within around 5 minutes. I've increased the cache size to 2048 MB (it has to be a geeky number indeed) and the timeout to 3600. That's no big deal, I hav a plenty of space and the caching seems to work fine; besides, I only work with non-doc files over fuse and the docs don't really change often.

[thmxv]

Just to summarize my experience. We use a TeamDrive to collaborate on an 'medium sized' asset collection (at the moment 150.000 files totaling 100GB, but destined to grow a lot). We have some custom tools that needs to go over the whole asset repository and get a listing of every directory (using a in house optimized version of python's os.walk() function).

• Getting he first listing is painfully slow, even if not having any Google docs
• Using ls -lR or tree to build a warm cache helps a lot
• Even with a warm cache, listing the whole drive is still 10x slower than on windows running the same python code on a FileStream mount.

[FabioPedretti]

Do you have many Google Docs (office-like files)? Those files can't be efficiently cached, so they slow down a lot the resource fetching. If you don't need them on your desktop, you can turn them completely off, putting download_docs=false in the config file. Otherwise, you can turn your docs into links, putting:

document_format=desktop
presentation_format=desktop
spreadsheet_format=desktop
drawing_format=desktop
form_format=desktop

in the config file, and then using -cc command line option to clear the cache.

Could you think about making this the default? Or at least have a command line option to force this for all formats?
Thanks.

[hmaarrfk]

Would you accept a PR to change the defaults?

[astrada]

In version 0.7.5, I added a command line option (-docsmode) to quickly set Google Docs config options. See Usage.

[hmaarrfk]

Nice that should do it!

Would you want to take PRs to add google drive relevant icons from wikipedia to the installed files. That way the desktop stuff gets nice icons

[astrada]

OK, thanks, just check that the icon license permits to embed it in another software.

[astrada]

Should be fixed in the latest versions.