Google-drive-ocamlfuse: Slow Downloads using stream_large_files

[stickenhoffen]

Hi, if I set stream_large_files to true, downloads max out at 200KiB/s, when set to false, the download speeds are very fast, but I have to wait for the whole file to be cached; in this example, 10GB.

Is this a restriction with Google Drive?

[stickenhoffen]

I tested with the ( -m ) option, and the download speed maxes out at 400KiB/s.

[astrada]

Is this a restriction with Google Drive?

No, there is no such restriction. I think the problem is that FUSE is not optimized to stream remote contents. Maybe you can play with FUSE options. But unfortunately I don't think there is an easy solution.

[stickenhoffen]

Thanks for getting back, I tried pretty much all combinations of options.

Since the files download very quickly when stream is switched off, is there anyway to use the partial file from the cache while it is still downloading?

[astrada]

Well, it's not easy. The file name in the cache directory is the file ID in Google Drive. It is saved in column remote_id of table resource in the SQLite DB.

[stickenhoffen]

Thanks. I tried recompiling the Fuse kernel module and setting FUSE_MAX_PAGES_PER_REQ to higher than 32, but I still get the same chunk size on download:

..
Stream resource (id=147,offset=34701312,finish=34832383,length=131072)...done
Stream resource (id=147,offset=34832384,finish=34963455,length=131072)...done
...

Stumped for now.

Cheers.

[astrada]

I don't know if this applies to you, but maybe this link is useful: http://permalink.gmane.org/gmane.comp.file-systems.fuse.devel/14350

[stickenhoffen]

Thanks, yes I already saw that, very helpful.

I checked a few other Fuse packages and don't seem to have the same restriction, is it possible it could be related to OCaml?

Also, if I use the direct_io fuse parameter, it drops even further to 16384 sized chunks.

[astrada]

I checked a few other Fuse packages and don't seem to have the same restriction, is it possible it could be related to OCaml?

Probably. The FUSE binding I'm using defines FUSE_USE_VERSION 26, so maybe it uses an old version of the API that doesn't support custom sized chunks. I can make a couple of test upgrading FUSE but I don't know if it introduces incompatibilities.

[dellipse]

I would also like to see an improvement in streaming large video files. What kind of performance tuning options would you suggest? I am aware of many others looking for this as well, so if we are able to come up with a working solution, I would be happy to create a Wiki page with the recommended options.

[astrada]

Sorry, I didn't have time to test a recompiled FUSE module because I have to recompile the kernel too.

[stickenhoffen]

I can help test here if you like, let me know what you need.

[astrada]

I looked into libfuse source code, and it seems that FUSE_USE_VERSION 26 is high enough to use the newest API.

From fuse-2.9.4/include/fuse.h:

/** @file
 *
 * This file defines the library interface of FUSE
 *
 * IMPORTANT: you should define FUSE_USE_VERSION before including this
 * header.  To use the newest API define it to 26 (recommended for any
 * new application), to use the old API define it to 21 (default) 22
 * or 25, to use the even older 1.X API define it to 11.
 */

Maybe the problem is MIN_BUFSIZE (see http://sourceforge.net/p/fuse/mailman/message/19952379/). If you have time, maybe you could recompile libfuse with a larger value for MIN_BUFSIZE. Thanks for helping out!

[stickenhoffen]

Hi, I built a new kernel with:

#define FUSE_MAX_PAGES_PER_REQ 256

And a libfuse2 with:

#define MIN_BUFSIZE 0x101000

Same results. Not sure else where to look.

Edit: Also, fuse is built as a module; I read that was necessary somewhere else.

[stickenhoffen]

Sorry, I just re-read that last post, and it sounds a little negative! :)

I'm going to tinker with as many things with Fuse as I can, will get back with my findings.

Thanks.

[astrada]

Sorry, I just re-read that last post, and it sounds a little negative! :)

No problem. :)

I'm going to tinker with as many things with Fuse as I can, will get back with my findings.

Thanks for your time!

[zenjabba]

I'm finding this problem. When I use the cache, I can get 1GB/s transfers, but when I tell it to stream, I get no more than 400Kib/s

Any thoughts?

[astrada]

I think the problem is that FUSE reads at most blocks of 128KB, so probably there is a lot of latency streaming vs downloading to the cache. I have to test if there is some way to further increase the block size.

[iplor]

I look forward to hearing the results of the testing. The very low streaming speed is killing my use-case, too.

It seems to me that Google drive is at its slowest when dealing many small requests. For those of us trying to stream large media files, I wonder if increasing the blocksize up into the megabyte range might be more efficient?

[zenjabba]

I have not been able to increase the block size but I know I can get >1gb/s from Google Drive when I copy files down from it to this machine, so it should in theory be wickedly fast.

On 13 Mar 2016, at 5:04 PM, iplor [email protected] wrote:

I look forward to hearing the results of the testing. The very low streaming speed is killing my use-case, too.

It seems to me that Google drive is at its slowest when dealing many small requests. For those of us trying to stream large media files, I wonder if increasing the blocksize up into the megabyte range might be more efficient?

—
Reply to this email directly or view it on GitHub https://github.com/astrada/google-drive-ocamlfuse/issues/128#issuecomment-196054372.

[iplor]

This might be of interest. I found another project called HTTPFS, which allows you to mount a single file over HTTP, using fuse.

I get transfer speeds of around 15 Mbits/s when accessing the mounted file. This is still well below the full speed I get from a straight web download, but it is around 3 times faster than what I'm getting with google-drive-ocamlfuse. Since both are using fuse, and the speeds differ significantly, I thought it might suggest some clues.

[astrada]

Unfortunately, it seems that getting a larger block size requires a kernel recompilation (see http://comments.gmane.org/gmane.comp.file-systems.fuse.devel/14335). So I don't think it's a viable solution. I think I have to implement an in-memory buffer, but it's not easy because it needs to support multi-threading mode (so concurrent access, out of order reads/writes), and there is the risk of using too much memory, so it needs some policy to free resources.

[zenjabba]

I’m happy to recompile the kernel… I’ll setup a test machine where we can setup this if it helps.

On 15 Mar 2016, at 5:43 PM, Alessandro Strada [email protected] wrote:

Unfortunately, it seems that getting a larger block size requires a kernel recompilation (see http://comments.gmane.org/gmane.comp.file-systems.fuse.devel/14335 http://comments.gmane.org/gmane.comp.file-systems.fuse.devel/14335). So I don't think it's a viable solution. I think I have to implement an in-memory buffer, but it's not easy because it needs to support multi-threading mode (so concurrent access, out of order reads/writes), and there is the risk of using too much memory, so it needs some policy to free resources.

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub https://github.com/astrada/google-drive-ocamlfuse/issues/128#issuecomment-197036431

[astrada]

Nice, thanks. It looks like you have to modify at least FUSE_MAX_PAGES_PER_REQ (https://github.com/torvalds/linux/blob/master/fs/fuse/fuse_i.h#L27). In the thread on comp.file-systems.fuse.devel, someone tried to set it to 2560 (10MB). Probably you have also to modify MIN_BUFSIZE (https://github.com/libfuse/libfuse/blob/6adcb719a933a31013c73fda8e0ccb0e13b45e58/lib/fuse_kern_chan.c#L86) in libfuse.

[astrada]

And this post suggests to update VM_MAX_READAHEAD too (https://github.com/torvalds/linux/blob/710d60cbf1b312a8075a2158cbfbbd9c66132dcc/include/linux/mm.h#L2053).

[zenjabba]

email me [email protected] for access to test system

[jant90]

Very interesting thread, so how is the solution working out? And will it make it into the main branch of google-drive-ocamlfuse?

Also, not sure if it helps as I'm no developer, but acd_cli can stream files at high speed (~30MB/s), so that's already about 100 times faster. Not sure if that's a FUSE speedlimit or if the connection is simply saturated but maybe it's worth looking into their implementation/solution?

[iplor]

My suspicion is that the main problem is not Fuse itself. I also used another project in the past, GDriveF4JS, and that used to have no problems streaming full HD. It was always a bit unstable, though, and now broken and abandoned. But I can't see how these other Fuse systems can stream efficiently if the problem is fundamental to the default kernel Fuse parameters.

Perhaps they are using a partial caching method, rather than relying on caching entire files?

[astrada]

Yes, buy GDriveF4JS uses an in-memory buffer to optimize streaming performances. I'm not saying that's not feasible, but that building a robust in-memory buffer is not easy.

[iplor]

I certainly appreciate the hard work you've already put it to this. If you have time to implement such a solution in the future, that would be terrific.

Have you discovered if tweaking the Fuse parameters in the kernel also solves the problem? It's obviously not an ideal solution, but it'd be fine for those of us that don't mind tinkering with things.

[stickenhoffen]

I finally got back to this, and built a new kernel, increasing FUSE_MAX_PAGES_PER_REQ to 2560 and VM_MAX_READAHEAD 10240, and reached streaming speeds of around 2MiB/s, so we're on the right track. I will increase again and rebuild, and let you know how it goes.

Edit: Bytes not bits.

[astrada]

Great! Thanks!

[stickenhoffen]

No luck increasing the values, is it possible there's something in ocaml-fuse that can be tweaked? Something seems to be capping the upper limit at 2MB chunks.

It's not all bad news though, with the -m switch, it's performing very nicely for me! :)

I have a couple of questions.

If I use streaming, the cache.db increases, is this caching directory structure and file size etc.?
If so, what is the TTL of the cache, and/or file/directory listings?
If I mount read only, and add a file directly to Google Drive, will the new file be picked up instantly? If not, how long will it take for the file to appear?
If I use -m, what's the maximum threads I can reach?

Thanks so much.

[astrada]

No luck increasing the values, is it possible there's something in ocaml-fuse that can be tweaked?

I don't think so. Did you also modify MIN_BUFSIZE (https://github.com/libfuse/libfuse/blob/6adcb719a933a31013c73fda8e0ccb0e13b45e58/lib/fuse_kern_chan.c#L86) in libfuse?

If I use streaming, the cache.db increases, is this caching directory structure and file size etc.?

Yes, the sqlite db stores file entries from Drive API.

If so, what is the TTL of the cache, and/or file/directory listings?

There is no hard TTL. File entries are deleted when some kind of changes are detected (from the API).

If I mount read only, and add a file directly to Google Drive, will the new file be picked up instantly? If not, how long will it take for the file to appear?

It depends on metadata_cache_time in the configuration file. It's expressed in seconds.

If I use -m, what's the maximum threads I can reach?

There is no limit (and no pooling). So it depends on the application accessing the filesystem. Usually there are not many concurrent threads.

[stickenhoffen]

I don't think so. Did you also modify MIN_BUFSIZE (https://github.com/libfuse/libfuse/blob/6adcb719a933a31013c73fda8e0ccb0e13b45e58/lib/fuse_kern_chan.c#L86) in libfuse?

Yes, I increased substantially but it didn't seem to make any difference at all.

[stickenhoffen]

By the way, I was hesitant to use the -m switch, but it is really quite stable for me. Also, debug seems to kill it eventually, the curl log was >1GB so I assume it's my crappy dev box, but I plan to test on a system with more memory and SSD in the next couple of days, so can pass you logs.

[iplor]

When you say it's performing very nicely with the -m switch, what sort of speeds are you getting?

[astrada]

Also, debug seems to kill it eventually, the curl log was >1GB

Yes. With -debug turned on, it dumps every byte from/to GDrive to curl.log. So if you are streaming video/music, it gets in the way. -verbose should keep gdfuse.log but not dump CURL requests/responses.

[stickenhoffen]

@iplor It's hard to say with any accuracy, I would say I can reach around 50-60mbit/s. I also tried a quick test with seven simultaneous video streams, and it seems to cope nicely. I'll try a few more tests over the weekend.

[seiferma]

Thanks for sharing your experience with us @stickenhoffen. Did you get any new results or even improvements?

[stickenhoffen]

Hi, I've been travelling so haven't had much time to look at this. Increasing the numbers didn't seem to increase performance that much for me after a point, but I'm in Singapore, perhaps I'm just hitting the limits with Google.

Perhaps if I build a new stock Ubuntu kernel with the changes, someone can install and help test?

[seiferma]

I would be happy to support you in testing your approach. You can share the compiled kernel with us or provide some short instructions on how to change and compile the kernel.

[iplor]

Do you think the idea of an in-memory buffer to get efficient streaming is something that will be implemented? I tried playing around with compiling a tweaked kernel, but I still wasn't getting good performance.

[stickenhoffen]

Sorry for the delay, been so busy with work. Packages for Ubuntu 16.04 x86_64 are here:

https://gofile.io/?id=J6Zb6h

I've also found that using the following command line parameters appears to work well in multithreaded mode:

google-drive-ocamlfuse -verbose -m -o allow_other -o sync_read drive

[iplor]

That's great - thanks very much. I'll give it a try and see if it works better than my own attempts.

I'm still curious to hear if the idea of a memory buffer, as mentioned by Astrada, is on the roadmap, or if that's something that's not being considered at the moment.

[iplor]

I installed those packages, and booted with the custom kernel. The difference is clearly noticeable, but for me the transfer speed now jumps wildly from 0 to full speed, and everywhere in between. The average rate is still too low, and the erratic speed means that streaming is still not working well.

[stickenhoffen]

Curious, how fast is your Internet there?

[iplor]

Officially 75 mbits, in practice more like 60. When downloading files using the google client or google-drive-ocamlfuse in non-streaming mode, I get almost full speed, at 5 - 6 megabytes per second.

[joemarshalljm]

I'm new to all this, can someone explain what I need to do with the Packages for Ubuntu in order to see the increased speed for video?

[astrada]

I created a new PPA with a new version (0.6.0) of my application that should address some of the issues reported. Mainly, it enables multi-threading by default, and add a read-ahead buffer when streaming (stream_large_files=true), using new config options: memory_buffer_size (defaults to 1MB) to specify the minimum size of download buffers, read_ahead_buffers (defaults to 3) to pre-fetch blocks of the same file, and max_memory_cache_size (defaults to 10MB) that specifies the maximum memory occupation of read-ahead buffers (before it starts to deallocate the oldest ones). You should consider this a beta version. I'm not yet publishing it to the main PPA until I've tested it a bit more. If you don't use Ubuntu, let me know, so I can give you instructions on how to install it. Anyway the code is in the new beta branch.

[iplor]

Thanks so much for doing this. So far, it's looking really encouraging. I'm seeing smooth playback of large videos, and the available bandwidth being well-utilised. I'm still messing around with the config options, and I'll report back after using it for a while.

[zenjabba]

I’m planning on testing it today, I hope I see such amazing throughput :)

On 25 Nov 2016, at 2:22 PM, iplor [e-mail] wrote:

Thanks so much for doing this. So far, it's looking really encouraging. I'm seeing smooth playback of large videos, and the available bandwidth being well-utilised. I'm still messing around with the config options, and I'll report back after using it for a while.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub https://github.com/astrada/google-drive-ocamlfuse/issues/128#issuecomment-263013474, or mute the thread https://github.com/notifications/unsubscribe-auth/AApfuPY6uxC3dcreOxh4ZANZ2SFHfnftks5rBzVpgaJpZM4FIUPN.

[zenjabba]

Worked great under Ubuntu, now trying to compile for Centos and having many problems getting the beta edition to be found.

On 25 Nov 2016, at 2:22 PM, Stephen Thompson [e-mail] wrote:

I’m planning on testing it today, I hope I see such amazing throughput :)

On 25 Nov 2016, at 2:22 PM, iplor <[email protected] [e-mail]> wrote:

Thanks so much for doing this. So far, it's looking really encouraging. I'm seeing smooth playback of large videos, and the available bandwidth being well-utilised. I'm still messing around with the config options, and I'll report back after using it for a while.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub https://github.com/astrada/google-drive-ocamlfuse/issues/128#issuecomment-263013474, or mute the thread https://github.com/notifications/unsubscribe-auth/AApfuPY6uxC3dcreOxh4ZANZ2SFHfnftks5rBzVpgaJpZM4FIUPN.

[astrada]

With OPAM, you can compile the beta version this way:

opam pin add google-drive-ocamlfuse https://github.com/astrada/google-drive-ocamlfuse.git#beta
opam upgrade google-drive-ocamlfuse

Then, to restore the standard version:

opam pin remove google-drive-ocamlfuse
opam upgrade google-drive-ocamlfuse

[zenjabba]

I’m getting this error with the beta version

[43.522596] TID=202: END: Checking MD5 checksum (path=/Backups/plexpy.log, cache path=/root/.gdfuse/default/cache/0B61rZZCN5Xw1VXNWaEUtYW9qNzA, hash=e7904832b3b44acdcd713ef287fc4d9f): Computed MD5 checksum: 02c6e6c047a2f9a4bd68efb3ac637ca6
[43.522610] TID=202: BEGIN: Downloading resource (id=86) to /root/.gdfuse/default/cache/0B61rZZCN5Xw1VXNWaEUtYW9qNzA
[43.522617] TID=202: BEGIN: Updating cache size (delta=18981) in db
[43.526199] TID=202: END: Updating cache size (new size=78467) in db
[43.526217] TID=202: BEGIN: Updating resource state in db (id=86, state=Downloading)
[43.529946] TID=202: END: Updating resource state in db (id=86)
[44.204101] TID=202: BEGIN: Updating resource state in db (id=86, state=ToDownload)
[44.208368] TID=202: END: Updating resource state in db (id=86)
[44.208659] TID=202: Error during request: hd
[44.208681] TID=202: Giving up
[44.208694] TID=202: Exception:Failure("hd”)

On 29 Nov 2016, at 2:59 AM, Alessandro Strada [e-mail] wrote:

With OPAM, you can compile the beta version like this:

opam pin add google-drive-ocamlfuse https://github.com/astrada/google-drive-ocamlfuse.git#beta
opam upgrade google-drive-ocamlfuse
Then, to restore the standard version:

opam pin remove google-drive-ocamlfuse
opam upgrade google-drive-ocamlfuse
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub https://github.com/astrada/google-drive-ocamlfuse/issues/128#issuecomment-263501559, or mute the thread https://github.com/notifications/unsubscribe-auth/AApfuHxSEOBmjn51zzK4LeJ7H8bCcg28ks5rC9tVgaJpZM4FIUPN.

[astrada]

@zenjabba can you please send gdfuse.log and curl.log to my email ([email protected])?

[iplor]

After quite a bit of testing, I've had no problems at all. Streaming full Bluyray backups with no hiccups. Brilliant.

When you have the time, could you give a bit more detail on how the config options work?

[astrada]

After quite a bit of testing, I've had no problems at all. Streaming full Bluyray backups with no hiccups.

Great!

When you have the time, could you give a bit more detail on how the config options work?

Sure. These new options work only if stream_large_files=true. Basically, files to stream are split in blocks of memory_buffer_size bytes (if it's above 0), and read_ahead_buffers specifies how many parallel threads are spun to pre-fetch next blocks. For example, using the default values (with memory_buffer_size = 1MB and read_ahead_buffers = 3), when you access the first bytes of a file, my app starts downloading the first block (of 1MB) and spins 3 new parallel threads to fetch blocks 2, 3, and 4. Then, when you access the second block, my app starts downloading block 5 (blocks 3, and 4 are already downloaded/downloading), and so on. Finally max_memory_cache_size specifies how many blocks can be kept in memory at the same time. So (using the default value of 10MB), in the example, when you access the 8th block, my app starts downloading block 11, and deallocates block 1. This way, I'm optimizing sequential streaming of big files, but making other access patterns (random seeks/reads) slower. Keep in mind that max_memory_cache_size is global and not per file, so if you stream multiple files concurrently, you should probably raise that value. Let me know, if you need further details.

[iplor]

Thank you, that's clear.

One more question - does this method generate a lot of API requests, like a separate request for each small block that is cached? I'm just wondering if the Google API limits could become a problem.

[astrada]

When streaming, each block (of memory_buffer_size bytes) requires an API request, if not cached.

I'm just wondering if the Google API limits could become a problem.

I don't think so. The limit is 1 billion requests/day.

[iplor]

I'm sure you know more about it than I.

I've just seen the issue come up in a couple of other projects that use the Drive API (I think rclone was one of them). I believe the issue was around the rate limit, rather than the absolute limit. I've had no trouble so far, so hopefully it's a non-issue here.

[astrada]

During testing, I never hit that limit. Anyway, if a request returns a 403, it is retried with exponential backoff, so this should not be an issue.

[tchey290]

@iplor Can you share you config/setup that is letting you stream full bluray backups without any issues? I can't seem to stream even smaller video files.