Android: Slow Directory Listing with Dropbox

[0xDAFE]

Basic Info

I'm using Android in version: 10

I'm running Cryptomator in version: 1.5.2

Description

Hi everyone,

I have noticed that my Vault on Dropbox is taking a significant amount of time to list folders even with only ~20 files or subfolders.

Steps to reproduce:

• Create a vault in Dropbox
• Create a folder with 20 subfolders
• Try to browse the folder

When comparing the loading times of the same vault on a WebDAV Host and Dropbox, the WebDAV Vault loads nearly immediately while the Dropbox Vault takes several seconds to load a directory.

I assume, Dropbox is rate limiting 3rd party apps in general or the integrated API Keys for the Cryptomator Android have reached some limit.

Can you check if it is possible to get either a API limit increase from Dropbox or the possibility to add one's own Dropbox API key in the app for advanced users?

[kojid0]

Having the same problem, but I'm on self-hosted Nextcloud WebDAV. Connection speed is fast when I download files in cryptomator android app, but loading progress of folder structure takes forever.

178 would probably help, I guess.?

[yiyang-dev]

@jwsp1 I think #178 only helps if you have long filenames or folder names. I think this is an issue introduced in 1.5.x.

Loads at the moment are insanely slow on any folder containing subfolders on version 7. For context my folder looks like this:

acp_comms/2020-04-21_id/<lots of files>
acp_comms/2020-04-22_id/<lots of files>
acp_comms/2020-04-23_id111/<lots of files>
acp_comms/2020-04-23_id22/<lots of files>
acp_comms/2020-04-25_id/<lots of files>
acp_comms/2020-04-26_id/<lots of files>
acp_comms/2020-04-27_id/<lots of files>
.. so on

Upon opening the folder acp_comms it takes at least a few minutes to load, whereas previously on vault format 6 the folder loads pretty much instantly.

I'm not sure why this is so slow. In theory, all the app needs to do to load directory names is to decrypt the encrypted directory names if the name is a c9r extension. I do not have any c9s directories.

My theory is that for some reason the app is trying to walk into each directory to figure out the type of the object (symlink or directory?), which has the exact same problem with .lng files, except worse now because you cannot cache this information.

Despite it being a bit dodgy I thought having the metadata in the filename itself wasn't terrible. Maybe we'll have a "vault format 8" where the first letter again denotes the type of the file. I mean we're already burning a few characters on the .c9r extension, so it's not that bad right? Or why not .c9d for directories .c9s for symlinks .c9r for files, and so on? Or for 24 bits of data you could just base64 it and represent it as 4 characters and permanently ditch extensions.

Anyway my plan/workaround is to downgrade to vault format 6. If my theory is correct, the app is either going to have to assume everything is a folder to get speedy again, or it's going to be this slow until the next vault format because we're trying to support symlinks. (or maybe asynchronously load the metadata?)

[tobihagemann]

My theory is that for some reason the app is trying to walk into each directory to figure out the type of the object (symlink or directory?), which has the exact same problem with .lng files, except worse now because you cannot cache this information.

This should not be the case. We will have to do some more investigation but vault format 7 _should not_ cause slow directory listings and it _should_ be equivalent to vault format 6 in speed. If the app sees a folder with .c9r suffix, it'll assume that it's a folder even though it could be a symlink. As soon as a user navigates inside this folder and it detects a symlink instead, it will show a "this is a symlink" error.

Despite it being a bit dodgy I thought having the metadata in the filename itself wasn't terrible. Maybe we'll have a "vault format 8" where the first letter again denotes the type of the file. I mean we're already burning a few characters on the .c9r extension, so it's not that bad right? Or why not .c9d for directories .c9s for symlinks .c9r for files, and so on? Or for 24 bits of data you could just base64 it and represent it as 4 characters and permanently ditch extensions.

The reason, why we didn't want to do that, is because we wanted to make the filename unambiguous. In vault format 6, e.g. a cleartext filename foo could be

• E5O6ESS776DXXOHXYCMM55RAVWMRPYI=,
• 0E5O6ESS776DXXOHXYCMM55RAVWMRPYI= (directory, 0 prefix), or
• 1SE5O6ESS776DXXOHXYCMM55RAVWMRPYI= (symlink, 1S prefix).

In vault format 7, it's now _always_ 2tSaUkzXDEfe6fu3700xcIxYew==.c9r. This simplifies certain checks and should actually reduce the number of requests for certain operations. Furthermore, shortened filenames are less common now as well because the filename is now encoded with base64url instead of base32 and the filename threshold has been increased.

Again: In theory, the new vault format _should not_ cause the slowdowns but in practice we have to double-check the implementation. @SailReal Could you please check if this might be the case that Cryptomator for Android prematurely checks the existence or even loads the contents of dir.c9r?

[SailReal]

@SailReal Could you please check if this might be the case that Cryptomator for Android prematurely checks the existence or even loads the contents of dir.c9r?

In Cryptomator for Android we check the existence of the dir.c9r to determine if the dir id cache is up to date but it is true, this should not happen during listing but asynchronously or when opening the folder. Sorry for the inconvenience caused by this issue.

Will fix this problem in a 1.5.4 ASAP

[SailReal]

1.5.4 is coming soon 🎉

[SailReal]

This should be fixed now, if you want to test it, join our beta: https://play.google.com/apps/testing/org.cryptomator