Client: Virtual Filesystem (VFS)

[MTRichards]

This is a research task, to look into and define what it would take to make this happen. Basic user story looks like this:

As a user, I want to see all of my files on my desktop, regardless of if they have synced or not so that I can access my files at any time, from anywhere.

Acceptance Criteria:

• A threshold can be set for the "cache" that is allowed on the desktop
• Favorites are synced automatically with the client.
• The most recently accessed files are synced until the cache is full, then files are removed from the local device.
• If favorites are bigger than the cache, only the recently accessed favorites are kept

And much more.

@karlitschek @dragotin @danimo for your continued discussion and additional requirements.

[jturcotte]

Since the source problem motivating this solution is that managing very large shares is cumbersome with the client, here are some specific problems/solution to consider that could be incrementally improving the quality of the sync client to also solve the issue instead of a risky and costly file system solution:

I'm subscribed to a large shared folder, but from time to time huge files that I don't need appear on my disk

Do shared folders need to be synced by default? Your own files, or files that you favorited in the web interface should be included in the selective sync by default. Other files could simply not be synced until you ask for them. The folder was shared with you, congratulations you have access, but in most cases you might not want all of them on your disk. A notification when you gain access could already ask you if you want to sync the new shared folder in totality.

I want to work on my own files in the company shared folder, but activating it in the selective sync now means that I get every new files included and I need to keep deselecting them

The selective sync UI currently only allows excluding files. If you have the structure a/b/c, and you want the directory a/b/c/d1, but not a/b/c/d2, the UI will allow you to deselect d2, but if d3 appears it will be synced by default. There is no way to say that you _just_ want a/b/c/d1, you can only say I don't want all other files, and you need to update this list every time a new folder appears.

We could provide a way to include and exclude folders, so that you can exclude a/b/c, but override this by then including only its a/b/c/d1 child, thus also excluding d3 by default when it appears. The control panel for the indexing service on Windows is an example of a UI that allows this. It's a bit more complicated, but allows to have both an include and exclude hierarchy.

For example:

The selective sync UI is slow and it shows arrows to expand folders even if they don't have children

We currently PROPFIND the server each time you click in the UI, we could keep the whole file structure and store it in the database, updating it on each sync so that the UI is fast and complete.

I want to have access to all files quickly even though I just want to access it once, I'll see later if I want to sync it

The selective sync UI could allow to jump quickly to the web interface with a context menu in the selective sync UI "Open in the ownCloud web interface" where the user can download the files in this folder.

The files themselves could also be shown in the selective sync UI and an "Open locally" menu could allow opening the file without syncing it, but there should be something making the user aware that modifications won't be synced back if it was excluded (either the file is readonly, or a popup tells him).

[MTRichards]

There is another one:

I have files that I create on my desktop, that are in my sync directory, that I don't want to sync - but I still want them to remain on my desktop.

Example: I rip a copy of a DVD I own, and it is in my video folder, but it is 45 GB (blue ray) and I don't want to sync that...but I don't want it deleted.

[karlitschek]

Great feedback. Thanks a lot @jturcotte There is definitely something that we could do to make selective sync more intelligent. But this might lead to even more config options that users need to understand. I also think that this is not the best option for a lot of scenarios.
I think we have to explore the 'final' solution even if it is complex. So I suggest that we look into the filesystem option as described above.

[arjamizo]

Got here from #2633 (thanks to @guruz @RealRancor).

I think the best way of having this is http://www.cs.nmsu.edu/~pfeiffer/fuse-tutorial/
and link it contains: http://www.cs.nmsu.edu/~pfeiffer/fuse-tutorial/html/index.html

view-source:http://www.cs.nmsu.edu/~pfeiffer/fuse-tutorial/src/bbfs.c

The solution on the client would look in a following way:

// pseudo-c++
class FilesystemOnDemand : FSCache // [0:cache-fs] | [1:github:/contantine/fscache] | [2:offlinefs] | [3:bcache]
{

    FilesystemOnDemand(const char* path):path(path)
    {

    }

    bb_readdir(path)
    {
        require('zipfs');
        if (folder~='*zip' && listZipFilesInsteadOfDownloading) return Functor(SmartZipFilesListing(owncloudDataProvider, path));
        else return Functor(readingFolderGenerator, path);
        // functor returned because in fact listing directories is stateful on low-level;\
        note that this could allow even smart listing
    }
    bb_open(path, ...)
    {
        return  super.createFile(path);   // opening a file requires creating a file in a caching filesystem

    }

    bb_filetype(path)
    {
        return filename.isDir(ptah) or fileExtension(path)=='zip' && DIR &&
    }

    bb_read(file, begin, end, ...)
    {
        file=super.open(getPathOfFile(file));
        super.write(file, begin, end, owncloudDataProvider([user, login] | ssid, relativePath(path), begin, end);
        file.close();
        // smart-read, retrieving only needed part of a requested file
    }

    // in first releases read-write access might break a lot, so I suggest first versions allow read-only access
    int bb_access(const char *path, int mask)
    {
        return 0440 // owner+r,group+r;
    }
}

(btw. is even writing posts like this allowed here? if no, then forgive me, but I need to highlight that the only acceptable way of implementing such a solution is filesystem-level file access, otherwise accessing files will not be transparent from the user point of view and will be not compatible with all the programs available out there) and I think we want to allow for example users to be able to watch not only their 20KB text-files, but also their 4GB videos, which - if stored in webm format, can be played even if partly-fetched RFC-6386 or webm-docs.

I think in first version this could be mounting additional folder Read-only like:

+ ~/owncloud (30GB)
  - --placeholder for disabled sync to indice that in this case there are non-sync-ed files--
  - Videos (10GB)
  - Read-only (which, when asked by filesystem what's its filesize, returns 20GB)

and whenever somebody tries to watch photos from ~/owncloud/Read-only/Photos/*.jpg, data get's fetched and cached.

What do you think?

---

Those are technologies to be researched:
[0] https://www.kernel.org/doc/Documentation/filesystems/caching/fscache.txt
[1] https://github.com/cconstantine/CacheFS 2010
[2] http://offlinefs.sourceforge.net/wiki/ 2011
[3] https://bcache.evilpiepirate.org/ Jan 4, 2016!
[4] http://openafs.org/ (from #4614 thanks to @NykauToutcourt) project alive, last update 16-March-2016
and more I think

[dragotin]

I think the first step is done which was the requirement for the 2.2.0 milestone. I will move this on.

[jon-tlmi]

Hi guys, I expect you've already seen this, but Dropbox are previewing this at the moment it seems.
http://www.theregister.co.uk/2016/04/26/dropbox_debuts_project_infinite/
(if you want to see how they're selling it)

[MTRichards]

Thanks for the link! We are looking, just to see how they are thinking about the topic.

[arjamizo]

@dragotin are you working on this feature now or it's on your bucket list??

[guruz]

@arjamizo It's a research topic currently. I think @dragotin @ogoffart have the source somewhere if you want to contribute/enhance/check it out.

[guruz]

https://blogs.dropbox.com/tech/2016/05/going-deeper-with-project-infinite/

[hodyroff]

This is getting more popular. From @dragotin:
Microsoft did something interesting, they implemented a VFS for git! That sounds to me as if it would be worth a look how they did it and if it might be adaptable to VFS for oC:
https://blogs.msdn.microsoft.com/visualstudioalm/2017/02/03/announcing-gvfs-git-virtual-file-system/
https://github.com/Microsoft/gvfs

This project seems even more interesting:
https://github.com/viciousviper/DokanCloudFS
https://github.com/viciousviper/CloudFS

This project implements VFS for various clouds, and it should be relatively easy to adopt it to ownCloud, because there is already a backend implementation for WebDAV. Worth a check?

[hodyroff]

Another option would be to combine an open source WebDAV client with the oC Desktop Client.

[Andreas-Kainz]

combine online with offline data was also what I suggested with my bug report https://help.nextcloud.com/t/show-online-files-in-local-storage/11149

user story: we want to switch from samba shares to owncloud.
the benefit is that you have always your data on your device (in an encrypted drive) and you can take the data you need, not everything from all other users.
the problem is that when the files are all in the cloude you can't connect fast to files the user didn't sync, which is a problem, cause you sometimes only want to search in a folder and open there one file. so a combination of online and offline data (like in a mail storage where you see the header/folder and file structure, but only when you open the file you will download it).

[pilot33]

As a user, I want to be able to find unsynced files quickly via Spotlight (Mac OS X), regardless of whether they are in the cache, or only in the cloud.

[AndrewBucklin]

DropBox also has this now with "selective sync".

ExpanDrive just released this feature in their product (they call it “offline-sync mode”).

ODrive has had this feature for a while (they call it "progressive sync").

I’d be over the moon if it came to Nextcloud. I'd even offer some bounty for this feature request; anyone else with me?

[Andreas-Kainz]

The features look quite good. Thunderbird offline mode give also the feature offline mails/files from the last x months. Maybe store offline up to xx gb store latest edited files

[jon-tlmi]

Interesting news today:
Google is also now starting to offer the equivalent of placeholders/ file stream/cache on demand as part of their move to Google File Stream from the standard Google Drive app
https://www.theregister.co.uk/2017/09/11/google_deprecates_drive/
https://techcrunch.com/2017/09/07/google-launches-drive-file-stream-to-replace-the-google-drive-desktop-app-for-g-suite-users/
https://support.google.com/a/answer/7491633

[ckamm]

I've started prototype exploration in https://github.com/owncloud/client/tree/vfs-prototype . I've assigned the issue to 2.5 to make it more clear that it's active.

The aspect I'm experimenting with is what effects "placeholders" would have on the synchronization algorithm. That should be mostly independent of whether we go with a basic file approach or write a fuse-like filesystem.

I think we should start out building a simple and clunky but functional cross-platform solution. That would very likely use placeholder-files that users can activate to trigger the download of the actual file.

To be specific the very basic first-iteration functionality I have in mind is this:

• On initial sync, no files are downloaded. Instead a placeholder ".owncloudplaceholder" file is created for each file.
• Double clicking or shell integration activation removes the file and downloads the actual file.
• From then on the file is synced as usual.

At the same time we can investigate creating a filesystem layer that makes it more streamlined. I'm currently working on the former. For the latter something like http://dokan-dev.github.io/ might be useful?

[Andreas-Kainz]

Sound good and would work for our office where we'd like to replace samba shares with the cloud server.

for image folders placeholder files could be a problem cause the user like to scroll through the folder.

[SamuAlfageme]

First experimental release with @ckamm's idea from https://github.com/owncloud/client/issues/4341#issuecomment-353009296 and explained in https://owncloud.org/news/owncloud-client-goes-virtual-sync-files-demand/

Grab it while it's 🔥

[pascalBokBok]

Cool!

Den 6. juni 2018 16.59.35 CEST, Samuel Alfageme [e-mail] skrev:

First experimental release with @ckamm's idea from
https://github.com/owncloud/client/issues/4341#issuecomment-353009296
and explained in
https://owncloud.org/news/owncloud-client-goes-virtual-sync-files-demand/

Grab it while it's 🔥

--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/owncloud/client/issues/4341#issuecomment-395100001

--
Dette er sendt fra min mobiltelefon. Undskyld at jeg fatter mig i korthed.

[guruz]

Discussion also in https://central.owncloud.org/t/owncloud-virtual-file-system-discussion/13823/10

[XA21X]

The new Cloud Filter API seems to be the modern way to achieve this on Windows 10 (Version 1709+) without implementing a full FUSE filesystem. I'm assuming that this is how OneDrive's Files On-Demand feature was implemented in the Fall Creators Update.

[ckamm]

@XA21X Thanks for the link!

[HanaGemela]

@ckamm what exactly should be tested here? Could you please provide the scenarios?

[ckamm]

@HanaGemela This is a high-level feature issue, I can only outline a few scenarios:

• Creating vfs folders from account wizard works (and non-vfs folders continue to work)
• Creating vfs folders from folder wizard works (and non-vfs folders continue to work)
• Vfs can be enabled/disabled on a per-folder basis
• When vfs is enabled placeholder/virtual files are created where appropriate
• These files can be hydrated on use
• The files can be explicitly marked as available, or reverted to placeholders
• When this is done to folders it has recursive effect
• This can also be done to the whole folder from the folder settings "..." menu
• New remote files are propagated according to the local parent directory availability
• Renames, moves, deletions of placeholder files have the expected effect

[HanaGemela]

@michaelstingl is there any chance we'll have documentation on VFS?

@ckamm I don't think it works on macOS 2.6.0alpha2 (build 12128) but hard to decide without documentation. Please confirm if this is the intended behaviour

Scenario 1
1) Folder has VFS on
2) User opens one file to view it -> the file has been downloaded, opened and immediately dehydrated again
Is that OK?

Scenario 2
1) User makes one file in VFS folder always available locally
2) User opens another VFS file from that folder -> the hydrated file that is supposed to be always available locally gets dehydrated
Is that OK?

Scenario 3
1) User deletes a placeholder -> it is recreated again
2) User deletes a folder -> it is deleted
Is this inconsistency OK? But when the folder is deleted then I get the notification that the files which were inside has been added. But they were not so this is a bug

[ckamm]

Both scenarios 1 and 2 would be the same bug. (does the pin state not apply properly on OSX?) (EDIT: Can reproduce, likely my fault in a recent change, am surprised the tests didn't catch that and will fix.)

Bad notifications in scenario 3 would also be a bug.

[HanaGemela]

@ckamm
Scenario 1 https://github.com/owncloud/client/issues/7322
Scenario 2 https://github.com/owncloud/client/issues/7323
Scenario 3 https://github.com/owncloud/client/issues/7324

Is there any good reason for inconsistency in Scenario 3?

[ckamm]

I will need to look at that later.

[HanaGemela]

@ckamm what is the expected result when I copy a virtual file? Currently the copy disappears. But it should behave as normal file which means to create a copy, am I right?

[HanaGemela]

@ckamm on Windows when I turn on VFS, the icons show 'Sync pending'. What does that mean? When I force sync now, it is still Pending.

[HanaGemela]

@ckamm What should happen when I try to download a placeholder offline? Could it give me a nice message? Currently macOS doesn't do anything, Windows gives Error 0x800701AA: time-out

[ckamm]

@HanaGemela

Delete: Also discussed here: https://github.com/owncloud/client/issues/6875
Copy: Currently that's the expected behavior. It's not clear how the suffix-vfs plugin could detect that a new file is a copy of another one.
cfapi sync pending: That's not what I saw last time I tried. The "sync pending" state should go away after the sync. Ticket please.
download while offline: yeah, that seems good. Not for 2.6.0 though.

[HanaGemela]

@ckamm
Delete: Thanks, I've moved my question about inconsistency there
Copy: OK
cfapi sync pending: https://github.com/owncloud/client/issues/7329
download while offline: What do you mean?

[ckamm]

@HanaGemela Download while offline: Currently it should be queued to be downloaded when the next sync happens. I was saying a warning message would be good to have, but that it should be targeted for 2.6.1.

[HanaGemela]

Closing this high level issue. Bugs and enhancements will be solved in separate issues

[theJian]

Is it possible to implement it without the .owncloud extension for placeholder files on MacOS?
Maybe using sparse files? OneDrive on mac also have a files on-demand feature and its placeholder files don't need a special extension.

[michaelstingl]

@theJian macOS 10.15 Catalina will come with a "all-new FileProvider API". Plan is, to use this for the ownCloud desktop sync client. No ETA yet.

https://www.apple.com/macos/catalina-preview/features/

Third-party cloud service integration
An all-new FileProvider API for cloud storage providers delivers a new way to seamlessly integrate their services into the Finder without requiring a kernel extension, helping to maintain the security of your Mac…

Here you can find the technical details:
https://developer.apple.com/documentation/fileprovider

[theJian]

@michaelstingl This API was removed from Catalina features page. Does Apple decide to not to bring this API to macOS Catalina?

[michaelstingl]

@michaelstingl This API was removed from Catalina features page. Does Apple decide to not to bring this API to macOS Catalina?

Sadly no information from Apple. They also rolled back their iCloud improvements during beta, but those are planned for Spring 2020:
https://www.macrumors.com/2019/10/08/apple-delays-icloud-folder-sharing-catalina-spring/