Desktop: Ability to select particular files to synchronize / placeholder files

[vitaliis]

Right now it is only possible to select the whole folder to sync.
It should be possible to select some particular files to sync, instead of only whole folders.

This can be useful in case of a multi-gigabyte folder where you want to synchronize only some files.
An example is a folder with podcasts. It is a big archive and has a lot of gigabytes, but you need only couple of them. It is a waste (of traffic, electricity, time) to download everything.
Right now it is not possible to select only some files to sync from a folder.

[camilasan]

@juliushaertl any thoughts on this? Would it make sense?

[juliushaertl]

I'd actually say that this is a very specific use case. It is also kind of hard to get this right UI-wise. The sync selection is already quite complex and adding even more options there would make it hard to get what is going on there.

cc @nextcloud/designers for other opinions.

[stefan-niedermann]

Maybe it is a better solution for you @vitaliis to not sync the folder at all but to mount that folder via webdav (of course only if you have a good internet connection)?

[vitaliis]

@stefan-niedermann thank you for the proposed solution. This is one of the options. And it is a possible workaround for now instead of having this functionality implemented.

I'd actually say that this is a very specific use case.

I disagree. I'd actually say the case described in the ticket description is a normal and valid use case.

When I first started using the client I expected this functionality to be there. The fact that it is complicated to implement is not a reason to reject it. It might be a good opportunity to refactor the code and make it more readable (and easier to understand).

[jancborchardt]

Currently we do it like Dropbox and other clients, where by default everything is synced. You could have subfolders in your Podcasts folder and use the integrated selective sync (which already exists based on folders) to exclude specific folders for syncing.

Excluding things on a per-file basis would be much too complicated and also confusing when you don’t remember what you enabled, and what you deleted. This then results in confusion as to which files are actually still there.

The proper way to do this would be a kind of virtual filesystem where the non-downloaded files are still shown as half-transparent in the file manager, and downloaded on opening. This unfortunately is some future vision and not really possible right now (not to speak of making it work cross-platform).

[errotu]

It seems like owncloud is implementing this feature to their desktop client. Does this increase the chances of an implementation in the nextcloud client?

[WhiteBahamut]

As stated in my initial request about placeholder files/virtual folders, it is for me a core scenario of cloud storage.
I do not which to have the file synchronized to all my local machines, only those I realy need (on demand).
I did try the ownCloud Client (with my nextcloud) and my first test made me quite happy. Would love to see nextcloud go this way as well and enable palce holders.

[jancborchardt]

@WhiteBahamut yes, we will consider it for certain. :) Just need to make sure it works in general usage and doesn’t result in confusion because files are not synced when you then do go offline.

[WhiteBahamut]

agreed. It must be easy to see whether the file was synced with the cloud or is still just local. But icons next the file will certainly help. E.g. in OneDrive has different Symbols for each state (only local, syncing, only cloud, cloud + local)

[jospoortvliet]

Another downside of this approach is that you can't open files from applications. So the 'recent files' or file open dialog in your apps won't work - they either don't see the placeholders or open broken files.

So this solution is a bit of a hack, we have to see if it is good enough to ship or not.

[BenBergman]

I'm excited to see what this finally materializes as. I like how Google Drive File Stream and others work where you don't even really notice that files aren't local and you can explicitly request files be saved for offline use.

One use case I can see for myself is for editing video. I would like to have my very large video files archived on my remote storage but be able to work on them on my local machine with the full performance of local files. Once I've stopped working on a project, I don't want to have to manually go in and tell it to stop caching files locally or reorganize things, and when I start a new project using files already on my remote storage I want it to automatically pull those files locally for me for performance and mobility.

[derWalter]

I want to propose something:

To implement a VFS (like OwnCloud) which allows to
see all files on the server, but to choose, filebased instead of only folderbased (!), between:

1. remotely accessing files like over NFS/SMB/WebDAV
2. downloading/syncing them on demand
3. sync them completely (standard mode)

[isuckatscripting]

I would love to have this functionality. Has there been anything else on this?

[jospoortvliet]

@BenBergma yes, something like that would be great, but the selective sync thing oC did isn't that I'm afraid.

@derWalter oC didn't implement a VFS, otherwise it would not have the downsides I mentioned. A VFS (which would allow those features) would be the proper solution, yes.

So, to be clear, we COULD integrate the oC solution with temporary placeholder files. I am personally not convinced, let's see. Note that developing a full VFS would be a big project, something to tackle once E2EE is out and stable - so help stabilize 2.5.0 beta so we can move to a new big feature :smile_cat:

[leafcutterant]

I'd actually say that this is a very specific use case.

I disagree as well and I very much need this functionality.

I want to access most part of folder X on phone, so by default I'd set it up to be synced to my Nextcloud's X-cloud folder. However, X also contains folder private and file private.jpg which I would never trust my Nextcloud provider with. So I either have to modify my file storage logic to fit Nextcloud's whims, or I can choose to not sync any of the folder at all.

This is quite a sad state. And I'd feel stupid if it stayed that way because OwnCloud seems to have implemented a solution already, and back then I've fully migrated from OwnCloud to Nextcloud because everyone said that there's much more development here.

[camilasan]

@leafcutterant

I'd feel stupid if it stayed that way because OwnCloud seems to have implemented a solution already,

I would not call what they implemented a solution but a workaround or hack.

because everyone said that there's much more development here.

This is still very much true. Specially on the server side and mobile clients. Desktop client it is still catching up but we will get there.

[vitaliis]

Hi @camilasan and team,
:+1: :+1: for work on the server side and mobile clients!

I would not call what they implemented a solution but a workaround or hack.

But listen, even if it is just a workaround, it solves the problem. There are always some disadvantages to each particular solution. OwnCloud guys have solved the issue in some way. Respect them for it.

Now here, at @nextcloud/desktop...
Let's get a least some solution that covers what users need. It doesn't have to be perfect. It has to be present. After solving the problem "in some way" improvements to it can be further discussed.

[camilasan]

@vitaliis

Hi @camilasan and team,
+1 +1 for work on the server side and mobile clients!

:heart:

But listen, even if it is just a workaround, it solves the problem.

I don't want to maintain a workaround of this size ;-)

There are always some disadvantages to each particular solution.

Sure.

OwnCloud guys have solved the issue in some way. Respect them for it.

A solution for me would be to use libfuse. I didn't mean to disrespect them.

Now here, at @nextcloud/desktop...
Let's get a least some solution that covers what users need. It doesn't have to be perfect. It has to be present. After solving the problem "in some way" improvements to it can be further discussed.

Priorities ;-)

[vitaliis]

@camilasan fair enough :smiley:

Priorities ;-)

I am curious.. Does this mean that we won't get any traction on this for a while?
Do you have an approximation for current priorities if they are somehow defined?

[camilasan]

I am curious.. Does this mean that we won't get any traction on this for a while?
Do you have an approximation for current priorities if they are somehow defined?

Current priorities are simple: get 2.5 out of the door. Then we have some further E2EE work to be done, like adding sharing & improving performance, as well as implementing many server features like comments, trash and versioning.

Now a big project like fuse could be done alongside, though it would still take a major release, if not two - so, think next year at the earliest.

[jospoortvliet]

Note that priorities are set in two ways:

• This is an open source project. So whatever volunteers do gets done, of course. And our engineers are supposed to give high priority to reviewing volunteer PR's and helping out.
• Besides the above, our paid team is supposed to spend their engineering time to implement our internal roadmap which is set by customers and our product vision. Internal -> as in, not public and not guided by public, but we try to work in the open so much of it is public anyway ;-)

The company pays our engineers to do 'housekeeping', like, keep the auto-builds running, manage releases, stuff like that. And they should fix bugs: those reported by customers, and if they have time left, those from home users. Feature requests from home users is really last, a almost 100% volunteer-only thing that our engineers rarely if ever have paid time for.

[derWalter]

@BenBergma yes, something like that would be great, but the selective sync thing oC did isn't that I'm afraid.

@derWalter oC didn't implement a VFS, otherwise it would not have the downsides I mentioned. A VFS (which would allow those features) would be the proper solution, yes.

So, to be clear, we COULD integrate the oC solution with temporary placeholder files. I am personally not convinced, let's see. Note that developing a full VFS would be a big project, something to tackle once E2EE is out and stable - so help stabilize 2.5.0 beta so we can move to a new big feature

@jospoortvliet

Hey thank you for replying!

Thanks for being on it in a quality first way!!!
I am in desperate need for a maximal dynamic solution regarding file handling,
as I need many different ways of accessing and syncing files right now
and those do not work together and trough this regularly creating chaos.

As I am waiting for many years now, I guess I can wait some more years,
or dream for ever and see features which are more important for the company being released :)

If I had ten more versions of me, maybe one of them would find time to code
a little bit in spare time and help out ^ ^

[ben-page]

I'm new to Nextcloud, but this feature is vital for my organization. We have about 500 GB of data on our current file server. I'd like to migrate to Nextcloud, but obviously we can't replicate the entire file server to every user. Using WebDAV isn't great for a few reasons.

Why not pull in the oC placeholder solution until a better solution (fuse) can be implemented?

[KieferMacK]

I'm also new to Nextcloud. After some time playing with it I can say that this feature (dynamic syncing with placeholder files shown) is the only feature critical for me that Nextcloud seems to be missing, and that most other "major" cloud storage solutions already support.

I suspect my situation is similar to @ben-page above, where we have several hundred GB of data and it's not reasonable to only sync select folders: access to all types of files is needed, just not to all the data simultaneously. I've encountered this situation personally as well as at a handful of companies.

Sorry I don't think I have much to offer on this issue technically. I just wanted to voice my desire for this feature.

Best,

[alex-tee]

I need this feature too. I'm surprised it's not implemented. I want to sync configuration files in linux, for example .vimrc. I don't want to include the whole /etc or home dirs. I guess a workaround would be symlinks

[guillermofbriceno]

I need this feature too. I'm surprised it's not implemented. I want to sync configuration files in linux, for example .vimrc. I don't want to include the whole /etc or home dirs. I guess a workaround would be symlinks

This is also my use case. Unfortunately symlinks don't seem to be supported either with the same reasoning.

[jacky35]

I think if the webdav function had been fast. There will not be that kind of request. But the webdav protocol is much too slow, at least in my case of Windows network sharing.

I am surprised that users have not made more insistent demands on this in 2018.

Hoping that messages from users like us react and include an option like owncloud.

Thank you very much and congratulations for all the efforts already made.

[bmarwell]

I am surprised that users have not made more insistent demands on this in 2018.

A lot of users just subscribed to not pollute this thread I guess. But if comments are needed, here is my +1.

[AndreasLukas1]

I am very happy about the discussion about this important feature.

In my opinion it has huge advantages if I could open files as easy as local files to edit with local software. With a mounted webdav folder I have this opportunity, but there are problems with big files (e.g. graphics, powerpoints etc...). The editing software itself is loading and saving through the mount direct from and to the server wich can be slow and lead the editing software to crash sometimes. Some softwares create temporary files for recovery - wich are also stored on the server and can make editing slower.

Cyberduck Client has the ability to browse the files and open these with an editor. The file is then downloaded to local temp-folder and opened with editor (e.g. photoshop, word, exc el....). Cyberduck is watching the changes to the local file and is uploading if it is changed.
= Great workflow for me!

But: In Cyberduck I do not see nice thumbnails and comments, markers, labels when browsing files. Wich I have when browsing online.

The future is online - this is why my favourite Solution would be: A Firefox addon wich allows me to open files from the nextcloud UI with my local editor software. This addon should create a local copy and syncs it when the local file is changed.
If I could mark files as "offline-available" with this Firefox-addon - this would be marvelous.

What are your thoughts about this idea?

[thibaultamartin]

We consider migrating from a misused Alfresco to Nextcloud at my company, and a sort of VFS definitely is a must have for us. I, as a home user, would also benefit a lot from this feature.

Does Nextcloud have a bounty program of some sort so home users, and companies willing to do so, can financially contribute to this feature?

[WhiteBahamut]

Looks like it is coming https://nextcloud.com/blog/nextcloud-introduces-virtual-drive-in-desktop-client-to-simplify-desktop-integration/ !

[Dennis360ict]

I've tried to get this function in, but I could get my message heard. So kudos to you guys that you succeeded where I failed..

We have large implementations at businesses and they really need this kind of feature. We have really bad workarounds based on jobs running nextcloudcmd, but it's ugly as hell. I'm looking forward to get this solution in place at my customers. Expect lots of feedback on this feature! ;-)

Cyberduck and mountainduck do not offer enough solutions for this. Cyberduck only browsed and downloaded the file. Which is fine, but not an solution where you have a team with 10 ppl who are working on the same files: even though sync is fast, its not instantaneous which left all kinds of duplicate files and need to merge them manually. Like a planning sheet being used by 3 frontdesk ppl who needed to fill in planningsheets. Mountainduck should be able to do this, but it did all kinds of stuff with caching, which actually corrupted files. My experience is that mountainduck does not understand mulit-user Windows like RDS. We could have gone through support, but as it was a paid client and the amount of work involveld, we opted to do scheduled syncs on a networkfolder.

This was also tested internally with a team of 10 ppl working on a Windows 2012R2 RDS server.

[Banderi]

I agree either via local symlink support or via selective file/folder selection, this is a much needed critical feature missing.

Currently the only option is to sync a folder as-is and only refuse remote files to be synced, or sync each individual folder/file inside the master structure which is not gonna happen for larger directories.
The added cons to that is that all the manually synced folder come with an automatic folder icon change and addition to the "favorites" folder list (in Windows) aside from also bloating the client's sync list.

[solarkraft]

Yes, Virtual Drive is exactly the functionality I'm looking for. But:

• Where can I follow the project? How is Linux support coming along? When can I expect it to be shipped (yes, there's the branch, but I fail to read the exact status from it)?
• Why use Dokan and not just Fuse (commits do mention fuse)?

Thanks. I really hope to be able to use this soon.

[tmolitor-stud-tu]

@camilasan Any progress regarding virtual drive feature? What is missing to get this feature rolling? Is there something I can help with?

[claell]

@jospoortvliet Is this the issue to track progress on the Virtual Drive feature? I searched for it and did not find an explicit one. Maybe the issue can be adjusted accordingly to reflect that more clearly or another issue can be created to track the progress, ideas and discussion for Virtual Drive.

[derWalter]

how to push this, give it a higher priority??

[claell]

@derWalter I don't really know, but you could try to contribute yourself to this feature. Else other developers have to be motivated to continue work on this. Currently this repo doesn't seem to support bounties, but maybe there are other ways you can find to motivate.

[derWalter]

@derWalter I don't really know, but you could try to contribute yourself to this feature. Else other developers have to be motivated to continue work on this. Currently this repo doesn't seem to support bounties, but maybe there are other ways you can find to motivate.

mhhh, I myself cant do it technically ... but bounties I would support :) I would instantly put some money on this feature!

[claell]

Since this issue does not seem to have gotten more attention in between and it should be adjusted probably to better reflect the current plans for virtual drives, a short ping. @jospoortvliet @er-vin @camilasan @misch7

[derWalter]

as they listed a hire for a C++ developer, for the desktop client, I still have my fingers crossed, that they implement this feature , to close this gap to all other cloud solutions asap.

[er-vin]

Since this issue does not seem to have gotten more attention in between and it should be adjusted probably to better reflect the current plans for virtual drives, a short ping. @jospoortvliet @er-vin @camilasan @misch7

It is under work. It's almost the only thing I'm working on for the past few weeks actually (well, when I'm not trying to deal with the tons of duplicates we have here, the customer requests, etc.).

[claell]

@er-vin Thanks for the update! Maybe it would be nice to have an issue about that feature to track the progress there or adjust this issue's description accordingly? See my previous comment (https://github.com/nextcloud/desktop/issues/184#issuecomment-609048079)

[paddywaan]

I see a lot of comments here about this feature request that are pointed towards syncing a sub-selection of directories/files on a remote sync, however I come here from the opposite requirement. I recently experienced a "bug"? / lack of feature in my attempt to sync my /home/ directory with nextcloud. The intended outcome is for me to sync /home/user/.config and /home/user/.dotfiles, with maybe a few important directories which i would wish to maintain between environments, however when creating the sync, and selecting the /home/user/ directory, there is no prompt/UI to select which sub directories should and should not be maintained in sync. The sync will start and only after the files have been uploaded? do the sub-directories show in the sync-explorer expand/drop-down.

Now, from a Linux perspective, this is problematic/impossible to achieve the desired effect for three reasons:

The first being that .ssh and other sensitive directories would be automatically synced when this may not be intended.
The second is that 100,000's of files are needlessly synced before even obtaining the option to choose not to sync those very same files. Chicken and egg, am I right?
The third is that manually selecting sub directories inside of /home/user/* disables the ability to sync .dotfiles inside of the users root: /home/user.

I have been hesitant to post this as a bug as I found this feature request, however I am now not so sure if this eventual feature will go any length to resolve the issue which I have described. Does anyone have any clarification to the state of this feature request and how it might relate to the situation I have described? I looked over the blog post however everything seems focused on sync selection server->client.

[claell]

@paddywaan Best open a new issue about this, this seems to be independent from the things in this issue.

[k1n6b0b]

What about implementhign this similar to MS OneDrive's FIles on Demand feature? Waiting on this feature is really the only thing keeping me from moving 100% to NextCloud. As per others, I have a ton of old folders/files/large files I don't need replicated everywhere and the concept of 'not seeing' them in the filesystem (al'a IMAP style subscriptions) seems antiquated to me.

The Files on Demand feature is slick and intuitive. Right click "Allways sync offline" or "Free up space" (Or similar) for enabling/disabling the placeholder vs actual files/folders.

[usatiuk]

One argument for google file stream-like approach: it would make it possible to, well, stream files. If you want to open a large file (e.g. video, archive, some image) with OneDrive or other placeholder file solutions, you have to download the entire file, and while it's downloading everything that tried to access it completely hangs, which is not a problem with google file stream. 🤔

[dumblob]

One argument for google file stream-like approach:...

Second argument is simply, that large files are presumably the main driver behind this whole functionality.