Android: [TOP] Enhance sync folders

[tobiasKaminsky]

As discussed on conf this is a collaboration between @AndyScherzinger and me.
Purpose:

Increment 1 - instant upload - #358 - release 1.4.0

• replace current instant upload
• full support for Android 6,7

Increment 2 - enhancement auto upload - #797 - release 2.0.0

• support video folders
• allow any kind of folder pair (custom folders)
• show jobs in upload pane
  
  
  
  • track jobs
  
  
  • get job result to update upload pane
  
  
  • if job fails: allow user to restart
  
  
• automatic retry
• upload updated files

Increment 3 - upload existing files

• add "upload/sync all existing images/videos/files" as an option to the auto upload folder three-dot menu.

Increment 4 - folder upload

• allow folder hierarchies :heavy_check_mark:
• smart UI to semi-automatically detect and notify about potential new media folders the user can add/active for instant upload #2406

Increment 5 - sync one way

• sync - one way
• free up space

Increment 6 - sync two way

• sync - two way

[aparcar]

Does this include a constant sync of a todo.txt file on change locally and periodically checked on server?

[tobiasKaminsky]

@aparcar currently this will only replace instant upload.
But this approach is designed to also handle a real two way sync and a constant sync of files.

[AndyScherzinger]

This issue/implementation is to be considered the top prio since we now already have > 1,300 Android 7 installations according to the play store :exclamation:

[tobiasKaminsky]

Camera should be top of the list as this is the most wanted "folder", I guess.
At least I was searching a while where it is ;-)

[tobiasKaminsky]

Uhh, and caching is not working properly, right?

[tobiasKaminsky]

I know it is kind of reduntant, but I think that it should be possible to enable/disable sync also in the setting of a folder (and not only in the folder sync view).
[folder sync looks fantastic! :+1: :tada: ]

[AndyScherzinger]

Yeah, we can put camera on top, need to think how to put this into the comparator. Doable :)
As for the caching, don't know actually. The queries need to be tweaked also since they need to order by date.

[AndyScherzinger]

And yes, the enable/disable can also be put into the settings. UI is close to feature complete. Then I will publish some screenshots for the discussion with the designers.

[AndyScherzinger]

Camera folder at the top (of the inactive): DONE

[AndyScherzinger]

Let's move v1 discussions to the newly created PR #358

[tobiasKaminsky]

V1.1:

• show jobs in upload pane
  
  
  
  • track jobs
  
  
  • get job result to update upload pane
  
  
  • if job fails: allow user to restart
  
  

[jasonbayton]

Do version numbers and milestones need to be adjusted here?

[AndyScherzinger]

Yes, removed since this is a long term roadmap

[rugk]

Is there any ETA for this?

[ShaunCurrier]

Auto upload is also not working for me on Google Pixel XL with Android 7. Release version of Android app on Play Store.

[jasonbayton]

@shauncurrier please create a new issue. Ensure you add the troubleshooting steps you've taken also and for how long files have been waiting to upload (or if they don't make it to upload at all).

[ShaunCurrier]

My mistake! I was in a hurry and thought I was posting in a forum thread I had open in another tab. I'll take it elsewhere, apologies.

[madorian]

At what increment are we now at? Shouldn't we have a separate bug for each of them?

[AndyScherzinger]

@madorian I just updated the top description and we are actually implementing what I described in Increment 2 now - #797. We try to keep the discussion here in general but it would make sense to still create separate issues for the different functions but keep this one as an umbrella issue referencing the detail issues and to reflect the complete road map of this feature in one place.

[Tormen]

Hi,
Any idea when the "upload existing files" feature might become user testable ?
Tormen

[Bullnados]

Yes, and also slowly start including android 8 support 😃

[W-Hamra]

First off, thanks a lot for your great work on this app, it's awesome.
On which increment are we standing now?

[AndyScherzinger]

Increment 2 has been implemented and will ship with the next stable release

[mario]

Well, we also have:

show jobs in upload pane
track jobs
get job result to update upload pane
if job fails: allow user to restart

• automatic retry
• upload updated files ;)

[mario]

Do note that this roadmap/issue will be changed once @tobiasKaminsky gets the time to put notebook notes in the issues :) cc @AndyScherzinger

[Orybon]

Does the work on increment 3 has started ?

[AndyScherzinger]

@Orybon not yet. Priority is bugfixing 2.0 and e2e encryption.

[Orybon]

Thank you for your answer @AndyScherzinger

[rgarrigue]

About increment 3, autoupload existing : is there a workaround to force upload of the existing ? maybe moving out / in via an android file manager ?

[mario]

Yes, that will work.

On Wed, Jan 3, 2018 at 2:44 PM, Rémy G. [e-mail] wrote:

About increment 3, autoupload existing : is there a workaround to force
upload of the existing ? maybe moving out / in via an android file manager ?

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/nextcloud/android/issues/285#issuecomment-355014915,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAAWsvuGVv9GX_fBSERTpZeaiO4Tlufxks5tG4RUgaJpZM4KDF9B
.

[dan0xii]

Increment 4 - folder upload

allow folder hierarchies

Does this mean that "auto upload" will also be able to upload sub-directories and their contents? I'm looking for any issues related to that functionality and this is the closest I can find.

I have an application that writes files to dir x and numerous sub-directories under that path so I pointed auto-upload at x and it only uploads files in that directory, not sub-directories and their contents.

[mario]

This should already work really.

On Mon, 15 Jan 2018 at 18:11, dan0xii [e-mail] wrote:

Increment 4 - folder upload

allow folder hierarchies

Does this mean that "auto upload" will also be able to upload
sub-directories and their contents? I'm looking for any issues related to
that functionality and this is the closest I can find.

I have an application that writes files to dir x and numerous
sub-directories under that path so I pointed auto-upload at x and it only
uploads files in that directory, not sub-directories and their contents.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/nextcloud/android/issues/285#issuecomment-357741956,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAAWsp6cznIuDvDVhN-Gzivn1qcVVO2Kks5tK4bXgaJpZM4KDF9B
.

[Bullnados]

Is old folder syncing now possible?

[AndyScherzinger]

Hi @Bullnados, the implementation for the upload of already existing files hasn't been started yet. Sorry for the sad news.
I hope that this can be achieved with 3.1 or 3.2 though.

[Bullnados]

@AndyScherzinger
No prob. Nice to know that u are working on that feature 👍

[holocronweaver]

Assuming work hasn't begun, can I implement increment 3? Just need some broad strokes on what changes need to be made.

[mario]

First to clarify that we'll (hopefully) never support touching outside folders (e.g. we won't delete or replace things outside the Nextcloud data folder) via 2-way-sync.

That being said, increment 3 has in part started by doing this via this PR: https://github.com/nextcloud/android/pull/2253

The next step would be to completely remove the need for Sync adapter (check the code in syncadapter folder).

If you can do that, while keeping existing functionality (but please let me know you're working on it!), then I can take over and finish increment 3.

Awesome that you want to contribute @holocronweaver btw! :)

[holocronweaver]

First to clarify that we'll (hopefully) never support touching outside folders (e.g. we won't delete or replace things outside the Nextcloud data folder) via 2-way-sync.

Yeah, that sounds like asking for trouble. Though maybe we could support creating symlinks for users so they get the same effect?

As for removing dependence on Sync adapter, I will start looking at the code this week and will try to get out a WIP PR this weekend.

[flohall]

I wait for this better auto upload feature for some time. Currently I use FolderSync Pro for 2-way-sync of important images and data which is handy somehow, but makes it necessary to have two separate applications and also is not free software.

What I am not sure about so far, if you also considered to not just sync media files, but also any other folder with files (like chat backup files etc.). This would make it possible to backup all important data on the phone in a very handy way.

First to clarify that we'll (hopefully) never support touching outside folders (e.g. we won't delete or replace things outside the Nextcloud data folder) via 2-way-sync.

But if you don't want to implement it in that way, how the following is meant?

Increment 6 - sync two way

Somehow files outside nextcloud folder must be replaced/deleted so someone can order/delete photos on the computer on a bigger screen without having the need to do the same again on the device.
Unlike on a desktop/laptop not all applications can work with files just stored in nextcloud.
The application FolderSync can do the 2-way-sync of any folder in android.

Symlinks sound like a reasonable solution - just not sure if it works.

Thanks for the great work so far - looking forward to use the new features.

[mario]

@TheCod-e you can already sync custom folders if you enable expert mode in settings.

As for why I don't want to touch outside folders - it's a bad, bad thing :D Android is going the way of restricting access to your own folder (which will happen sooner or later), and it's increasingly hard to manage outside folders.

[flohall]

@mario thanks for the tip with expert options

ok you say no touching of outside folders - but is there maybe another way of avoiding redundancy of files? just believe it will be confusing to manage files twice (on the phone) and (in nextcloud)

[tobiasKaminsky]

First to clarify that we'll (hopefully) never support touching outside folders (e.g. we won't delete or replace things outside the Nextcloud data folder) via 2-way-sync.

As far as I know we never really talked about this, so please do not rush and let us talk about the overall sync concept before trying to start implementing things.

[holocronweaver]

@tobiasKaminsky @mario To clarify, this has nothing to do with increment 3, which I am working on, right? Seems like an increment 4+ issue.

[mario]

@tobiasKaminsky we talked about this between you, @AndyScherzinger and me months ago. Or did I mistake this for something else? cc @AndyScherzinger

[mario]

@holocronweaver it doesn't. While removing sync adapter isn't a real requirement for stage 3 either, it helps with cleaning up overall sync code.

[mario]

Btw. to clarify ... even one part of increment 4 is already implemented - we already support folder hierarchies in MASTER.

[AndyScherzinger]

That being said, increment 3 has in part started by doing this via this PR: #2253

Just my 2 cents...

2253 doesn't implement the upload existing files option as fare as I can tell while I'd also say that whenever the users activate the auto upload via the cloud icon for a folder the client should check for the existing files' existence on the server _and_ if not present simply upload them. That would imho be the simplest solution for probably >80% of the cases where users are reporting "issues". And if this then works we could then also integrate it into the 3-dot menu to retrigger such logic.

While I am aware that this isn't a real 1-way sync (which isn't defined for increment 3 anyways) it would in my personal opinion still solve most peoples reports and as such help with the overall satisfaction.

And if this would be in place I'd say the next steps afterwards should be the "free up space" feature since this + the "free up space" will solve most use cases people are having (e.g. for the reports of "delayed deletion" because that could than me handled by manually triggering the "free up space" whenever one likes or needs to do that).

[mario]

Well, even 1-way-sync is damn near impossible for outside folders. You never know if server file == local file if it doesn't have the damn etag, and files in random folders won't have them :)

[AndyScherzinger]

true, while for the "simple upload existing files" we can simply upload them, no harm done to the original file on the client :)

[korelstar]

Other sync apps (e.g. Synchronize Ultimate) solve this problem by building a local database which tracks meta data of files in the sync folders when synchronizing. They compare files using file size and modified time (which is what an ETag is often generated from). In nearly all cases, this is sufficient for synchronizing files outside the app dir.

It would be very cool, if a full two-way sync of any folder could be a mid-term goal of the Nextcloud app. This would be nice in order to create backups from local files with the possibility to archive files using Nextcloud.

[AndyScherzinger]

@korelstar with Android limiting external file access more and more with each OS release the issue is that sync would also mean that you can sync a deletion (or not sync a deletion) since many people might want the files to be backed up but no action upon local deletion. So having a real sync on external folders is likely to break sooner or later due to OS changes.

[bam80]

Sorry for interpose, has anybody considered SyncML as a solution for 2-way sync?

[AndyScherzinger]

@bam80 no, everything is based on WebDAV/CalDAV/CardDAV

[ThaDaVos]

If this is done for Android, will this also be done for iOS? To keep them the same?
As of now Android is at 3.x and iOS 2.x last time I checked

[altek86]

any luck yet with Increment 3?

[mario]

No. I have instead implemented increment 4.

[javaisbetterthanpython]

@mario @AndyScherzinger is there a sketched-out plan for implementing Increment 3? What would be necessary to complete this? Any pseudocode available? I recall reading one comment on here somewhere that references a specific class that would be a good place to start, but I checked and didn't see that class existing in the current source.

[javaisbetterthanpython]

cc @tobiasKaminsky

[jospoortvliet]

@mario @AndyScherzinger is there a sketched-out plan for implementing Increment 3? What would be necessary to complete this? Any pseudocode available? I recall reading one comment on here somewhere that references a specific class that would be a good place to start, but I checked and didn't see that class existing in the current source.

Isn't that what the re-scan feature does? @tobiasKaminsky

[jj-uk]

"Uploads -> Force rescan" appears to do nothing at all for me.

Android 5.1.1 / Samsung J3 2016

Nextcloud app v3.2.0

Nextcloud server v13 (can't see actual version in GUI anywhere?)

EDIT:
13.0.5 (found in admin account)

[ftastisch]

As the app description in the Playstore/FDroid includes "Keep your favorite files and folders synced" I was surprised that there is still no auto-sync for mobile devices. This would be increment 6, right?
Are there current plans for implementation?

[jj-uk]

Sync is there and it works. Menu - all files - ... - sync

Auto upload doesn't seem to work correctly though. And foreceive rescan does nothing all all.

[ftastisch]

@jj-uk [Either] I'm blind or meant something different: Is there currently an option to have the android app update (automatically in the background) the files on the smartphone if there have been changes on the server?

[Janhouse]

Auto upload in Android app is currently buggy and doesn't allow pushing all of the files after initial setup. Also if you change settings for auto upload folders, they won't be used for already existing files in the queue and since there is no "push all files to cloud" button, there seems to be no good way of uploading those files.
This "roadmap" has been here for almost two years now. Not sure if Nextcloud lacks developers and funding or they don't prioritize Android because this functionality works great in the iPhone app.

It is weird because I thought that Android is the most used mobile platform in the world and that most people (not organizations though) using private cloud want to automatically back up photos from their phones to their private cloud.

There is a pull request #2873 that should fix auto uploading but it seems to be stuck there for now.

[rugk]

Not sure if Owncloud lacks developers and fundin

You mixed up companies here. :smile:

[Janhouse]

Haha, sorry, I still mix up the names. :laughing:

[jj-uk]

All the database tables still use oc_ prefixes... someone should open a enhancement request for that lolz

[m1cm1c]

Auto upload in Android app is currently buggy and doesn't allow pushing all of the files after initial setup.

This is currently my biggest problem with it. Is there a workaround? Or perhaps a different app to do the job in the meantime? Furthermore, when I set up a custom folder (requires "expert mode" to be enabled), it sometimes doesn't even sync additions.

[madorian]

Auto upload in Android app is currently buggy and doesn't allow pushing all of the files after initial setup.

This is currently my biggest problem with it. Is there a workaround? Or perhaps a different app to do the job in the meantime? Furthermore, when I set up a custom folder (requires "expert mode" to be enabled), it sometimes doesn't even sync additions.

syncthing is a different app that works great, while we wait;)

[Bugsbane]

syncthing is a different app that works great, while we wait;)

Will Nextcloud server be aware of files copied to the server by Syncthing? Do you set up a cron job to scan for new files every 5 minutes, or is there a more elegant way?

[ajvsol]

2-way sync is especially necessary now that Nextcloud 14 supports client-side encrypted folders. I've been using FolderSync for a long time but it can only support unencrypted files.

[Fabian42]

@neomodern Do you mean you used FolderSync with NextCloud? I don't find an option for that in my FolderSync app.

[m1cm1c]

@Fabian42 I simply chose the latest version of OwnCloud supported by FolderSync. It works.

[ajvsol]

^ yes, what he said

[Fr3shF1sh]

Autoupload does not work stable on Android 7. App is excempt from battery optimization.
Even when I force a rescan under Uploads it will not work. Probably because I don't start with empty folders.
But even when I start with empty folders it only works very seldom automatic. Most of the time I have to force a rescan in the Uploads menu.

[Dmitra]

same for Android 8. In addition a had an issue with uploading photos to two folders simultaneously. Specified custom location and the folder which was in the setting by default: "InstantUpload/Camera"

[thrdroom]

At the current state of development, how far away are we from getting fully functional Increment 5 and Increment 6? I want to go fully FOSS(fdroid apps only) and as FolderSync Pro is a paid PlayStore only app, i can't use it nor can i use the nextcloud app without fully functional Increment 5 and Increment 6 features. I don't want to push anyone, im just curious about the status of this two most important features.

[apalacheno]

Fully OT: You may use Syncopoli (from F-Droid) as a temporary workaround. Granted; it is a very minimal solution and not really comparable to the wealth of features of NextCloud or FolderSync, but it allows for basic, automatic and robust one-way-sync.

Setting up syncopoli is a bit tedious; the following tips may get you started:

In the settings of Syncopoli the following additional options have proven to be useful:

-vHrlt --no-perms --delete-excluded --progress

Careful: with --delete-excluded this does a true sync, i.e. all files on the target will get deleted that are not present on the source!
Another gotcha is that the private key needs to be in dropbear format.

Set up a profile with the following options:

• Direction: Local -> Remote
• Profile name: [arbitrary name]
• Source: /sdcard/
• Target: [name of the folder on your server]
• Additional rsync options [modify to fit your personal use case]:

--exclude=.cache/ --exclude=.thumbnails --exclude=cache/ --exclude=pending_blobs --exclude=net.osmand.plus/files/*.obf --exclude=net.osmand.plus/files/roads/ --exclude=net.osmand.plus/files/srtm/ --exclude=net.osmand.plus/files/tiles/ --exclude=net.osmand.plus/files/travel/ --exclude=net.osmand.plus/files/voice/ --exclude=net.osmand.plus/files/wiki/ --exclude=BACKUPS/

Sorry for being off-topic, but having a basic and FOSS-compatible way to sync things between phone and desktop/server is something people reading this bug report may be interested in until Increments 5+6 have been implemented.

[JorisFRST]

Hi,

is there a timeline for ?

Increment 3 - upload existing files
add "upload/sync all existing images/videos/files" as an option to the auto upload folder three-dot menu.

[PetrVladimirov]

It's probably worth to mention that there is a bounty for this particular issue, another for very similar one and one more for another very similar, totalling $500+

@tobiasKaminsky @AndyScherzinger do you think it may help to put additional focus on this issue? Probably at least adding relevant tags and links may increase donations... BTW many thanks for making these features real!

[AndyScherzinger]

Don't know actually... I don't have enough spare time at the moment to spend on developing larger features and I can't tell what @tobiasKaminsky or @ezaquarii might be focusing on.

[derekblankmccoy]

Would this include a simple "keep folder in sync" option? For folders that contain important and time critical files, it is important for those files to be delivered ASAP.
It might be worth opening a new feature request, but maybe like a "high priority" option where files or folders with that label are synced instantly no matter what.

[klawdhfzasjhaa]

Would this include a simple "keep folder in sync" option? For folders that contain important and time critical files, it is important for those files to be delivered ASAP.
It might be worth opening a new feature request, but maybe like a "high priority" option where files or folders with that label are synced instantly no matter what.

That's what I would expect when "starring" a file or folder in the app.

[cantalupo555]

Auto upload in Android app is currently buggy and doesn't allow pushing all of the files after initial setup. Also if you change settings for auto upload folders, they won't be used for already existing files in the queue and since there is no "push all files to cloud" button, there seems to be no good way of uploading those files.
This "roadmap" has been here for almost two years now. Not sure if Nextcloud lacks developers and funding or they don't prioritize Android because this functionality works great in the iPhone app.

It is weird because I thought that Android is the most used mobile platform in the world and that most people (not organizations though) using private cloud want to automatically back up photos from their phones to their private cloud.

There is a pull request #2873 that should fix auto uploading but it seems to be stuck there for now.

I have this same problem, it only performs auto upload of the files that are after the installation and configuration of the APP, the old photos and videos (created before the installation) are not being uploaded.

[Janhouse]

This is why I don't use the Nextcloud App on Android devices. What is the point of the app if it can't sync already existing files?
iOS app devs have solved this issue long, long time ago.

If the already existing pull request contains bad code, why don't you just close it and remove the "has pull request" tag from this case? It seems like someone is working on it, while in reality there is only one commit made a year ago and no one is actually looking into it.

[nazzario]

The Thread is open 2 Years now. I Could remember the nextcloud android automatically never has worked. 2 Years later I have the same issue. Please fix that on android. I want to backup my Photos.
Device: Xiaomi Mi 9 / Android 9 / Nextcloud Version 3.6.2 Please fix

[NESC1US]

that is not exactly true, nextcloud android client automatically uploads data appearing in a setup directory after its setup, this is a limitation which i can live with - to work around it, you just need to manually upload already existing directory content, however i agree that this bug is specially confusing and annoying for new users who then could have easily give up on the nextcloud.

[Janhouse]

@tobiasKaminsky how about removing the "pr exists" tag and rejecting it finally?

[mgutt]

Still waiting for "Increment 3 - upload existing files" and "Increment 4 - folder upload". Uploading existing files should be a default setting. I do not see a reason why it should exist a setting to disable this. I would pay for both features. Maybe you think about a paid version of the client that gets such features x month before the free version.

[javaisbetterthanpython]

tobiasKaminsky opened this issue on Sep 21, 2016

Wow this has been open for a long time. What is required to make this actually get done? It seems like a very popular and commonly requested feature.

[nazzario]

That's true that the sync working after first install. But it's not a really user experience. The function that you can select the Folders for sync are great and give the name too. But I have a lot of work to look which Photos etc. I want to sync because the older one will not automatically sync that is not really nice. I really hope that you can fix but as the comments from other people told the issue is open for two years.

[Henry78]

What is required to make this actually get done?

You could increase the bounty: https://github.com/nextcloud/android/issues/285#issuecomment-491605209

[mgutt]

What is required to make this actually get done?

You could increase the bounty: #285 (comment)

Thanks for the info. Why aren't bountysource links part of the issue description? I sent 15$ and created a google play store review with a hint to bountysource and will post in relevant forum threads.

[Janhouse]

@Henry78, @mgutt , I doubt it will help in any way since existing developers are already employed by Nextcloud and they get paid more than that. There is an open pull request with bad code that hasn't been rejected or reworked in years, suggesting that current Android developer team has too much to do and they are drowning in cases, unable to deal with all the things going on in these cases.
There are probably also no project managers that would go through the ton of bug reports that exist for different Nextcloud projects and set some priorities. I could not even find any publicly available roadmaps, except for the desktop app, which suggests that there is no clear common plan on where things are going and teams might be working in silos.

Nextcloud marketing team keeps introducing some gimmicky features like live video confirmation to gain access to folders, and some other useless features that make it seem like Nextcloud has it all, while in reality it has too many bugs to be used for anything serious.

I guess the only way to get actual support on any of these cases is if you are a paying enterprise customer, which makes sense but it is still sad that they can't hire more people to manage the projects and fix such essential features as folder sync. Their sales must be really low. 😢

@AndyScherzinger, please correct me if I am wrong.

I'll set up this project tonight to see how bad the codebase is and how hard cookie this particular case is but I suspect it isn't great.
Meanwhile if you need a working folder sync on Android - try Syncthing.

[mgutt]

@Janhouse
I hope you are wrong.

I'm using FolderSync (sadly a dead project) by myself, but apps like this (f.e. Syncthing or Synchronize Ultimate) aren't usable for noobs as they do not automatically select camera folders etc. My problem is: I have unlimited cloud storage and it's a nextcloud server. So I need the (working) native nextcloud app or a similar app that supports WebDAV.

Does Syncthing use android backup for it's settings? Maybe I could create all the sync rules for my users and they survive a switch to their next android device. But as the SD card has a random folder number I have a sad feeling about that although it uses the backup function.

[AndyScherzinger]

@AndyScherzinger, please correct me if I am wrong.

Hard to tell since I am a community contributor. Nextcloud as a company likely focuses on providing a good service to its customers so if they don't have an issue with the auto upload then this isn't of a high priority to the dev team (just a guess, nothing ever stated to me in person or else).

I created the UI part for the Auto Upload feature (no ligic, processing etc.) but don't have much time these days for contributing code to the app since my day job is keeping me quite busy these days.

[wweich]

Does Syncthing use android backup for it's settings? Maybe I could create all the sync rules for my users and they survive a switch to their next android device. But as the SD card has a random folder number I have a sad feeling about that although it uses the backup function.

Syncthing can backup / restore it's config to / from /storage/emulated/0/backups/syncthing/. But in most cases you won't be able to write to SD Card, as it currently needs direct access to /storage/... but on modern Android one has to use the file access API to get write access to the SD Card.

[Janhouse]

Oh, sorry @AndyScherzinger, I mistakenly mentioned you because this task is assigned to you. I guess the question was to @tobiasKaminsky.
Andy, but if you don't plan working on this, please give it back to Tobias.

[mase76]

I wonder, the keep-in-sync-function worked as expected in the past? I think about a year ago. Now it is completely broken. This should be the main feature of the app.

[Henry78]

@mase76 You refer to folder keep in sync, i suppose.
In my experience: Files are kept in sync well. Folder sync only lacks detectio and syncing if newly added files.

[mgutt]

@mase76
Try to open recent apps, hold your finger on the NextCloud app and use the lock symbol to forbid the OS to kill the app in the background. I'm using this for FolderSync for a year now without any problems and FolderSync's "Instant Sync" was broken, too.

[mase76]

I disabled battery optimization and allow background activity for the nc app. I also see, that there were syncing processes in tbe android account settings. But nothing happened. Even changes of existing files are not synced, until I trigger it manually.

[mgutt]

@mase76
I did everything what you set up too and Folder Sync never synced files until I used the locking. You should give it a try.

[thrdroom]

@Henry78, @mgutt , I doubt it will help in any way since existing developers are already employed by Nextcloud and they get paid more than that. There is an open pull request with bad code that hasn't been rejected or reworked in years, suggesting that current Android developer team has too much to do and they are drowning in cases, unable to deal with all the things going on in these cases.
There are probably also no project managers that would go through the ton of bug reports that exist for different Nextcloud projects and set some priorities. I could not even find any publicly available roadmaps, except for the desktop app, which suggests that there is no clear common plan on where things are going and teams might be working in silos.

Nextcloud marketing team keeps introducing some gimmicky features like live video confirmation to gain access to folders, and some other useless features that make it seem like Nextcloud has it all, while in reality it has too many bugs to be used for anything serious.

I guess the only way to get actual support on any of these cases is if you are a paying enterprise customer, which makes sense but it is still sad that they can't hire more people to manage the projects and fix such essential features as folder sync. Their sales must be really low. cry

Meanwhile if you need a working folder sync on Android - try Syncthing.

THIS is sadly SO TRUE!

Instead of fixing bugs or extend the features of basic-apps and funktions, nextcloud is only introducing more more features no regular user is using on a day-to-day basis.

I hope they start to change this...Nextcloud has a lot of potential. But it seems like they have a lack of focus.

[nazzario]

The sync folder is very unstable. Sometimes the android client has a message and not found the server and then comes the message Upload failure. But the auto upload is really bad. I hope you can fix it. The Problem is many years now. And when I looking for picture on my Photo folder the preview mostly not really load 70% loading and the rest not. Relaod again sometimes comes sometims not! The Client on Mac and Windows is really good but the android client is disappointed. Thanks for the work for that app but please fix that 2 issues.

[mgutt]

I tested it by myself now:

• installed Nextcloud
• activated auto upload for the camera folder
• the app asks for whitelisting regarding battery management
• shot a photo
• it auto uploads

Next try:

• i kill all apps in the background through the recent apps tab
• shot a photo
• nothing happend (whitelisting does not work!)

Next try:

• I start Nextcloud
• I open recent apps again and lock Nextcloud through the lock icon
• i kill all apps in the background through the recent apps tab (Nextcloud disappears)
• shot a photo
• it auto uploads the photo(!)

This is the same trick I need to use for FolderSync, Tasker, etc This is not really related to Nextcloud. It is something in the Android OS since Android 8 I think (I'm using Android 9 at the moment). But you are right. Other apps like Dropbox do not have this problem. I think they use a database and the android api to wake up the app randomly and check if there is a new file in the folder.

But as long the Nextcloud app is not updated the lock icon trick works perfectly. Try it by yourself.

[Janhouse]

@mgutt try this:

• Uninstall app
• Take photos
• Install app
• Mark folder for auto upload
• Nothing gets uploaded from before auto upload was enabled

I think the app only uses Cursor to get new files. It doesn't list all of the files in them.

[AndyScherzinger]

@mgutt of course, simply since that isn't implemented (yet) - The auto upload implementation at this very moment only upload pictures/videos taken _after_ the folder has been activated. Like you mentioned with the cursor, we query Android's Media Index

[mgutt]

@Janhouse & AndyScherzinger
Ok, this was a definition problem. I answered @mase76 and @nazzario because they were complaining a non working "auto upload" function. They think its a bug, but its not a bug its the aggressive battery managment since Android 8 killing the Nextcloud app in the background so it can not auto upload anymore.

You are talking instead of the feature request "upload existing files". Of course this does not work as it was never implemented.

[mase76]

There is no problem with auto-upload. This works all the time. The problem is to keep changed files, like my keepass db or the notes textfiles in sync.

[mgutt]

Do you have Android 9? There are problems. Look at the app reviews. The main complains are:

• auto upload is broken
• login problems
• does not sync existing files
• interrupting up- and downloads

[AndyScherzinger]

@mase76 did you define a custom folder within auto upload?

@mgutt My device is Android 9 and it works like a charm, no issues ever. It is not specific to an Android version imho but rather OEM or/and device/model issues. So they are really hard to reproduce :/

[mase76]

@AndyScherzinger Auto-upload works as expected. The problem is with changed files in synced folders or new files in these folders. I have a keepass db file synced between Linux desktops and Android. Also a notes folder full of txt files. These are permanently changed by the client machines. The desktops get the changes, but Android doesn't, until I trigger the sync of the folder(s) manually. Even when the file changes were made by the Android machine.
This worked fine in the past, but stopped working during Android 8. Now I am on 9. Because auto-upload works on my device, the nc process was not killed. I allow bg activity and disabled battery optimizing. I see the account synced in Android's account settings, but no changes were applied.

[Henry78]

@mase76 perfect summary! It worked a long time, since OwnCloud(!) but ia broken now. Keepass and notes. Exactly my use case.

[korelstar]

For synchronizing notes, you could alternatively use the notes app (for Nextcloud server and for Android).

[swaechter]

By accident I found a workaround how you can add existing images etc. for example phone screenshots:

1. Attach your phone via USB cable to your PC and allow the MTP connection
2. Somewhere on your PC, create a directory so you can Ctrl + X & Ctrl + V files into this directory
3. Navigate to the image location on the phone via explorer on the PC
4. Ctrl + X and Ctrl + V all images on your phone to the temporary directory on the PC
5. Ctrl + X and Ctrl + V all images from your PC directory back to your phone

After this, the Nextcloud sync kicks in and will upload all images. I tried this for my phone screenshots only

[mgutt]

@swaechter
You do not need a pc. You can move all files through an android file manager to an other folder and copy them back (if you have enough space). But both ways overwrite the file creation date. This is not a problem if your filename contains the date, but if not, you loose this data. If the video filenames does not contain the date you loose the record date forever as video files do not contain EXIF data like photos.

[Kraowe]

What I don't understand is that the iOS app has this functionality, you can check to upload all existing photos in it. Why not just look at how it's done in the iOS app? And then implement the same in the Android app?

[trymeouteh]

I would like to have Auto Upload changed to Sync Connections and work just like on the desktop client. Instead a uploading it will upload and download and you can set it on how often it will scan for uploads and downloads.

[clawoflight]

Seriously, how is this feature not prioritized more instead of all the gimmicks? This is the core functionality of a cloud offering!

[szaimen]

Yes, I think so, too.
But in the moment there are so many bugs in this app, that need to get fixed before any enhancement can be made, imho.
I also cannot understand while just one developer from Nextcloud is actively working on this app?

[W-Hamra]

What's so strange about that? If you know people willing to code for free, invite them.

[szaimen]

Because this should be one of the most important apps for Nextcloud and therefore shouldn't have that much bugs.

[Janhouse]

And Nextcloud GmbH is a company with paid employees.

[lemmy04]

I'm kind of opposing to "increments" 5 and 6... unless the actual "sync" was optional.

My personal workflow regarding the picture sync folders goes like this:

• take picture
• picture goes in nextcloud
• desktop client pulls pictures off the cloud
• i move pictures into my final image storage (fileserver, managed with digikam on linux)
• desktop client deletes pictures off the cloud

at this point, a two way sync would delete the pictures off my phone - which I might not want to happen just yet
also, sometimes I delete pictures off the phone before I move them into digikam on my desktop, which also would not be possible with a two way sync.
So if those two additions get implemented they would need to be optional for me...

[clawoflight]

@lemmy04 simple, distinguish between auto-upload and sync, which are completely different concepts with distinct use-cases.

[lemmy04]

@lemmy04 simple, distinguish between auto-upload and sync, which are completely different concepts with distinct use-cases.

that is exactly what I'm saying, I'd have auto upload turned on but full sync turned off.

[clawoflight]

And I'm saying they should be distinct features.
Auto-upload based on local folders like it is now, and sync as extension of the sync functionality we have now, based on remote folders.

[lemmy04]

And I'm saying they should be distinct features.

exactly my point.

Auto-upload based on local folders like it is now, and sync as extension of the sync functionality we have now, based on remote folders.

But WITH actually uploading what is in the folder when you initially turn on auto upload, see #4430

[Bugsbane]

Is there any kind of roadmap for increment 3? It seems that it was skipped for increment 4. Wondering if increment 3 or 5 will likely be next,

[tobiasKaminsky]

Increment 3 needs a server side support for checksums, otherwise we cannot make sure which files should be uploaded.

[lemmy04]

...how are the desktop client and the iOS client avoiding this problem then, if it can't be done without server side checksums?

[invertium]

So is there any ETA for increment 3? It seems crazy that normal sync is possible but the auto-upload feature doesn't have it.

[jospoortvliet]

...how are the desktop client and the iOS client avoiding this problem then, if it can't be done without server side checksums?

They work rather differently. They fully control and monitor a folder, and keep it in sync all the time, storing information about each file in a database. They use a variety of heuristics, some very platform dependent, to determine when a file has to be synced. Even then, this has issues, which is why work on a virtual drive is happening.

I can indeed imagine that on a mobile phone, this is much harder and has more risks of data loss. You can of course say "if a local file has the same name and size as one on the server, don't sync." but that will occasionally mean data loss. So how else do you know if a file is the same as a local one? Download and compare, or checksums, are the only way.

The first is processing and bandwidth intensive, the second needs server support.

Of course, it could 'just' upload all missing photos, purely based on file name and size. I personally think that that is enough - photos are typically stored with a date after all.

For files, though, this would be risky. The chance that you have a few documents with the same size and the same name, at some point, is not anywhere near close to zero.

WRT the people who complain this was not high enough priority: we are indeed a company with paid employees. Paid by our customers. If you're not a customer, you're not paying us and you really can't expect us to work for you.

We set our priorities based on our own long term goal and on the needs of our customers. If you'd like to influence the choices we make, either contribute code, or become a customer...

[Janhouse]

We set our priorities based on our own long term goal and on the needs of our customers. If you'd like to influence the choices we make, either contribute code, or become a customer...

I guess Germans do it differently but I usually pay for software that actually works. 😆 Corect me if I am wrong, but Nextcloud GmbH is a software product company, not a consulting company that mainly does custom development.
If the product has lots of bugs and roadmap doesn't mention solutions to those problems, I don't use such software.
I donate more to open source projects yearly than I pay for SaaS subscriptions but in cases like private cloud there aren't any good self hosted options for private use. (I keep looking though)

I guess Nextcloud is mainly intended for use by small/medium companies and public organizations but that should be clearly stated in the marketing materials. People who are used to iCloud/Google cloud have some feature expectations which clearly are not met in case of Nextcloud, and seems like those features are not planned any time soon.

[jospoortvliet]

@Janhouse It seems we find ourselves in a chicken-egg situation, then. You want us to first fix all your issues for free, then you might want to 'donate'. We would like to get paid for our work and have some guarantee we get our salaries each month.

We don't ask you to pay for a software license. Nextcloud is open source, you can download our software for free, use it how you like and we don't solicit donations, least of all from small businesses or private users.

We're a serious business and our customers are serious, usually large enterprises. Those typically do a Proof of Concept before they decide on any solution. We help with that, advising on the installation and fixing issues they might encounter, or we commit to fixing them (or implementing features they need) after the contract is signed.

Of course, you can try to do a PoC without us, but then, yes, we won't commit to fix your issues for free and you typically won't get as good an idea what Nextcloud can do for you as you would when working with us.

In short, we make Nextcloud work for our customers. Everybody has different needs and different priorities and all software has bugs, so we can't make Nextcloud work perfectly for everyone on github in a reasonable time (before the the sun engulfs earth). That is the service our customers pay us for (among other things).

If you are serious about using Nextcloud in your business, you can contact our sales team to get our help with a PoC. If you don't want that, you can always contribute your code or wait until somebody else does.

[alkemyst]

@jospoortvliet I think you just clearly summarised the position of your company on this point.
Since this appears to be a quite popular request for enhancement, though, would it be possible to crowdsource it somehow? I mean, if it were requested by a large company that company could pay for a similar feature: would it be possible to sum the requests of several private users, somehow?

Thanks for your feedback
Stefano

[AndreGeddert]

Since this appears to be a quite popular request for enhancement, though, would it be possible to crowdsource it somehow? I mean, if it were requested by a large company that company could pay for a similar feature: would it be possible to sum the requests of several private users, somehow?

i would pay a small amount for that.

[AndreGeddert]

Maybe we can use this? https://www.bountysource.com/teams/nextcloud/bounties

[derekblankmccoy]

Surely the upload timestamp tells you if the file needs to be synced or not, no need for checksums. Am I missing something?

[ajvsol]

"Increment 6 - sync two way" is referenced by this issue here and it already has a Bountysource for it, of which $200 was added by Nextcloud themselves so far. I'd suggest contributing more here for funding this long-awaited feature.

[jospoortvliet]

@alkemyst @AndreGeddert Bounties are generally useful to attract volunteers to a subject. The suggestion from @ajvsol is excellent. Hopefully somebody wants to get on that!

And yes, in theory, our company could pick up these tasks, but that is unlikely for 3 reasons:

• it's quite a bit of money to actually pay a realistic amount for something like this. Esp as I think this item is a LOT of work. If you'd pay somebody a fair salary by the hour, I'm betting this would be in the 10K area (!)
• we are rather flooded with customer projects already
• worse, as a internal rule we only do consulting (custom development) if it goes together with a subscription, as that is our main source of (recurring) revenue. That is of course not the case/relevant for a bountysource-style payment so this would be on the bottom of our priority list...

Thinking about it, though, there are a third and fourth way to help get this issue done faster.

As third, just think: the bigger Nextcloud is as a community and user base, the more customers we tend to get, and the more people we can hire. And thus - we can do more work like this. So yes, your advocacy and spreading the world has a real effect on what we can do to improve the product. ALSO for home users, not just for customers. While we, as a reality, have to focus on earning our salaries, we personally all very much want to make Nextcloud great for private users, and we spend time on that. After all, we started Nextcloud to help people protect their privacy. And companies don't need to have privacy :smile:

Last but not least, you can simply be NICE. You'd be surprised how much grumpyness there is directed at our developers from users of a product they don't even pay for - and that is demotivating and absolutely not good for their productivity. A few nice words can be very helpful!

Of course, these last two are rather long term and indirect ways of helping. But we appreciate it!

:hugs:

[Janhouse]

I suggest that you remove "pr exists" tag and reject the pr, if that code is not relevant anymore.
I doubt anyone will start working on this if they think someone is already on it.

Or even better - split this into smaller cases because description lists multiple things, and some of them seem to be working.

[poiNt3D]

I would love to recommend Nextcloud to all my friends, but it just doesn't work on Android.
Correct me if i'm wrong, but Nextcould is presented like an open source alternative to Dropbox (Wikipedia agrees). File sync should be the main feature. It's like building a car without wheels. You say you want bigger community? Android is the most popular mobile platform. So you want this tiny community to find you 10k. It's not how Open Source works. First you need your product to be useful for everyone and then money will find you.
I apologize if i sound too harsh, but it's just how it is. Not trying to hurt anyone's feelings.

[lemmy04]

I would love to recommend Nextcloud to all my friends, but it just doesn't work on Android.

...and the attitude of some devs definitely does not improve it's standing.

[lemmy04]

...how are the desktop client and the iOS client avoiding this problem then, if it can't be done without server side checksums?

They work rather differently. They fully control and monitor a folder, and keep it in sync all the time, storing information about each file in a database. They use a variety of heuristics, some very platform dependent, to determine when a file has to be synced. Even then, this has issues, which is why work on a virtual drive is happening.

I can indeed imagine that on a mobile phone, this is much harder and has more risks of data loss.

if it works on iOS (which is a mobile phone) why can't it work the same way on Android?

I'm being serious here - I use the auto upload feature for the pictures on my phone so that I don't have to take the SD card out or find a USB cable, especially now that the USB port is usually blocked by the headphones. I would be perfectly fine with "fully control and monitor", even if that means pictures will be deleted from my phone if i delete them from the sync folder on my PC - that's how I do it anyway: look at pictures, move them FROM the nextcloud folder TO either trash or my picture library.

All this stalling and meta discussing is NOT helping the public image that the user has of nextcloud.

But I guess that is a problem that a LOT of open source projects have...

[Ablu]

I really do not want to spam this topic more than it already is... But all the negativity really makes me sad. For me Nextcloud on Android definitely is working! Enhancement to the syncing would of course be very useful, but for a freely usable project it already works better than what I could hope for!

I get all the comments as mails because I am interested into further development on this topic. The negative comments do not really contribute to that...

[lemmy04]

For me Nextcloud on Android definitely is working!

Are you saying that you can take a photo with your phone, and no matter what connectivity state your phone is in you can be always 100% sure that the photo will be uploaded to your cloud as soon as your phone reaches the right "connectivity state"?
Because frankly, I can't - and on my phone the client even is allowed to upload through cellular data...

[Ablu]

Are you saying that you can take a photo with your phone, and no matter what connectivity state your phone is in you can be always 100% sure that the photo will be uploaded to your cloud as soon as your phone reaches the right "connectivity state"?

Exactly :). I only upload on Wifi. If wifi goes away during upload it will try again later and eventually succeed. Never had a failure with that. If it does not work for you, I fear you hit a different issue. This issue seems to be about adding more advanced synchronisation features.

[AndyScherzinger]

I am with @Ablu, the feature has always worked for me on my OP3T which runs on Lineage OS nightlies.

[Cyber1000]

For me (Honor 8 with Emui 8/Android 8), it works without problems too, 100% for the last 6 months with mainly photos/videos. But I upload only on wifi and if connected to power and have disabled the aggressive power saving of honor 8 for this app.

[lemmy04]

I'm sorry to be negative about it, but on all my devices "auto" upload means that for about 50% of all the pictures I take I have to manually open the nextcloud app, go to the "uploads" tab, and swipe down to refresh, to get the app to actually see the new pictures and realize that there are uploads to be done.

And there seems to be no logic behind then and why it happens...

The only connecting factor seems to be the fact that all my devices store their pictures on SD card, I could reactivate my old S6 that does not have one and see what happens on that one.

[Janhouse]

I noticed that on new Android devices if app doesn't have the "autostart" permission (which is not given by default), and if Nextcloud is off while the photo is taken, it never figures out that it has to upload something unless you open the app and try to refresh something. (Tested on Xiaomi device)
So you have to give every sleep related permission to the app manually and then it seems to work more reliably.

Meanwhile on iOS the Nextcloud app uses some sort of geofence feature that works around the lack of "autostart" on iOS. Basically when the location changes, it is allowed to wake up and do some work for limited time.

I guess Android app has to be updated to be more in line with newer Android versions.

[lemmy04]

I checked: on my most current device, a Samsung Tab S5e running Android 9, the app does have the autostart permission.
What I don't have anymore on that device is the option of "locking" the running app so that "close all apps" leaves it alone, which sort of seems to be a working workaround on my phone...

[rugk]

I guess you issues generally are the heavy optimization of Android and especially some vendors trying to prevent high battery usage and thus stop/kill many, many apps. See https://dontkillmyapp.com/ for potential solutions or similar.

[jospoortvliet]

I think @rugk is right. This auto-upload feature works fine for the vast majority of phones, and isn't the thing we are discussing here - this topic was about full folder synchronization.

Sadly some phone vendors do crazy stuff that breaks auto upload, and it is near impossible to work around every weird thing every single vendor does PLUS keep up with all new weird things they do on new versions of their phones. If your phone is one where things don't work well and you want it fixed, complaining won't do much. Determine what is wrong and provide information - because we don't have your phone so we don't know what is wrong and how to fix it... At the very least, create an issue with as much info as possible - like, in what situations DOES it sync (when the app is running? When you disabled all powermanagement? etc), when it doesn't, and of course exactly what phone and software you use. Maybe a bit of googling can tell you or somebody else how to fix it and that fix might possibly be applied to our app.

[Janhouse]

@jospoortvliet, no, it does not really work fine on Android. It uploads only files created after auto upload was enabled. This means that if I have phone with lots of photos and I install Nextcloud, none of those images will be automatically uploaded.
iOS app doesn't have this problem.

[AndyScherzinger]

@Janhouse this is known and has never been implemented and thus doesn't work on any device running Android.
So this isn't even a Sync discussion while your point then simply is implementing existing pictures when auto upload is being activated.

What I don't know is how iOS client is handling this (technically speaking). Maybe somebody can shed some light on this?

[jospoortvliet]

@Janhouse yeah, I personally would also like that feature to be implemented, no doubt that'd be cool.

A full 'sync' that is two-way is very hard, but a one-time upload-all for a folder would be nice. Or even a dumb "upload unless file exists" or even just bloody overwrite them... would all be fine for me.

[AndyScherzinger]

A full 'sync' that is two-way is very hard, but a one-time upload-all for a folder would be nice. Or even a dumb "upload unless file exists" or even just bloody overwrite them... would all be fine for me.

As for "upload existing", there is #2873 while is a starting point but nobody has been able to get the people involved to an common understanding and agreement hwo this should wor exactly in the first place thus this feature is simply "pending".

[Henry78]

What I don't know is how iOS client is handling this (technically speaking). Maybe somebody can shed some light on this?

I also am really interested!

[Janhouse]

Probably keeps list of files locally and marks which files don't have to be uploaded again.
Android app on the other hand does this weird Cursor thing.

[jospoortvliet]

let me link to the successor PR: https://github.com/nextcloud/android/pull/4788

[mjflower]

I can't believe this is still an issue after all this time. I recently installed the app on the kids (have to have an apple because the school mandates it) devices and it uploaded all previously taken images without issue. Why can the android apps have this option, I don't get it.

[jospoortvliet]

I can't believe this is still an issue after all this time. I recently installed the app on the kids (have to have an apple because the school mandates it) devices and it uploaded all previously taken images without issue. Why can the android apps have this option, I don't get it.

Oh, it isn't so hard to understand. Why did you not write this feature?

No time? I can imagine. Happens to be true for everyone here, otherwise it had be done by somebody. I personally would love this feature, but I don't have time to learn programming and then do it. Hope that helps explain why it isn't done.

If you really want this done, you can do it, of course, your kids and everyone else will be very grateful if you did!

Until then, I'll close this discussion as it isn't going anywhere.