Android: Scanning all files on first start needed?

[aronovgj]

Actual behaviour

• Scans all files in all subfolders
• Continues to do so even if in background
• The loading indicator constantly is moving
  

  Expected behaviour

1. Load files and folders in /

2. Nothing until you open a new folder

   Steps to reproduce

3. Open app with an accout having tens of thousands of files

   Environment data

Android version:
cm 12.1

Device model:
I9505

Stock or customized system:
cusom

Nextcloud app version:
1.1.0 RC

Nextcloud server version:
9.0.51

Logs

Nextcloud log (data/nextcloud.log)

{"reqId":"xv9o12FjV\/7jg9PH4UGh","remoteAddr":"xxx.xxx.xxx.xxx","app":"admin_audit","message":"Login successful: \"me\"","level":1,"time":"2016-06-28T20:44:13+02:00","method":"PROPFIND","url":"\/remote.php\/webdav\/2635978\/PSE-Info_WS09-10\/Aufgabenblatt_5\/Aufgabe_5\/images\/","user":"me"}
{"reqId":"p4PJN7Kd9mPvIlUh0jKX","remoteAddr":"xxx.xxx.xxx.xxx","app":"admin_audit","message":"Login attempt: \"me\"","level":1,"time":"2016-06-28T20:44:14+02:00","method":"PROPFIND","url":"\/remote.php\/webdav\/2635978\/PSE-Info_WS09-10\/Aufgabenblatt_6\/","user":"--"}
{"reqId":"p4PJN7Kd9mPvIlUh0jKX","remoteAddr":"xxx.xxx.xxx.xxx","app":"admin_audit","message":"Login successful: \"me\"","level":1,"time":"2016-06-28T20:44:14+02:00","method":"PROPFIND","url":"\/remote.php\/webdav\/2635978\/PSE-Info_WS09-10\/Aufgabenblatt_6\/","user":"me"}
{"reqId":"msC7Cjijp4wciCwNupfi","remoteAddr":"xxx.xxx.xxx.xxx","app":"admin_audit","message":"Login attempt: \"me\"","level":1,"time":"2016-06-28T20:44:14+02:00","method":"PROPFIND","url":"\/remote.php\/webdav\/2635978\/PSE-Info_WS09-10\/Aufgabenblatt_6\/","user":"--"}
{"reqId":"msC7Cjijp4wciCwNupfi","remoteAddr":"xxx.xxx.xxx.xxx","app":"admin_audit","message":"Login successful: \"me\"","level":1,"time":"2016-06-28T20:44:14+02:00","method":"PROPFIND","url":"\/remote.php\/webdav\/2635978\/PSE-Info_WS09-10\/Aufgabenblatt_6\/","user":"me"}
{"reqId":"A0PL+4fcjfz+qLKa4QoJ","remoteAddr":"xxx.xxx.xxx.xxx","app":"admin_audit","message":"Login attempt: \"me\"","level":1,"time":"2016-06-28T20:44:15+02:00","method":"PROPFIND","url":"\/remote.php\/webdav\/2635978\/PSE-Info_WS09-10\/Aufgabenblatt_6\/Aufgabe_1\/","user":"--"}
{"reqId":"A0PL+4fcjfz+qLKa4QoJ","remoteAddr":"xxx.xxx.xxx.xxx","app":"admin_audit","message":"Login successful: \"me\"","level":1,"time":"2016-06-28T20:44:15+02:00","method":"PROPFIND","url":"\/remote.php\/webdav\/2635978\/PSE-Info_WS09-10\/Aufgabenblatt_6\/Aufgabe_1\/","user":"me"}
{"reqId":"azIPa\/nuRkzRwrd0gNwO","remoteAddr":"xxx.xxx.xxx.xxx","app":"admin_audit","message":"Login attempt: \"me\"","level":1,"time":"2016-06-28T20:44:15+02:00","method":"PROPFIND","url":"\/remote.php\/webdav\/2635978\/PSE-Info_WS09-10\/Aufgabenblatt_6\/Aufgabe_1\/","user":"--"}
{"reqId":"azIPa\/nuRkzRwrd0gNwO","remoteAddr":"xxx.xxx.xxx.xxx","app":"admin_audit","message":"Login successful: \"me\"","level":1,"time":"2016-06-28T20:44:15+02:00","method":"PROPFIND","url":"\/remote.php\/webdav\/2635978\/PSE-Info_WS09-10\/Aufgabenblatt_6\/Aufgabe_1\/","user":"me"}

[AndyScherzinger]

@tobiasKaminsky is this to be considered a bug? It's a sync and needs to check the whole thing so it can detect changes for favorite files, doesn't it? Else we would have to implement a more fine grained sync but I would consider it an enhancement. One that makes sense but an enhancement nonetheless.

Thanks for reporting thus @aronovgj

[aronovgj]

If you ask me, if it drains my battery for an hour (and sends data back and forth, although it is not too much) while running in the background, I consider it a bug.

[AndyScherzinger]

That is true since it should not do that but that doesn't necessarily have to do with syncing the whole file tree info, not the files themselves though.
Yes draining the battery, sync taking an hour, I'd also consider that a bug.

[przybylski]

This is expected behavior. When I was implementing this there was no way to tell is subtree modified. I don't know is it still true. If so, then it would be nice to introduce smarter way of refreshing mechanism.

[przybylski]

To be clear it's not expected to drain battery but to refresh the entire file tree.

[AndyScherzinger]

@przybylski totally agree with you that's also what I tried to express with my comment, but you put it way clearer :+1:

[aronovgj]

That behavior stopped for whatever reason. No idea when though.

[aronovgj]

Oh it is back again. New finding: it only syncs for hours when freshly installed. I was testing the beta lately which involved a lot of reinstalling the app. So to avoid the issue: install the app, open it, force stop it, reopen it. And now it does not sync forever anymore. Could not see any other issues after aborting the filetree scan (if that is what it really does).

[tobiasKaminsky]

As far as I know there is a once a complete server sync (metadata, not file content) directly after installation. Depending on your account size it might take a while.

[aronovgj]

Yes, I understand. My question is why this is done in the first place if I don't need to let it finish syncing and everything works anyway. Why is this needed? Even the folder sizes are displayed somewhat correctly without finishing the sync

[tobiasKaminsky]

I updated the title. We will have a look into this.

[tobiasKaminsky]

Sorry for coming back so late.
I think the initial file scan is for favorited files which are kept in sync. As the system does not know which files are favorited, it must scan the whole file tree.

With our new approach (thanks @mario) we can now search directly for favorites and thus omit the inital file scan.
Or is there something I miss?
@mario @AndyScherzinger @przybylski

[AndyScherzinger]

I do think so @tobiasKaminsky, we should be able to gather that info via the search, yes :)

[mario]

Only on NC12+.

[tobiasKaminsky]

Right, then we can enhance this for >=NC12

[tobiasKaminsky]

Thinking about this again, I do not know why there is an initial scan at all

• "Favorites" on the server are not the same as "set as available offline" so my idea above is wrong
• as this is about adding a new account/first start then there also cannot be files "set as available offline" -> no scan needed
• right after adding a new account there is a sync for the root folder

@przybylski

When I was implementing this there was no way to tell is subtree modified

For what do you need to know if the subtree is modified?

[NiklasRosenstein]

I can confirm this behaviour, unfortunately. 😅 I just checked my NGinx log and was wondering where all these PROPFIND requests were coming from, for every single file in my cloud (and there's a goddamn lot of it). It must have been issuing all these requests for days now and hasn't finished yet.

92.218.241.146 - niklas [12/Sep/2017:17:01:04 +0200] "PROPFIND /remote.php/webdav/data/Cinema%204D/Plugin%20Prototypes/c4dpl-libload-test/ HTTP/1.1" 207 798 "-" "Mozilla/5.0 (Android) ownCloud-android/1.4.3"
92.218.241.146 - niklas [12/Sep/2017:17:01:04 +0200] "PROPFIND /remote.php/webdav/data/Cinema%204D/Plugin%20Prototypes/c4dpl-libload-test/ HTTP/1.1" 207 5217 "-" "Mozilla/5.0 (Android) ownCloud-android/1.4.3"
92.218.241.146 - niklas [12/Sep/2017:17:01:05 +0200] "PROPFIND /remote.php/webdav/data/Cinema%204D/Plugin%20Prototypes/c4dpl-libload-test/.git/ HTTP/1.1" 207 803 "-" "Mozilla/5.0 (Android) ownCloud-android/1.4.3"
92.218.241.146 - niklas [12/Sep/2017:17:01:05 +0200] "PROPFIND /remote.php/webdav/data/Cinema%204D/Plugin%20Prototypes/c4dpl-libload-test/.git/ HTTP/1.1" 207 9589 "-" "Mozilla/5.0 (Android) ownCloud-android/1.4.3"
92.218.241.146 - niklas [12/Sep/2017:17:01:05 +0200] "PROPFIND /remote.php/webdav/data/Cinema%204D/Plugin%20Prototypes/c4dpl-libload-test/.git/branches/ HTTP/1.1" 207 807 "-" "Mozilla/5.0 (Android) ownCloud-android/1.4.3"
92.218.241.146 - niklas [12/Sep/2017:17:01:06 +0200] "PROPFIND /remote.php/webdav/data/Cinema%204D/Plugin%20Prototypes/c4dpl-libload-test/.git/branches/ HTTP/1.1" 207 902 "-" "Mozilla/5.0 (Android) ownCloud-android/1.4.3"
92.218.241.146 - niklas [12/Sep/2017:17:01:06 +0200] "PROPFIND /remote.php/webdav/data/Cinema%204D/Plugin%20Prototypes/c4dpl-libload-test/.git/hooks/ HTTP/1.1" 207 808 "-" "Mozilla/5.0 (Android) ownCloud-android/1.4.3"
92.218.241.146 - niklas [12/Sep/2017:17:01:07 +0200] "PROPFIND /remote.php/webdav/data/Cinema%204D/Plugin%20Prototypes/c4dpl-libload-test/.git/hooks/ HTTP/1.1" 207 7596 "-" "Mozilla/5.0 (Android) ownCloud-android/1.4.3"
92.218.241.146 - niklas [12/Sep/2017:17:01:07 +0200] "PROPFIND /remote.php/webdav/data/Cinema%204D/Plugin%20Prototypes/c4dpl-libload-test/.git/info/ HTTP/1.1" 207 805 "-" "Mozilla/5.0 (Android) ownCloud-android/1.4.3"
92.218.241.146 - niklas [12/Sep/2017:17:01:07 +0200] "PROPFIND /remote.php/webdav/data/Cinema%204D/Plugin%20Prototypes/c4dpl-libload-test/.git/info/ HTTP/1.1" 207 1631 "-" "Mozilla/5.0 (Android) ownCloud-android/1.4.3"

etc. etc. It Seems like the only way to disable it is to remove the account from the App.

I think it should really just load the contents of a directory when you want to view it (which it does when first starting the app anyway, you even need to swipe down to actually get the list of files in the root directory at first).

[aronovgj]

@NiklasRosenstein have you tried killing the app? That "fixed" it for me.

[Mickael-Martin]

Hi !
Same issue one my phone (last update) and NC 12 (last update too). All my directories are scanned. Moreover, the operation is very long (many directories) and it's drain my battery.

[tobiasKaminsky]

@mario @AndyScherzinger @przybylski has any of you more insights than me and can answer my question? https://github.com/nextcloud/android/issues/82#issuecomment-312211742

For me it seems as if we can disable this completely...

[AndyScherzinger]

No idea, sorry. :(

[mario]

I will take a look next week

On Thu, 12 Oct 2017 at 17:05, Andy Scherzinger [e-mail]
wrote:

No idea, sorry. :(

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/nextcloud/android/issues/82#issuecomment-336166315,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAAWsov_g36gZaSqpsEHgU-RfJ2t1ucRks5srirIgaJpZM4JAax6
.

[xXSTrikeXx]

right after adding a new account there is a sync for the root folder.
Maybe another layer adding to "WELCOME_WHATS_NEW"-MSG a button to open a layer to bring the sync status up to date. So it´s really only on first start, where no uplaods has been taken. Then the behavior should be to upload all stuff and then the autouploader should do the rest of work.
refs: #285 #1328