Android: New autoupload not respecting custom directories & errors

[jasonbayton]

Actual behaviour

• When a photo is automatically uploaded, it creates a default instantupload folder in NC root and populates files within that folder (and respective subfolders it creates)

Expected behaviour

• As I've selected custom folders for device folders to upload to, I expect it to respect those.

Environment data

Android version: 6.0.1

Device model: Vodafone VFD 900

Stock or customized system: Stock

Nextcloud app version: 20170206

Nextcloud server version: 11.0.1

Logs

The following shows up when I click on the uploads menu item:

************ CAUSE OF ERROR ************

java.lang.RuntimeException: Unable to start activity ComponentInfo{com.nextcloud.android.beta/com.owncloud.android.ui.activity.UploadListActivity}: java.lang.NullPointerException: Attempt to invoke interface method 'java.util.Iterator java.util.List.iterator()' on a null object reference
at android.app.ActivityThread.performLaunchActivity(ActivityThread.java:2452)
at android.app.ActivityThread.handleLaunchActivity(ActivityThread.java:2512)
at android.app.ActivityThread.access$900(ActivityThread.java:158)
at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1364)
at android.os.Handler.dispatchMessage(Handler.java:102)
at android.os.Looper.loop(Looper.java:148)
at android.app.ActivityThread.main(ActivityThread.java:5515)
at java.lang.reflect.Method.invoke(Native Method)
at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:764)
at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:654)
Caused by: java.lang.NullPointerException: Attempt to invoke interface method 'java.util.Iterator java.util.List.iterator()' on a null object reference
at com.owncloud.android.datamodel.UploadsStorageManager.getPendingJobs(UploadsStorageManager.java:402)
at com.owncloud.android.datamodel.UploadsStorageManager.getCurrentAndPendingUploads(UploadsStorageManager.java:390)
at com.owncloud.android.ui.adapter.ExpandableUploadListAdapter$1.refresh(ExpandableUploadListAdapter.java:150)
at com.owncloud.android.ui.adapter.ExpandableUploadListAdapter.loadUploadItemsFromDb(ExpandableUploadListAdapter.java:620)
at com.owncloud.android.ui.adapter.ExpandableUploadListAdapter.<init>(ExpandableUploadListAdapter.java:185)
at com.owncloud.android.ui.fragment.UploadListFragment.onStart(UploadListFragment.java:103)
at android.support.v4.app.Fragment.performStart(Fragment.java:2106)
at android.support.v4.app.FragmentManagerImpl.moveToState(FragmentManager.java:1146)
at android.support.v4.app.FragmentManagerImpl.moveToState(FragmentManager.java:1290)
at android.support.v4.app.FragmentManagerImpl.moveToState(FragmentManager.java:1272)
at android.support.v4.app.FragmentManagerImpl.dispatchStart(FragmentManager.java:2154)
at android.support.v4.app.FragmentController.dispatchStart(FragmentController.java:212)
at android.support.v4.app.FragmentActivity.onStart(FragmentActivity.java:610)
at android.support.v7.app.AppCompatActivity.onStart(AppCompatActivity.java:178)
at com.owncloud.android.ui.activity.BaseActivity.onStart(BaseActivity.java:189)
at com.owncloud.android.ui.activity.FileActivity.onStart(FileActivity.java:181)
at android.app.Instrumentation.callActivityOnStart(Instrumentation.java:1250)
at android.app.Activity.performStart(Activity.java:6476)
at android.app.ActivityThread.performLaunchActivity(ActivityThread.java:2415)
... 9 more

************ DEVICE INFORMATION ***********
Brand: Vodafone
Device: VFD900
Model: VFD 900
Id: MMB29M
Product: Smart_platinum7

************ FIRMWARE ************
SDK: 23
Release: 6.0.1
Incremental: vH8H-0

[mario]

Thank you @jasonbayton :)

[jasonbayton]

To add, it's also uploading from other folders where autoupload is disabled. Camera for example is currently backed up by the non-beta and doesn't need to be done twice, which is exactly what it's doing in a different folder structure to how I set it up!

[mario]

@jasonbayton right, so you have camera auto-upload enabled on the non-beta app, but not on beta?

Unsure if the behaviour you describe is something that can be avoided at the moment :-/

[jasonbayton]

Let's ignore the non-beta app. So:

Currently camera upload is disabled through the autoupload menu, it has a line through the cloud icon so I know it shouldn't work, and yet it's uploading to a directory I didn't specify anyway.

[mario]

Ok, I've isolated the issue of not uploading to the proper directory, but can't reproduce the issue where it uploads stuff even if that folder is disabled for autouploads :-/

[jasonbayton]

Even if it's the camera folder? Is there anything special about the camera folder historically that may attempt to upload even if disabled?

I've got a new camera folder created this morning to a folder directory I've never had (as it's all custom locations) and over a gig of camera files uploaded so far...

Unless because it _hasn't_ been working, it's been queuing it all up? I imagine the camera backup was selected for a while but I ignored it since it didn't work.

Oh, and it's not creating subfolders for month/year either.

[jasonbayton]

Mario I think there's been a queue forming, with the locations of files to be backed up stored against the files in the queue. It's now switched to uploading them where they should be going (therefore respecting the settings against each directory).

Literally seconds after installing this update it started uploading and with the error on the upload screen I pasted I've been having trouble seeing where and how things are being uploaded.

It's been destroying my battery uploading ~7GB of queued files so far!

[mario]

@jasonbayton I'm investigating - sorry about the trouble, and thank you for helping me investigate this!

[jasonbayton]

Besides the error in the upload area, it looks like it's behaving as it should be actually. I wasn't aware it was queuing all this time (I've had the non-working beta on the device for several months) so it's caught me by surprise. I blame @tobiasKaminsky :)

What I have noticed is it's not respecting the date of the whatsapp media. There are images from 03/2016 being uploaded to 02/2017 for example. Is it not set to interrogate image data on non-camera images?

[mario]

@jasonbayton can I please ask you to open a separate issue for that? Reason being I probably won't be able to fix this particular issue for 1.4.2 (no time :()

[jasonbayton]

Sure, I'll do that shortly.
I think I've probably mentioned about 3 issues on this ticket, shall I break out the error message into its own as well?

[mario]

@jasonbayton no, everything else should be fixed in a build tomorrow :)

[mario]

This should be fixed. Confirm please? :)

[jasonbayton]

Storage path was reset back to "unknown" for the first time in several upgrades and all previously selected autoupload folders have been deselected (nothing will autoupload by default).

The error is certainly gone on the uploads status screen and it's uploading perfectly moments after I receive a new image via WhatsApp (my test folder).

Looking good!

[mario]

Hey,

sorry about the deselection - it's a bug, and has been fixed already shrug. As for "unknown" that part puzzles me (and shouldn't have anything to do with this).

So you can confirm that:

• there are no duplicate uploads?
• custom folders work?
• all other use cases you've tried work? :)

Tomorrow I'd like you to test (if possible) if uploads happen even if you deselect all folders.

[jasonbayton]

It has stopped trying to upload everything as soon as I open it, which is excellent.
It's not trying to upload the same new files repeatedly, just the once.
Custom folders are respected.

When I intentionally try to manually upload the same file a second time, it doesn't prompt me, just uploads it as file(2).jpg - so there's no duplicate detection prompting happening yet.

The larger folders such as the WhatsApp one after yesterday's upload fest take forever to load and scrolling around them is very slow, though this is likely due to the size of the directory and the device itself.

I've tested screenshots which are now deselected, they don't upload

[mario]

Yes, the prompts will take a bit of time to implement - dunno when I can come to that.

Ok. thanks for testing. I would appreciate if you'd do the same round of testing tomorrow :)

[jasonbayton]

Sure, just ping me when it's ready :)

[mario]

Btw. the slow loading and scrolling is also a problem with the implementation. Will be fixed eventually - fix is known, just takes time.

[mario]

I'm going to close this issue since it's solved, but let me know if it pops up again as I re-arrange the code :)

[jasonbayton]

Oh god it's uploading everything again.

I moved all the unique (non-duplicate) files out of the February folder (2017/02) as only a tiny percentage of them are for Feb, the bulk being 2016.

Now it has seemingly noticed and is uploading every file in my watched WhatsApp folder all over again.

[mario]

What did you move where? Can you clarify please?

[mario]

Aka, please give me a list of steps so I can reproduce.

[jasonbayton]

On the server, I deleted all the duplicates from the all-day upload session 2 days ago, then all of the unique remaining files that were uploaded to the 2017/02 folder (because it organises by month uploaded, not month of file modified) I moved around into organised folders by date.

So:

On the server I moved every file that isn't 2017/02 to other respective folders (2017/01, 2016/04, etc) leaving a tiny fraction of files in the 2017/02 folder.

It appears now the app has done a comparison locally to the remote dir and seen the files aren't there anymore, so has uploaded them all again.

It's basically like 1-way sync at the moment..

[mario]

Cool, I wasn't even aware I implemented that :D Should I just extend it to support 2-way sync? xD

Thanks, will try to debug!

[mario]

Question: did it only upload the missing files, or did you get duplicates again?

[jasonbayton]

You're half way there, why not? ;)

Here's a scenario to consider as well. In a week or so I'll be swapping phones, at which point media from WhatsApp will be automatically restored to the new phone as part of setup. If this happens in a different month I fear it'll consider those all new files in a new month and duplicate them all again.

No duplicates - it literally just finished now: 624 files @ 103MB during which the phone got super hot and unresponsive (via WIFI with internal DNS redirect to local server, so very fast uploads).

[mario]

Yes, it will consider them as new files, there's no way to do MD5/SHA256/whatever and check if files are the same. Ah.

[mario]

Thank you, I will look into this all.

[mario]

I believe it is easier to implement full 2 way sync than it is to fix
existing problems, but time constraints ... eh :)

One day, one day - I now have a fairly good idea how the perfect 2 way sync
should look like.

On Thu, 9 Feb 2017 at 22:27, Jason Bayton [e-mail] wrote:

You're half way there, why not?

Here's a scenario to consider as well. In a week or so I'll be swapping
phones, at which point media from WhatsApp will be automatically restored
to the new phone as part of setup. If this happens in a different month I
fear it'll consider those all new files in a new month and duplicate them
all again.

No duplicates - it literally just finished now: 624 files @ 103MB during
which the phone got super hot and unresponsive.

—
You are receiving this because you modified the open/close state.

Reply to this email directly, view it on GitHub
https://github.com/nextcloud/android/issues/630#issuecomment-278779117,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAAWsun6gUukPk5wEs-Qwvsmc-s7WiWlks5ra4TcgaJpZM4L4BdS
.

[jasonbayton]

Can't you not write a flag to uploaded files in the app database to not look at them if they've been uploaded already?

If the fix for this isn't feasible, how's the fix for organising files based on modified/filename instead? One of these needs to be fixed I guess before it leaves beta.

[mario]

Its a bit more complicated - we shouldnt store every updated file in the DB.

And there will be a fix for directories based on date available in beta
soon.

On Thu, 9 Feb 2017 at 23:08, Jason Bayton [e-mail] wrote:

Can't you not write a flag to uploaded files in the app database to not
look at them if they've been uploaded already?

If the fix for this isn't feasible, how's the fix for organising files
based on modified/filename instead? One of these needs to be fixed I guess
before it leaves beta.

—
You are receiving this because you modified the open/close state.

Reply to this email directly, view it on GitHub
https://github.com/nextcloud/android/issues/630#issuecomment-278790236,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAAWsjJwZk2beqB5xXAA08yKxSb6UtAyks5ra45igaJpZM4L4BdS
.

[jasonbayton]

soon :)

Directory organisation will sort it I guess, whether it's used or not by the end-user the app will then see the files either without organisation or with on the server and will be able to compare accordingly.

[jasonbayton]

This needs to be reopened, unless the new version resolves..

I had an image come through on WhatsApp last night and it went mad overnight while I slept :)

[mario]

How do I reproduce?
On Fri, 10 Feb 2017 at 10:36, Jason Bayton [e-mail] wrote:

This needs to be reopened, unless the new version resolves..

[image: image]
https://cloud.githubusercontent.com/assets/3178412/22821739/58e9355a-ef74-11e6-9bb7-fb47ac83e103.png

—
You are receiving this because you modified the open/close state.

Reply to this email directly, view it on GitHub
https://github.com/nextcloud/android/issues/630#issuecomment-278898693,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAAWsn4gI0Fr6hoee8l58JJdcb41IB1wks5rbC-KgaJpZM4L4BdS
.

[jasonbayton]

My environment hasn't changed throughout this issue:

• Select a folder for backup (add a custom directory if you want to be complete)
• Wait for an image to hit that directory
• Watch it duplicated 23 times on the server

[mario]

Thanks
On Fri, 10 Feb 2017 at 10:38, Jason Bayton [e-mail] wrote:

My environment hasn't changed throughout this issue:

• Select a folder for backup (add a custom directory if you want to be
  complete)
• Wait for an image to hit that directory
• Watch it duplicated 23 times on the server

—
You are receiving this because you modified the open/close state.

Reply to this email directly, view it on GitHub
https://github.com/nextcloud/android/issues/630#issuecomment-278899268,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAAWshMAOamR9YhfKuqJCTJBhqX-SlN6ks5rbDAzgaJpZM4L4BdS
.

[mario]

Btw. if you're doing backup from old phone & restore to another phone operations starting next beta (whenever that is), you won't (shouldn't) get duplicated uploads :)

[jasonbayton]

It's currently doing an upload again, I'm up to 17 duplicates per file in the watched folder tday (of up to 600 files)

[mario]

@jasonbayton just built a new beta, please try.

[jasonbayton]

Thank you - installed during another upload session and the update appears to have stopped it for now.
Still no logs past the 6th I'm afraid!

[mario]

@jasonbayton another build coming in 10 minutes - tiny changes to auto upload + it should upload to proper directories based on date modified.

[jasonbayton]

Bugs or not, this is super fast development of an incredibly useful feature. Thank you :)

[mario]

@jasonbayton that's what you get when you have a super-receptive audience that tests your every failure and celebrates every success :)

[mario]

Alert alert, new build ready :D

[jasonbayton]

I'm not sure if I pulled the apk down too soon before your announcement or not, but I just did once more to be sure - I'll let you know!

• Keeping storage path
• Keeping selected backup folders selected

Looking good so far!

[mario]

@jasonbayton no duplicate uploads yet?

[jasonbayton]

Nothing as yet. Two things though.

1. It's occasionally confusing months 09 file in 10 folder?)
2. The app continues to terminate and not restart (I'll open another issue)

[mario]

@jasonbayton do you have a file explorer of sorts on Android? Check what it says for last modified?

[jasonbayton]

Yes, here's an example file uploaded to 2016/10 with a modified date of 2016/09.

How come you've opted for last modified instead of filename?

[mario]

Because filenames are tricky - you have infinite variations of how a file can be called :)

[mario]

@tobiasKaminsky since you've changed this code -> any idea why this would happen? :)

[jasonbayton]

Understood. I can envision mis-organised files when 2 way sync comes in on full folder uploads (if a picture in 2015 was modified or - more likely - has its metadata stripped through being moved between systems or programs that don't respect it) but obviously no problem for new files immediately uploaded.

Too bad android can't work with date created.

[mario]

@tobiasKaminsky I have attempted a fix, so @jasonbayton can test it tomorrow ;)

[mario]

@jasonbayton let me know when you open the "app was terminated" and how to reproduce.

[mario]

@jasonbayton starting tomorrow, we use exif tag date created where available, or as a fallback last modified from android.

[jasonbayton]

Duplicates are going again.

Edit: no picture as the phone crashed.
Edit edit: got one.

[jasonbayton]

It's weird in how it's doing this - almost like it's not checking the server for existing files first.
I've just pulled down the latest apk and will see what it does today.

[mario]

@jasonbayton it does, but it decided to create a new file anyway (that's by design currently, since we can't know if it's the same file). The bigger issue is why it decides to create two jobs for the same file...

[mario]

Today's beta should fix the date created (from EXIF), and not this - working on this today if I can find the real cause.

[jasonbayton]

we can't know if it's the same file

why it decides to create two jobs for the same file

Did you answer your question? It should know what it's uploaded shouldn't it? Isn't there a local db in the app to keep track of these things, or MD5, or .. something else that lets the app know what it's dealing with?

[mario]

We could use hashing algorithm for local files, but can't compare them to remote files because we don't have hashes of remote files (and we can't have them). While in theory one could get uploaded files from jobs issued, you can also clear jobs and what then?

[mario]

New beta in 10 minutes, hopefully fixing this :P

[jasonbayton]

I guess it'd again be good to see how Google Photos handles it, but then their implementation is different as it's a manual backup on request (backup all now) and wouldn't compare to a real two-way sync like clients such as bittorrent (resilio) sync do.

[jasonbayton]

@mario Btw. if you're doing backup from old phone & restore to another phone operations starting next beta (whenever that is), you won't (shouldn't) get duplicated uploads :)

Just tried this, WhatsApp restored from Google Drive as per usual when logging in on a new device for the first time and NC beta pushed every restored file from 2015 -> Present to /2017/02

[mario]

Ah, so this is different to the backup -> restore strategy I had in mind :-/

Ok, so I guess currently the only solution is to temporarily disable sync of that folder.

[mario]

@jasonbayton closing this as the original issue is solved.

[mario]

@jasonbayton oh wait, I just re-read your issue! Can you check in file explorer:

• filetype of those images, and do they have relevant EXIF tags
• last modified attribute of those images

[jasonbayton]

Yes sure, they're mostly jpg and it looks like last modified is yesterday's date.. I don't see any other EXIF (created, etc) and by being backed up and restored it would appear it's wiped out the original modification dates.

[mario]

Cool. I already wondered if exif tags reading or last modified stopped working.

[jasonbayton]

It's a restore of _another backup_ so as long as I install NC and configure it after the WhatsApp (or any other) restore it'll be fine.

Until 2way sync.

Then it's going to see there are a heap of duplicate filenames with different modified dates and litter me with duplicates.

Unless by then I install NC first and set up sync before restoring.

Hmm.