Android: Instant upload never started - ANDROID 6

[M0ses]

Actual behaviour

When taking a picture its not getting uploaded

Expected behaviour

pictures should be uploaded, at least when connecting WIFI next time

Steps to reproduce

1. Configure InstantUpload
2. Take picture
3. Check files on server

Environment data

Android version:
multiple Versions

Device model:
BQ Aquaris e4.5
BQ Aquaris X5 Plus
Huawei P8 Lite

Stock or customized system:
Stock rom's / mostly rooted

ownCloud app version:
2.1.1

ownCloud server version:
9.1.0
(problem exists since a year with multiple versions

Logs

No relevant logs on server side ATM. Please let me know what to provide exactly.

Noticed this messages in 'adb logcat'

12-30 21:48:31.886  1871  3900 I ActivityManager: Killing 5166:com.owncloud.android/u0a95 (adj 15): empty #17
12-30 21:48:43.908  1871  1884 I ActivityManager: Start proc 6979:com.owncloud.android/u0a95 for broadcast com.owncloud.android/.files.BootupBroadcastReceiver
12-30 21:48:59.192  1871  1898 I ActivityManager: Waited long enough for: ServiceRecord{7a8fd19 u0 com.owncloud.android/.services.observer.FileObserverService}

[M0ses]

I could fix it on my BQ Aquaris X5 Plus

It seems to be a permissions problem, but I could not fix it by revoking and granting permissions again.
I had to deinstall and install the owncloud app again.

I will do some further investigation on the other devices

[jesmrec]

@M0ses is Android 7 your OS?

If you can provide us the following info about your devices:

• Device model
• Android version
• App version: 2.1.1 (2.2.0 is already available)
• Instant upload config (upload path, only wifi, camera folder, kept/moved...)
• Have you ever upgraded the app from another version, or is installed from scratch
• Camera app

will be helpful.

[houseofbugs]

I am having issues where nothing is triggering on my Android app running 2.2.0. I have the latest OwnCloud server installed (just installed today too).

[M0ses]

@jesmrec

Sorry for delay. To give one example:

• BQ Aquaris X5 Plus
• Android Version:: 6.0.1
• App Version: 2.1.1
  
  
  
  • Upgrade to 2.2.0 did not help
  
  
  • Deinstall/Install app again did the trick
  
  
• Config Pictures/Videos:
  
  
  
  • Path: /SofortUpload
  
  
  • Wifi only
  
  
  • Camera directory: /storage/emulated/0/DCIM/Camera
  
  
  • keep original
  
  
• AFAICR : App was upgraded (from Version unknown)
  
  
  
  • App taken from F-Droid
  
  
• Its the default camera app by BQ (Kamera 2.3.1) - Unfortunatly I could not find out more about the camera app - but if you guide me, i will try to find out.

I think it was a permissions problem. I found some log entries with logcat. Unfortunatly I did not store them, sorry for that. These problems where not raised to the user, only visible in logcat. After reinstalling the App from scratch, the were gone.

[bkraul]

Installing/uninstalling did not resolve the issue for me.

[smmilani]

I upgraded my Samsung Galaxy s7 edge from 6.0.1 to 7.0

After the upgrade instant upload stopped.

My owncloud app version is 2.2.0

I changed the camera directory to /storage/9c33-6bbd (external sd)/DCIM/Camera
and that did not help either.

I also have a Oneplus 3 running android 7.0 and instant upload works fine with both owncloud 2.1.2 and 2.2.0

[jensd0e]

Instant Upload not working. Reinstalled the app but that did not fix the issue.
Device: Samsung Galaxy S7
OS Version: Android 7.0
App Version: 2.2.0
Configuration:
Upload Path: /Kamera
WiFi only: checked
[Video has same config]
Camera Folder: /storage/0000-0000/DCIM/Camera

I can manually upload pictures from this folder.

[davivel]

@Moses, if I read correctly this is fixed for you, right?

[davivel]

@houseofbugs , what do you mean with:

I am having issues where nothing is triggering

?

Are you talking about instant uploads? Video and / or picture uploads?

[davivel]

@smmilani , @jensd0e , and everybody with problems in Android 7 devices: Please, copy your comment in #1829. Let's keep that as the only issue for devices with Android 7 where instant uploads don't work.

[M0ses]

@davivel Yes, correct - removing the app and reinstalling fixed this issue for me - but this might be not a good solution for everyone - It is not obvious that a permissions problem might be the solution. I think the owncloud app should complain if it does not get read access to the configured folder

[M0ses]

And the error message should be visible to the user!

[davivel]

@michaelstingl , the problem described here is different than what we discussed yesterday. Not about interrupted uploads.

@M0ses, I guess that before reinstalling the app the instant uploads never started, right? It's not about uploads that were interrupted and listed as failed in the 'Uploads' view, it's about pictures never detected.

[davivel]

This log

Waited long enough for: ServiceRecord{7a8fd19 u0 com.owncloud.android/.services.observer.FileObserverService}

leads me to https://github.com/android/platform_frameworks_base/commit/9210bc85545f31973c957b5179e6a82d05f473c6 , added in Android 4.4.

Seems more difficult to fix than a problem with permissions. The system is preventing that the observer service (the one watching for there appearance of new pictures in the camera folder) starts when the device is rebooted. That means that no picture will be detected at least until the app is opened by the user.

We need to move to a solution based in periodical scanning of camera folder(s). There is no other safe way to fix this. In Android 7 we can try also to use JobScheduler to be notified about new pictures in the MediaStore, but that's not possible in previous versions of the system.

[davivel]

@M0ses , your devices are both Android 6, right?

[M0ses]

@davivel

• Yes, both were Android 6 - Now my "BQ Aquaris X5 Plus" is on Android 7.
• Yes, pictures where never detected
• Same issue on a "Huawei P8 Lite" with Android 6.0 with updated owncloud app (2.3.0)

If I got you right, the workaround for my Problem is

• start the owncloud app manually after each reboot
• otherwise no images get uploaded

correct?

I will test this, to ensure, that your assumptions are right.
Is there any possibility trigger a sync from the "Gallery" to "Instant Upload again?

[M0ses]

After some further testing, I could observe the following behavior:

As long as the OC app is started when a picture is taken _and_ it doesn`t get closed till you connect to WIFI again (which is able to reach your owncloud server of course) - InstantUpload works fine, but if you do the following steps:

• start owncloud
• disconnect WIFI
• take a picture
• stop owncloud
• start wireless
• start owncloud

nothing gets transferred

This is really an annoying behavior and a warning in the dialog for Android < 7 if it takes longer to fix the root cause.

[davivel]

From Android 4.4 Google is more and more aggressively killing services that try to live forever. Our current solution to detect instant uploads is partially based in this, what means that stability of operation is worse the bigger the Android version is.

Even further: Android O will officially introduce time limits to operation of background services, so our solution is just doomed.

We need to move to a solution based in periodic scanning of camera folder once for all.

[davivel]

Hi all @M0ses, @houseofbugs, @bkraul, @smmilani and @jensd0e .

Let me invite you to try the last release of the ownCloud Beta app that includes a candidate fix for your issue with instant uploads. This beta app can be installed in your devices besides of the regular ownCloud app, and both will work independently.

You will notice that uploads do not start immediately, we set an interval of 5 minutes to poll for new pictures in the camera folder. Please, try this fix and tell us your impressions about it. There is room for many improvements yet, and any feedback from you will be very appreciated.

For the moment you can find ownCloud Beta in F-Droid. You will need to install the F-Droid client in your device from https://f-droid.org/FDroid.apk, and then search "ownCloud Beta" in it.

Thank you so much.

[ghost]

You will need to install the F-Droid client in your device from https://f-droid.org/FDroid.apk, and then search "ownCloud Beta" in it.

For testing purposes the ownCloud Beta client APK is also available directly at:

https://f-droid.org/packages/com.owncloud.android.beta/
-> https://f-droid.org/repo/com.owncloud.android.beta_20400000.apk

[bkraul]

Could the interval become a configurable feature?

[jensd0e]

Testing Beta 2.5.0-beta.1.

• Had some photos on my phone that did not get uploaded with the non-beta-version (three days old).
• Set-up "instant" upload as on non-beta-version. Waiting more then ten minutes, but nothing gets uploaded. Since this is not instant sync, that might be as desired?
• Took a photo while on WiFi, waited another 10 minutes, but photo didn't get uploaded.
• Switched WiFi off/on, still no upload after another ten minutes.

Phone:
Model : SM-G930F
Brand : samsung
Product : heroltexx

ownCloud:
Server: ownCloud 9.1.6-1.1 on Debian 8.8
Client: ownCloud beta 2.5.0-beta.1

Settings:
Instant Photo Upload: checked
Upload-Path: /Kamera
Instant Upload for Photos on WiFi only: unchecked
Instant Video Upload: checked
Upload-Path: /Kamera
Instant Upload for Videos on WiFi only: checked
Camera-Folder: /storage/0000-0000/DCIM/Camera/
Keep original file: checked

Logfile available. What strings to search for?

[davivel]

Hi, @jensd0e , thanks for your feedback.

Set-up "instant" upload as on non-beta-version. Waiting more then ten minutes, but nothing gets uploaded. Since this is not instant sync, that might be as desired?

We expected that, with the current implementation only pictures that are taken from the moment that the feature is enabled are being detected and uploaded. Pictures taken before are not considered. We know many users want we change that behaviour, and seems that finally we will be able to do it.

Took a photo while on WiFi, waited another 10 minutes, but photo didn't get uploaded.

We did not expect that at all. Bad news.

Switched WiFi off/on, still no upload after another ten minutes.

If the picture wasn't detected in the previous step, turn Wi-Fi off & on will not fix it.

Logfile available. Where to look for?

In the app, in 'Settings', scroll down and find and extry named "Logs". Tap on it to see the app logs. There button 'Send log' will allow you to share the log file to other apps. Send it to [email protected] via e-mail if you wish, including the text 'Android issue #1870' in the subject.

See my answer here, too: https://github.com/owncloud/android/issues/1829#issuecomment-314802104

Seems very strange that you don't get any single picture is uploaded. But since we will have to fully rewrite the feature (explained in the link above), probably it is not worth that we analyse too hard the details of your concrete problem at this moment. In any case, is your camera storing pictures in an external SD Card, or in internal storage in the device?

[bkraul]

My pictures are uploading after 5 minutes as expected. However, on some instances there seem to be issues uploading when connected to WiFi, but uploaded fine while on mobile data (WiFi-only unchecked).

Again, it would be cool to have the interval as a configurable option.

Specs of my setup:

Phone:
Model : SM-G930A
Brand : samsung

ownCloud:
Server: ownCloud 9.1.6 (stable) on CentOS 7 (1611)
Client: ownCloud beta 2.5.0-beta.1

Settings:
Instant Photo Upload: checked
Upload-Path: /InstantUpload
Instant Upload for Photos on WiFi only: unchecked
Instant Video Upload: checked
Upload-Path: /InstantUpload
Instant Upload for Videos on WiFi only: checked
Camera-Folder: /storage/emulated/0/DCIM/Camera/
Keep original file: checked

[davivel]

Again, it would be cool to have the interval as a configurable option.

That is feasible, but if you want to have an smaller interval, it will just not work.

The current solution is based in a background service that tries to live forever, and in it we could set whatever poll interval we want. But the service is being killed by the system too often (see details in https://github.com/owncloud/android/issues/1829#issuecomment-314802104), so we have to move to a solution based in scheduling the poll via the Android framework, using JobScheduler API or something based on it.

The problem is that JobScheduler doesn't grant accuracy for the requested timings, and no matter how small is the period you choose, it will never allow periodic jobs with times lesser than 9 minutes (if I recall correctly).

[bkraul]

Just an idea from the limited understanding I get from what you wrote down. You say that the background jobs are killed often. Why not run 2 jobs, where the first one starts the second one close to the perceived kill time out, and so on and so forth (daisy-chaining).

Then handle all the scheduling logic internally by maintaining a running list (flat or db) of queued files to poll from, that any of the jobs can access? This would give you virtual permanence, even if the background jobs are eventually killed.

Or perhaps, if you can detect when the background job is killed, you would have it start the next one. Again, I apologize for any naivety in my comments but, being a programmer myself its just a flow that makes sense to me.

[davivel]

Just an idea from the limited understanding I get from what you wrote down. You say that the background jobs are killed often. Why not run 2 jobs, where the first one starts the second one close to the perceived kill time out, and so on and so forth (daisy-chaining).

It is a great idea, but we have no clue about what's the "perceived kill time out" in Android 4, 5, 6 and 7. Maybe if we dig into Android source code? Sounds like too much effort, and it is possible that it is not a timeout, but a set of conditions based in other things.

There is no warning also via framework, no notification about the service being killed, since we are talking about a process kill.

Precisely through the API of JobScheduler, jobs (that are just a special kind of background services) have a new onStop method that is called to warn that we need to finish.

Don't apologise, please, any help is welcome :)

[davivel]

Also, in Android 8... not a single chance, I need to double check, but I read recently that background jobs will not be allowed to start other background jobs -> will result in an exception.

[bkraul]

I got you. I have read really old threads about suggesting owncloud's use of REST over WebDav. I know it is a humongous task, but with REST, this would likely be a non-issue, or at least not as complicated as the present situation, because of the power of server push events. Thanks for trying to find a solution to this issue.

I've just noticed something odd with the uploads. Between yesterday (which is when I first installed the Beta), and today, I have taken a few pictures. They all were eventually automatically uploaded. However, when I looked at my Windows owncloud folder, and looked at the uploads folder, those files seem to have been duplicated, like File.jpg, File (1).jpg. That tells me that it was not a mobile action, as it is Windows that uses the (1,2) system. Anyway, just thought I'd mention it.

[davivel]

Mmm, the OC Android app also uses that scheme when finds names already .

@bkraul , is it possible that you have instant uploads enabled in both beta and production app, both using the same target folder? Could you check the 'Uploads' view in the app and see if the duplicated files appear there?

Thanks.

[jensd0e]

Thanks for all the hard work on the OC Client. I really appreciate this! :-)
I don't want to sound too heretical, but in nextCloud, the instant upload does work as expected (but the settings do not fit my needs. Photos and Videos are handled alike, where I would prefer only photos to be uploaded on mobile connection). IIRC it took them a while to get it sorted out, also. I'm not sure, if you could clone their code, giving them credits.

And as a final note: Google Photos does always manage to get my Photos uploaded. Is there something to learn/borrow from?

Again, I can't emphasis how much I rely on OC.

[davivel]

I understand you @jensd0e , I know you want to help, and really appreciate :)

This has been the fist time we had a look to NextCloud code for some inspiration. Indeed, the fix in OC beta is using the same external library they use to poll for changes in the camera folder.

Nevertheless, we need to take their code with a grant of salt. If you check their Android repository, you will find quite a bunch of issues complaining about auto uploads draining their battery, lost uploads, uploads going on when Wifi connection is lost although wifi-only option is set, etc... All that after they took their while to implement it, as you said.

I'm not saying they did nothing useful, on the contrary. I know some of the contributors there and I have a deep respect for them. I just want you to understand that there will be no easy-and-fast solution for this.

Besides, it is not that we don't know how to fix this. The point is that we have been trying to avoid going through "the hard way", but now that is the only way left.

[davivel]

And in the end will be a good thing, because the result will be much more stable.

[bkraul]

@davivel, no man, I uninstalled the prod app, and only have the beta app running. But, that said, the uploads view does show the duplicate files.

[jesmrec]

Feature already released in 2.5.0 with other improvements. Please follow up in #2028 and give us there your feedback.

Thanks to all contributors here.

[davigonz]

Hi @M0ses, @houseofbugs, @bkraul, @smmilani and @jensd0e ,

we released yesterday the 2.6.0 Android app with the new Camera uploads feature, which will replace Instant Uploads and will upload automatically your pictures and videos in 15 minutes at most, focusing on reliability instead of immediacy and avoiding battery draining caused by excessive checking of the folder.

I invite you to try it out and share with us your impressions, thank you.

[arnwas]

Even with 2.7.0 on a Samsung S7 Edge with Oreo it does still not work. Both SD and internal storage