Android: Sometimes Pictures are not uploaded

[CordlessWool]

Actual behaviour

• Some itmes pictures are not uploaded, they are shown in "Auto Upload"-Settings but not in Uploads. My flatmate has the same problem on her older phone

Environment data

Android version:
7.1.2 (newest Lineage) (Nexus 5)
6.x.x (some Samsung)

Device model:
Nexus 5
Samsung XXX (Model for Indian Market)

Stock or customized system:
Lineage on Nexus 5

Nextcloud app version:
1.4.3

Nextcloud server version:
11.0.3

[jasonbayton]

To verify, are you saying they hang in the uploads area, or are never detected?

[CordlessWool]

It is not shown in the uploads area (there are not shown as uploaded, failed or something else), but in the area I can configurate uploads I see the pictures. So the app can show the pictures but do not upload them or show them in the uploads/upload stream and also not upload them

[jasonbayton]

@mario this is what I mentioned a while back. Sometimes it just doesn't detect a picture to add to uploads

[mhzawadi]

Evening, I get the same issue. Picture taken and visible in auto upload config screen, but not picked up to upload.

One plus 3T
android 7.1.1
Android app 1.4.3
Nextcloud server 12

[mario]

I know. It will all be fixed in 1.5.

[OlivierS1]

@mario , if it is going to be fixed in 1.5, shouldn't this be part of the Dev version as well?

[mario]

It will be as soon as I write relevant code

On Fri, 23 Jun 2017 at 04:01, OlivierS1 [e-mail] wrote:

@mario https://github.com/mario , if it is going to be fixed in 1.5,
shouldn't this be part of the Dev version as well?

—
You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub
https://github.com/nextcloud/android/issues/1098#issuecomment-310550168,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAAWsqIQccJfswBy_TKCLrugUfteRnLCks5sGxyFgaJpZM4N_vot
.

[OlivierS1]

@mario No need to have it soon on Dev version. I thought Dev was supposed to get it before Release!
Doesn't make much sense at the moment I guess.

[jasonbayton]

Dev will get it before release, but that requires the code is written, added to Dev, validated and then pushed the release.. so it's not just going to magically be there :p

[mario]

Dear users,

thank you for your patience. After lots of work, we’re super excited to be able to give you a test version with new auto upload implementation. Keep in mind we will only accept bugs related to Auto Upload in this APK, as other things might be broken.

https://cloud.nextcloud.com/s/ftZraTruRQjTe0R

This is a dev APK so it will not overwrite your existing play store/f-droid APK unless you’re already running dev version. Also, you might need to remove your existing dev version before installing this one.

Explanation of the current functionality:

• added support for custom folders
• separated image and video folders
• checks for new files happens every 15-20 minutes
• automatic retry every 15-20 minutes or when you force close/open the app
• automatic pause once network is changed (untested!)
• significantly less battery usage

All and any testing is more than appreciated, especially with large and ever-going files like Videos. (like, try to record a 30 min video).

Let me know if you’ve got any questions.

Cheers,
Mario

[mhzawadi]

Thanks @mario
I have downloaded and setup, will feedback on how it works
:)

[OlivierS1]

@mario @tobiasKaminsky What are the fundamental differences between the two APK files?
nextcloud-dev-20170726.apk (downloaded last night)
and
nextcloud-android-generic-debug.apk (Your version)

[mario]

Rewritten auto upload.

And I had to remove some thumbnail generation code.

On Wed, 26 Jul 2017 at 17:24, OlivierS1 [e-mail] wrote:

@mario https://github.com/mario @tobiasKaminsky
https://github.com/tobiaskaminsky What are the fundamental differences
between the two APK files?
nextcloud-dev-20170726.apk (downloaded last night)
and
nextcloud-android-generic-debug.apk (Your version)

—
You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub
https://github.com/nextcloud/android/issues/1098#issuecomment-318088045,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAAWsoUQLCzJKtEIjOWGSOPC_gdj3Amqks5sR1otgaJpZM4N_vot
.

[OlivierS1]

@mario Yep, no thumbnail previews.
I have asked the question way back, but I am asking again, would have it been better to create a new branch so that both regular Dev and your version be installed separetly in different location so both can still be used concurrently?
How long do you plan on having your version up and running before I can switch back to the normal Dev?

[mario]

As soon as Google fixes their toolchain this will go into dev.

On Wed, 26 Jul 2017 at 17:36, OlivierS1 [e-mail] wrote:

@mario https://github.com/mario Yep, no thumbnail previews.
I have asked the question way back, but I am asking again, would have it
been better to create a new branch so that both regular Dev and your
version be installed separetly in different location so both can still be
used concurrently?
How long do you plan on having your version up and running before I can
switch back to the normal Dev?

—
You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub
https://github.com/nextcloud/android/issues/1098#issuecomment-318091692,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAAWssWFTBi8UjUYHUnGq5e5uGqlznlBks5sR1zsgaJpZM4N_vot
.

[OlivierS1]

@mario Under settings, I can select to check for updates, and one seems available. Would it be yours or the normal Dev?
Also, more to the point of your testing, what are the criteria for the app to differentiate the types of files, ie video vs picture vs any other file type?
Are pictures only targeting JPG, TIFF, NEF...

[mario]

Its the normal dev probably.

image: mimetype is image/*
Video: mimetype is video/*

On Wed, 26 Jul 2017 at 17:47, OlivierS1 [e-mail] wrote:

@mario https://github.com/mario Under settings, I can select to check
for updates, and one seems available. Would it be yours or the normal Dev?
Also, more to the point of your testing, what are the criteria for the app
to differentiate the types of files, ie video vs picture vs any other file
type?
Are pictures only targeting JPG, TIFF, NEF...

—
You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub
https://github.com/nextcloud/android/issues/1098#issuecomment-318095366,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAAWsv77BwNG8ubdgyhncGnlD0eYqc6Uks5sR1-hgaJpZM4N_vot
.

[OlivierS1]

I unsintalled Dev to install your version, now I can't find my regular Nextcloud app (Non Dev). Did I do something wrong or did your app create a conflict and removed the normal app?

[mario]

Afaik it should overwrite the dev version unless I have merged something
wrong. I can check it out a bit later.

On Wed, 26 Jul 2017 at 17:52, OlivierS1 [e-mail] wrote:

I unsintalled Dev to install your version, now I can't find my regular
Nextcloud app (Non Dev). Did I do something wrong or did your app create a
conflict and removed the normal app?

—
You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub
https://github.com/nextcloud/android/issues/1098#issuecomment-318096866,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAAWslutwfvddLCpjSk84oOxfKtT-mjrks5sR2CkgaJpZM4N_vot
.

[OlivierS1]

It couldn't overwrite the dev version as it was throwing an failed message. So I uninstalled manually. It happens when APK doesn't have same signature or other element to validate the install. Not an issue for me though.

[mario]

That is ok - that is why my comment about removing existing dev version.

Will check if it could overwrite official app.

On Wed, 26 Jul 2017 at 17:57, OlivierS1 [e-mail] wrote:

It couldn't overwrite the dev version as it was throwing an failed
message. So I uninstalled manually. It happens when APK doesn't have same
signature or other element to validate the install. Not an issue for me
though.

—
You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub
https://github.com/nextcloud/android/issues/1098#issuecomment-318098695,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAAWsjPR8NBHliUoy7MK-zu_vJ1of4Xqks5sR2H6gaJpZM4N_vot
.

[OlivierS1]

For Autoupload, as soon as a picture or video is taken, how soon will NC starts sending the file?

[mario]

15-20 mins.

On Wed, 26 Jul 2017 at 18:01, OlivierS1 [e-mail] wrote:

For Autoupload, as soon as a picture or video is taken, how soon will NC
starts sending the file?

—
You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub
https://github.com/nextcloud/android/issues/1098#issuecomment-318099993,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAAWsq5be8ijW3vG-74B28M92N9GMlnEks5sR2LfgaJpZM4N_vot
.

[OlivierS1]

Not good. Why can't we have the picture upload being processed as before, meaning having pictures uploaded instantly on 1.4.3 or with a slight delay on NC Dev 20170722.
It used to work fine at some point for Videos, and I think this is the reason why you added a longer delay as the Dev had issues sending partial files.

[mario]

reliable uploads
Quick uploads
Least battery usage

Pick 1st and 3rd or just 2nd :) simple as that.

On Wed, 26 Jul 2017 at 18:06, OlivierS1 [e-mail] wrote:

Not good. Why can't we have the picture upload being processed as before,
meaning having pictures uploaded instantly on 1.4.3 or with a slight delay
on NC Dev 20170722.
It used to work fine at some point for Videos, and I think this is the
reason why you added a longer delay as the Dev had issues sending partial
files.

—
You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub
https://github.com/nextcloud/android/issues/1098#issuecomment-318101572,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAAWsoC0smmzBxlxmOqzTn-VwrKvZcPqks5sR2QBgaJpZM4N_vot
.

[jasonbayton]

The battery consumption was easily 10+% on a charge with 1.4.3 due to it polling for pictures so often. With custom folders now available that would only get worse.

Unless you're worried about your phone exploding 7 mins after taking a photo this shouldn't in reality pose a concern once you're using the app day to day. As long as it uploads them - and the uploads pane continues to show everything we need - it'll be OK.

Thumbnails would be nice too, eventually, but I'll take streaming media as a priority over downloading the content!

[OlivierS1]

You can't say you now have quick uploads.
My concern now is all about sharing files.
If I take a picture to share it, I will have to wait for the app t upload it to the server. If I am on Whatsapp or if I need to share a photo right away, this is no longer possible. Which means I would have to take a picture and share it within whatspapp which I don't like.
When I do work for myself, I take pictures and they get uploaded to my sever which then are being downloaded on my PC. I have no need to connect my phone to the PC via USB.
Would it be possible, to have a check box or setting that would allo transmit as soon as the pictures are taken?
Later, there will be introduction to 2 way sync as far as I understand and this feature could correct possible corrupted uploads.

[jasonbayton]

I had your situation earlier and just uploaded them from within the app for fast sharing. Auto upload them didn't try to upload them later which is cool (but I'd that intentional?)

[mario]

Of course not. I am saying I picked reliable uploads and less battery usage
over speedy uploads.

There is no "instant" unless you poll every second which kills battery and
potentially the SD card - so I will not do this, sorry.

On Wed, 26 Jul 2017 at 18:16, OlivierS1 [e-mail] wrote:

You can't say you now have quick uploads.
My concern now is all about sharing files.
If I take a picture to share it, I will have to wait for the app t upload
it to the server. If I am on Whatsapp or if I need to share a photo right
away, this is no longer possible. Which means I would have to take a
picture and share it within whatspapp which I don't like.
When I do work for myself, I take pictures and they get uploaded to my
sever which then are being downloaded on my PC. I have no need to connect
my phone to the PC via USB.
Would it be possible, to have a check box or setting that would allo
transmit as soon as the pictures are taken?
Later, there will be introduction to 2 way sync as far as I understand and
this feature could correct possible corrupted uploads.

—
You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub
https://github.com/nextcloud/android/issues/1098#issuecomment-318104482,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAAWshfFqavoZBhfrclOdZvAQ024K2xgks5sR2ZJgaJpZM4N_vot
.

[OlivierS1]

If battery usage is a concern then I would at least have the ability to do a force sync to upload the files when I request it.
I would hate the app stall transfer because I would move from wifi to wifi during the course of my day or due to unreliable connection reset each trnasfer attempt 15 minutes later. I will never have uploads in that case.

[OlivierS1]

If instant upload is a concern because of battery usage, then why even bother having "Only upload when charging" option?
If SD card wear is a concern, I doubt it is really an issue as wear leveling would take care of that, beside the memory endurance should cover the device for many years to come.

[jasonbayton]

Because some people want a balance between uploading when the photo has been taken and 10 hours later when it's put on charge? 15 mins is a reasonable delay, though a force scan could be good.

[mario]

@OlivierS1 I just updated the APK to be proper "beta"/"dev" app rather than replace the original app. As for other things:

• manual re-scan could be an option in 1.5.1

The way current architecture works:

• check for new files every 15-20 mins (either via filesystem or media providers)
• initiate/retry uploads every 15-20 mins (or on app startup) if conditions are favorable (aka, phone is charging or is on wifi, depending on your setup)

"Only upload when charging" and "Only upload on wifi" were really a lie all up till now, mostly because Google promises stuff they can't (don't want to?) deliver. Via the mechanism explained above we get this functionality, and most importantly reliable uploads.

Polling FS every second is super inefficient, prone to errors, shutdowns and various other things that have caused issues in 1.4.2/3 and that the current implementation avoids.

[OlivierS1]

@mario Based on my usage of the app over the past month, year, I really find the need to retain at least for the Camera the ability to upload the photo shortly after it has been taken (Videos on the other end can be postponed to 15-20 minutes I think). For NC Dev, there was already a 30s to 1minute delay. I would be fine with this value and going into a force update just for one photo is not really efficient.
15-20 minute scan period also makes debugging more challenging.
What about documents, I would hate to work on a files, txt or other and only see the updated version show up after 15-20 minutes on my PC to find the document has not be synced yet.

[mario]

@OlivierS1 I understand the issue. But also (currently) don't see a way to change this.

[OlivierS1]

@mario Very difficult to test.
I am able to take video at 720P and size seems limited to 4GB. The video seems to have recorded 51.29 minutes.
Start of recording 4:01PM, end of recording 4:52PM.
I got a failed upload message, but now the files is 20% uploaded. It seems it started from scratch.
Time is currently 6:19PM

[mario]

The phone limits your video recording abilities, not me (if it's the upload that's limited to 4GB, then maybe you web server setup?) :) I guess you're uploading on Wifi? If so, the chunking is disabled there to speed up uploads and therefore resumes would start from scratch.

What's the error? Why did it fail?

[OlivierS1]

@mario I am definitely not a happy camper since release of NC Dev 1.4.3.
I was able to take a 52 minutes of recording but it would fails over and over.
As I was using the phone, I accidentaly switch "Data" on, and I thought I disabled it.
Since I did a reset of the phone a few weeks ago, not all my settings were restored. One of them is the "Data" limit, which was not enabled,
So to put it bluntly, my "Data" usage went over the roof and exceeding it's quota six times over. Here in Canada cellphone services are among if not the worst in the world and it takes arms and leg to get decent services.
Anyway, my Data plan as been automatically blocked by my provider due to overage.
About the video, I am not concerned about the limit size of 4GB, however, sending 4GB doesn't take 5 minutes and I suspect it got into conflict with the auto retry feature.
My server has a higher upload limit so it shouldn't be the issue. If it was, there should be a warning before the transfer takes place and prevent the transfer to proceed further.

Are you saying that doing upload over Wifi doesn't support chunking of data?
If this is a workaround to make transfer faster, I can tell you this is the worst idea ever.

I can tell you I am very disapointed.
Simply put, I am really not happy about it.
I just need to blow up steam. Sorry.

[mario]

Auto-retry would not attempt a retry of in-progress upload, so that would seem strange. As for the server supporting or not supporting 4GB (and up) uploads - we can't detect that. It's a combination of web server + PHP setup that we have no way of knowing about.

Yes, I did indeed disable chunking over wifi to speed uploads significantly as a temporary hack which will be remedied in 1.5.1 hopefully just by using larger chunks for wifi, and smaller chunks for 4g.

I am sorry you're disappointed. We're working in a VERY fragmented space where OEMs and Google decide what we can or can't do.

That being said, you never told me what the error was when the upload failed?
(Others have reported files over 4GB uploaded without a problem)

[mario]

(Btw. new apk at the same URL has expert mode that you can turn on in Settings - if you do that, Uploads view gets a "Force rescan" option)

[OlivierS1]

I tried to get a log but as long as transfer is in progress the log will not be updated.
I did upload the log to my server, but waiting to see if the file can complete as I am now home and transfer is much faster.

[mario]

Olivier: I just wanted the error the uploads screen shows when it fails for the file. But yup, cool.

[OlivierS1]

There wasn't much details on the message, just saying that upload failed.

[OlivierS1]

Unfortunately, it seems the log was not updated since this morning, somewhere around the time I updated the app I would imagine.
I tried to make a few bugreports, I don't know if they will be useful.

[mario]

Thanks. In the mean time, this issue contains a bit more info why "instant" uploads are unrealistic:
https://github.com/nextcloud/android/issues/1234

[OlivierS1]

@mario I have sent you my bugreports to your email address.

[OlivierS1]

@mario I couldn't see the link to Instant upload you are referring to.

[mario]

Thanks. Probably because you read this via email, and I forgot to add it initially so I edited the post. Here it is: https://github.com/nextcloud/android/issues/1234

[OlivierS1]

Well, I was on the forum, but it seems page update doesn't update the link yet until the page is being refreshed.
email sent with bugreport files.

[jasonbayton]

@CordlessWool can you let us know if this Dev version resolves your issue? If so we can close it off.

[CordlessWool]

@jasonbayton I tested it on the nexus 5x for some days now. It seems to be working correct. Thank you very much. I will let the dev-version installed on my phone for some more they for testing. Cloud it be a problem if dev and normal version of nextcloud app is installed?

[AndyScherzinger]

I'll close it for now then, please re-open in case it is malfunctioning again.

[CordlessWool]

Yesterday afternoon I took some picture and getting the same issue. In dev and normal version the same pictures are not uploaded but shown in auto-upload.

But It looks like as was one of the pictures not uploaded shown in the uploaded area in normal version. Dev-Version uploaded the picture. This picture was one of three taken at same time and appears in the uploaded area

[CordlessWool]

@AndyScherzinger can you reopen it, see my post before

[AndyScherzinger]

@CordlessWool re-opened

[mario]

@CordlessWool do you have 2.0.0RC2?

[CordlessWool]

At the moment on my phone the dev-version (20170807) and "normal"-version 1.4.3 is installed.
Should I test it with 2.0.0RC2? Are there differences to dev-version? If I search in F-Droid I only see this two versions. Do I have to uninstall version 1.4.3 before I could install release candidate?

[AndyScherzinger]

The release candidate can be installed on top and will upgrade 1.4.3 but since you mentioned f-droid: The RC is only available via Google Play at the moment since we have to wait for f-droid to update their build pipeline to even support the build of version 2.0.0. The dev build differ a lot from 2.0.0 at the moment since the auto upload changes than have been done for 2.0 require the new build pipeline which is why the latest dev build can also not be build by f-droid until then.
The only way to test the 2.0 auto upload would be to manually update the dev version (there is a check in the settings screen) which might be the best option for the moment. Else I'd say try the one from Google play but wait until Friday when 2.0.0RC3 will be released since it contains some important database upgrade fixes.

[CordlessWool]

Ok I installed the release candidate via google play store and remove dev version. I will test it the next days and will report if errors occur or not.

[mario]

@CordlessWool news?

[CordlessWool]

Yes, sorry I was a little bit busy the last days. The upload works now. Only some times there are messages that there is an error but all is uploaded. After the pictures were uploaded there appear a black tile for each picture in my local library which was uploaded. I am not sure if this happens every time but mostly

[mario]

OK, thanks.