Aws-sdk-ios: Best practice for pausing upload in AWSS3TransferManger and AWSS3TransferUtility on network loss?

[jeffjvick]

• What service are you using?
  AWSS3TransferManger
• In what version of SDK are you facing the problem?
  2.6.6
• Is the issue limited to Simulators / Actual Devices?
  I'm testing on an actual device
• Can your problem be resolved if you bump to a higher version of SDK?
  There is none higher.
• Is this problem related to specific iOS version?
  Does not appear to be.
• How are you consuming the SDK? CocoaPods / Carthage / Prebuilt frameworks?
  CocoaPods
• Can you give us steps to reproduce with a minimal, complete, and verifiable example? Please include any specific network conditions that might be required to reproduce the problem.

I start a large upload with TransferManager and part way through I turn on airplane mode to kill the wifi connection. This gives a "The Internet connection appears to be offline." error as expected since it does not appear that TransferManager handles network drops on its own like TransferUtility does.

To combat this I'm trying to catch the network loss myself and pause the uploads before TransferManager errors. I'm using the Reachability cocoapod https://github.com/ashleymills/Reachability.swift to detect network loss and it seems like it's a race to see if I can pause before TransferManager errors. Usually I lose.

I have not seen any official sdk documentation on how to handle pausing on a network loss so I wanted to see if someone could point to another source. Or possibly I should be handling this in a different manner?

Also, please don't tell me to use TransferUtility because TransferManger is "old". TransferUtility does not support multipart uploads so whenever the network is lost it just restarts from the beginning and my app will be uploading large (100+ mb) files on less than ideal networks and I can't force users to keep restarting. I don't understand why multi-part wasn't a priority for the new code base but that's the way it is so it appears I'm stuck with the old TransferManager.

Thanks

[chrisscholly]

Exactly same problem here...

[chrisscholly]

TransferUtility should support multi-parts, otherwise it is definitely useless.

[kvasukib]

Hi @jeffjvick,

Thanks for reporting to us. We are taking it as a feature request to introduce multi-part upload in TransferUtility. We will update this thread when we release it. Thank you for the understanding.

[billykahl]

Hi @kvasukib,
Any timeline on when we can expect TransferUtility to have multi-part and pausing uploads instead of restarting the upload on network loss? I am having the same issue as @jeffjvick

Thank you for your help!

[kvasukib]

Hello @jeffjvick, @billykahl, @chrisscholly,

We have implemented Mulit-part for the uploads in TransferUtility in 2.6.13. Please try it and let us know. See https://docs.aws.amazon.com/AWSiOSSDK/latest/Classes/AWSS3TransferUtility.html#//api/name/uploadDataUsingMultiPart:key:contentType:expression:completionHandler:NS_SWIFT_NAME:

[jeffjvick]

@kvasukib Oh wow, so first, thanks, I seriously didn't think you guys were actually working on this. :)

Unfortunately though I literally (today) just finished pushing the feature complete version of my app for the client and had been forced to make do with Transfer Manager. Doing so required a myriad of band-aids and work-arounds to deal with its issues (this was just one of many).

I spoke with them and they are willing to at least do some tests with Transfer Utility. We've gotten the old Transfer Manager stable to the point where most users can complete their uploads but there could be room for improvement so it's worth investigating.

I will let you know how those tests go.

[chrisscholly]

@kvasukib Thank you very much! Will take some time asap to test this new awesome feature :)

[swaminator]

Hi @jeffjvick. I'm a PM at AWS Mobile. Appreciate your contributions to this project. I'd like to chat with you sometime to get your feedback on our updates to TransferUtility. You can reach me at https://twitter.com/TheSwaminator

[kvasukib]

@chrisscholly Thank you for the response! Please try it and let us know any feedback you have.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[fantapop]

we'll be implementing this very soon. Probably within this week. Will let
you know how it goes.

On Sun, Apr 15, 2018 at 10:07 PM, stale[bot] [e-mail]
wrote:

This issue has been automatically marked as stale because it has not had
recent activity. It will be closed if no further activity occurs. Thank you
for your contributions.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/aws/aws-sdk-ios/issues/769#issuecomment-381481092,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AGWbqKFromPoNGLTsIDGFyuRaBNsX_n_ks5tpCcSgaJpZM4QrVxT
.

[jeffjvick]

@kvasukib, @swaminator I've finally started to test the new multi-part for transfer utility. First, please update your aws-sdk-ios-samples to included a multi-part example. This would be helpful for others using it. Also, please add enumerateToAssignBlocksForUploadTask to the swift examples. They are there for the objc examples but not for swift. That being said, I'm still a little unclear how to use this with multipart uploads. AWSS3TransferUtilityMultiPartUploadTask does seem to have a taskIdentifier and I sometimes get cases where multiple progress blocks get "piled on each other".

So far I'm having some issues with crashes when doing tests with network connectivity loses and moving in and out of background mode. One thing I'm unclear on, does TransferUtility pause the multiparts on network loss or is that something we still need to do on our own? According to this it looks like it does not handle pausing https://github.com/aws/aws-sdk-ios/issues/891. I've tested pausing on my own with reachability but I'm still get crashes regarding putting null objects in a dictionary.

I'll try to file some bugs for the specific issues although I'm not sure how much longer they want me working on this as it's starting to seem like it's not much better than Transfer Manager.

[cbommas]

@jeffjvick

Sorry to hear that you are still facing issues. We fixed a couple of the crashes in 2.6.18. Could you try with that version - Also, I would appreciate it very much if you could file the specific issues and will work to get them resolved.

[jeffjvick]

@cbommas Thanks. I just tried 2.6.18 and it's definitely better. I think I've only seen it crash once so far but I'm still testing.

I've filed #924 as I'm not clear if we are required to suspend/resume tasks on network failure but from my testing it appears TransferUtility is doing this itself.

I am still having some issues with the file progress getting messed up when returning from the background. This was much worse prior to 2.6.18 (it would often crash) but now I'm still having issues with the progress not fulling hitting 1.0 when the upload completes. This may be due to my miss use of enumerateToAssignBlocks which is why I commented on #595.

I will try to file other issues as I discern a pattern.

[cbommas]

@jeffjvick

I will investigate the enumerateToAssignBlocks topic and get back to you on this thread.

[jeffjvick]

thanks @cbommas

Here is a copy of the code I made for enumerateToAssignBlocks for multipart uploads (I didn't see an exact example for multipart, just normal uploads).

   @objc func applicationDidBecomeActive(notification: NSNotification) {
        let transferUtility = AWSS3TransferUtility.default()

        transferUtility.enumerateToAssignBlocks(forUploadTask: {
            (task, progress, completion) -> Void in

            }, multiPartUploadBlocksAssigner: {
            (task: AWSS3TransferUtilityMultiPartUploadTask, progress, completion) -> Void in
                print("task " + task.transferID + " prog " + String(task.progress.fractionCompleted))
                print(self.progressBlock.debugDescription)

                let progressPointer = AutoreleasingUnsafeMutablePointer<AWSS3TransferUtilityMultiPartProgressBlock?>(&self.progressBlock)

                let completionPointer = AutoreleasingUnsafeMutablePointer<AWSS3TransferUtilityMultiPartUploadCompletionHandlerBlock?>(&self.completionHandler)

                // Reassign your progress feedback
                progress?.pointee = progressPointer.pointee

                // Reassign your completion handler.
                completion?.pointee = completionPointer.pointee

                // needed??
                //task.suspend()
                //task.resume()
        }, downloadBlocksAssigner: {
             (task, progress, completion) -> Void in
        })
    }

One thing I noticed is that AWSS3TransferUtilityMultiPartUploadTask does not have a taskIdentifier while AWSS3TransferUtilityUploadTask does. Why? I'm not sure if why I'm doing is correctly reassigning the progress and completion handlers as I sometime see the progress running at multiples of 1, 2, etc when coming out of the background.

[jeffjvick]

@cbommas Have you had a chance to look at the code snip I added above? Am I handling this correctly?

[cbommas]

@jeffjvick

Apologies for the delay in my response. There has been a bunch of confusion on about how to use the enumerateToAssignBlocks from other folks as well. So, I have a taken a crack at simplifying getting the latest status of a transfer, how to associate/re-associate progressBlocks and connectionHandlers to ongoing transfers in the latest version of the SDK (2.2.23).

I have added three convenience methods to the TransferUtilityTask

• status: to get the current status of the transfer. Returns an enumeration of type AWSS3TransferUtilityTransferStatusType
• setCompletionHandler: to set a completion handler to the transfer
• setProgressBlock: to set a progress block to the transfer for tracking.

If the app transitions from background to foreground, you don't need to re-associate or setup connection handlers. The ones that you setup before the app went to the background will be in place and will work fine.

To re-associate connectionHandlers and progressBlocks after app restarts, I'd recommend the following approach (demonstrated for uploadTasks below)

 let uploadTasks = transferUtility.getUploadTasks().result
         for task in uploadTasks! {
             task.setCompletionHandler(completionHandler!)
             task.setProgressBlock(progressBlock!)
         }

You can repeat this for downloadTasks and MultipartUploadTasks.

Would love to get your feedback on how this works out for you.

[cbommas]

@jeffjvick

Regarding - _"One thing I noticed is that AWSS3TransferUtilityMultiPartUploadTask does not have a taskIdentifier while AWSS3TransferUtilityUploadTask does"_

The Mulitpart transfer mechanism uses a NSURLSessionTask per part to upload the file in pieces. That is the reason why there is no taskIdentifier on it. All tasks, however, have a unique TransferID, that you should use instead of taskIdentifier to keep track of transfers.

[jeffjvick]

@cbommas Thanks for these additions, I appreciate it. Adding status is great as that was something present in the UploadRequests of TransferManager that was not, up til now, present in TransferUtility. setProgressBlock should also be helpful as I was previously having to keep a list of expressions so I could later set the progress block in a different view but now I can just keep track of the tasks.

I will try this out and report back. Thanks.

[SanCHEESE]

@cbommas
Ok, I've tried it out, but its not working. What am I doing wrong?
When app terminates, I suspend all ongoing tasks using suspend.
Then when app launches after creating AWSS3TransferUtility instance, I enumerate utility's getDownloadTasks.result tasks and there is none of them in returned array.

[cbommas]

@SanCHEESE

The transferUtility uses the value passed in for the key (see below) to link back to the transfers that were in progress in the previous run of the app

_AWSS3TransferUtility.register(with: self.configurationAWSfor(video: video)!, forKey: key_

Is the value of the key in your case a constant value? As a best practice, I would suggest is to register the TransferUtility in your app delegate. Once registered, you can look up the client wherever needed using the _AWSS3TransferUtility.s3TransferUtility(forKey: key)_ method.

There is one nuance that I'm working on. The TransferUtility recovery mechanism happens asynchronously. So if you make the getDownloadTasks call immediately after registering the TransferUtility, there may be zero results based on timing ( there will essentially be 2 threads - the thread that the app is working on and the thread that is being used by the TransferUtility recovery mechanism).

In my testing so far, registering the TransferUtility in the app delegate and looking it up in a view controller subsequently seemed to always work; but I understand that this is a hit or miss approach. I will be working on a callback mechanism that will get a notification when the TransferUtility recovery is completed to address this in a more deterministic manner.

[jeffjvick]

@cbommas

I have tested setProgressBlock on previously created AWSS3TransferUtilityMultiPartUploadTask and it seems to work fine. This is a nice convince method as previously I had to keep track of all of my expressions so I could later attach a progressBlock to them. I have not yet test setCompletionHandler.

The auto re-assignment of progressBlocks when going in/out of background seems to work fine as well. I have not yet experienced any issues of mismatches when uploading multiple files at once.

I have noticed some issues with status though. For one thing, if you issue a cancel()to a AWSS3TransferUtilityMultiPartUploadTask while the network is down the status doesn't change to AWSS3TransferUtilityTransferStatusCancelled until the network is back up, it just stays in AWSS3TransferUtilityTransferStatusInProgress. Was this by design or a bug? TransferManager would sometimes get stuck if you tried to cancel while there was no network but I have not experienced that with TransferUtility.

Another issue I've run into is that now that TransferUtility handles pausing/resuming the upload when the network goes in and out it is not firing errors in the completionHandler like TransferManager would do. I was using these errors to change status bars regarding the upload being "paused" due to network. While I should be able to replace this with some Reachability code, I noticed that the status does not change to AWSS3TransferUtilityTransferStatusPaused, it's still AWSS3TransferUtilityTransferStatusInProgress. Is the paused status only being for manual suspends of uploads and not the network dropping? That's fine if so I just want to clarify this is not a bug.

[jeffjvick]

@cbommas @SanCHEESE

I also tested out transferUtility.getMultiPartUploadTasks().result.

When applicationWillTerminate fires I run though my list of mutlipartUploadTasks and suspend them all. Then when the app is restarted I check transferUtility.getMultiPartUploadTasks().result using let transferUtility = AWSS3TransferUtility.s3TransferUtility(forKey: "my-advanced-options"). Thank you @cbommas for pointing out that the key is needed but please update this in the docs as it's not clear.

I can see tasks in the array however since this is a multipart upload I often see more than one task for the same transferID. I'm not clear if I am supposed to resume all of these or would just one suffice? Also, when I start a multipart upload for the first time the completion handler normally just returns me one AWSS3TransferUtilityMultiPartUploadTask which I use to keep track of my uploads but in the case of this resuming, which of the many tasks should I pick? When I just pick the first one and try to setProgressBlock on it never updates the progress. Can you please offer more guidance on how to handle multipart for this case?

Also, I'm seeing the asynchronous nature of registering transferUtility as sometimes transferUtility.getMultiPartUploadTasks().result is empty when it should not be. I am registering transferUtility in the app delegate so I'm doing it as early as possible. Some type of notification appears necessary.

Honestly though I'm considering not using this since if a user force quits the app or powers off their phone while uploading they shouldn't really expect it to continue where they left off and as long as I'm not a memory hog I shouldn't get terminated while in the background.

[SanCHEESE]

@cbommas
Still no luck. But sometimes there are download tasks in utility on app launch, but when I resume those downloads, download progress begins from 0, which means that already downloaded data was lost.
I followed your advice and replaced [AWSS3TransferUtility defaultS3TransferUtility] with custom registered utility with constant key which is registered and restored using +S3TransferUtilityForKey. Knowing that utility registering is async I do wait until utility gets the tasks in bg thread:

NSTimeInterval totalWait = 0;
NSArray *downloadTasks;
while ((downloadTasks = utility.getDownloadTasks.result).count == 0 && totalWait < 5) {
[NSThread sleepForTimeInterval:1];
totalWait += 1;
}
Sometime there are tasks, sometimes there are no. If there are, resuming download begins it anew, which I don't understand.

Let me explain what am I doing. First of all, I have 2 utilities for downloading. The first one is for normal downloads, second is for accelerated downloads. All utilities registered with constant keys. All the tasks created are retained and associated with downloading files and download path is simply a bucket key. All the tasks ongoing in the moment of termination are being suspended. When app is moved to an active state, utility's -(void)enumerateToAssignBlocksForUploadTask:downloadTask: is called.

[cbommas]

@jeffjvick

Let me address your points as best as I can

_The_ auto re-assignment of progressBlocks when going in/out of background seems to work fine as well. I have not yet experienced any issues of mismatches when uploading multiple files at once._ Just to clarify, for this I am not auto-assigning anything. It is just that when the app goes into the background and transitions back to the foreground, the OS will make sure that all objects and references that were in memory before will remain as-is after. So it simply works :)

_if you issue a cancel()to a AWSS3TransferUtilityMultiPartUploadTask while the network is down the status doesn't change to AWSS3TransferUtilityTransferStatusCancelled until the network is back up_ - I will look into this. The behavior is that the transfer task will go into a cancel status immediately, so there may be a bug.

_Is the paused status only being for manual suspends of uploads and not the network dropping?_ Yes, the paused status is only for explicit suspends of uploads/downloads and not network drops. The TransferUtility will continue its attempt to transfer ( and retry as per the retry threshold) if the network drops intermittently.

_I can see tasks in the array however since this is a multipart upload I often see more than one task for the same transferID. I'm not clear if I am supposed to resume all of these or would just one suffice?_ I haven't seen this in my local testing. The expectation is that you will only have one TransferUtilityTask per TransferID ( So only one task, no matter how many how many parts it has). There may be a bug and I will investigate and post back on it. If you can dump the contents of the array into a log and post the log here, it will be super helpful.

_Some type of notification appears necessary._ Agreed. I am working on this.

_Honestly though I'm considering not using this since if a user force quits the app or powers off their phone while uploading they shouldn't really expect it to continue where they left off and as long as I'm not a memory hog I shouldn't get terminated while in the background._ I see your point of view, though iOS may have a slightly different take based on how they have positioned background transfers. In any case, your strategy of suspending all transfers when applicationWillTerminate will enable you to fully control this as you see fit.

[cbommas]

@SanCHEESE

sorry about the troubles you are having on the timing issue. Unfortunately, the best solution for that is the callback (good news is that I'm working on it) as putting a delay can be hit or miss.

_Why does the download start anew_ - I see that you are suspending the tasks when you get a signal that the app is going to be terminated. When I get the callback working, you will see all the tasks properly loaded up as the timing issue will be gone then.

For files that were being downloaded that were paused, if you resume them within a time limit ( typically 20 seconds based on a S3 server side timeout), it will continue where it left off. Otherwise, the transfer will error with a server time out and will be retried. For downloads, it will cause it to start from 0.

[jeffjvick]

@cbommas Thanks for the response

Her is the dump of transferUtility.getMultiPartUploadTasks().result. It seems to usually be five and they all appear to be the same:

savedMultipartUploadTasks   NSArray 5 elements  0x00000001c0241d70
[0] AWSS3TransferUtilityMultiPartUploadTask *   0x1c012b0e0 0x00000001c012b0e0
[1] AWSS3TransferUtilityMultiPartUploadTask *   0x1c012b0e0 0x00000001c012b0e0
[2] AWSS3TransferUtilityMultiPartUploadTask *   0x1c012b0e0 0x00000001c012b0e0
NSObject    NSObject    
_cancelled  BOOL    NO  false
_temporaryFileCreated   BOOL    NO  false
_retryCount int 0
_partNumber int 0
_transferID __NSCFString *  @"B9ADDF8F-8F76-4186-9FA8-D03BECB0F654" 0x00000001c0279180
_bucket __NSCFString *  @"test-upload"  0x00000001c005f3b0
_key    __NSCFString *  @"0F893573-B15F-407A-85DD-CAEA6D1DF9F7_76Kc8kuucDeFKirts"   0x00000001c008ec40
_progress   NSProgress *    0x1c012b180 0x00000001c012b180
_status AWSS3TransferUtilityTransferStatusType  AWSS3TransferUtilityTransferStatusPaused
_expression AWSS3TransferUtilityMultiPartUploadExpression * 0x1c005f380 0x00000001c005f380
_uploadID   __NSCFString *  @"YoSN8eN4gRHvbwc6VGc25Jz2iqhYbxEw4.AD_Ai7BNBO48LEAcJXbqaoUuCVyq9ZjOIrRF_BWvtWgDRJhdv3oKoQeMwBjmrJeHQ2ovZP8S0CbG6LAQdx462cTi44RRet9uBMgKop0FSIkKvP85VtSP3Ejrk4PIR6dB919mQn6hE-" 0x00000001c017a100
_waitingPartsDictionary __NSDictionaryM *   0 key/value pairs   0x00000001c0225420
_completedPartsDictionary   __NSDictionaryM *   0 key/value pairs   0x00000001c0225520
_inProgressPartsDictionary  __NSDictionaryM *   5 key/value pairs   0x00000001c0225480
_file   __NSCFString *  @"/private/var/mobile/Containers/Data/Application/1D76DF01-9470-4AC8-8B28-7C3091F594D3/tmp/F2FF6EEF-1F08-4BA5-8A1B-2CDFC83C329D.MOV"    0x00000001c012b040
_transferType   NSString *  nil 0x0000000000000000
_nsURLSessionID __NSCFString *  @"com.amazonaws.AWSS3TransferUtility.Identifier.transfer-utility-with-advanced-options" 0x00000001c00d6ff0
_databaseQueue  AWSFMDatabaseQueue *    0x1c0240030 0x00000001c0240030
_error  NSError *   nil 0x0000000000000000
_contentLength  __NSCFNumber *  (int)39211963   0xb000000025653bb2
[3] AWSS3TransferUtilityMultiPartUploadTask *   0x1c012b0e0 0x00000001c012b0e0
NSObject    NSObject    
_cancelled  BOOL    NO  false
_temporaryFileCreated   BOOL    NO  false
_retryCount int 0
_partNumber int 0
_transferID __NSCFString *  @"B9ADDF8F-8F76-4186-9FA8-D03BECB0F654" 0x00000001c0279180
_bucket __NSCFString *  @"test-upload"  0x00000001c005f3b0
_key    __NSCFString *  @"0F893573-B15F-407A-85DD-CAEA6D1DF9F7_76Kc8kuucDeFKirts"   0x00000001c008ec40
_progress   NSProgress *    0x1c012b180 0x00000001c012b180
_status AWSS3TransferUtilityTransferStatusType  AWSS3TransferUtilityTransferStatusPaused
_expression AWSS3TransferUtilityMultiPartUploadExpression * 0x1c005f380 0x00000001c005f380
_uploadID   __NSCFString *  @"YoSN8eN4gRHvbwc6VGc25Jz2iqhYbxEw4.AD_Ai7BNBO48LEAcJXbqaoUuCVyq9ZjOIrRF_BWvtWgDRJhdv3oKoQeMwBjmrJeHQ2ovZP8S0CbG6LAQdx462cTi44RRet9uBMgKop0FSIkKvP85VtSP3Ejrk4PIR6dB919mQn6hE-" 0x00000001c017a100
_waitingPartsDictionary __NSDictionaryM *   0 key/value pairs   0x00000001c0225420
_completedPartsDictionary   __NSDictionaryM *   0 key/value pairs   0x00000001c0225520
_inProgressPartsDictionary  __NSDictionaryM *   5 key/value pairs   0x00000001c0225480
_file   __NSCFString *  @"/private/var/mobile/Containers/Data/Application/1D76DF01-9470-4AC8-8B28-7C3091F594D3/tmp/F2FF6EEF-1F08-4BA5-8A1B-2CDFC83C329D.MOV"    0x00000001c012b040
_transferType   NSString *  nil 0x0000000000000000
_nsURLSessionID __NSCFString *  @"com.amazonaws.AWSS3TransferUtility.Identifier.transfer-utility-with-advanced-options" 0x00000001c00d6ff0
_databaseQueue  AWSFMDatabaseQueue *    0x1c0240030 0x00000001c0240030
_error  NSError *   nil 0x0000000000000000
_contentLength  __NSCFNumber *  (int)39211963   0xb000000025653bb2
[4] AWSS3TransferUtilityMultiPartUploadTask *   0x1c012b0e0 0x00000001c012b0e0

[cbommas]

@jeffjvick
This is definitely a bug. I will fix it in an upcoming release. You are correct that all of these are references to the same object. You should only resume one of them.

[cbommas]

@jeffjvick
The bug fixes for the issues you pointed out and the completion handler feature are included in the latest rev of the SDK. See my post on the https://github.com/aws/aws-sdk-ios/issues/759 thread to see how to setup the completion handler.

It will be great if you could give this a go and let me know your thoughts.

[cbommas]

@jeffjvick
Just checking back to see if the fixes helped in resolving the issues you were encountering

[jeffjvick]

@cbommas
I was able to try out the new version. I only see one task now instead of 5 so that appears to be fixed. Adding a completion handler onto the resumed task also works fine. There seems to be an issue with the progress block though as progress.fractionCompleted is always 1.0. Do you have an idea of why this may be occurring?

[cbommas]

@jeffjvick

Interesting. I haven't observed that in my testing. Is this happening for all types of transfers? Only after the transfer is resumed/continued after app restart?

[jeffjvick]

@cbommas
It was only happening for resumes after app restart. I just tried the new 2.6.26 and it works fine. It looks like 2.6.25 has this fractionCompleted issue. Something must have been changed/fixed in 26.

I'm noticing another issue now. Do we still need to call AWSS3TransferUtility.interceptApplication(application, handleEventsForBackgroundURLSession: identifier, completionHandler: completionHandler) in func application(_ application: UIApplication, handleEventsForBackgroundURLSession identifier: String, completionHandler: @escaping () -> Void) {?

It seems when I do that the completion handler for a task is not firing when the upload completes in the background. If I omit the interceptApplication call though it works. Did something change because I thought that was required to notify Transfer Utility the background upload has completed.

[cbommas]

@jeffjvick
You don't need to call the interceptApplication any more. With the recent changes in the TransferUtility, I have put in some measures to ensure that the completion handler is always fired when the transfer reaches an end state (i.e., completed successfully or ended up in an error)

[jeffjvick]

@cbommas

Okay that seems to align with what I’m seeing. You should probably update the documents about this and the many other things that have changed (like not needing to enumerate, etc)

I’m still having issues though where the completion handler does always fire when the upload finishes in the background. It seems to happen only when the debugger is not attached which makes debugging it difficult. Are there normal cases in which it will not fire?

[jeffjvick]

@cbommas

I'm still having an issues where background multipart uploading is not working when I'm NOT connected to the debugger and the upload time is longer than ~1 min. I think I'm not seeing it when the debugger is attached because the debugger never lets the app actually suspend: https://forums.developer.apple.com/message/42353#42353

I'm starting to think I never tested background uploading without the debugger connected. Do you have an idea of what may be causing this?

[cbommas]

@jeffjvick
I am looking into this issue. I have made some changes to fix the resume issue for MultiPartUploads #1015 that should get out in the next release.

I think I can explain what you are seeing. For Multipart uploads, the TransferUtility uses a "relay" style architecture. The file to be uploaded is divided into 5MB chunks and uploaded as n requests in parallel. (n is determined by the multiPartConcurrencyLimit attribute in AWSTransferUtilityConfiguration). When one of the n requests is completed, then the next one i.e., n+1 is issued by the TransferUtility. And this process continues until all parts are uploaded and a final CompleteMultiPartUpload operation is executed to finalize the upload.

For the relay architecture to work, the app has to be running ( either in the foreground or background). If the app is closed/restarted, you will need to instantiate the TransferUtility and it will reattach to the ongoing transfers and continue them.

Based on your comments on this thread and in a couple of others, my understanding is that you are expecting the app to "woken up from background" or "restarted " when a transfer is completed. Is that right?

I'd like to jump on a call with you to discuss this further. Let me know if you are open to it and I can setup a conference call.

[jeffjvick]

@cbommas

Since you are using URLSession I was under the impression that urlSessionDidFinishEvents(forBackgroundURLSession session: URLSession) could be called to run our completionHandlers once all the parts have completed. But maybe there is something different about how a multi-part upload is handled that I'm not understanding.

Yes a call would be good, I'm free today, 8/31.

[cbommas]

@jeffjvick

How about 4:00 PM PST at the following conference bridge

United States Toll-Free: [phone]
Meeting PIN: 9376 87 1968

[jeffjvick]

@cbommas That works, thanks.

[jeffjvick]

@cbommas Do you have an update on progress towards a version that allows multipart background uploads with more than n parts to complete?

[cbommas]

@jeffjvick

Thanks for your patience on this issue. I am working on this and am targeting to get this into the next rev of the SDK. I will post back on this thread once I have more info.

[cbommas]

@jeffjvick

Here is an update as promised. I did a bunch of tests using the TransferUtility and was able to consistently get uploads ( including multipart uploads) and downloads to complete successfully in the background. In my tests, I foregrounded and backgrounded the app multiple times and the transfer finished successfully every time.

My setup included

• Running my transfer test app using XCode on a simulator.
• Running the transfer test app using XCode on a physical device
• Running the transfer test app on a physical device ( without XCode)

I had the following settings for the transferUtilityConfiguration

        transferUtilityConfiguration.retryLimit = 10 //I set this high to guard against server side timeout errors 
        transferUtilityConfiguration.multiPartConcurrencyLimit = 1 //I set it to 1 to ensure the relay style logic is triggered multiple times
        transferUtilityConfiguration.timeoutIntervalForResource = 15*60

The only change I had to make to my app was to add the following in the AppDelegate.

func application(_ application: UIApplication, handleEventsForBackgroundURLSession identifier: String, completionHandler: @escaping () -> Void) {
        print ("handleEventsForBackgroundSession method called for session Identifier:" + identifier)
        AWSS3TransferUtility.interceptApplication(application, handleEventsForBackgroundURLSession: identifier, completionHandler: completionHandler)
    }

I think I may have added to the confusion on this topic by suggesting earlier that this callback was no-longer required. However, I was mistaken. Adding this callback to the App suggests to iOS that it should provide CPU cycles to the app when an ongoing NSURLSession task is completed. With this callback in place, I noticed that the OS automatically woke up the app in the background.

The speed of the upload/download varied over the course of the tests, depending on how the OS prioritized the app and associated transfers; however, the transfers finished successfully each time.

I was also able to find the issue in multipart uploads where the progress tracking was being under-reported when the app moves from background to the foreground. I have a fix for it, which I am hoping to get included in the next rev of the SDK.

Can you please try this on your side when you have a moment and let me know how it goes.

[jeffjvick]

@cbommas Thanks I will try this out.

I'm guessing it has to do with the interceptApplication part as I had removed that a while ago at your request. I thought I had tried a test with adding it back in but I was trying a lot of things at the time.

[cbommas]

@jeffjvick

Thank you. Please let me know how it goes. By the way, the latest rev of the SDK (2.6.31) contains the fix for the under-reporting of the mulitpart transfer progress bug

[cbommas]

@jeffjvick
Just checking back. Let me know if you have any questions/issues that I can help with.

[jeffjvick]

@cbommas
So I have been testing it out and I can see that the progress under-reporting is better (which was part of the reason I was thinking it wasn't working before) but I sometimes see it over-report by a few percent.

However, it still seems like the uploads are not continuing when going to the background or are possibly just progressing extremely slowly. I have see a few times where it finished in the background but usually it seems to have made minimal progress after many minutes of waiting.

I understand that iOS adjusts the priority of the background tasks but this seems excessive so I'm not sure if I am still missing something or is this really the speed at which background uploads take. My experience with other apps doing background downloads is that it is much faster.

[cbommas]

@jeffjvick

I ran a bunch more uploads using my testapp and the upload speeds were varying rather widely; I couldn't spot a pattern, other than what you observed - in some cases the transfer had completed, in other cases it had stalled. However, in all cases, the transfers did eventually finish, which led me to believe that the transfers were progressing extremely slowly.

I looked for information on how iOS schedules & prioritizes the transfers and I couldn't find anything definitive. What I have seen on the forums is it depends on a bunch of factors - is the device on Wifi or cellular, is the device fully charged, is it plugged in etc.

I also ran some tests by the setting the discretionary flag to Yes ( this is currently not set by the TransferUtility and defaults to No). With the discretionary flag set, it was actually even slower.

Regarding the progress over-reporting, is this showing up in your progress bars? I have seen that if you use a DispatchQueue.main.async to call your UI logic, it can have weird results from a sequencing perspective. I have had to implement logic as shown here (https://github.com/awslabs/aws-sdk-ios-samples/blob/0a301ad6bdf1cb2e1ecdda9354ea76daa9ea8555/S3TransferUtility-Sample/Swift/S3BackgroundTransferSampleSwift/UploadViewController.swift#L38) to prevent the progress bar from skipping forwards and backwards.

[jeffjvick]

@cbommas Thanks for looking at this. I had looked as well into how iOS priorities background transfers and like you said there is no definitive answer. I did read though that the isDiscretionary flag set to true is supposed to make things slower as you are telling iOS to leave it up to its discretion of how to prioritize your upload, which appears to always be "low" 😂. So that makes sense with what you saw when changing this.

https://developer.apple.com/documentation/foundation/nsurlsessionconfiguration/1411552-discretionary

One interesting thing to note though in that documentation is in the last paragraph: "For transfers started while your app is in the background, the system always starts transfers at its discretion". It's possible that when your scheduler starts new parts in the background the discretionary flag is getting set by iOS regardless of the initial setting and thus why it progresses so slowly in the background. This kind of make sense since I always see the first part(s) started in the foreground completing quickly in the background but then subsequent parts started in the background complete significantly slower.

I also did some tests using single part uploads and they all progress very quickly in the backend. This could also support the theory that the parts started in the background are getting hit with discretionary flag since in this the case there is only one "part" which is started in the foreground and thus has this flag set to false. You could try setting discretionary to true and doing a background single part upload and see if it is also slow

Could you check in your scheduler if this is possibly what was happening? (new parts getting started in the background have the discretionary flag set).

Regarding the progress over-reporting, I'm just doing a print(Float(progress.fractionCompleted)) in the progress block and I see it over-reporting. It seems to have to do with the file size, possibly the further the size is from a multiple of 5 the worse the over-reporting. For instance a 42.9Mb file ended with progress.fractionCompleted like so (using a concurrency of 1 part):

1.3723482
1.3752619
1.3781755
1.3810892
1.3840029
1.3869166
1.3898304
1.392744
1.3956577
1.3985714 <--- this is when it finished

Larger files exhibit less over-reporting, potentially since the ratio of the small difference in the one part is less. You might not notice it.

[cbommas]

@jeffjvick

_Could you check in your scheduler if this is possibly what was happening? (new parts getting started in the background have the discretionary flag set)._

Very nice analysis! I agree with your findings. This is exactly what is happening - The discretionary flag is set per session and defaults to No. In my code, I am using the default value when the NSURLSession is created. For transfers initiated with the app in the foreground, the system is using the value of No ; for transfers initiated with the app in the background, the OS is overriding what was set in the NSURLSession and treats it as discretionary, and therefore they progress slower. Let me look into this and see if I can find a way around it.

I will investigate the over-reporting issue and post back here in a few days.

[cbommas]

@jeffjvick

Here is an idea that may help get around this - there is a property in the AWSS3TransferUtilityConfiguration called multiPartConcurrencyLimit. This sets an upper limit in the number of parts that are being uploaded at any given point of time.

Can you adjust that up to accommodate your typical file sizes - if your typical filesize is 50 MB or lesser, then adjust it to at least 10 so that all the parts will be uploaded in parallel. It should result in each part being initiated when the app is in the foreground and speed things up. We don't need to worry about overloading the device as the background NSURLSession prioritization mechanism will kick in and only allow a subset of transfers to proceed at any given point.

[cbommas]

@jeffjvick
Just a quick update - I ran some experiments over the weekend. The speeds are definitely much improved with a higher setting for the multiPartConcurrencyLimit. Let me know how it goes on your side once you've had a chance to try it out.

[jeffjvick]

@cbommas

I was able to try increasing the multiPartConcurrencyLimit. I had considered this but I thought I read somewhere that this had a max limit of 5. Anyway, for our use case the uploads are in the 100's of MB so we would need a very large limit to get everything initiated in the foreground. I tried setting it to 999 and tried some files around 200 MB and it seemed to work well (as in, not slowing down in the background) but this seems like "cheating". I wonder if iOS's prioritization may start to throttle the app's background transfers if it detects us not using discretionary on too many background transfers. I need to do more testing.

Are you aware of any max number on this limit? Is the 5 MB per part set in stone or can that be adjusted as well?

For another data point, I have tested Google's drive app and it does background uploads rapidly and appears to support multi-part. I'm not sure what they are doing under the hood to accommodate this.

[jeffjvick]

@cbommas

Hi, also I have another follow-on question for you. Why did you implement a background scheduler if it seems you can just create all the NSURLSession parts at once and let it do the scheduling? I assume to limit the number of concurrent parts uploading but it seems even in the foreground there is some sort of internal limit placed on this.

[Jane930525]

@jeffjvick

Regarding - _"One thing I noticed is that AWSS3TransferUtilityMultiPartUploadTask does not have a taskIdentifier while AWSS3TransferUtilityUploadTask does"_

The Mulitpart transfer mechanism uses a NSURLSessionTask per part to upload the file in pieces. That is the reason why there is no taskIdentifier on it. All tasks, however, have a unique TransferID, that you should use instead of taskIdentifier to keep track of transfers.

Thank you for the detailed explanation.

[cbommas]

@jeffjvick

_Why did you implement a background scheduler if it seems you can just create all the NSURLSession parts at once and let it do the scheduling?_

The thinking essentially was to provide the controls to the developer on how "rich" of a mix they wanted to run using the multiPartConcurrencyLimit to tune. The one thing that did not occur to me was the impact of the OS prioritization logic that automatically applied the discretionary flag to the parts that were initiated when the app was in the background.

One of the ideas I am working on is to create all the NSURLSession tasks upfront, but only start up to the multiPartConcurrencyLimit. I will let you know in a few days how this goes.

[rromanchuk]

Are you any of you folks having issues with keeping your credentials alive? I'm getting some failed uploads with "The operation couldn’t be completed. (NSURLErrorDomain error -1001.)" I don't have a lot of insight yet, it sure feels like expired cognito credentials though, it looks like it happens when a user brings the app back up to the foreground after after some period of time.

I also feel like i'm abusing this with a very lazy singleton i'm using

 lazy var transferUtility: AWSS3TransferUtility = {
-        AWSS3TransferUtility.s3TransferUtility(forKey: "my-upload-manager")
 }()

This is probably abusive type of use, does anyone have any intimate knowledge of how transfer utility and cognito lifecycle works? When does the countdown begin, and is it smart enough to refresh on the next retry? I could see a number of ways this type of singleton abuse https://gist.github.com/rromanchuk/ca7b994be0fcf69387c8c6c74f7c36fb could be burning me

[minbi]

Hi @jeffjvick @Jane930525 @rromanchuk ,

We have just released version 2.8.2 of the SDK. Please check if this fixes the issue for you.

[cbommas]

@jeffjvick

This version has the change to create all the NSURLSession tasks upfront. It also has a new approach for calculating progress for multipart transfers - which should resolve the progress tracking issue that you have been encountering. Please upgrade to this when you have some time and let me know if you run into issues.

[jeffjvick]

@cbommas Hi, thanks, I will try testing it this evening.

[stale[bot]]

This issue has been automatically closed because of inactivity. Please open a new issue if are still encountering problems.

[jeffjvick]

@cbommas, Hi, so when I said I will test this in the "evening" I apparently meant 4 months from now... ;)

I am testing this using version 2.9.7 although I believe not much has changed regarding AWSS3 since 2.8.2. I am on an iOS 12.2 device.

I'm still have some issues after upgrading. I can open a new issue if you'd like but I'll just list them here for now:

1)
Regarding the progressBlock, I have noticed that if while you are uploading you disable the network and then reenable it (causing the currently upload parts to restart) the progressBlock stops getting updates until the progress has surpassed the point where it left off. This is especially evident if you do a non-multipart upload. So for example you are uploading a 50 mb file and at 30% you turn on/of the network, it will restart the upload but the progress will just sit at 30% until the restarted upload passes back over 30%.

I looked into the AWS3TransferUtility.m code and see the issue. It is only updating transferUtilityUploadTask.progress.completedUnitCount if it is less than'totalBytesSent but this doesn't take into account if totalBytesSent gets reset when parts (or the whole) upload needs to get restarted. I think the easiest way to fix this would be to just use !=. I'm not sure why you have <. I made these two changes and it is working:

if (transferUtilityUploadTask.progress.completedUnitCount != totalBytesSent) {
    transferUtilityUploadTask.progress.completedUnitCount = totalBytesSent;

    if (transferUtilityUploadTask.expression.progressBlock) {
        transferUtilityUploadTask.expression.progressBlock(transferUtilityUploadTask, transferUtilityUploadTask.progress);
    }
}

if (transferUtilityMultiPartUploadTask.progress.completedUnitCount != totalSentSoFar ) {
    transferUtilityMultiPartUploadTask.progress.totalUnitCount = [transferUtilityMultiPartUploadTask.contentLength longLongValue];
    transferUtilityMultiPartUploadTask.progress.completedUnitCount = totalSentSoFar;

    //execute the callback to the progressblock if present.
    if (transferUtilityMultiPartUploadTask.expression.progressBlock) {
        AWSDDLogDebug(@"Total %lld, ProgressSoFar %lld", transferUtilityMultiPartUploadTask.progress.totalUnitCount, transferUtilityMultiPartUploadTask.progress.completedUnitCount);
        transferUtilityMultiPartUploadTask.expression.progressBlock(transferUtilityMultiPartUploadTask, transferUtilityMultiPartUploadTask.progress);
    }
}

2)
Also regarding the progressBlock, I'm seeing now where if you start an upload and go the background (hit home button) and come back the progress reporting has stopped. It is still uploading though because if you wait a bit the upload will complete (completionHandlers will fire and progressBlock will get called once with a fractionComplete of 1.0). Sometimes if you go in and out of the background a few times you can get the progressBlock to start working again. Have you seen this behavior?

I haven't been able to test the new changes you made for multipart yet because I'm stuck on these issues. I'm not sure how someone didn't notice these problems with the progress reporting.

[palpatim]

@jeffjvick

Please do open a new issue with the repro steps and details of your setup so we can investigate.

[jeffjvick]

@palpatim @cbommas

I have made https://github.com/aws-amplify/aws-sdk-ios/issues/1512 and https://github.com/aws-amplify/aws-sdk-ios/issues/1513

[jeffjvick]

@palpatim @kvasukib

Hi, does someone have an update for #1512 or #1513 ?

It's been over one and half years since we tried to integrate TransferUtility into our project and it's still too broken for us to release to production. Our app is still using TransferManager.

[navasiloy]

Hi @jeffjvick

Looks like you did a great job finding all these small details!
I've got the same task (100Mb upload).
Can you tell, please, what settings you ended with for

• multiPartConcurrencyLimit
• part file size
• do you start all parts foreground (discretionary = no)

Thank you in advance!

[Guolanlan]

I encountered a similar problem. After the app is killed and restarted, the upload will not resume, because the status of the fragmentation task has changed to error. I debugged it in the source code. When creating a temporary subtask, the path of my fragmentation task Not found,
-(NSString ) createTemporaryFileForPart: (NSString *) fileName
partNumber: (long) partNumber
dataLength: (NSUInteger) dataLength
error: (NSError *) error{

 //Check if the file exists.
 if (![[NSFileManager defaultManager] fileExistsAtPath:fileName]) {
     NSString *errorMessage = [NSString stringWithFormat:@"Local file not found. Unable to process Part #: %ld", partNumber];
     NSDictionary *userInfo = [NSDictionary dictionaryWithObject:errorMessage
                                                          forKey:@"Message"];
     if (error) {

and But I still don’t understand, why can’t I find this path? If uploadDataUsingMultiPart is used, this problem will not exist? ?