Aws-sdk-ios: [S3] Completion Handler is sometimes not invoked when the transfer errors due to timeout.

[simeondrage]

Describe the bug
A clear and concise description of what the bug is.

To Reproduce
Steps to reproduce the behavior:

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

Which AWS service(s) are affected?

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Environment(please complete the following information):

• SDK Version: [e.g. 2.6.29]
• Dependency Manager: [e.g. Cocoapods, Carthage]
• Swift Version : [e.g. 4.0]

Device Information (please complete the following information):

• Device: [e.g. iPhone6, Simulator]
• iOS Version: [e.g. iOS 11.4]
• Specific to simulators:

Additional context
Add any other context about the problem here.

[simeondrage]

@pablogeek could you add in the details of what you have found out please and any technical information above. Thanks

[simeondrage]

This ticket is a continuation from https://github.com/aws-amplify/aws-sdk-ios/issues/1065 - please let me know if we should continue to use that one

[pablogeek]

We are experiencing issues with low internet connections. For some reason the completion callback is not called sometimes in case of timeout or connection error.

We are using transfer utility to upload files to S3

[pablogeek]

I get this error in debug mode (from logs) and the completion callback isn’t being called.

Task <E78A1C37-7EE8-4337-9F80-093CDF664B5F>.<3> load failed with error Error Domain=NSURLErrorDomain Code=-1001 "The request timed out." UserInfo={_kCFStreamErrorCodeKey=-2102, NSUnderlyingError=0x283dd1290 {Error Domain=kCFErrorDomainCFNetwork Code=-1001 "(null)" UserInfo={_kCFStreamErrorCodeKey=-2102, _kCFStreamErrorDomainKey=4}}, _NSURLErrorFailingURLSessionTaskErrorKey=LocalDataTask <E78A1C37-7EE8-4337-9F80-093CDF664B5F>.<3>, _NSURLErrorRelatedURLSessionTaskErrorKey=(
    "LocalDataTask <E78A1C37-7EE8-4337-9F80-093CDF664B5F>.<3>"
), NSLocalizedDescription=The request timed out., NSErrorFailingURLStringKey=https://s3-eu-west-1.amazonaws.com/eu-hermes-discovery-kit/eu/main/2014_07_14_11_21_27/videos/44/421551/vid/1234567890_part_1.mov?uploads=, NSErrorFailingURLKey=https://s3-eu-west-1.amazonaws.com/eu-hermes-discovery-kit/eu/main/2014_07_14_11_21_27/videos/44/421551/vid/1234567890_part_1.mov?uploads=, _kCFStreamErrorDomainKey=4} [-1001]
2019-02-09 00:09:44.257860+0100 IrisConnect_BETA[5406:614114] Task <E78A1C37-7EE8-4337-9F80-093CDF664B5F>.<3> HTTP load failed (error code: -999 [1:89])

[kvasukib]

@simeondrage @pablogeek Sorry for the inconvenience caused.
Can you share a code snippet where you construct the callback object?
Can you try with the latest version of the SDK?
What version of iOS are you using?
What is the state of the app (foreground/background/suspended) when the request times out?

[pablogeek]

@kvasukib sure :)
iOS 12
SDK 2.8.4
Always foreground

I have been able to reproduce this when I use the networking conditioner (bad network or 100% loss).

let completionBlockUpload: AWSS3TransferUtilityMultiPartUploadCompletionHandlerBlock = {
            [weak self, weak video] task , error in
           //Never enters here when timeout
            NSLog("completionBlockUpload!!")
                guard let video  = video else { return }
                switch task.status {
                case .completed:
                    self?.checkS3File(video: video) { length, error in
                        guard error == nil else {
                            DispatchQueue.main.async {
                                self?.errorVideo(error, video: video)
                            }
                            return
                        }
                    }
                default:
                    DispatchQueue.main.async {
                        self?.errorVideo(error, video: video)
                    }
                }
        }

AWSTransfer(video: video) { [weak self] transferUtility, error in
            if let error = error {
                Slim.error(error.localizedDescription)
                print("Error: \(error.localizedDescription)")
                self?.errorVideo(error, video: video)
                return
            }
            let fileURL = URL(fileURLWithPath: video.localFilePath)

            transferUtility?.uploadUsingMultiPart(fileURL: fileURL,
                                                             bucket: video.amazonBucket,
                                                             key: video.uploadKey,
                                                             contentType: "video/mov",
                                                             expression: expression, completionHandler: completionBlockUpload).continueWith(executor: AWSExecutor.mainThread()) {
                task -> AnyObject? in
                // sometimes it enters here, but error is nil.
                if let error = task.error as NSError? {
                    print("Error: \(error.localizedDescription)")
                    self?.errorVideo(error, video: video)
                }
                NSLog("continueWith!! \(String(describing: task.error))")

                return nil
            }
        }

[pablogeek]

@kvasukib I have been playing a little bit and basically the problem is the sdk doesn't provide info about timeouts. The sdk keeps retrying to upload in case of timeout, this is good and it should be in that way, the only thing is the sdk doesn't notify anything about this timeouts. So we can't provide info to the users about the connection status.

[kvasukib]

@pablogeek Thank you for the information. To sum up, the issue is that the AWSS3TransferUtility does not provide information about timeout due to bad network conditions. I have few questions and suggestion before jumping on to a fix.

1) Do you get a error in the completion handler when all the retries are eventually completed? If so, is the information in the completion handler helpful?
2) Have you tried changing the retry configuration?
3) Will it be useful if the SDK notified the completion handler of all the retries?

[pablogeek]

@kvasukib

After a few checks I think it was caused by another problem.

1. Yes I get an error in the completion handler.
2. Changing the retry configuration helped
3. There is no need for that I think.

In our project we have to create differents AWSS3TransferUtility for each upload, because each upload have different configuration + credentials. So if there is a timeout and you retry again you register a new AWSS3TransferUtility.register and it gives you this log A background URLSession with identifier com.amazonaws.AWSS3TransferUtility.Default.Identifier.yourkey already exists!. Then it gets stuck.

The solution was to remove the AWSS3TransferUtility in every retry. I guess we could just call resume() but seems there is an issue which the file seems not completed in S3, but the sdk confirms it has been uploaded

[kvasukib]

@pablogeek

Thank you for the quick response.

1) When you register a AWSS3TransferUtility object again for a new transfer or for retry, you need to use a new key identifier for the AWSS3TransferUtility object. Are you using a new identifier every time you register?

2) When you try to resume the transfer, can you post a code snippet of how you are resuming the transfer? What are the updates you get from the SDK when you resume the transfer?

[pablogeek]

@kvasukib
1 - Yes that's what fixed my problem, use a different key or just remove that before the upload starts
2- We are not doing this yet. I will share you the code I use.

The main issue now is seems the callback doesn't work sometimes when you have a bad connection. I tried the timeouts and it works well, it enters to the callback. But if you are uploading a file, and then you turn off your router, the upload gets stuck, it doesn't enter to the completion callback. Is this expected? We have more scenarios where the completion callback is not called.

[kvasukib]

@pablogeek Thank you for the quick response.

To sum up, you are facing an issue where the completion handler is not getting invoked under certain circumstances. One of them is when the network is turned off when an upload is in progress. Can you describe other scenarios where the completion handler is not invoked?

[pablogeek]

@kvasukib Hi!
We have some random cases where completion handler is not getting called. All of them normally happens with bad internet connection.

[simeondrage]

@kvasukib To expand on what Pablo is describing, what we experience a lot with customers is they attempt an upload (possibly with bad internet connection but not a certainty). the status of our app shows an uploading status but no transfer takes place. The upload doesnt timeout or error. We have to force the app to error by turning wifi off and hard quitting the app. Once the upload has gone into error status, and the WiFi connection is reconnected then most times the upload then commences.

I tested moving a device from bad wifi to good wifi (with a stalled upload) and the upload wouldnt start until I did the steps described.

Hopefully that helps.

[kvasukib]

@simeondrage @pablogeek Thank you for the detailed analysis on this issue. I am looking into reproducing this issue. Either @cbommas or I will follow up with more findings.

[simeondrage]

Thanks. Just to explain its the same issue as https://github.com/aws-amplify/aws-sdk-ios/issues/1065 we just tried to use the new SDK again and are having the same problems

[pablogeek]

Sorry for this big low, we were able to get a stuck upload. We got lots of these logs. Maybe this gives you some clue. Completion block wasn't called

Thanks!

2019-02-19 15:56:22.024948+0000 IrisConnect_BETA[1288:44589] Task <616594A2-290C-486E-9EDD-7ECEF7143F6A>.<219> load failed with error Error Domain=NSURLErrorDomain Code=-999 "(null)" UserInfo={NSURLErrorBackgroundTaskCancelledReasonKey=0, NSErrorFailingURLStringKey=https://eu-hermes-discovery-kit.s3-eu-west-1.amazonaws.com/eu/main/2014_07_14_11_21_27/videos/117546/425029/vdr/1234567890_part_2.mov?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIAXWCSGGUDW7FXSSUY%2F20190219%2Feu-west-1%2Fs3%2Faws4_request&X-Amz-Date=20190219T155434Z&X-Amz-Expires=2999&X-Amz-SignedHeaders=content-type%3Bhost&uploadId=oBdqda77Xkgl4Am.JU86mgNKyAmbPzwKB654UfuJ90zBTsGXscUgVMBchb5Lt.O0Q_0soGbDnPGYvka6CiJ.HWBbamiQXGPwJn0VFEAAj.p9Pi_zouIEDSDUmqhav6RZ6Fy8XHzx.4YogDKNFIFIGQ--&partNumber=1&X-Amz-Security-Token=AgoGb3JpZ2luEFQaCWV1LXdlc3QtMSKAAkyO0PsI65alPCFwIk0FkoIBR6tHDH9u1l6K8ME%2BJO%2Fq2z7PhOSWOXB47tQJL1D40hKyGQW451s6gZ9EWfI81shEfAGfui9OeIzoNGQq5WIaIHRCxoiQaQfAFyP0NBhkb0Vas2x%2FX5TgN8TBxwAsGUuzM26QodUJtljIHG47gj75KTV%2F%2FC%2B6osb5e3iqhz0cjGhmPikmu0LLYCYD00PU6t7nulHRojziiR%2FEbJyDQZmUfFrcbKQEsbR9nh9LTvbR8bd5XVvKPBrvxBxnfHvMgxnc6v9j71locQNISyyiYZvSCbsX2lCdr7qlhTPJBwR9ROnbQ76R8Kmnz8NzGJrMbksqpgMIeRAAGgw1Mjg0NTMzNTA2NjMiDGmCeCgE6ieoHibpvSqDA2FXuNMg9vpXPck4d5gkVYB0jfflNNwpPXrPfl%2F8FW3b5UK9b%2F%2Bn9ISVR991P5nkBwSkLbYj%2FN6DdyL2x9IHW%2BXXqTB5jACNOzypjb65frOUNsxcxovmBxHcA%2B7XdNM%2BwszUAVdAzO%2B2%2F2rYKwRU4W1Gtp4M8EB51507LuR0WEJUMMdJD9UCilvXaWc5pBoRfjB9YuWoxtJ7b6Dxx6XOrbOsW0gMl%2Br8q0UyFYb051lpXdsWZsJcnX5LllO2fHYC0q0y7IDWfREyd%2FPuXc%2BtuJ2%2B2QLbKGLqH%2FkMycBC01Izsbs0ms0FcGhIuqxYw6SrbHAE1PQXA%2Bnp0FnRYCLKi69edSUaC3dCEhMIMi6ZulXF8K%2BpQ3HWGS68ltEIHmDGnerwDHb1sOv6fKkToP27ZxD0zo8aTdr8W4gwXwja%2FpVXVdSgSMdDqNKtaZuZZ3IKfgYa86nYLtbcCQik0yItVLUe6%2BS5LEKrFeehsknzmbK09qKY6Cra8T%2BUI5kr%2BPXYyS5TeTC6zbDjBQ%3D%3D&X-Amz-Signature=7e0b62b82b8268383d5c4fa7821c5cd7decacc37dc40bd077750b9d691b2915b, NSErrorFailingURLKey=https://eu-hermes-discovery-kit.s3-eu-west-1.amazonaws.com/eu/main/2014_07_14_11_21_27/videos/117546/425029/vdr/1234567890_part_2.mov?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIAXWCSGGUDW7FXSSUY%2F20190219%2Feu-west-1%2Fs3%2Faws4_request&X-Amz-Date=20190219T155434Z&X-Amz-Expires=2999&X-Amz-SignedHeaders=content-type%3Bhost&uploadId=oBdqda77Xkgl4Am.JU86mgNKyAmbPzwKB654UfuJ90zBTsGXscUgVMBchb5Lt.O0Q_0soGbDnPGYvka6CiJ.HWBbamiQXGPwJn0VFEAAj.p9Pi_zouIEDSDUmqhav6RZ6Fy8XHzx.4YogDKNFIFIGQ--&partNumber=1&X-Amz-Security-Token=AgoGb3JpZ2luEFQaCWV1LXdlc3QtMSKAAkyO0PsI65alPCFwIk0FkoIBR6tHDH9u1l6K8ME%2BJO%2Fq2z7PhOSWOXB47tQJL1D40hKyGQW451s6gZ9EWfI81shEfAGfui9OeIzoNGQq5WIaIHRCxoiQaQfAFyP0NBhkb0Vas2x%2FX5TgN8TBxwAsGUuzM26QodUJtljIHG47gj75KTV%2F%2FC%2B6osb5e3iqhz0cjGhmPikmu0LLYCYD00PU6t7nulHRojziiR%2FEbJyDQZmUfFrcbKQEsbR9nh9LTvbR8bd5XVvKPBrvxBxnfHvMgxnc6v9j71locQNISyyiYZvSCbsX2lCdr7qlhTPJBwR9ROnbQ76R8Kmnz8NzGJrMbksqpgMIeRAAGgw1Mjg0NTMzNTA2NjMiDGmCeCgE6ieoHibpvSqDA2FXuNMg9vpXPck4d5gkVYB0jfflNNwpPXrPfl%2F8FW3b5UK9b%2F%2Bn9ISVR991P5nkBwSkLbYj%2FN6DdyL2x9IHW%2BXXqTB5jACNOzypjb65frOUNsxcxovmBxHcA%2B7XdNM%2BwszUAVdAzO%2B2%2F2rYKwRU4W1Gtp4M8EB51507LuR0WEJUMMdJD9UCilvXaWc5pBoRfjB9YuWoxtJ7b6Dxx6XOrbOsW0gMl%2Br8q0UyFYb051lpXdsWZsJcnX5LllO2fHYC0q0y7IDWfREyd%2FPuXc%2BtuJ2%2B2QLbKGLqH%2FkMycBC01Izsbs0ms0FcGhIuqxYw6SrbHAE1PQXA%2Bnp0FnRYCLKi69edSUaC3dCEhMIMi6ZulXF8K%2BpQ3HWGS68ltEIHmDGnerwDHb1sOv6fKkToP27ZxD0zo8aTdr8W4gwXwja%2FpVXVdSgSMdDqNKtaZuZZ3IKfgYa86nYLtbcCQik0yItVLUe6%2BS5LEKrFeehsknzmbK09qKY6Cra8T%2BUI5kr%2BPXYyS5TeTC6zbDjBQ%3D%3D&X-Amz-Signature=7e0b62b82b8268383d5c4fa7821c5cd7decacc37dc40bd077750b9d691b2915b, _NSURLErrorRelatedURLSessionTaskErrorKey=(
    "BackgroundUploadTask <616594A2-290C-486E-9EDD-7ECEF7143F6A>.<219>",
    "LocalUploadTask <616594A2-290C-486E-9EDD-7ECEF7143F6A>.<219>"
), _NSURLErrorFailingURLSessionTaskErrorKey=BackgroundUploadTask <616594A2-290C-486E-9EDD-7ECEF7143F6A>.<219>} [-999]
2019-02-19 15:56:22.069174+0000 IrisConnect_BETA[1288:44576] Task <84FD82A7-C219-4802-8CF6-34596343E2EA>.<220> load failed with error Error Domain=NSURLErrorDomain Code=-999 "(null)" UserInfo={NSURLErrorBackgroundTaskCancelledReasonKey=0, NSErrorFailingURLStringKey=https://eu-hermes-discovery-kit.s3-eu-west-1.amazonaws.com/eu/main/2014_07_14_11_21_27/videos/117546/425029/vdr/1234567890_part_2.mov?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIAXWCSGGUDW7FXSSUY%2F20190219%2Feu-west-1%2Fs3%2Faws4_request&X-Amz-Date=20190219T155434Z&X-Amz-Expires=2999&X-Amz-SignedHeaders=content-type%3Bhost&uploadId=oBdqda77Xkgl4Am.JU86mgNKyAmbPzwKB654UfuJ90zBTsGXscUgVMBchb5Lt.O0Q_0soGbDnPGYvka6CiJ.HWBbamiQXGPwJn0VFEAAj.p9Pi_zouIEDSDUmqhav6RZ6Fy8XHzx.4YogDKNFIFIGQ--&partNumber=20&X-Amz-Security-Token=AgoGb3JpZ2luEFQaCWV1LXdlc3QtMSKAAkyO0PsI65alPCFwIk0FkoIBR6tHDH9u1l6K8ME%2BJO%2Fq2z7PhOSWOXB47tQJL1D40hKyGQW451s6gZ9EWfI81shEfAGfui9OeIzoNGQq5WIaIHRCxoiQaQfAFyP0NBhkb0Vas2x%2FX5TgN8TBxwAsGUuzM26QodUJtljIHG47gj75KTV%2F%2FC%2B6osb5e3iqhz0cjGhmPikmu0LLYCYD00PU6t7nulHRojziiR%2FEbJyDQZmUfFrcbKQEsbR9nh9LTvbR8bd5XVvKPBrvxBxnfHvMgxnc6v9j71locQNISyyiYZvSCbsX2lCdr7qlhTPJBwR9ROnbQ76R8Kmnz8NzGJrMbksqpgMIeRAAGgw1Mjg0NTMzNTA2NjMiDGmCeCgE6ieoHibpvSqDA2FXuNMg9vpXPck4d5gkVYB0jfflNNwpPXrPfl%2F8FW3b5UK9b%2F%2Bn9ISVR991P5nkBwSkLbYj%2FN6DdyL2x9IHW%2BXXqTB5jACNOzypjb65frOUNsxcxovmBxHcA%2B7XdNM%2BwszUAVdAzO%2B2%2F2rYKwRU4W1Gtp4M8EB51507LuR0WEJUMMdJD9UCilvXaWc5pBoRfjB9YuWoxtJ7b6Dxx6XOrbOsW0gMl%2Br8q0UyFYb051lpXdsWZsJcnX5LllO2fHYC0q0y7IDWfREyd%2FPuXc%2BtuJ2%2B2QLbKGLqH%2FkMycBC01Izsbs0ms0FcGhIuqxYw6SrbHAE1PQXA%2Bnp0FnRYCLKi69edSUaC3dCEhMIMi6ZulXF8K%2BpQ3HWGS68ltEIHmDGnerwDHb1sOv6fKkToP27ZxD0zo8aTdr8W4gwXwja%2FpVXVdSgSMdDqNKtaZuZZ3IKfgYa86nYLtbcCQik0yItVLUe6%2BS5LEKrFeehsknzmbK09qKY6Cra8T%2BUI5kr%2BPXYyS5TeTC6zbDjBQ%3D%3D&X-Amz-Signature=a5b9ecbbf71ea2f27ecf04516f1c1872de05deb539427bceb55827e7dab29f79, NSErrorFailingURLKey=https://eu-hermes-discovery-kit.s3-eu-west-1.amazonaws.com/eu/main/2014_07_14_11_21_27/videos/117546/425029/vdr/1234567890_part_2.mov?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIAXWCSGGUDW7FXSSUY%2F20190219%2Feu-west-1%2Fs3%2Faws4_request&X-Amz-Date=20190219T155434Z&X-Amz-Expires=2999&X-Amz-SignedHeaders=content-type%3Bhost&uploadId=oBdqda77Xkgl4Am.JU86mgNKyAmbPzwKB654UfuJ90zBTsGXscUgVMBchb5Lt.O0Q_0soGbDnPGYvka6CiJ.HWBbamiQXGPwJn0VFEAAj.p9Pi_zouIEDSDUmqhav6RZ6Fy8XHzx.4YogDKNFIFIGQ--&partNumber=20&X-Amz-Security-Token=AgoGb3JpZ2luEFQaCWV1LXdlc3QtMSKAAkyO0PsI65alPCFwIk0FkoIBR6tHDH9u1l6K8ME%2BJO%2Fq2z7PhOSWOXB47tQJL1D40hKyGQW451s6gZ9EWfI81shEfAGfui9OeIzoNGQq5WIaIHRCxoiQaQfAFyP0NBhkb0Vas2x%2FX5TgN8TBxwAsGUuzM26QodUJtljIHG47gj75KTV%2F%2FC%2B6osb5e3iqhz0cjGhmPikmu0LLYCYD00PU6t7nulHRojziiR%2FEbJyDQZmUfFrcbKQEsbR9nh9LTvbR8bd5XVvKPBrvxBxnfHvMgxnc6v9j71locQNISyyiYZvSCbsX2lCdr7qlhTPJBwR9ROnbQ76R8Kmnz8NzGJrMbksqpgMIeRAAGgw1Mjg0NTMzNTA2NjMiDGmCeCgE6ieoHibpvSqDA2FXuNMg9vpXPck4d5gkVYB0jfflNNwpPXrPfl%2F8FW3b5UK9b%2F%2Bn9ISVR991P5nkBwSkLbYj%2FN6DdyL2x9IHW%2BXXqTB5jACNOzypjb65frOUNsxcxovmBxHcA%2B7XdNM%2BwszUAVdAzO%2B2%2F2rYKwRU4W1Gtp4M8EB51507LuR0WEJUMMdJD9UCilvXaWc5pBoRfjB9YuWoxtJ7b6Dxx6XOrbOsW0gMl%2Br8q0UyFYb051lpXdsWZsJcnX5LllO2fHYC0q0y7IDWfREyd%2FPuXc%2BtuJ2%2B2QLbKGLqH%2FkMycBC01Izsbs0ms0FcGhIuqxYw6SrbHAE1PQXA%2Bnp0FnRYCLKi69edSUaC3dCEhMIMi6ZulXF8K%2BpQ3HWGS68ltEIHmDGnerwDHb1sOv6fKkToP27ZxD0zo8aTdr8W4gwXwja%2FpVXVdSgSMdDqNKtaZuZZ3IKfgYa86nYLtbcCQik0yItVLUe6%2BS5LEKrFeehsknzmbK09qKY6Cra8T%2BUI5kr%2BPXYyS5TeTC6zbDjBQ%3D%3D&X-Amz-Signature=a5b9ecbbf71ea2f27ecf04516f1c1872de05deb539427bceb55827e7dab29f79, _NSURLErrorRelatedURLSessionTaskErrorKey=(
    "BackgroundUploadTask <84FD82A7-C219-4802-8CF6-34596343E2EA>.<220>",
    "LocalUploadTask <84FD82A7-C219-4802-8CF6-34596343E2EA>.<220>"
), _NSURLErrorFailingURLSessionTaskErrorKey=BackgroundUploadTask <84FD82A7-C219-4802-8CF6-34596343E2EA>.<220>} [-999]
2019-02-19 15:56:22.088382+0000 IrisConnect_BETA[1288:44576] Task <CA4AF760-7ED9-4B99-B104-D947D02E3FBA>.<221> load failed with error Error Domain=NSURLErrorDomain Code=-999 "(null)" UserInfo={NSURLErrorBackgroundTaskCancelledReasonKey=0, NSErrorFailingURLStringKey=https://eu-hermes-discovery-kit.s3-eu-west-1.amazonaws.com/eu/main/2014_07_14_11_21_27/videos/117546/425029/vdr/1234567890_part_2.mov?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIAXWCSGGUDW7FXSSUY%2F20190219%2Feu-west-1%2Fs3%2Faws4_request&X-Amz-Date=20190219T155434Z&X-Amz-Expires=2999&X-Amz-SignedHeaders=content-type%3Bhost&uploadId=oBdqda77Xkgl4Am.JU86mgNKyAmbPzwKB654UfuJ90zBTsGXscUgVMBchb5Lt.O0Q_0soGbDnPGYvka6CiJ.HWBbamiQXGPwJn0VFEAAj.p9Pi_zouIEDSDUmqhav6RZ6Fy8XHzx.4YogDKNFIFIGQ--&partNumber=9&X-Amz-Security-Token=AgoGb3JpZ2luEFQaCWV1LXdlc3QtMSKAAkyO0PsI65alPCFwIk0FkoIBR6tHDH9u1l6K8ME%2BJO%2Fq2z7PhOSWOXB47tQJL1D40hKyGQW451s6gZ9EWfI81shEfAGfui9OeIzoNGQq5WIaIHRCxoiQaQfAFyP0NBhkb0Vas2x%2FX5TgN8TBxwAsGUuzM26QodUJtljIHG47gj75KTV%2F%2FC%2B6osb5e3iqhz0cjGhmPikmu0LLYCYD00PU6t7nulHRojziiR%2FEbJyDQZmUfFrcbKQEsbR9nh9LTvbR8bd5XVvKPBrvxBxnfHvMgxnc6v9j71locQNISyyiYZvSCbsX2lCdr7qlhTPJBwR9ROnbQ76R8Kmnz8NzGJrMbksqpgMIeRAAGgw1Mjg0NTMzNTA2NjMiDGmCeCgE6ieoHibpvSqDA2FXuNMg9vpXPck4d5gkVYB0jfflNNwpPXrPfl%2F8FW3b5UK9b%2F%2Bn9ISVR991P5nkBwSkLbYj%2FN6DdyL2x9IHW%2BXXqTB5jACNOzypjb65frOUNsxcxovmBxHcA%2B7XdNM%2BwszUAVdAzO%2B2%2F2rYKwRU4W1Gtp4M8EB51507LuR0WEJUMMdJD9UCilvXaWc5pBoRfjB9YuWoxtJ7b6Dxx6XOrbOsW0gMl%2Br8q0UyFYb051lpXdsWZsJcnX5LllO2fHYC0q0y7IDWfREyd%2FPuXc%2BtuJ2%2B2QLbKGLqH%2FkMycBC01Izsbs0ms0FcGhIuqxYw6SrbHAE1PQXA%2Bnp0FnRYCLKi69edSUaC3dCEhMIMi6ZulXF8K%2BpQ3HWGS68ltEIHmDGnerwDHb1sOv6fKkToP27ZxD0zo8aTdr8W4gwXwja%2FpVXVdSgSMdDqNKtaZuZZ3IKfgYa86nYLtbcCQik0yItVLUe6%2BS5LEKrFeehsknzmbK09qKY6Cra8T%2BUI5kr%2BPXYyS5TeTC6zbDjBQ%3D%3D&X-Amz-Signature=bc4096ac09e5c5c2ade8419a7aff9922882b132adf0c981b687eb6fa0b3c7770, NSErrorFailingURLKey=https://eu-hermes-discovery-kit.s3-eu-west-1.amazonaws.com/eu/main/2014_07_14_11_21_27/videos/117546/425029/vdr/1234567890_part_2.mov?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIAXWCSGGUDW7FXSSUY%2F20190219%2Feu-west-1%2Fs3%2Faws4_request&X-Amz-Date=20190219T155434Z&X-Amz-Expires=2999&X-Amz-SignedHeaders=content-type%3Bhost&uploadId=oBdqda77Xkgl4Am.JU86mgNKyAmbPzwKB654UfuJ90zBTsGXscUgVMBchb5Lt.O0Q_0soGbDnPGYvka6CiJ.HWBbamiQXGPwJn0VFEAAj.p9Pi_zouIEDSDUmqhav6RZ6Fy8XHzx.4YogDKNFIFIGQ--&partNumber=9&X-Amz-Security-Token=AgoGb3JpZ2luEFQaCWV1LXdlc3QtMSKAAkyO0PsI65alPCFwIk0FkoIBR6tHDH9u1l6K8ME%2BJO%2Fq2z7PhOSWOXB47tQJL1D40hKyGQW451s6gZ9EWfI81shEfAGfui9OeIzoNGQq5WIaIHRCxoiQaQfAFyP0NBhkb0Vas2x%2FX5TgN8TBxwAsGUuzM26QodUJtljIHG47gj75KTV%2F%2FC%2B6osb5e3iqhz0cjGhmPikmu0LLYCYD00PU6t7nulHRojziiR%2FEbJyDQZmUfFrcbKQEsbR9nh9LTvbR8bd5XVvKPBrvxBxnfHvMgxnc6v9j71locQNISyyiYZvSCbsX2lCdr7qlhTPJBwR9ROnbQ76R8Kmnz8NzGJrMbksqpgMIeRAAGgw1Mjg0NTMzNTA2NjMiDGmCeCgE6ieoHibpvSqDA2FXuNMg9vpXPck4d5gkVYB0jfflNNwpPXrPfl%2F8FW3b5UK9b%2F%2Bn9ISVR991P5nkBwSkLbYj%2FN6DdyL2x9IHW%2BXXqTB5jACNOzypjb65frOUNsxcxovmBxHcA%2B7XdNM%2BwszUAVdAzO%2B2%2F2rYKwRU4W1Gtp4M8EB51507LuR0WEJUMMdJD9UCilvXaWc5pBoRfjB9YuWoxtJ7b6Dxx6XOrbOsW0gMl%2Br8q0UyFYb051lpXdsWZsJcnX5LllO2fHYC0q0y7IDWfREyd%2FPuXc%2BtuJ2%2B2QLbKGLqH%2FkMycBC01Izsbs0ms0FcGhIuqxYw6SrbHAE1PQXA%2Bnp0FnRYCLKi69edSUaC3dCEhMIMi6ZulXF8K%2BpQ3HWGS68ltEIHmDGnerwDHb1sOv6fKkToP27ZxD0zo8aTdr8W4gwXwja%2FpVXVdSgSMdDqNKtaZuZZ3IKfgYa86nYLtbcCQik0yItVLUe6%2BS5LEKrFeehsknzmbK09qKY6Cra8T%2BUI5kr%2BPXYyS5TeTC6zbDjBQ%3D%3D&X-Amz-Signature=bc4096ac09e5c5c2ade8419a7aff9922882b132adf0c981b687eb6fa0b3c7770, _NSURLErrorRelatedURLSessionTaskErrorKey=(
    "BackgroundUploadTask <CA4AF760-7ED9-4B99-B104-D947D02E3FBA>.<221>",
    "LocalUploadTask <CA4AF760-7ED9-4B99-B104-D947D02E3FBA>.<221>"
), _NSURLErrorFailingURLSessionTaskErrorKey=BackgroundUploadTask <CA4AF760-7ED9-4B99-B104-D947D02E3FBA>.<221>} [-999]

[simeondrage]

@kvasukib @cbommas did you have an update on this issue at all please?

[cbommas]

@simeondrage @pablogeek

Hey guys, sorry for the delay in my response. I ran a bunch of experiments on my side to reproduce the issue you are seeing and had partial success. These were done using a test app, the network link conditioner, the latest version of the SDk, timeoutIntervalForResource set to 60 seconds, and on both simulator and physical device. Here are my findings

• The uploads would start at an arbitrary time after it has been initiated.
• In all cases, the upload eventually started when the device transitioned to a good network.
• I simulated the network going bad during the transfer by enabling the network link conditioner. In this case, the upload would stall and make little to no progress ( based on the network link conditioner profile chosen). When the network was in a bad state, I wouldn't get the callback, even if the timeoutIntervalForResource was exceeded. _It is likely that this is what is happening when you are seeing the transfer appearing to be stuck and the callback not being invoked_
• However, once the device was back on a good network, the transfer would continue. It would finish successfully if the timeoutIntervalForResource was not exceeded. Otherwise, it would finish with an error. But in all of my tests, bar none, the callback was eventually invoked.

Regarding the question about uploads not starting, (#1065) I have observed at various times that the upload does not start immediately - I believe this is fundamental to how NS Background URL sessions work. My understanding is that when an transfer is requested in a Background URL session, it will be _eventually_ completed by the OS at an optimal time for the device based on network, battery and other conditions. This scheduling may be further impacted by your app architecture where each transfer is started in a new TransferUtility ( which internally will create a new Background session).

I will be happy to jump on a screenshare/call with you guys to dive further into this. If you are open to this, please send me your contact info and times that work for you ( my email is chabom at amazon.com)

[simeondrage]

@cbommas we have finally got a device back from a customer with the issue and also another device from one of my colleagues. I am sending the devices to Pablo to investigate and then i'll get him to update this ticket. It will probably be Friday/early next week

Are there any plans to improve the callback on a bad network?

[cbommas]

@simeondrage

I will look into see what options we have to improve the callback behavior on bad networks. As it stands, this behavior is largely driven by the underlying NSURLSession, but I will continue to look into it.

[kvasukib]

@pablogeek @simeondrage Are you still facing this issue?

[pablogeek]

@kvasukib yes we are still having this. I can see errors in the logs sometimes, due timeouts, no space, etc and the callback is not called

[inzinga]

Was wondering if there was a resolution to this issue? I'm seeing it randomly with latest iOS AWS SDK. Similar symptoms that others note above. Only occurs randomly.

[stevensanborn]

just turn off wifi, and try and make a transfer download and completion hander never fires with an error. Yet outputs with :
Task ...<1> finished with error - code: -1009
load failed with error Error Domain=NSURLErrorDomain Code=-1009 "The Internet connection appears to be offline." UserInfo={_kCFStreamErrorCodeKey=50, NSUnderlyingError=0x2830efa80 {Error Domain=kCFErrorDomainCFNetwork Code=-1009 "(null)" UserInfo={_kCFStreamErrorCodeKey=50, _kCFStreamErrorDomainKey=1}}, _NSURLErrorFailingURLSessionTaskErrorKey=.....<1>, _NSURLErrorRelatedURLSessionTaskErrorKey=( "LocalDataTask " ), NSLocalizedDescription=The Internet connection appears to be offline., NSErrorFailingURLStringKey=https://cognito-identity.us-east-1.amazonaws.com/, NSErrorFailingURLKey=https://cognito-identity.us-east-1.amazonaws.com/, _kCFStreamErrorDomainKey=1} [-1009] 2019-08-28 11:08:28.203608-1000 Lerner[846:76597] Could not signal service com.apple.WebKit.Networking: 113: Could not find specified service

[lawmicha]

hi @stevensanborn here's what I found regarding "turn off wifi, try to make transfer download and completion handler never fire with an error"

Setup

• I am running the sample app https://github.com/awslabs/aws-sdk-ios-samples/tree/master/S3TransferUtility-Sample/Swift
• added extra code for uploadDataUsingMultiPart, and AWSS3TransferUtilityConfiguration.timeoutIntervalForResource set to 30 seconds.
• launched the app on my device, turned off wifi and cellular, adding some breakpoints in https://github.com/aws-amplify/aws-sdk-ios/blob/master/AWSS3/AWSS3TransferUtility.m

1. downloadData
   Result: I noticed that the download task is created successfully and I wait until the 30 seconds is up until the error is sent back through the completionHandler as "The request timed out".

2. uploadData
   Result: Same case as downloadData

3. uploadDataUsingMultiPart
   Result: I was able to reproduce that the completionHandler is not called. The task is immediately returned with an error in a fail fast approach with the error "The Internet connection appears to be offline" (reference where it is returned: https://github.com/aws-amplify/aws-sdk-ios/blob/master/AWSS3/AWSS3TransferUtility.m#L1286-L1291 )
   I would say this is the right behavior, even though it's not the same behavior as in case 1 and 2. The error after creating the task explains why it failed to create the task and no work was done and the completionHander's error should explain what failures occurred given that it actually started performing the work.

4. Turn wifi on, uploadDataUsingMultiPart(), turn off wifi while it is in progress to create timeout. wait 30 seconds for timeout.
   Result: The callback does get called with an error for me (in this path https://github.com/aws-amplify/aws-sdk-ios/blob/master/AWSS3/AWSS3TransferUtility.m#L2157-L2166). However it looks the error object gets allocated with some existing error before returning it in the completionHandler. When i print out the localized error, it is "Error Domain=NSURLErrorDomain Code=-1001 "(null)"

@simeondrage @pablogeek can you provide some more specifics to the scenario you are facing issues with the most? are you using multipart upload?

timeoutIntervalForResource is set to 50 minutes by default, could this be the issue for not seeing a timeout, thus not seeing completionHandler get called for some cases? If timeout is being called, the transferUtility code implements NSURLSessionTaskDelegate's didCompleteWithError(), so there could be a bug in there that it's not calling the completionHandler.

[pablogeek]

@lawmicha thanks for the details. I'm also using configuration?.timeoutIntervalForResource = 30
I use multipart upload because we normally upload big files. I can easily reproduce with multipart uploads but no with non multipart.

[lawmicha]

@pablogeek can you reiterate on the repro steps again?

version of SDK?
how large is a typical file size you are testing with?
on device/simulator?
actions you took? used network conditioning at which step of the process?
and any logs and code snippet will be helpful

[stevensanborn]

@lawmicha

Yes looking back at the samepl you are correct. When i formatted the download request just like the sample i did get a proper completion handler call back . I believe i originally had the code:

 transferUtility.download(to: urlTemp,
                                 bucket: self.bucket, key:"media/json/sitemap.xml",
                                 expression:expression,
                                 completionHandler: { (task, location, data, error) -> Void in

and then i converted to:

``` let completionHandler:AWSS3TransferUtilityDownloadCompletionHandlerBlock? = { (task, location, data, error) -> Void in
DispatchQueue.main.async(execute: {
if let error = error {
NSLog("Caught with error: (error)")

            }

        })
    }

    transferUtility.download(to: urlTemp,
                             bucket: self.bucket, key:"media/json/sitemap.xml",
                             expression:expression,
                             completionHandler:completionHandler).continueWith { (task) -> AnyObject? in
                                print("COMPLETED")
                                    if let error = task.error {
                                        print("caught error: \(error.localizedDescription)")```

which worked . TY

[lawmicha]

I can easily reproduce with multipart uploads but no with non multipart.

@pablogeek this issue may have some out dated repro steps since it's been so long, can you provide the latest more details on the scenario you are doing? This will help me focus on the issue that you are seeing exactly because right now I would have to try and identity variations of the usability for multipart upload hoping to get a repro.

Another suggestion is we can set up some sort of shared sample app that shows the issue when applying certain actions. I've been making local changes to https://github.com/awslabs/aws-sdk-ios-samples/tree/master/S3TransferUtility-Sample/Swift to try and reproduce. if you can reproduce on the sample app and push the code to your own branch, I can continue to look at it and debug through the transfer utility code.

[stale[bot]]

This issue has been automatically closed because of inactivity. Please open a new issue if are still encountering problems.

[lawmicha]

The latest correspondence here is about multipart upload completion handler not being called.

On resumable network errors, transfer utility is creating a new sessionTask but is not resuming it, the fix here basically adds a call to resume it https://github.com/aws-amplify/aws-sdk-ios/pull/1944