Sdwebimage: Crash on SDWebImageDownloaderOperation.m line 407

[iamZoltanVaradi]

Crashed: NSOperationQueue 0x156c6b7d0 :: NSOperation 0x15cc7c2e0 (QOS: LEGACY)
0 CFNetwork 0x184743bbc -[NSCachedURLResponse _reestablishInternalCFCachedURLResponse:] + 48
1 CFNetwork 0x1847449b0 -[NSURLCache cachedResponseForRequest:] + 144
2 SDWebImage 0x101edbe88 -SDWebImageDownloaderOperation URLSession:task:didCompleteWithError:
3 SDWebImage 0x101ed9c10 -SDWebImageDownloader URLSession:task:didCompleteWithError:
4 CFNetwork 0x18473c930 __51-[NSURLSession delegate_task:didCompleteWithError:]_block_invoke170 + 72
5 Foundation 0x1849e8540 __NSBLOCKOPERATION_IS_CALLING_OUT_TO_A_BLOCK__ + 16
6 Foundation 0x18493a870 -[NSBlockOperation main] + 96
7 Foundation 0x18492ae48 -[__NSOperationInternal _start:] + 604
8 Foundation 0x1849ea934 __NSOQSchedule_f + 224
9 libdispatch.dylib 0x183a7947c _dispatch_client_callout + 16
10 libdispatch.dylib 0x183a854c0 _dispatch_queue_drain + 864
11 libdispatch.dylib 0x183a7cf80 _dispatch_queue_invoke + 464
12 libdispatch.dylib 0x183a87390 _dispatch_root_queue_drain + 728
13 libdispatch.dylib 0x183a870b0 _dispatch_worker_thread3 + 112
14 libsystem_pthread.dylib 0x183c91470 _pthread_wqthread + 1092
15 libsystem_pthread.dylib 0x183c91020 start_wqthread + 4

Issue Info

| Info | Value |
| --- | --- |
| Platform Name | ios |
| Platform Version | 9.0 |
| SDWebImage Version | 3.8.1 |
| Integration Method | cocoapods |
| Xcode Version | Xcode 7.3.1 |

[yshrkt]

this issue is happening on my app too.

Crashed: NSOperationQueue 0x17d8b8e0 :: NSOperation 0x17dcc990 (QOS: LEGACY)
0  libobjc.A.dylib                0x24bbba86 objc_msgSend + 5
1  CFNetwork                      0x259e58e7 -[NSURLCache cachedResponseForRequest:] + 114
2  SDWebImage                     0x10538b5 -[SDWebImageDownloaderOperation URLSession:task:didCompleteWithError:] + 460
3  SDWebImage                     0x1051913 -[SDWebImageDownloader URLSession:task:didCompleteWithError:] + 92
4  CFNetwork                      0x259de6fd __51-[NSURLSession delegate_task:didCompleteWithError:]_block_invoke170 + 76
5  Foundation                     0x25c0684d __NSBLOCKOPERATION_IS_CALLING_OUT_TO_A_BLOCK__ + 8
6  Foundation                     0x25b6787f -[NSBlockOperation main] + 146
7  Foundation                     0x25b59f8f -[__NSOperationInternal _start:] + 774
8  Foundation                     0x25c08b0d __NSOQSchedule_f + 192
9  libdispatch.dylib              0x24f8de7f _dispatch_queue_drain + 1762
10 libdispatch.dylib              0x24f86e17 _dispatch_queue_invoke + 282
11 libdispatch.dylib              0x24f8f20d _dispatch_root_queue_drain + 400
12 libdispatch.dylib              0x24f8f07b _dispatch_worker_thread3 + 94
13 libsystem_pthread.dylib        0x25122e0d _pthread_wqthread + 1024
14 libsystem_pthread.dylib        0x251229fc start_wqthread + 8

iPhone 5c / iOS 9.3.2 / SDWebImage 3.8.1 / Xcode 7.3.1

[zaczh]

We encountered same crash.
crash_log.txt

[GBoyadziev]

I am encountering the same problem in one of my projects and it is reproducing quite often.

[worthbak]

Seeing a similar crash in my app - can't speak the severity, as only a small portion of our user base is using SDWebImage v3.8.1. Here's a sample stack trace:

Crashed: NSOperationQueue 0x15785ce90 :: NSOperation 0x15a9c13b0 (QOS: LEGACY)
0  libobjc.A.dylib                0x182349b9c objc_msgSend + 28
1  CFNetwork                      0x1833e899c -[NSURLCache cachedResponseForRequest:] + 124
2  SDWebImage                     0x101753e88 -[SDWebImageDownloaderOperation URLSession:task:didCompleteWithError:] + 428
3  SDWebImage                     0x101751c10 -[SDWebImageDownloader URLSession:task:didCompleteWithError:] + 128
4  CFNetwork                      0x1833e0930 __51-[NSURLSession delegate_task:didCompleteWithError:]_block_invoke170 + 72
5  Foundation                     0x18368c540 __NSBLOCKOPERATION_IS_CALLING_OUT_TO_A_BLOCK__ + 16
6  Foundation                     0x1835de870 -[NSBlockOperation main] + 96
7  Foundation                     0x1835cee48 -[__NSOperationInternal _start:] + 604
8  Foundation                     0x18368e934 __NSOQSchedule_f + 224
9  libdispatch.dylib              0x18271d47c _dispatch_client_callout + 16
10 libdispatch.dylib              0x1827294c0 _dispatch_queue_drain + 864
11 libdispatch.dylib              0x182720f80 _dispatch_queue_invoke + 464
12 libdispatch.dylib              0x18272b390 _dispatch_root_queue_drain + 728
13 libdispatch.dylib              0x18272b0b0 _dispatch_worker_thread3 + 112
14 libsystem_pthread.dylib        0x182935470 _pthread_wqthread + 1092
15 libsystem_pthread.dylib        0x182935020 start_wqthread + 4

[bestwnh]

I don't know is this related issue, but more information always helpful, right? and seem the v3.8.1 actually have this issue.
https://github.com/pinterest/PINRemoteImage/issues/92#issuecomment-163073075
a fix for the issue I mention is easy, but the bug is hard to reproduce by myself.(But it does reproduce again and again by user 😵 )

[anafinotti]

Anyone had lucky?

[Zigzag968]

same problem here, with 3.8.1 version

[nearllopez]

Hi guys,

Same problem (3.8.1). Any updates about it?

Thanks!

[bpoplauschi]

I will take a proper look at this issue this weekend.

[bpoplauschi]

@bestwnh I looked at the PINRemoteImage issue you linked, but their crash log looks very different.
theirs:

0   libdispatch.dylib  0x0000000199e22b48 dispatch_async + 76
1   PINRemoteImage     0x0000000100952904 -[PINURLSessionManager URLSession:task:didReceiveChallenge:completionHandler:] (PINURLSessionManager.m:79)

ours:

0  libobjc.A.dylib 0x2393fa66 objc_msgSend + 5
1  CFNetwork       0x2478877f -[NSURLCache cachedResponseForRequest:] + 114
2  SDWebImage      0x23bd795 -[SDWebImageDownloaderOperation URLSession:task:didCompleteWithError:] (SDWebImageDownloaderOperation.m:407)

Thanks anyway, it might have been of help. I have looked at the crash for a while, but can't figure out the cause.

[bestwnh]

Sorry I post the wrong information. Just want to make some help for this weird issue. At least you get rid of one possibility, right? Let's stay positive. 😅

[bpoplauschi]

Guys, I'm in a bit of dead-end here. I need help. I've added 1 commit on master with a small change, could you try it and see if the issue still occurs? Also, if anyone can reproduce the issue constantly, could you create a small demo project so I can debug it properly? (maybe @GBoyadziev you said it happens quite often)

_Note_: I was able to simulate the same crashlog by using a deallocated instance of NSURLRequest passed to [NSURLCache cachedResponseForRequest:], but I forced that. Not able to properly reproduce with the current code.

[iamqk]

same crash here

SIGSEGV
SEGV_ACCERR
0 libobjc.A.dylib 0x0000000180865b90 objc_msgSend + 16
1 CFNetwork 0x000000018190499c -[NSURLCache cachedResponseForRequest:] + 124
2 MicroRun 0x0000000100683a38 -[SDWebImageDownloaderOperation URLSession:task:didCompleteWithError:] SDWebImageDownloaderOperation.m:407

[bpoplauschi]

I have made an attempt fix in e7bd5ab: Apparently there is a race condition on NSURLCache and we avoid making those checks unless necessary (basically we will query the NSURLCache only when SDWebImageRefreshCached is used and the image cannot be cached by the system when it's too big or behind authentication). Will release a new version soon.

[bpoplauschi]

Guys, 3.8.2 is now available via CocoaPods, please try it out and let me know if the issue still happens or not.

[smogun]

Thanks. Will test it. However this issue very hard to reproduce.

I also did some investigation and found that there are different NSOperation objects being used by SDWebImageManager and SDWebImageDownloader. The Manager uses NSOperation while Downloader uses sublcass of it and it has implementation very similar to a workaround suggested here: http://stackoverflow.com/questions/9409994/cancelling-nsoperation-from-nsoperationqueue-cause-crash

I'm using Manager object and I have crashes there. So I think this might be the problem.

[luohui8891]

Hey guys, does any one confirm the 3.8.2 version about the fix?

[zheck]

Just deployed with 3.8.2, still got the same error:

Crashed: NSOperationQueue 0x1706217a0 :: NSOperation 0x174654460 (QOS: DEFAULT)
0  libobjc.A.dylib                0x193082f30 objc_msgSend + 16
1  SDWebImage                     0x1011ebdd0 -[SDWebImageDownloaderOperation URLSession:task:didCompleteWithError:] (SDWebImageDownloaderOperation.m:405)

[bpoplauschi]

@zheck does it have the same frequency? We did not fix the issue, but try to reduce the calls to [[NSURLCache sharedURLCache] cachedResponseForRequest:], since we considered that is the cause (a race condition on NSURLCache). Right now it should be called only if SDWebImageRefreshCached is explicitly sent as option and the image cannot be cached (since it's too big or requires authentication). Can you confirm any of this?

[zheck]

@bpoplauschi unfortunately, we are using SDWebImage without any option, only simple calls: sd_setImageWithURL() and sd_cancelCurrentImageLoad()

Since the issue is very hard to reproduce, we are removing the completionBlock for production, and continue to investigate in dev/pre-prod

Will post if we find any ways to reproduce the issue (network condition, activity etc.)

[zaczh]

Hey guys. I think the problem is that when calling [[NSURLCache sharedURLCache] cachedResponseForRequest:, the iOS system will do some cache merging. See the last line of the crash log:
-[NSCachedURLResponse _reestablishInternalCFCachedURLResponse:]
This implies that every time you call this method, the response caches will be reestablished. I think this is the cause of our problem. If we are caching multiple responses on different threads, then it may cause a crash.

After digging into the code, I found that this call is reductant. If we specify to use system cache, then the system will do it correctly, which means what you get in - (void)URLSession:(NSURLSession *)session task:(NSURLSessionTask *)task didCompleteWithError:(NSError *)error is already handled. If you specified to only use cached response, and there is no local data available, then you will get nil.

Business level cache is always a bad idea, which we should avoid.

I have submit a patch to workaround this. Any idea on it is welcome.

This is the patch

[bpoplauschi]

Thanks to @zaczh, this issue will entirely go away in version 4.1.0.

[jz709u]

when is version 4.1.0 going public

[dreampiggy]

OK...I found the reason. The thing Apple docs says is correct:

In iOS 8 and later, and macOS 10.10 and later, NSURLCache is thread safe.
Although NSURLCache instance methods can safely be called from multiple execution contexts at the same time, be aware that methods like cachedResponseForRequest: and storeCachedResponse:forRequest: have an unavoidable race condition when attempting to read or write responses for the same request.

I try to decompile the CFNetwork framework(NSURLCache class is placed in CFNetwork but not Foundation). But it looks seems safe except that the value stored in r14 from _CFURLCacheCopyResponseForRequest. If this address becomes a dangling pointer, then the crash will appear.

If you try to call this method in URLSessionDelegateQueue while downloading, you may face the race condition while URLSession internal use another queue (not your delegate queue, sadly) to save the cache. It will crash because that _internal address may be released.

So, the best way to avoid this crash it's NOT to call that two method during URLSessionDelegate method. We need another approach to obey current behavior. Or I think, we can just remove that code and let SDWebImageRefreshCached call twice completion block with image whenever it hit the cache or not because it's more logical (I don't know why we need this twice call, maybe NSURLConnection historical reason). This change will also keep HTTP cache control according to the Apple's doc [NSURLSession's cache policy]. 🤔

[dreampiggy]

The decompiled pseudo code is here:

void * -[NSURLCache cachedResponseForRequest:](void * self, void * _cmd, void * arg2) {
    rdx = arg2;
    rdi = self;
    if (rdx != 0x0) {
            rax = rdi->_internal;
            if (rax != 0x0) {
                    r14 = _CFURLCacheCopyResponseForRequest(rax->_cacheRef, [rdx _CFURLRequest], rdx, *_OBJC_IVAR_$_NSURLCacheInternal._cacheRef);
                    if (r14 != 0x0) {
                            rbx = _CFCachedURLResponseGetNSCachedURLResponse(r14);
                            if (rbx != 0x0) {
                                    rax = [rbx retain]; // dead here!
                                    [rax autorelease]; // here
                                    [rbx _reestablishInternalCFCachedURLResponse:r14]; // and here
                            }
                            else {
                                    rax = objc_getClass("NSCachedURLResponse");
                                    if (rax == 0x0) {
                                            rax = _faultInFoundation("NSCachedURLResponse");
                                    }
                                    rbx = [[[rax alloc] _initWithCFCachedURLResponse:r14] autorelease];
                            }
                            CFRelease(r14);
                    }
                    else {
                            rbx = 0x0;
                    }
            }
            else {
                    rbx = 0x0;
            }
    }
    else {
            rbx = 0x0;
            _NSLog(@"called cachedResponseForRequest: with nil request");
    }
    rax = rbx;
    return rax;
}

void -[NSCachedURLResponse _reestablishInternalCFCachedURLResponse:](void * self, void * _cmd, struct _CFCachedURLResponse * arg2) {
    r14 = arg2;
    rbx = self;
    rax = rbx->_internal;
    if ((rax != 0x0) && (rax->_cachedURLResponse == 0x0)) {
            if (r14 != 0x0) {
                    CFRetain(r14);
                    rax = rbx->_internal;
            }
            rax->_cachedURLResponse = r14;
    }
    return;
}

[bpoplauschi]

2032 and #1994 should take care of this crash here, even though there are some chances it will bring it back :) let's keep an eye open.

[bpoplauschi]

4.1.1 is now released.

[dreampiggy]

Well...Thing is not so simple. Some user point that there are still crash in cachedResponseForRequest: #2156 . So I try to figure out the actual reason. I find that this should be safe accroding to the decompiled code above. But the crash log seems only about iOS 9. So I guess this crash is the iOS version realated. I try to compare the CFNetwork decompiled code from iOS 9 & iOS 10 & iOS 11, the different show the reason:

iOS 11:

int _CFCachedURLResponseGetNSCachedURLResponse(int arg0) {
    rdi = arg0;
    if (rdi != 0x0) {
            rbx = *(rdi + 0x10);
            _os_nospin_lock_lock(rbx + 0x50); // Good, a lock with OSSpinLock
            rbx = [[*(rbx + 0x60) retain] autorelease]; // grab the ivar and autorelease
            (*__NtlmCreateClientResponse)(__NSConcreteStackBlock); // seems a NTLM HTTP Auth support ? but this is not related to this issue
    }
    else {
            rbx = 0x0;
    }
    rax = rbx;
    return rax;
}

iOS 9 & iOS 10:

int _CFCachedURLResponseGetNSCachedURLResponse(int arg0) {
    rdi = arg0;
    rax = 0x0;
    if (rdi != 0x0) {
            rax = *(rdi + 0x10); // grab the ivar without lock
            rax = *(rax + 0x60);
    }
    return rax;
}

It seems that NSCachedURLResponse is created using CFCachedURLResponseRef , the collee code is nearly the same in both iOS version, use a global _CFURLCacheCopyResponseForRequest to grab the CFCachedURLResponse, and the crash stacktrace seems never show this, so it's safe.

In iOS 11, there is a global lock _os_nospin_lock_lock call, to ensure that the create process is thread-safe. However, in iOS 9 & 10, there contains nothing lock, just grab the ivar and return back to the callee. So if different queue call on a same NSURLCache instance with the same request(For example, call same URL on the shared URLCache), this ivar will be in a really danger situation because the retain and autorelease call is not protected by the lock.

So what can we do ? To say, our SDWebImageDownloader design can ensure the same URL will not be downloaded twice. But we can not to protect this situation at all. For example, if the user outside code call the cachedResponseForRequest: on shared cache with the same URL(may be not the direct call, load UIWebView page with the same URL may also trigger this). We can not do anything lock inside our framework to protect this.