Firebase-ios-sdk: GULNetworkURLSession crash

[dangthaison91]

Environment

Xcode version: 10.2
Firebase/Google Pod Instal log:

Using Firebase (6.2.0)
Using FirebaseABTesting (3.0.0)
Using FirebaseAnalytics (6.0.1)
Using FirebaseAnalyticsInterop (1.2.0)
Using FirebaseCore (6.0.2)
Using FirebaseDynamicLinks (4.0.0)
Using FirebaseInstanceID (4.1.1)
Using FirebaseMLCommon (0.16.0)
Using FirebaseMLVision (0.16.0)
Using FirebaseMLVisionBarcodeModel (0.16.0)
Using FirebasePerformance (3.0.0)
Using FirebaseRemoteConfig (4.0.0)

Using GTMSessionFetcher (1.2.2)
Using GoogleAPIClientForREST (1.3.9)
Using GoogleAnalytics (3.17.0)
Using GoogleAppMeasurement (6.0.1)
Using GoogleMobileVision (1.6.0)
Using GoogleSymbolUtilities (1.1.2)
Using GoogleTagManager (7.1.2)
Using GoogleToolboxForMac (2.2.1)
Using GoogleUtilities (6.2.0)
Using GoogleUtilitiesLegacy (1.3.2)

Problem

App crashes in GULNetworkURLSession, line 172. Code:

Steps to reproduce

We cannot reproduce it but about 8k users are affected.

Crash log from Firebase Crashlytics:

Fatal Exception: NSInvalidArgumentException
0  CoreFoundation                 0x181a75d04 __exceptionPreprocess
1  libobjc.A.dylib                0x180cc4528 objc_exception_throw
2  CoreFoundation                 0x181a831c8 __methodDescriptionForSelector
3  CoreFoundation                 0x181a7b6b0 ___forwarding___
4  CoreFoundation                 0x18196101c _CF_forwarding_prep_0
5  libobjc.A.dylib                0x180cc543c CALLING_SOME_+initialize_METHOD
6  libobjc.A.dylib                0x180cc5804 _class_initialize
7  libobjc.A.dylib                0x180ce0c34 objc_loadWeakRetained
8  libobjc.A.dylib                0x180ce0d3c objc_loadWeak
9  Foundation                     0x18239daa4 readARCWeakAt
10 Foundation                     0x1823854dc -[NSConcreteMapTable grow]
11 Foundation                     0x182365bc0 -[NSConcreteMapTable setObject:forKey:]
12 GoogleUtilities                0x101fa199c +[GULNetworkURLSession setSessionInFetcherMap:forSessionID:] (GULNetworkURLSession.m:699)
13 GoogleUtilities                0x101f9f688 -[GULNetworkURLSession sessionIDFromAsyncPOSTRequest:completionHandler:] (GULNetworkURLSession.m:172)
14 GoogleUtilities                0x101f9d744 -[GULNetwork postURL:payload:queue:usingBackgroundSession:completionHandler:] (GULNetwork.m:143)
15 XXXXX                       0x10418a668 -[FIRClearcutLogger sendNextPendingRequestWithCompletionHandler:]
16 XXXXX                       0x10418a8d0 __65-[FIRClearcutLogger sendNextPendingRequestWithCompletionHandler:]_block_invoke
17 libdispatch.dylib              0x1813f9088 _dispatch_call_block_and_release
18 libdispatch.dylib              0x1813f9048 _dispatch_client_callout
19 libdispatch.dylib              0x181406378 _dispatch_root_queue_drain
20 libdispatch.dylib              0x181405f10 _dispatch_worker_thread3
21 libsystem_pthread.dylib        0x18169f120 _pthread_wqthread
22 libsystem_pthread.dylib        0x18169ec20 start_wqthread

0  JavaScriptCore                 0x1ba9b1160 Inspector::JSGlobalObjectInspectorController::JSGlobalObjectInspectorController(JSC::JSGlobalObject&) + 906
1  JavaScriptCore                 0x1bac53af8 JSC::JSGlobalObject::init(JSC::VM&) + 268
2  JavaScriptCore                 0x1bac6b3c4 JSC::JSGlobalObject::finishCreation(JSC::VM&) + 104
3  JavaScriptCore                 0x1bac6b320 JSC::JSGlobalObject::create(JSC::VM&, JSC::Structure*) + 160
4  JavaScriptCore                 0x1ba4e93f4 JSGlobalContextCreateInGroup + 316
5  JavaScriptCore                 0x1ba2a05e8 -[JSContext initWithVirtualMachine:] + 108
6  VinID                          0x100eb2728 -[TAGRuntime initWithContainerId:resource:hitStore:dispatchScheduler:] + 4311426856
7  VinID                          0x100ea8d38 __55-[TAGContainer handleLoadOfContainerData:errorMessage:]_block_invoke + 4311387448
8  libdispatch.dylib              0x1b2bd0a38 _dispatch_call_block_and_release + 24
9  libdispatch.dylib              0x1b2bd17d4 _dispatch_client_callout + 16
10 libdispatch.dylib              0x1b2b7a320 _dispatch_lane_serial_drain$VARIANT$mp + 592
11 libdispatch.dylib              0x1b2b7ae3c _dispatch_lane_invoke$VARIANT$mp + 428
12 libdispatch.dylib              0x1b2b834a8 _dispatch_workloop_worker_thread + 596
13 libsystem_pthread.dylib        0x1b2db1114 _pthread_wqthread + 304
14 libsystem_pthread.dylib        0x1b2db3cd4 start_wqthread + 4


GULMutableDictionary
0  libsystem_kernel.dylib         0x1b2d2e9d4 __ulock_wait + 8
1  libsystem_platform.dylib       0x1b2d9f348 _os_unfair_lock_lock_slow + 220
2  libobjc.A.dylib                0x1b2384604 objc_destroyWeak + 144
3  libobjc.A.dylib                0x1b236a7cc object_cxxDestructFromClass(objc_object*, objc_class*) + 148
4  libobjc.A.dylib                0x1b237a6b8 objc_destructInstance + 68
5  libobjc.A.dylib                0x1b237a720 object_dispose + 16
6  GoogleUtilities                0x1018aec4c -[GULObjectSwizzler dealloc] (GULObjectSwizzler.m:155)
7  libobjc.A.dylib                0x1b2380754 _object_remove_assocations + 468
8  libobjc.A.dylib                0x1b237a6d4 objc_destructInstance + 96
9  libobjc.A.dylib                0x1b237a720 object_dispose + 16
10 libdispatch.dylib              0x1b2bd0a38 _dispatch_call_block_and_release + 24
11 libdispatch.dylib              0x1b2bd17d4 _dispatch_client_callout + 16
12 libdispatch.dylib              0x1b2b7a320 _dispatch_lane_serial_drain$VARIANT$mp + 592
13 libdispatch.dylib              0x1b2b7ae3c _dispatch_lane_invoke$VARIANT$mp + 428
14 libdispatch.dylib              0x1b2b834a8 _dispatch_workloop_worker_thread + 596
15 libsystem_pthread.dylib        0x1b2db1114 _pthread_wqthread + 304
16 libsystem_pthread.dylib        0x1b2db3cd4 start_wqthread + 4

[google-oss-bot]

I found a few problems with this issue:

• I couldn't figure out how to label this issue, so I've labeled it for a human to triage. Hang tight.
• This issue does not seem to follow the issue template. Make sure you provide all the required information.

[maksymmalyhin]

@dangthaison91 Thank you for the issue report. Would you be able to provide additional details:

1. What Cocoapods version do you use?
2. Which flags do you use in your Podfile, e.g. use_frameworks!, generate_multiple_pod_projects, etc. Ideally, the Podfile itself would help.
3. What are iOS versions for the crash (is it a specific iOS version)
4. Does the crash happen on a specific device?
5. Is there any Objective-C runtime logic in your app you are aware of? E.g. NSURLSession/NSURLSessionDelegate method swizzling, etc.

[dangthaison91]

Hi @maksymmalyhin This bug is having Velocity Alert on our Firebase Crashlytics

1. Cocoapods version: We are using Cocoapods 1.7.1 with use_frameworks!, nothing special.

1. What are iOS versions for the crash (is it a specific iOS version)

Crash appears on every iOS version we support from iOS 10 - 12. But 86% on iOS 11 (reported by Firebase Crashlytics)

1. Does the crash happen on a specific device?

No specific device

1. Is there any Objective-C runtime logic in your app you are aware of?

AFAIK, There are no swizzle in our app.

FYI, We have 30-50 crash on the earlier version used Swift 4.2 but for the current app version using Swift 5.0 we get 14K crash logs.

[dangthaison91]

Added more crash logs that may be help to trace this bug:

0  JavaScriptCore                 0x1ba9b1160 Inspector::JSGlobalObjectInspectorController::JSGlobalObjectInspectorController(JSC::JSGlobalObject&) + 906
1  JavaScriptCore                 0x1bac53af8 JSC::JSGlobalObject::init(JSC::VM&) + 268
2  JavaScriptCore                 0x1bac6b3c4 JSC::JSGlobalObject::finishCreation(JSC::VM&) + 104
3  JavaScriptCore                 0x1bac6b320 JSC::JSGlobalObject::create(JSC::VM&, JSC::Structure*) + 160
4  JavaScriptCore                 0x1ba4e93f4 JSGlobalContextCreateInGroup + 316
5  JavaScriptCore                 0x1ba2a05e8 -[JSContext initWithVirtualMachine:] + 108
6  VinID                          0x100eb2728 -[TAGRuntime initWithContainerId:resource:hitStore:dispatchScheduler:] + 4311426856
7  VinID                          0x100ea8d38 __55-[TAGContainer handleLoadOfContainerData:errorMessage:]_block_invoke + 4311387448
8  libdispatch.dylib              0x1b2bd0a38 _dispatch_call_block_and_release + 24
9  libdispatch.dylib              0x1b2bd17d4 _dispatch_client_callout + 16
10 libdispatch.dylib              0x1b2b7a320 _dispatch_lane_serial_drain$VARIANT$mp + 592
11 libdispatch.dylib              0x1b2b7ae3c _dispatch_lane_invoke$VARIANT$mp + 428
12 libdispatch.dylib              0x1b2b834a8 _dispatch_workloop_worker_thread + 596
13 libsystem_pthread.dylib        0x1b2db1114 _pthread_wqthread + 304
14 libsystem_pthread.dylib        0x1b2db3cd4 start_wqthread + 4


GULMutableDictionary
0  libsystem_kernel.dylib         0x1b2d2e9d4 __ulock_wait + 8
1  libsystem_platform.dylib       0x1b2d9f348 _os_unfair_lock_lock_slow + 220
2  libobjc.A.dylib                0x1b2384604 objc_destroyWeak + 144
3  libobjc.A.dylib                0x1b236a7cc object_cxxDestructFromClass(objc_object*, objc_class*) + 148
4  libobjc.A.dylib                0x1b237a6b8 objc_destructInstance + 68
5  libobjc.A.dylib                0x1b237a720 object_dispose + 16
6  GoogleUtilities                0x1018aec4c -[GULObjectSwizzler dealloc] (GULObjectSwizzler.m:155)
7  libobjc.A.dylib                0x1b2380754 _object_remove_assocations + 468
8  libobjc.A.dylib                0x1b237a6d4 objc_destructInstance + 96
9  libobjc.A.dylib                0x1b237a720 object_dispose + 16
10 libdispatch.dylib              0x1b2bd0a38 _dispatch_call_block_and_release + 24
11 libdispatch.dylib              0x1b2bd17d4 _dispatch_client_callout + 16
12 libdispatch.dylib              0x1b2b7a320 _dispatch_lane_serial_drain$VARIANT$mp + 592
13 libdispatch.dylib              0x1b2b7ae3c _dispatch_lane_invoke$VARIANT$mp + 428
14 libdispatch.dylib              0x1b2b834a8 _dispatch_workloop_worker_thread + 596
15 libsystem_pthread.dylib        0x1b2db1114 _pthread_wqthread + 304
16 libsystem_pthread.dylib        0x1b2db3cd4 start_wqthread + 4

[maksymmalyhin]

@dangthaison91 Thank you for the provided details! The logs you added, I guess they are call-stacks of other threads at the time of the crash. It is correct?

[dangthaison91]

Yes, They are some of call-stacks of other threads at the time it crashs @maksymmalyhin

[dangthaison91]

Hi @maksymmalyhin , how is this going on? Could you share any progress?

[maksymmalyhin]

@dangthaison91 I am sorry for the long wait. We haven't found the exact reason of the issue yet. Most probably, the reason of the issue is in FIRClearcutLogger which is going to be removed (estimated release M52). In the meanwhile, I'll continue the investigation to find a hot fix for the crashes while waiting for FIRClearcutLogger to be replaced.

BTW, there is a newer Firebase version available - 6.3.0. If you had a chance to check if the crash is reproducible with it?

[HyunjoonKo]

Hi. I got the same error.
In my case occurs when loading the url in WKWebView. (Using GoogleUtilities 6.2.0, FirebaseCore 6.0.1)

[maksymmalyhin]

@HyunjoonKo Thank you for the additional details! It looks like you were able to reproduce the crash with the debugger attached. Will it be possible to share a sample project and steps to reproduce the issue? It will help a lot.

[maksymmalyhin]

@HyunjoonKo Another question for you. Could you share other pods you are using in your app? Do you use FirebasePerformance as well?

[maksymmalyhin]

I have a hypothesis that can explain the crash logs. It involves FirebasePerformance and GULNetwork interactions. The most probably chain of event is:

1. GULNetwork creates an instance of GULNetworkURLSession and stores it into GULNetwork *_requests ivar
2. GULNetworkURLSession stores itself into static NSMapTable *sessionIDToFetcherMap = [NSMapTable strongToWeakObjectsMapTable]
3. GULNetworkURLSession creates a NSURLSession instance and sets itself as a delegate. If FirebasePerformance is used, it leads to ISA swizzling of GULNetworkURLSession by GULObjectSwizzler
4. Steps 1 - 3 repeat enough number of times so -[NSConcreteMapTable grow] on sessionIDToFetcherMap will be called on the next setObject:..
5. One of previous requests is finished, GULNetwork->_requests removeObjectForKey: is called. It schedules actual removing the GULNetworkURLSession on GULMutableDictionary ->_queue
6. Another set of steps 1 - 2 is performed
7. [sessionIDToFetcherMap setObject: forKey:] triggers [sessionIDToFetcherMap grow] under the hood which leads to reading of the stored objects.
8. At the same time as 7 is happening, [NSMutableDictionary removeObjectForKey:] is called from GULMutableDictionary ->_queue. It triggers the corresponding GULNetworkURLSession instance deallocation which leads to the corresponding GULObjectSwizzler deallocation, which triggers removing the GULNetworkURLSession subclass created for ISA swizzling.
9. When the partially deallocated GULNetworkURLSession instance is accessed at step 7, there must be no original subclass that leads to the crash.

I am still working on the validation of the hypothesis and designing a proper fix.

@dangthaison91 If my assumptions are true, then as a workaround while waiting for the fix, you can try to disable FirebasePerformance temporary in your app to avoid undesirable crashes and restore it back once the fix is released.

[HyunjoonKo]

@maksymmalyhin Yes. I am using this feature. We are also using the following Firebase and Google services:

Using Crashlytics (3.12.0)
Using Fabric (1.9.0)
Using Firebase (6.1.0)
Using FirebaseABTesting (3.0.0)
Using FirebaseAnalytics (6.0.1)
Using FirebaseAnalyticsInterop (1.2.0)
Using FirebaseAuth (6.1.0)
Using FirebaseAuthInterop (1.0.0)
Using FirebaseCore (6.0.1)
Using FirebaseDynamicLinks (4.0.0)
Using FirebaseInstanceID (4.1.0)
Using FirebaseMessaging (4.0.1)
Using FirebasePerformance (3.0.0)
Using FirebaseRemoteConfig (4.0.0)
Using GTMSessionFetcher (1.2.2)
Using GoogleAnalytics (3.17.0)
Using GoogleAppMeasurement (6.0.1)
Using GoogleIDFASupport (3.14.0)
Using GoogleSignIn (4.4.0)
Using GoogleToolboxForMac (2.2.1)
Using GoogleUtilities (6.2.0)
Using Protobuf (3.8.0)
Using nanopb (0.3.901)
Using google-cast-sdk (4.4.1)

This problem does not always happen, so it is hard to find the exact flow.
However, the following could be found:

• 'GULNetwork' tried to remove the 'GULMutableDictionary _objects' with nothing(count = 0).
• Problems occur if delay or timeout occurs due to bad network conditions.

[HyunjoonKo]

@maksymmalyhin
Anyway, since it is a problem to approach the key value that has already disappeared, I would like to solve it like this.

- (void)removeObjectForKey:(id)key {
  dispatch_async(_queue, ^{
      if(self->_objects.count > 0 && [[self->_objects allKeys] indexOfObject:key] != NSNotFound) {
          [self->_objects removeObjectForKey:key];
      }
  });
}

[maksymmalyhin]

@HyunjoonKo Thank you for the provided data!

Thank you for the proposed fix as well. Unfortunately it won't work, because the crash happens at the time the object is being removed from the dictionary, which means that the check self->_objects.count > 0 would succeed.

[dangthaison91]

@maksymmalyhin Fortunately we temporary remove Firebase Performance for another reason from 2 release versions ago. Your assumption may be true as the crash goes away completely.

We will wait for the fix to add Firebase Performance again.

@HyunjoonKo Thanks for your very helpful info.

[maksymmalyhin]

The issue should not be reproducible after #3322, but I'll keep the issue open until #3300 landed.

The changes from #3322 are available at GoogleUtilities 6.2.2 that has been published recently. @HyunjoonKo @dangthaison91 would you have chance to test the issue from your side and confirm if the change fixes the crash?

[maksymmalyhin]

Sorry, we had to revert #3322 becuase GULMutableDictionary is used by other SDKs such FirebaseAnalytics which rely on its original implementation, so #3322 may cause some performance issues.

Currently we are testing #3300. We will push it once it is tested. It will be great if you have a chance to test it from your side. To do it add to your Podfile a line:

pod 'GoogleUtilities', :git =>'https://github.com/firebase/firebase-ios-sdk.git', :branch=>'mm/gul-swizzle'

and perform pod install.

[paulb777]

Closing since #3300 has merged and released.