Firebase-android-sdk: Native symbol generation produces OutOfMemoryError

[JessHolle]

Generation of native symbols fails with an OutOfMemory error for larger native libraries.

This issue is noted in https://stackoverflow.com/questions/57623793/crashlytics-generate-symbols-gradle-step-fails-gc-overhead-limit-exceeded for Fabric Crashlytics.

The issue is that now everyone is required to migrate to the Firebase Crashlytics SDK, this issue still exists when using the Firebase Crashlytics SDK, but the Fabric workaround no longer works and no other workaround is documented.

Ideally, of course, the processing would take place in a streaming fashion such that binaries of an arbitrary size can be processed without an OutOfMemoryError. Barring that, however, a workaround is needed before we can move to the Firebase Crashlytics SDK.

[google-oss-bot]

I couldn't figure out how to label this issue, so I've labeled it for a human to triage. Hang tight.

[JessHolle]

Also filed Firebase Support Case 00062742 on this prior to discovering this (more appropriate?) venue.

[JessHolle]

I spent some time trying to find support for this in the code via decompilation of the firebase-crashlytics-buildtools-2.1.1 jar (I couldn't find these sources -- are they available?) -- but as best I could follow there is no support to disable using debug information currently. Perhaps I misread...

[GlebMaltsev]

We have external native build, and one of our libraries takes 61-63 MB (depends on ABI) in unstripped state. If I remove only those 60+MB versions of libraries from the folders, everything works as exptected, If I return library back - it seems that task turns into infinite loop (I waited for 30+ minutes and task is still not completed, but CPU and RAM consumption is above 95% during all this time). Here my paths to the native library directories:

firebaseCrashlytics {
     nativeSymbolUploadEnabled true
     unstrippedNativeLibsDir "build/intermediates/merged_native_libs/platoRelease/out/lib/"
     strippedNativeLibsDir "build/intermediates/stripped_native_libs/platoRelease/out/lib/"
}

[JessHolle]

So with Fabric it seemed that strippedNativeLibsDir didn't need to be specified if it was the default.

I can try specifying that if that is supposed to help -- I'm 99% sure it didn't with Fabric.

[JessHolle]

Two issues here:

1. Specifying strippedNativeLibsDir did not fix the GC limit exceeded OOME
2. It's not immediately clear how to specify this properly when one has build flavors

I tested this with one of my build flavors and thus had:

strippedNativeLibsDir 'build/intermediates/stripped_native_libs/devRelease/out/lib'

Given that this is the default stripped library location, I would hope that the plugin would simply _assume_ this location if not otherwise directed -- as it is pointless, painful, and error prone to have to respecify this default.

In my case the library in question is built externally by another Gradle project and then consumed as a stripped library as part of an AAR via a Maven repository. The unstripped version of the library is separately published to Maven, downloaded to a local directory which is specified as unstrippedNativeLibsDir.

[HaiPhan2002]

Hi @JessHolle,
I just want to ask for more information.
What is the size of your SO without stripping?
What is the linker you used to link your SO?
PS: I'm not from Fabric/Firebase team, however, I met the issue before so I want to ask if I could help.

[JessHolle]

Leaving strippedNativeLibsDir specified as noted above, I edited gradle.properties to set:

org.gradle.jvmargs=-Xmx8536m

And I have 32GB of RAM, so this shouldn't put an undue burden on the system (though my CI runners have nowhere near this, so this is for testing purposes only).

And then I get a very slightly different error: Out of memory: Java heap space.

Also I should note that the ARM8 version of my unstripped library is 122MB in size. That said, as per the stackoverflow issue, I'm clearly not the only one to have run into this issue with Fabric Crashlytics and needed the workaround noted there. In fact I have 2 different projects using _different_ native libraries which need this workaround.

As for the linker I used, I've used whatever the default is for NDK r21 via CMake -- I'm not getting fancy here.

[JessHolle]

I have no Fabric plugin here. The project in question works fine with the Fabric Crashlytics SDK (thanks to the workaround noted in the stackoverflow issue noted above). The issue is that it fails with the Firebase Crashlytics SDK (which does not support the workaround we used with Fabric), which we have been told we must move to by November.

So instead of a Fabric plugin classpath, I have:

classpath 'com.google.firebase:firebase-crashlytics-gradle:2.1.1'  // Firebase Crashlytics SDK

[HaiPhan2002]

In my case, I still use the fabric plugins so I found the new version of Fabric (1.31 in my case) is incompatible with bfd linker. I suggest you should try to switch to gold linker in cmake and test it again

[emreakcan]

Having the same issue here.
I've followed this guide https://firebase.google.com/docs/crashlytics/ndk-reports.
(Shouldn't we all using that instead of fabric?)

I'm trying to upload around 150mb, it was working when it is 20mb. Dedicated 10gb to gradle but still no luck. It gives heap error after 12 mins.

edit : dedicated 14 gb took 30 min to fail.

Setup:

classpath 'com.google.firebase:firebase-crashlytics-gradle:2.1.1'
implementation 'com.google.firebase:firebase-crashlytics-ndk:17.0.0'

* What went wrong:
Execution failed for task ':app:generateCrashlyticsSymbolFileAppname'.
> Java heap space

* Try:
Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Exception is:
org.gradle.api.tasks.TaskExecutionException: Execution failed for task ':app:generateCrashlyticsSymbolFileAppNameDebug'.
        at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.lambda$executeIfValid$1(ExecuteActionsTaskExecuter.java:205)
        at org.gradle.internal.Try$Failure.ifSuccessfulOrElse(Try.java:263)
        at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeIfValid(ExecuteActionsTaskExecuter.java:203)
        at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.execute(ExecuteActionsTaskExecuter.java:184)
        at org.gradle.api.internal.tasks.execution.CleanupStaleOutputsExecuter.execute(CleanupStaleOutputsExecuter.java:114)
        at org.gradle.api.internal.tasks.execution.FinalizePropertiesTaskExecuter.execute(FinalizePropertiesTaskExecuter.java:46)
        at org.gradle.api.internal.tasks.execution.ResolveTaskExecutionModeExecuter.execute(ResolveTaskExecutionModeExecuter.java:62)
        at org.gradle.api.internal.tasks.execution.SkipTaskWithNoActionsExecuter.execute(SkipTaskWithNoActionsExecuter.java:57)
        at org.gradle.api.internal.tasks.execution.SkipOnlyIfTaskExecuter.execute(SkipOnlyIfTaskExecuter.java:56)
        at org.gradle.api.internal.tasks.execution.CatchExceptionTaskExecuter.execute(CatchExceptionTaskExecuter.java:36)
        at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.executeTask(EventFiringTaskExecuter.java:77)
        at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.call(EventFiringTaskExecuter.java:55)
        at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.call(EventFiringTaskExecuter.java:52)
        at org.gradle.internal.operations.DefaultBuildOperationExecutor$CallableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:416)
        at org.gradle.internal.operations.DefaultBuildOperationExecutor$CallableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:406)
        at org.gradle.internal.operations.DefaultBuildOperationExecutor$1.execute(DefaultBuildOperationExecutor.java:165)
        at org.gradle.internal.operations.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:250)
        at org.gradle.internal.operations.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:158)
        at org.gradle.internal.operations.DefaultBuildOperationExecutor.call(DefaultBuildOperationExecutor.java:102)
        at org.gradle.internal.operations.DelegatingBuildOperationExecutor.call(DelegatingBuildOperationExecutor.java:36)
        at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter.execute(EventFiringTaskExecuter.java:52)
        at org.gradle.execution.plan.LocalTaskNodeExecutor.execute(LocalTaskNodeExecutor.java:41)
        at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$InvokeNodeExecutorsAction.execute(DefaultTaskExecutionGraph.java:372)
        at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$InvokeNodeExecutorsAction.execute(DefaultTaskExecutionGraph.java:359)
        at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareExecutionAction.execute(DefaultTaskExecutionGraph.java:352)
        at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareExecutionAction.execute(DefaultTaskExecutionGraph.java:338)
        at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.lambda$run$0(DefaultPlanExecutor.java:127)
        at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.execute(DefaultPlanExecutor.java:191)
        at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.executeNextNode(DefaultPlanExecutor.java:182)
        at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.run(DefaultPlanExecutor.java:124)
        at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
        at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:48)
        at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:56)
Caused by: java.lang.OutOfMemoryError: Java heap space
        at com.google.firebase.crashlytics.buildtools.ndk.internal.dwarf.DwarfDataParser.interleave(DwarfDataParser.java:355)
        at com.google.firebase.crashlytics.buildtools.ndk.internal.dwarf.DwarfDataParser.interleaveRanges(DwarfDataParser.java:330)
        at com.google.firebase.crashlytics.buildtools.ndk.internal.dwarf.DwarfDataParser.processChildDebugInfoEntries(DwarfDataParser.java:285)
        at com.google.firebase.crashlytics.buildtools.ndk.internal.dwarf.DwarfDataParser.processChildDebugInfoEntries(DwarfDataParser.java:286)
        at com.google.firebase.crashlytics.buildtools.ndk.internal.dwarf.DwarfDataParser.processCompilationUnit(DwarfDataParser.java:204)
        at com.google.firebase.crashlytics.buildtools.ndk.internal.dwarf.DwarfDataParser.readCompilationUnit(DwarfDataParser.java:183)
        at com.google.firebase.crashlytics.buildtools.ndk.internal.dwarf.DwarfDataParser.readCompilationUnit(DwarfDataParser.java:149)
        at com.google.firebase.crashlytics.buildtools.ndk.internal.dwarf.DwarfDataParser.parse(DwarfDataParser.java:73)
        at com.google.firebase.crashlytics.buildtools.ndk.internal.elf.ElfDataParser.parseElf(ElfDataParser.java:132)
        at com.google.firebase.crashlytics.buildtools.ndk.internal.elf.ElfDataParser.parse(ElfDataParser.java:101)
        at com.google.firebase.crashlytics.buildtools.ndk.internal.elf.ElfDataParser.parse(ElfDataParser.java:39)
        at com.google.firebase.crashlytics.buildtools.ndk.internal.csym.ElfCSymFactory.createCSymFromFile(ElfCSymFactory.java:92)
        at com.google.firebase.crashlytics.buildtools.ndk.internal.csym.NdkCSymGenerator.generateCodeMappings(NdkCSymGenerator.java:101)
        at com.google.firebase.crashlytics.buildtools.ndk.internal.csym.NdkCSymGenerator.generateCodeMappings(NdkCSymGenerator.java:90)
        at com.google.firebase.crashlytics.buildtools.ndk.internal.csym.NdkCSymGenerator.generateFromDirectories(NdkCSymGenerator.java:44)
        at com.google.firebase.crashlytics.buildtools.Buildtools.generateSymbolsFromDirectory(Buildtools.java:213)
        at com.google.firebase.crashlytics.buildtools.Buildtools$generateSymbolsFromDirectory$3.call(Unknown Source)
        at com.google.firebase.crashlytics.buildtools.gradle.tasks.GenerateSymbolFileTask.generateSymbols(GenerateSymbolFileTask.groovy:45)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at org.gradle.internal.reflect.JavaMethod.invoke(JavaMethod.java:104)
        at org.gradle.api.internal.project.taskfactory.StandardTaskAction.doExecute(StandardTaskAction.java:49)

[JessHolle]

As an aside, it is well past the time that external libraries were treated better, i.e. we should generate the crashlytics native symbols for an AAR once, upload those to Maven, and re-use those in all downstream apps. That may be possible today, but that's certainly not a documented path.

[JessHolle]

Oh, I reach the question on stackoverflow and I found that plugins in the question. So apology for it. I will delete this comment later. In my case, I still use the fabric plugins so I found the new version of Fabric is incompatible with bfd linker. I suggest you should try to switch to gold linker in cmake and test it again

If we need to use a specific linker to get this to work, then this needs to be clearly documented.

As it stands I don't know how to do what you've requested I try/test. I'm just using the default CMake+NDK21 linker.

[HaiPhan2002]

@JessHolle inside your app\src\main\CMakeLists.txt, set it
set(CMAKE_SHARED_LINKER_FLAGS "${CMAKE_SHARED_LINKER_FLAGS} -fuse-ld=gold")

[HaiPhan2002]

@emreakcan I'm not the Cmake master, just basic. Your cmake seems custom and difference with the one is generated by Android Studio. I suggest you should add the line at the end of file. To ask Cmake use the custom linker

[emreakcan]

@emreakcan I'm not the Cmake master, just basic. Your cmake seems custom and difference with the one is generated by Android Studio. I suggest you should add the line at the end of file. To ask Cmake use the custom linker
PS: I Oo Gitbhub as midnight ~~!

Still same error :(

[kevinkokomani]

Thanks @emreakcan, @seal2002, and @JessHolle for your discussion and feedback on this error. All the testing and reproduction efforts are super helpful to see.

To clear up something mentioned above in this thread, the new Crashlytics build tools should attempt to find the necessary stripped and unstripped directories as it did in Fabric - if it's unable to resolve on its own we do recommend setting paths using unstrippedNativeLibsDir and strippedNativeLibsDir.

This GC overhead limit exceeded/OOM issue is definitely a known bug to us and high on our list of improvements to make to the Crashlytics NDK component. I don't have a particular ETA for a fix to this but to be clear, we are considering this an important piece of work to pick up and prioritize. Unfortunately I don't know of any workarounds involving increasing the heap size or using a machine with more memory.

Thanks for reporting this and we hope to have better news for you soon!

[S1GSEGV]

I run into this issue and found two walk around.
1: use NDK r18 and Android Gradle plugin 3.4.2. (19 and later NDK will cause java heap issue when generate symbol for arm64-v8a)
2: use most recent NDK r21d and set linker flag -fuse-ld=lld
If use ndkBuild, set APP_LDFLAGS += -fuse-ld=lld in application.mk

Those two worked for me to get pass this issue.

[thedmd]

Adding to the pile. I have to deal with 982 MB of binaries with unstripped symbols (1.20 GB before switching to lld), rendering around 60 MB executable after stripping.

Assigning 24GB of RAM does not help. Disabling GC overhead limit -XX:-UseGCOverheadLimit does not help.

Tried with com.google.firebase:firebase-crashlytics-ndk:17.1.0.

I'm looking forward for fix to the issue. Without symbols crashlytics is no help, because Firebase display callstacks with "missing" text instead of an addresses I could manually try to symbolicate.

[JessHolle]

The r21d/lld workaround seems to work for me!

I run into this issue and found two walk around.
1: use NDK r18 and Android Gradle plugin 3.4.2. (19 and later NDK will cause java heap issue when generate symbol for arm64-v8a)
2: use most recent NDK r21d and set linker flag -fuse-ld=lld
If use ndkBuild, set APP_LDFLAGS += -fuse-ld=lld in application.mk

Those two worked for me to get pass this issue.

[HaiPhan2002]

Glad to heard the issue is solved @JessHolle with another linker :D. I am curious which the stacktrace on Firebase, does it show full stacktrace of an crash or missing some functions? And Do you use same Firebase tool at before?

[JessHolle]

I have not verified proper operation of Firebase Crashlytics symbolization with a native crash yet. Yes, I should have done so. I was frankly just too happy just to get build success.

[JessHolle]

Hmm... It is belatedly now clear that by moving to lld I can no longer set C++ line breakpoints in Android Studio (4.0). Without investigating further I'm going to count that workaround as a dead end and come back in from out-on-the-limb use of the quasi-experimental linker option.

[DanAlbert]

Depending on how you enabled LLD, you may have bypassed the workaround in https://github.com/android/ndk/issues/885.

[JessHolle]

Thanks @DanAlbert -- but I'm unclear on any "workaround" here when the bug you note has supposedly been fixed in r21.

To enable lld, I literally just added:
add_link_options( -fuse-ld=lld )
to my top-level CMakeLists.txt.

So I'm unclear as to how this could have loused things up.

[DanAlbert]

Yeah, that's wrong. If you do that the CMake toolchain doesn't know about it and won't set up the correct default arguments. Pass -DANDROID_LD=lld to CMake instead.

[jthrush]

Found a fix:
Had the same issue after migrating from Fabric Crashlytics to Firebase Crashlytics. Increasing to -Xms4G -Xmx8G did not work. Adding --no_daemon to the gradle command line fixed this issue for me (and don't need the -Xms/-Xmx).

[JessHolle]

So, I tried adding -DANDROID_LD=lld to the CMake arguments instead of add_link_options( -fuse-ld=lld )

Sure -- debugging works great now.

But I get an OutOfMemoryError again from the Firebase Crashlytics SDK. So overall, I'm still not sold on this workaround.

[DanAlbert]

Really? The only difference between those two approaches is the workaround for debugging, which is just -Wl,--build-id=sha1. The other option is that you only switched to LLD for a subset of your targets rather than all of them. Either way, you can set the build-id flag yourself as well.

[askamn]

@kevinkokomani Can you guys provide an update on when this issue will get fixed? Because the firebase crashlytics is pretty much useless if we are unable to generate and upload symbol files.
Our game (a very simple casual 2d game), which uses firebase crashlytics is not even that big and still, the process fails.

[jthrush]

@askamn Try adding --no_daemon to the gradle command line and let everyone know if that works for you like it did for me.

For whoever is fixing this issue, one thing to note is that I don't believe it is a true out-of-memory condition. I had the max heap size set to 8 GB with -Xmx8G, and watching the java.exe process in the task manager it never exceeded 1 GB before issuing the out-of-memory error. Could be the size of some object/array is exceeding some limit. It seems to be intermittent too, sometime it succeeds but most of the time fails.

[thedmd]

@jthrush Usually this is an java.lang.OutOfMemoryError: GC overhead limit exceeded exception.

https://stackoverflow.com/a/1393503

This message means that for some reason the garbage collector is taking an excessive amount of time (by default 98% of all CPU time of the process) and recovers very little memory in each run (by default 2% of the heap).

[askamn]

@jthrush I did and it did not help. Thanks though.

[mrwillis21]

Hi folks - wanted to let everyone know that we're aware of this issue and it's on our roadmap to improve.

[alexzatsepin]

it's unbelievable! We have no option to fix this OOM and read our native crashes. If I use the lld linker then OOM is gone, but breakpoints are stopped working in Android Studio and crash reports are shown incorrectly in Firebase console. And suggestions from the Firebase documentation about -Wl,--no-rosegment or -fno-omit-frame-pointer usage(not explaining where shoud i put these flags and how use it correctly) doesn't work, i.e. the OOM is came again. Look like Evil cycle!

Please, fix this OOM issue! When we use Fabric sdk we have at lease workaround to use old dependency, but now we have only useless firebase crashlytics integration.

[alexzatsepin]

Hi everyone!
Spending two days I could build our project and upload native symbols successfully for all ABIs (arm32, arm64, x86 and x86_64) with followings settings:

1. Add linker flags in your project CmakeLists.txt - required
   set(CMAKE_SHARED_LINKER_FLAGS "${CMAKE_SHARED_LINKER_FLAGS} -fuse-ld=gold"), @seal2002 thanks!
2. Increased max heap size in gradle.properties - I guess it's optional
   org.gradle.jvmargs=-Xmx8g
3. Don't forget to remove '-DANDROID_LD=lld' from build.gradle if you already added it there - required
4. Update NDK to last stable version (now, it's r21d, ndkVersion 21.3.6528147) - I guess it's optional
5. Update firebase-analytics and firebase-crashlytics-nsk to last stable versions - I guess it's optional

implementation 'com.google.firebase:firebase-analytics:17.4.4'
implementation 'com.google.firebase:firebase-crashlytics:17.1.1'
implementation 'com.google.firebase:firebase-crashlytics-ndk:17.1.1'

1. Update gradle plugin to 4.0.1 - I guess it's optional
   classpath 'com.android.tools.build:gradle:4.0.1'

Guys, I marked some steps as optional because I didn't check the build without those changes yet, because I spent a lot of time to make generating and uploading the symbols work. I let you know later if I check it. But, I'm sure that 1 and 3 steps are required, without these I get either OOM or stack traces were looked incorrectly in console and breakpoints weren't working when I tried to use ld=lld linker.

All my changes you can find here https://github.com/mapsme/omim/pull/13451. Good luck!

[JessHolle]

Can someone from the Google Firebase team provide an update as to when this issue is going to be fixed?

We really want to have our app moved to the Firebase Crashlytics SDK a month or two prior to Fabric Crashlytics SDK being disabled -- so that any user who has updated the app in the last month or two continues to produce Crashlytics data as needed.

At this point this is becoming tight -- and there's still no word of this issue being fixed.

At the very least an _officially_ documented _minimum_ set of steps to work around this issue is needed well prior to turning off data from the Fabric Crashlytics SDK!

[mrwillis21]

Hi everyone! Popping in here to let you all know we're actively working on this, and will be rolling out some optimizations in the very near future to help everyone get unstuck in the short term, as well as some broader overall improvements to Crashlytics native crash reporting as a whole in the medium term. So stay tuned!

[askamn]

Hello everyone! I just wanted to post an update regarding this.
So I managed to upload the symbols without any change to memory options by simply using the gold linker in CMakeLists.txt
set(CMAKE_SHARED_LINKER_FLAGS "${CMAKE_SHARED_LINKER_FLAGS} -fuse-ld=gold")

However, a few interesting points to note here:

1. Only works on my Windows(16GB) machine. Does not work on Mac(8GB) or Linux(32GB). _Mentioned the RAM all these systems have._
2. Only works 2-3 times. After that I again start getting either Java Heap Space or GC Overhead Limit exceeded. To make it work again, I have to change the linker (Either i remove the linker setup from CMake or just use lld instead. lld fails for me because I use some prebuilt libraries that just don't seem to work when linking with lld). It doesn't matter whether the build succeeds or not when using a different linker, I just have to build it once with a different linker, but after that, once I switch back to the gold linker, the symbol generation process starts working again like a charm!
3. The symbol generation process, when it works, _only takes like 2 minutes to complete_. My final native lib size is 167MB. And when the process does not work, it takes upto 30 min to report to me the error about Memory.

My crash reports are now coming properly on crashlytics with this workaround.
Just posting this here for the devs. Maybe this helps!

[mrwillis21]

Hello everyone! Great news - we've just released the Firebase Crashlytics Gradle plugin version 2.3.0 with some new optimizations that should improve memory usage quite a bit when generating symbol files. Please try it out and let me know how it goes for you!

And as I said before, we're still hard at work on improving our native crash reporting support, so be on the lookout for more news coming soon. :)

[thedmd]

I can confirm that 2.3.0 handle symbol generation better.

130MB of cSYM file was generated from 710MB unstripped (80MB stripped) so file built in release configuration.

Unoptimized build is around 834MB and this one still hit out-of-memory exception.
So far I tried to grant gradle more and more memory, but maybe there is a better way. Is there any property we can set to increase heap size for Crashlytic generator only?

Out of curiosity, what is more beefy so file you had tested?

Thanks for help!

[S1GSEGV]

2.3.0 fix our issue. I also reverted linker changes. Now I'm using the default ndk linker and there is no OOM error anymore.
Our obj folder size under ndkBuild is 1.49G. So far so good!
Thanks!

[QMagic]

I confirm that with 2.3.0 the issue has been fixed for us.
Thanks!