Firebase-android-sdk: Crashlytics console not showing any crashes anymore

[tspecht]

The Problem

All Crashlytic logs randomly stopped showing up on the Firebase Crashlytics console. The crash free rate percentage, however, does show up there. I assume that they are indeed getting some data, because in order to calculate this crash free percentage they need the crash log data (maybe not, idk).

To add to this, older builds where crashes used to be seen, also stopped showing the crash logs in the Firebase Crashlytics console. At this point i am convinced that it might not be a configuration issue on our end, if it was older builds shouldnt be affected.

There are no error logs either, telling us that something is wrong.

What i have tried

Checked to see if we changed anything related to firebase crashlytics (besides updating to the latest version)
Updating to the latest firebase Crashlytics version (multiple times)
Ive tried comparing the google-services.json file to the one we have in the project, and it is the same.
Contacting the firebase team, been at it for a few weeks now and still could not resolve, ive sent the logs, code snippets, everything they asked for. But still no solution.
Made sure we no longer have any references to the old crashlytics/fabric plugin/implementation.
Project info:

• Kotlin version: 1.4.10
• Gradle plugin version: 4.0.2
• firebase-crashlytics-gradle version: 2.3.0
• Firebase Crashlytics version: 17.2.2

Existing Crashlytics configuration:

Project level build.gradle

// Top-level build file where you can add configuration options common to all sub-projects/modules.
buildscript {

    repositories {
        google()
        jcenter()
        maven { url 'https://plugins.gradle.org/m2/' }
    }

    dependencies {
        classpath 'com.android.tools.build:gradle:4.0.2'
        classpath "com.apollographql.apollo:apollo-gradle-plugin:1.2.0"
        classpath group: 'commons-io', name: 'commons-io', version: '2.5'
        classpath 'com.google.gms:google-services:4.3.4'
        classpath "org.jetbrains.kotlin:kotlin-gradle-plugin:1.4.10"
        classpath 'com.google.firebase:perf-plugin:1.3.2'
        classpath 'com.google.firebase:firebase-crashlytics-gradle:2.3.0'
    }
}

allprojects {
    repositories {
        google()
        mavenCentral()
        jcenter()
        maven { url 'https://jitpack.io' }
    }
}

task clean(type: Delete) {
    delete rootProject.buildDir
}

App level build.gradle

import groovy.json.JsonOutput
import org.apache.commons.io.IOUtils

apply plugin: 'com.android.application'
apply plugin: 'com.google.firebase.firebase-perf'
apply plugin: 'com.google.firebase.crashlytics'
apply plugin: 'kotlin-android'
apply plugin: 'kotlin-kapt'
apply plugin: 'kotlin-android-extensions'

if (project.hasProperty("enableJacoco")) {
    apply from: '../jacoco.gradle'
}

repositories {
    maven { url 'https://www.jitpack.io' }
    flatDir {
        dirs 'libs'
    }
}

androidExtensions {
    experimental = true
}

android {
    compileSdkVersion 29
    buildToolsVersion "29.0.2"
    defaultConfig {
        applicationId "com.my-app"
        minSdkVersion 21
        targetSdkVersion 29

        testInstrumentationRunner "com.my-app.TestAppJUnitRunner"

        vectorDrawables.useSupportLibrary = true

        // for auto factory
        javaCompileOptions {
            annotationProcessorOptions {
                includeCompileClasspath = true
            }
        }
    }

    buildTypes {
        release {
            minifyEnabled true
            zipAlignEnabled true
            debuggable false
            testCoverageEnabled false

            proguardFiles getDefaultProguardFile('proguard-android.txt'), 'proguard-rules.pro'
        }
        debug {
            FirebasePerformance {
                instrumentationEnabled false
            }
            debuggable true
            testCoverageEnabled project.hasProperty("enableJacoco")

            firebaseCrashlytics {
                mappingFileUploadEnabled false
            }
        }
    }

    compileOptions {
        sourceCompatibility JavaVersion.VERSION_1_8
        targetCompatibility JavaVersion.VERSION_1_8
        // Flag to enable support for the new language APIs
        coreLibraryDesugaringEnabled true
    }

    useLibrary 'android.test.runner'
    useLibrary 'android.test.base'
    useLibrary 'android.test.mock'
}

dependencies {

    // Firebase
    implementation "com.google.firebase:firebase-messaging:20.1.0"
    implementation "com.google.firebase:firebase-core:17.2.0"
    implementation "com.google.firebase:firebase-perf:19.0.0"
    implementation "com.google.firebase:firebase-auth:19.2.0"

    implementation "com.google.firebase:firebase-analytics:17.6.0"
    implementation "com.google.firebase:firebase-crashlytics:17.2.2"

}

apply plugin: 'com.google.gms.google-services'

(cleaned up to show firebase related only)

Note: We have our google-services.json located in the proper location in our project, as it always been.

[google-oss-bot]

I found a few problems with this issue:

• I couldn't figure out how to label this issue, so I've labeled it for a human to triage. Hang tight.
• This issue does not seem to follow the issue template. Make sure you provide all the required information.

[luispereira]

This is still happening to me too.
Suddently our prod env stop reporting crashes. No changes were made to the google-services.json or in the project. However develop env works fine.

We have also 2 projects under the same firebase

[laalto]

We have also had an issue with one of our apps. Crash reports stopped appearing in crashlytics console mid-November with both old and new version of the app, with some of the newer versions updated to 17+ version of the Google Crashlytics SDK.

Debugging the app locally with logcat enabled shows Crashlytics initializing but not the usual logcat entries seen when a crash report is being sent.

I don't know if it's related but the app is listed under two different projects in Firebase i.e. the same package id is in two places - we migrated the app from one project to another some time ago and did not yet delete the old one to keep seeing crashes etc. from older app versions.

[mrwillis21]

Hi folks, we're aware of an issue with an older version of Dexguard that causes crashes not to be reported - are you using Dexguard at all?

[JosiasSena]

@mrwillis21 hey, no Dexguard on our end. Just plain old proguard.

[mrwillis21]

Thanks for letting me know @JosiasSena. Are you experiencing a sudden reduction in crash reports for an app in production?

@tspecht, @luispereira, @laalto?

[JosiasSena]

@mrwillis21 yes sorry, @tspecht and i are on the same team

[laalto]

@mrwillis21 Yes that's a plausible explanation for us. The app that stopped working wrt crashlytics reporting is dexguarded (currently with version 8.6.13); those apps that continued to report crashes have R8 optimization/obfuscation in place.

[mrwillis21]

For anyone experiencing this using Dexguard, please update to the latest 8.x version or add this to your obfuscation rules:

-keepresourcexmlelements manifest/application/service/meta-data@value=cct

That should address the issue for future versions of your app.

[mrwillis21]

@JosiasSena @tspecht Ah, no worries. :)

If you're not using Dexguard, that's very strange, because we've only seen what you described for Dexguard users.

Can you let me know your app ID so I can look into this further? Or if you're not comfortable with posting it here, please open a case at https://firebase.google.com/support/troubleshooter/contact and send it along? I recognize you may have done this once before, and I apologize that the issue wasn't resolved.

[JosiasSena]

@mrwillis21 yea we did open a case there over a month ago as well, but they still don't know what the problem is :(

Would giving you a list of all third-party libraries and their versions help you?

[laalto]

@mrwillis21 Thanks. After that dexguard rule change I can now see the D/TransportRuntime.CctTransportBackend: Making request to: https://crashlyticsreports-pa.googleapis.com/v1/firelog/legacy/batchlog in logcat that was missing before, and a number of test crashes appeared in dashboard too. This seems to resolve the issue for us.

[tspecht]

@mrwillis21 we have two open cases right now, numbers are 00100394 and 00095805 . Please let us know if these provide enough context & debug information, we're happy to provide more as needed. We've been working with the Firebase support on resolving these issues for well over a month now and still no progress either way, which is troubling.

[mrwillis21]

@tspecht Thanks for letting me know the case numbers. I'll follow up and see if I can get any more information for you. Definitely a bummer that we've not been able to get some sort of resolution for you yet!

[luispereira]

Thanks for letting me know @JosiasSena. Are you experiencing a sudden reduction in crash reports for an app in production?

@tspecht, @luispereira, @laalto?

Yes, dexguard might be the problem then, it is possible since we do not use it in develop and use it in release (production) environment. I will check this tomorrow and will let you know

[luispereira]

Just tested this and it seems to be reporting to the firebase console now

For anyone experiencing this using Dexguard, please update to the latest 8.x version or add this to your obfuscation rules:

-keepresourcexmlelements manifest/application/service/meta-data@value=cct

That should address the issue for future versions of your app.

[tspecht]

@mrwillis21 any luck looking into this internally? Would really love to get back to shipping updates to our Android users with confidence, right now we are still flying blind since all the crashes are missing ... let me know if there is any additional information necessary from our end!

[mrwillis21]

@tspecht Yup! Currently digging. When did crashes stop showing up for you, exactly?

[tspecht]

@mrwillis21 I unfortunately don't have an exact date for you, but we started noticing the issue around Oct 21, 2020.

[mrwillis21]

@tspecht and is this one particular version or multiple versions of your app?

[tspecht]

@mrwillis21 curiously literally every version of our Android application was affected by this, even those that were released way before that time-window and were showing crashes just fine. That's the reason we don't believe Dexguard (or some similar incompatibility) is at fault here, in that case I'd only expect crashes to stop showing up for a certain new version or any incoming data _after_ a certain date. We however can't even access historic data anymore at this point.

[mrwillis21]

@tspecht after the October 21 date, did crashes just suddenly drop off, or did all crashes (even those from before October 21) disappear?

[tspecht]

@tspecht after the October 21 date, did crashes just suddenly drop off, or did all crashes (even those from before October 21) disappear?

@mrwillis21 unfortunately the latter. So even a version that was released let's say in July and was showing data just fine the whole time, all data suddenly disappeared and never came back. It's almost like the underlying datastore got borked somehow 🤔

[mrwillis21]

@tspecht Alright cool. Well, not cool, but thanks for the info. 😝 I'll report back when I have more information for you.

[mrwillis21]

@tspecht @JosiasSena Hey there - I did some digging, and I think your app might have somehow ended up in a bad state. I've flipped some switches on our backend to try and resolve it, can you let me know if you start seeing crashes again going forward?