Google-cloud-java: Duplicate classes error when adding dialogflow dependency 0.98.0-alpha along with firebase

[h-amg]

i'm getting a duplicate classes error when building with android studio gradle after i added dialogflow to my android app which uses firebase.

DEPENDENCIES:
these are my dependencies:

implementation fileTree(dir: 'libs', include: ['*.jar'])
implementation 'androidx.appcompat:appcompat:1.0.2'
implementation 'androidx.constraintlayout:constraintlayout:1.1.3'
testImplementation 'junit:junit:4.13-beta-3'
androidTestImplementation 'androidx.test:runner:1.2.0'
androidTestImplementation 'androidx.test.espresso:espresso-core:3.2.0'
implementation 'pl.droidsonroids.gif:android-gif-drawable:1.2.3'
implementation 'androidx.multidex:multidex:2.0.1'
implementation 'com.google.android.material:material:1.1.0-alpha07'
implementation 'com.jakewharton:butterknife:10.1.0'
annotationProcessor 'com.jakewharton:butterknife-compiler:10.1.0'
implementation 'com.github.VaibhavLakhera:Circular-Progress-View:0.1.2'
implementation 'com.github.CardinalNow:Android-CircleProgressIndicator:v0.2'
implementation 'ai.api:sdk:2.0.7@aar'
implementation 'ai.api:libai:1.6.12'
implementation 'com.google.cloud:google-cloud-dialogflow:0.98.0-alpha'
implementation 'io.grpc:grpc-okhttp:1.21.0'
implementation 'com.google.apis:google-api-services-oauth2:v2-rev99-1.21.0'
implementation 'com.google.apis:google-api-services-dialogflow:v2beta1-rev9-1.23.0'
implementation 'com.andkulikov:transitionseverywhere:2.0.0-beta01'
implementation 'com.google.firebase:firebase-core:17.0.0'
implementation 'com.google.firebase:firebase-auth:18.0.0'
implementation 'com.google.firebase:firebase-firestore:20.1.0'

STACKTRACE:
This is part of the stacktrace as it continues to show duplicate classes arror (com.google.api.*) for another atleast 60 more classes:

Duplicate class com.google.api.Advice found in modules classes.jar (com.google.firebase:protolite-well-known-types:17.0.0) and proto-google-common-protos-1.16.0.jar (com.google.api.grpc:proto-google-common-protos:1.16.0)
Duplicate class com.google.api.Advice$1 found in modules classes.jar (com.google.firebase:protolite-well-known-types:17.0.0) and proto-google-common-protos-1.16.0.jar (com.google.api.grpc:proto-google-common-protos:1.16.0)
Duplicate class com.google.api.Advice$Builder found in modules classes.jar (com.google.firebase:protolite-well-known-types:17.0.0) and proto-google-common-protos-1.16.0.jar (com.google.api.grpc:proto-google-common-protos:1.16.0)
Duplicate class com.google.api.AdviceOrBuilder found in modules classes.jar (com.google.firebase:protolite-well-known-types:17.0.0) and proto-google-common-protos-1.16.0.jar (com.google.api.grpc:proto-google-common-protos:1.16.0)

UNSUCCESSFUL ATTEMPTS:
I have tried exludsing the duplicated classes unsuccessdully using the below methods:

implementation('com.google.cloud:google-cloud-dialogflow:0.93.0-alpha'){ 
exclude group: 'com.google.api.grpc' 
exclude group: 'com.google.protobuf' 
}

implementation('com.google.cloud:google-cloud-dialogflow:0.99.0-alpha'){ 
exclude group: 'com.google.api.' 
}

implementation('com.google.cloud:google-cloud-dialogflow:0.99.0-alpha'){ 
exclude group: 'com.google' , module: 'api'
}

android {
    configurations {

            all*.exclude group:'com.google.api.Advice', module: 'classes.jar'
            all*.exclude group: 'com.google.api.Advice', module: 'proto-google-common-protos-1.16.0.jar'

        }
}

[h-amg]

I have narrowed down cause of the duplication to firestore dependency (com.google.firebase:firebase-firestore:20.1.0) and dialogflow dependency (implementation 'com.google.cloud:google-cloud-dialogflow:0.98.0-alpha)

[h-amg]

Hi @sduskis, I till haven't figured out a way to sort this issue, do you have any tip on anything i should try?

[h-amg]

Hi @android10, I'd really appreciate your input on this. any thoughts on how to solve this problem?

[DavidBuzatu-Marian]

@h-amg I have the exact same problem as you and I tried all the steps you said. Still, it doesn't work. Did you find a workaround?

[h-amg]

@DavidBuzatu-Marian Nope, not yet. I'll post an update if i figured it out

[DavidBuzatu-Marian]

@DavidBuzatu-Marian Nope, not yet. I'll post an update if i figured it out

I've made a stackoverflow post and got a response, but I have an error. Maybe you can figure it out:

https://stackoverflow.com/questions/56922949/resolve-duplicate-classes-with-firebase-and-dialog-flow

[h-amg]

@DavidBuzatu-Marian Nope, not yet. I'll post an update if i figured it out

I've made a stackoverflow post and got a response, but I have an error. Maybe you can figure it out:

https://stackoverflow.com/questions/56922949/resolve-duplicate-classes-with-firebase-and-dialog-flow

@DavidBuzatu-Marian The snippet lance provided is unnecessary since android studio lists all the duplicate classes and jars where they are located in the build window (part of which is included in my post above under "stacktrace"), in other words there is a similar script built into android studio that does this. The challenge is how to exclude the duplicate classes?

[DavidBuzatu-Marian]

@h-amg I managed to exclude the classes, but I have something new in my code, maybe you did it differently and won't have it.

I added this to gradle (module) just above dependencies:

configurations {
implementation.exclude module:'proto-google-common-protos'
implementation.exclude module:'protobuf-java'
}

[h-amg]

Hey @DavidBuzatu-Marian, thank you so much. This method worked though i had to modify it a little bit. (for those landing on this page in search for a solution to the same problem. post the below code into your build.gradle (module app) file)

android {
    ... 


   configurations {
             implementation.exclude module:'proto-google-common-protos'
             implementation.exclude module:'protolite-well-known-types'
             implementation.exclude module:'guava'
             implementation.exclude module:'protobuf-lite'
    }

}

Let me know if this works for you.

[DavidBuzatu-Marian]

Well, it did not for me. Apparently, my firebase crashes if I do it like you did. But I will search for a workaround

[h-amg]

I got too excited too fast, the app crashes with the following error when it launches:

java.lang.ClassNotFoundException: Didn't find class "com.google.common.base.Preconditions"

[sduskis]

Maybe you can try to import com.google.guava:guava:27.0-android

[h-amg]

Thanks @sduskis , it didn't work.

[h-amg]

@DavidBuzatu-Marian remove guava exclusion like so:

android {
    ... 


   configurations {
             implementation.exclude module:'proto-google-common-protos'
             implementation.exclude module:'protolite-well-known-types'
             implementation.exclude module:'protobuf-lite'
    }

}

It should work without causing duplicate classes error or crashing the app

[h-amg]

So I ended up excluding only these two modules:

android {
    ... 


   configurations {
             implementation.exclude module:'protolite-well-known-types'
             implementation.exclude module:'protobuf-lite'
    }

}

However, when i try to retrieve data from firebase-firestore DB the app crashes with the following exception message:

Caused by: java.lang.VerifyError: Verifier rejected class com.google.firestore.v1.WriteResponse due to bad method java.lang.Object com.google.firestore.v1.WriteResponse.dynamicMethod(com.google.protobuf.GeneratedMessageLite$MethodToInvoke, java.lang.Object, java.lang.Object) (declaration of 'com.google.firestore.v1.WriteResponse' appears in /data/app/com.app.android.name-1/split_lib_dependencies_apk.apk:classes2.dex)

DebugLogs:

07-11 08:02:11.603 27666-27666/com.app.android.name W/DisplayListCanvas: DisplayListCanvas is started on unbinded RenderNode (without mOwningView)
07-11 08:02:11.613 27666-27666/com.app.android.name D/ViewRootImpl: MSG_RESIZED_REPORT: ci=Rect(0, 72 - 0, 0) vi=Rect(0, 72 - 0, 0) or=1
07-11 08:02:11.653 27666-28056/com.app.android.name I/DynamiteModule: Considering local module providerinstaller:0 and remote module providerinstaller:0
07-11 08:02:11.653 27666-28056/com.app.android.name W/ProviderInstaller: Failed to load providerinstaller module: No acceptable module found. Local version is 0 and remote version is 0.
07-11 08:02:11.653 27666-28056/com.app.android.name W/ResourcesManager: getTopLevelResources: /data/app/com.google.android.gms-1/base.apk / 1.0 running in com.app.android.name rsrc of package null
07-11 08:02:11.683 27666-27666/com.app.android.name I/Timeline: Timeline: Activity_idle id: android.os.BinderProxy@ff29e8d time:147037202
07-11 08:02:11.723 27666-29453/com.app.android.name I/art: Verification error in void com.google.firestore.v1.WriteResponse.mergeCommitTime(com.google.protobuf.Timestamp)
07-11 08:02:11.723 27666-29453/com.app.android.name I/art: void com.google.firestore.v1.WriteResponse.mergeCommitTime(com.google.protobuf.Timestamp): [0x10] couldn't find method com.google.protobuf.Timestamp$Builder.mergeFrom (Lcom/google/protobuf/GeneratedMessageLite;)Lcom/google/protobuf/GeneratedMessageLite$Builder;
07-11 08:02:11.723 27666-29453/com.app.android.name I/art: void com.google.firestore.v1.WriteResponse.mergeCommitTime(com.google.protobuf.Timestamp) failed to verify: void com.google.firestore.v1.WriteResponse.mergeCommitTime(com.google.protobuf.Timestamp): [0x10] register v3 has type Precise Reference: com.google.protobuf.Timestamp but expected Reference: com.google.protobuf.GeneratedMessageLite
07-11 08:02:11.723 27666-29453/com.app.android.name I/art: Verification error in java.lang.Object com.google.firestore.v1.WriteResponse.dynamicMethod(com.google.protobuf.GeneratedMessageLite$MethodToInvoke, java.lang.Object, java.lang.Object)
07-11 08:02:11.723 27666-29453/com.app.android.name I/art: java.lang.Object com.google.firestore.v1.WriteResponse.dynamicMethod(com.google.protobuf.GeneratedMessageLite$MethodToInvoke, java.lang.Object, java.lang.Object): [0x5C] couldn't find method com.google.protobuf.Timestamp.toBuilder ()Lcom/google/protobuf/GeneratedMessageLite$Builder;
07-11 08:02:11.723 27666-29453/com.app.android.name I/art: java.lang.Object com.google.firestore.v1.WriteResponse.dynamicMethod(com.google.protobuf.GeneratedMessageLite$MethodToInvoke, java.lang.Object, java.lang.Object): [0x73] couldn't find method com.google.protobuf.Timestamp$Builder.mergeFrom (Lcom/google/protobuf/GeneratedMessageLite;)Lcom/google/protobuf/GeneratedMessageLite$Builder;
07-11 08:02:11.723 27666-29453/com.app.android.name I/art: java.lang.Object com.google.firestore.v1.WriteResponse.dynamicMethod(com.google.protobuf.GeneratedMessageLite$MethodToInvoke, java.lang.Object, java.lang.Object) failed to verify: java.lang.Object com.google.firestore.v1.WriteResponse.dynamicMethod(com.google.protobuf.GeneratedMessageLite$MethodToInvoke, java.lang.Object, java.lang.Object): [0x73] register v5 has type Precise Reference: com.google.protobuf.Timestamp but expected Reference: com.google.protobuf.GeneratedMessageLite
07-11 08:02:11.723 27666-29453/com.app.android.name W/art: Verification failed on class com.google.firestore.v1.WriteResponse in /data/app/com.app.android.name-1/split_lib_dependencies_apk.apk because: Verifier rejected class com.google.firestore.v1.WriteResponse due to bad method java.lang.Object com.google.firestore.v1.WriteResponse.dynamicMethod(com.google.protobuf.GeneratedMessageLite$MethodToInvoke, java.lang.Object, java.lang.Object)
07-11 08:02:11.733 27666-27666/com.app.android.name D/AndroidRuntime: Shutting down VM
07-11 08:02:11.733 27666-27666/com.app.android.name E/AndroidRuntime: FATAL EXCEPTION: main
Process: com.app.android.name, PID: 27666
java.lang.RuntimeException: Internal error in Firestore (20.1.0).
at com.google.firebase.firestore.util.AsyncQueue.lambda$panic$5(com.google.firebase:firebase-firestore@@20.1.0:379)
at com.google.firebase.firestore.util.AsyncQueue$$Lambda$5.run(com.google.firebase:firebase-firestore@@20.1.0)
at android.os.Handler.handleCallback(Handler.java:739)
at android.os.Handler.dispatchMessage(Handler.java:95)
at android.os.Looper.loop(Looper.java:158)
at android.app.ActivityThread.main(ActivityThread.java:7224)
at java.lang.reflect.Method.invoke(Native Method)
at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:1230)
at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1120)
Caused by: java.lang.VerifyError: Verifier rejected class com.google.firestore.v1.WriteResponse due to bad method java.lang.Object com.google.firestore.v1.WriteResponse.dynamicMethod(com.google.protobuf.GeneratedMessageLite$MethodToInvoke, java.lang.Object, java.lang.Object) (declaration of 'com.google.firestore.v1.WriteResponse' appears in /data/app/com.app.android.name-1/split_lib_dependencies_apk.apk)
at com.google.firestore.v1.WriteResponse.getDefaultInstance(com.google.firebase:firebase-firestore@@20.1.0:1012)
at com.google.firestore.v1.FirestoreGrpc.getWriteMethod(com.google.firebase:firebase-firestore@@20.1.0:379)
at com.google.firebase.firestore.remote.WriteStream.(com.google.firebase:firebase-firestore@@20.1.0:74)
at com.google.firebase.firestore.remote.Datastore.createWriteStream(com.google.firebase:firebase-firestore@@20.1.0:104)
at com.google.firebase.firestore.remote.RemoteStore.(com.google.firebase:firebase-firestore@@20.1.0:186)
at com.google.firebase.firestore.core.FirestoreClient.initialize(com.google.firebase:firebase-firestore@@20.1.0:263)
at com.google.firebase.firestore.core.FirestoreClient.lambda$new$2(com.google.firebase:firebase-firestore@@20.1.0:117)
at com.google.firebase.firestore.core.FirestoreClient$$Lambda$2.run(com.google.firebase:firebase-firestore@@20.1.0)
at com.google.firebase.firestore.util.AsyncQueue.lambda$enqueue$4(com.google.firebase:firebase-firestore@@20.1.0:311)
at com.google.firebase.firestore.util.AsyncQueue$$Lambda$4.call(com.google.firebase:firebase-firestore@@20.1.0)
at com.google.firebase.firestore.util.AsyncQueue.lambda$enqueue$3(com.google.firebase:firebase-firestore@@20.1.0:287)
at com.google.firebase.firestore.util.AsyncQueue$$Lambda$3.run(com.google.firebase:firebase-firestore@@20.1.0)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:423)
at java.util.concurrent.FutureTask.run(FutureTask.java:237)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:154)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:269)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1113)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:588)
at com.google.firebase.firestore.util.AsyncQueue$DelayedStartFactory.run(com.google.firebase:firebase-firestore@@20.1.0:205)
at java.lang.Thread.run(Thread.java:818)
07-11 08:02:11.863 27666-27685/com.app.android.name W/art: Suspending all threads took: 62.326ms
07-11 08:02:11.893 27666-27685/com.app.android.name I/art: Background sticky concurrent mark sweep GC freed 78605(5MB) AllocSpace objects, 48(2MB) LOS objects, 18% free, 28MB/35MB, paused 63.394ms total 143.944ms
07-11 08:02:11.963 27666-28056/com.app.android.name V/NativeCrypto: Registering com/google/android/gms/org/conscrypt/NativeCrypto's 284 native methods...
07-11 08:02:12.073 27666-28056/com.app.android.name I/art: Rejecting re-init on previously-failed class java.lang.Class
07-11 08:02:12.083 27666-28056/com.app.android.name I/art: Rejecting re-init on previously-failed class java.lang.Class
07-11 08:02:12.083 27666-28056/com.app.android.name I/art: Rejecting re-init on previously-failed class java.lang.Class
07-11 08:02:12.083 27666-28056/com.app.android.name I/art: Rejecting re-init on previously-failed class java.lang.Class
07-11 08:02:12.133 27666-28056/com.app.android.name I/ProviderInstaller: Installed default security provider GmsCore_OpenSSL

[Laguses]

I have the exact problem as you, I'm running out of patience for over 1 week now and I still can't find a fix for this conflict between Firestore internal protoLite and other dependencies that use other versions of protobuf

https://stackoverflow.com/questions/56993238/android-studio-gradle-exclude-is-not-working-as-intended

[chingor13]

The core of the issue is that protobuf-java and protobuf-lite provide the same classes with different implementations, and that proto-google-common-protos do the same protolite-well-known-types. Firebase pulls in the lite flavor of protobuf whereas google-cloud-java provided libraries pull in the full protobuf-java versions.

Since it appears you are using android, you likely want to keep the protobuf-lite and protolite-well-known-types and excluding protobuf-java and proto-google-common-protos.

Also note that the google-cloud-X libraries are not guaranteed to fully support android at this time although they may in fact work.

[h-amg]

Thanks @chingor13 that was insightful, I tried excluding protobuf-java and proto-google-common-protos. I get missing com.google.protobuf classes error. I think this is a compatibility issue as you alluded. Not giving up yet though.

[Laguses]

@h-amg
I had a workaround, I tested it and it solved the problem.
The problem that some libraries depend on proto-java and Firestore depends on the proto-lite so that's why duplicate classes error appear because they both have some classes with the same names and package.
Excluding modules 'protolite-well-known-types' and 'protobuf-lite' from Firestore won't fix the problem because the project will build successfully but at run-time, the app will crash when making any Firestore operation.
Also excluding proto-java from other libraries will do the same crashes at run-time.
So the workaround is to combine the 3 jars 'protobuf-lite', 'protobuf-java' and 'protolite-well-known-types' into one jar
1) replace extension ".jar" with ".zip" for all the 3 jars and unzip each of them
2) copy "com" folder from the unzipped protobuf-java folder and paste it into the protobuf-lite folder, duplicate files name popup will appear to tell you that they are files with the same name choose to ignore as we don't need to override these files we need only to copy the extra files from protobuf-java to proto-lite
3) copy "com" folder from the unzipped protolite-well-known-types folder and paste it into the protobuf-lite folder
4) compress "protobuf-lite" folder
5) extra step if you are using macOS, the system creates ".__MACOSX" folder inside the final zip file which causes errors later while using the final jar, so we have to remove it, use this command

$ zip -d your-archive.zip "__MACOSX*"

6) replace extension ".zip" with ".jar"
7) add the jar to your android project as library

Now you can use this jar for both Firestore and any other library that depends on proto-java.
No need to exclude 'protolite-well-known-types' and 'protobuf-lite' from Firestore, just exclude the proto-java from all other libraries.

[h-amg]

@Laguses thanks so much, but the google java client library files are all .java files that import the .jar files i think. just to put things in context i use android studio and the buil.gradle file imports these libraries from their repositories and implements them. how do i get to protobuf-lite, protobuf-java, and protolite-well-known-type .jar files?

[Laguses]

@h-amg
You can get them from "External Libraries" section in your Android project by following these steps
1- Select "Project" as a view to display all the project files

2- Expand "External Libraries" section

3- Search for the needed library then right-click on the jar file and press on "Reveal In Finder" option in the popup menu to open the folder that contains it

[h-amg]

@Laguses , i was able to create the new jar file but:

1. I got duplicate classes error between the new jar file and protobuf-lite, protolite-well-known-type and proto-google-common .jar.

2. When I tried excluding all the above classes and merging com files from proto-google-common with the new jar then adding it back to the project as a library, I got a can't open zip file (new jar file) error.

[Laguses]

@h-amg
Send me your Gradle dependencies tree to check where the conflict comes from.
check this link if don't know how to do that
https://stackoverflow.com/questions/21645071/using-gradle-to-find-dependency-tree/35235229#35235229

[athakor]

@h-amg can you add below workaround in your app level build.gradle file. I have tested it and its working fine for me without crashing app or duplicate classes.

android {
...
...
configurations.all {
exclude group:'com.google.api.grpc',module:'proto-google-common-protos'
exclude group: 'com.google.protobuf', module: 'protobuf-java'
exclude group: 'com.google.guava',module: 'guava-jdk5'
}
packagingOptions {
exclude 'META-INF/INDEX.LIST'
exclude 'META-INF/DEPENDENCIES'
}
lintOptions {
checkReleaseBuilds false
}
}

[h-amg]

@Laguses thamks mate, i have decided to do away with firebase and use mongodb stitch instead. thanks so much for your help. i really appreciate it.

@athakor i think i tried your config before and it didn't work. but thanks anyways!

[shiran82]

I have the same issue, @h-amg I added what you suggested and there are no duplicates, however i still get: Verifier rejected class com.google.firestore.v1.WriteResponse
I tried adding protobuf 'com.google.protobuf:protobuf-java:3.6.1' and it works but then when i try to build in release mode it doesn't compile
Any suggestions?

[h-amg]

@shiran82 yeah thats what i got too, unfortunate i could't figure out how to get it to work. The fact is dialogflow is not supported for android so you are pretty much on your own. i ended switching to mongodb stitch.

[carlosrsabreu]

@h-amg did you tried to use the Realtime Database instead of the Cloud Firestore? It's working fine on my project and doesn't give the errors that you have: duplicate classes.

[kaniundungu]

Here's a great workaround for those who can't wait for an official fix:

https://github.com/protocolbuffers/protobuf/issues/7094#issuecomment-586532060

The idea behind it is to 'fatjar' the proto artifacts in a sequence that overwrites classes that do not work well with firesbase firestore or inappmessaging.

[ansh-mehta]

@Laguses I followed the steps you have mentioned but because I have copied the com folder from protolite-well-known-types, and not excluded it, I still get Duplicate class error between new protolite. jar and protolite-well-known-types.jar Can you please help me out? I am stuck on this for more than a week now.

@athakor I have added the exclusion but still, gives me duplicate class error

[aquadize]

The workarround just work if you are using protobuf-javalite. The title is confusing, be careful.

I am trying to use protobuf-java 3.11.4 (full version, no lite) with Firestore 21.4.3 and it seems to be impossible, due to internal incompatibilities with different protobuf versions.

I have to use protobuf-java because my app needs Any type message.

I would like to know if there is a way to accomplish that. If it is not, I will move away from Firestore.

[rakeshprojects]

lintOptions {
checkReleaseBuilds false
}

Thank you man!!!!! its working............i got stuck for last few days and you really help me. Thank you again

[giovappp]

I have the same problem, I followed some solutions proposed here, only I could not solve the problem of duplicate classes. Anyone have a solution?

[Yoginee47]

@h-amg I have followed some of the solutions given here.. I dont have any compile issues like duplicate classes but app crashes when performing any firestore operation..Is there a way to use both firestore and dialogflow at the same time or i have to change my db? If anyone could help really appreciate it..

[h-amg]

@Yoginee47 I had to change my db to mongodb at the end to resolve this issue. I hope this helps

[Yoginee47]

@h-amg Firestore provides the REST API to access the database... Should we consider using that in place of firestore sdk?
Would appreciate your views on this..

[h-amg]

@Yoginee47 hi, I think that will help. Since you no longer use the FIRE STORE SDK and therefore avoid the dependencies conflict.