Jackson-databind: Android Desugaring breaks serializating with NoClassDefFoundError

[retrodaredevil]

Describe the bug
When serializing with ObjectMapper#writeValueAsString() (and possibly other serialization methods) when Android's new desugaring feature is enabled (coreLibraryDesugaringEnabled true), there's a NoClassDefFoundError for java.util.function.Consumer. This does not occur when android's desugaring is disabled.

Stacktrace:

java.lang.NoClassDefFoundError: java.util.function.Consumer
        at libcore.reflect.InternalNames.getClass(InternalNames.java:55)
        at java.lang.Class.getDexCacheType(Class.java:479)
        at java.lang.reflect.ArtMethod.getDexCacheType(ArtMethod.java:236)
        at java.lang.reflect.ArtMethod.getParameterTypes(ArtMethod.java:176)
        at java.lang.reflect.Method.getParameterTypes(Method.java:174)
        at java.lang.Class.getDeclaredMethods(Class.java:802)
        at com.fasterxml.jackson.databind.util.ClassUtil.getClassMethods(ClassUtil.java:1172)
        at com.fasterxml.jackson.databind.introspect.AnnotatedMethodCollector._addMemberMethods(AnnotatedMethodCollector.java:117)
        at com.fasterxml.jackson.databind.introspect.AnnotatedMethodCollector.collect(AnnotatedMethodCollector.java:49)
        at com.fasterxml.jackson.databind.introspect.AnnotatedMethodCollector.collectMethods(AnnotatedMethodCollector.java:40)
        at com.fasterxml.jackson.databind.introspect.AnnotatedClass._methods(AnnotatedClass.java:382)
        at com.fasterxml.jackson.databind.introspect.AnnotatedClass.memberMethods(AnnotatedClass.java:322)
        at com.fasterxml.jackson.databind.introspect.POJOPropertiesCollector._addMethods(POJOPropertiesCollector.java:555)
        at com.fasterxml.jackson.databind.introspect.POJOPropertiesCollector.collectAll(POJOPropertiesCollector.java:323)
        at com.fasterxml.jackson.databind.introspect.POJOPropertiesCollector.getJsonValueAccessor(POJOPropertiesCollector.java:203)
        at com.fasterxml.jackson.databind.introspect.BasicBeanDescription.findJsonValueAccessor(BasicBeanDescription.java:252)
        at com.fasterxml.jackson.databind.ser.BasicSerializerFactory.findSerializerByAnnotations(BasicSerializerFactory.java:396)
        at com.fasterxml.jackson.databind.ser.BasicSerializerFactory.buildCollectionSerializer(BasicSerializerFactory.java:704)
        at com.fasterxml.jackson.databind.ser.BasicSerializerFactory.buildContainerSerializer(BasicSerializerFactory.java:646)
        at com.fasterxml.jackson.databind.ser.BeanSerializerFactory._createSerializer2(BeanSerializerFactory.java:196)
        at com.fasterxml.jackson.databind.ser.BeanSerializerFactory.createSerializer(BeanSerializerFactory.java:165)
        at com.fasterxml.jackson.databind.SerializerProvider._createUntypedSerializer(SerializerProvider.java:1474)
        at com.fasterxml.jackson.databind.SerializerProvider._createAndCacheUntypedSerializer(SerializerProvider.java:1442)
        at com.fasterxml.jackson.databind.SerializerProvider.findPrimaryPropertySerializer(SerializerProvider.java:652)
        at com.fasterxml.jackson.databind.ser.impl.PropertySerializerMap.findAndAddPrimarySerializer(PropertySerializerMap.java:72)
        at com.fasterxml.jackson.databind.ser.BeanPropertyWriter._findAndAddDynamic(BeanPropertyWriter.java:896)
        at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:706)
        at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:755)
        at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:178)
        at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:728)
        at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:755)
        at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:178)
        at com.fasterxml.jackson.databind.ser.std.CollectionSerializer.serializeContentsUsing(CollectionSerializer.java:171)
        at com.fasterxml.jackson.databind.ser.std.CollectionSerializer.serializeContents(CollectionSerializer.java:116)
        at com.fasterxml.jackson.databind.ser.std.CollectionSerializer.serialize(CollectionSerializer.java:107)
        at com.fasterxml.jackson.databind.ser.std.CollectionSerializer.serialize(CollectionSerializer.java:25)
        at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:728)
        at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:755)
        at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:178)
        at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider._serialize(DefaultSerializerProvider.java:480)
        at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:319)
        at com.fasterxml.jackson.databind.ObjectMapper._configAndWriteValue(ObjectMapper.java:4374)
        at com.fasterxml.jackson.databind.ObjectMapper.writeValueAsString(ObjectMapper.java:3629)
        at me.retrodaredevil.solarthing.android.prefs.saving.JacksonProfileHolder.getProfile(JacksonProfileHolder.kt:21)
        at me.retrodaredevil.solarthing.android.prefs.saving.BasicProfileManager.getActiveUUID(BasicProfileManager.kt:50)
        at me.retrodaredevil.solarthing.android.activity.ProfileHeaderHandler.<init>(ProfileHeaderHandler.kt:30)
        at me.retrodaredevil.solarthing.android.activity.SettingsActivity.onCreate(SettingsActivity.kt:83)
        at android.app.Activity.performCreate(Activity.java:6288)
        at android.app.Instrumentation.callActivityOnCreate(Instrumentation.java:1119)
        at android.app.ActivityThread.performLaunchActivity(ActivityThread.java:2646)
        at android.app.ActivityThread.handleLaunchActivity(ActivityThread.java:2758)
        at android.app.ActivityThread.access$900(ActivityThread.java:177)
        at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1448)
        at android.os.Handler.dispatchMessage(Handler.java:102)
        at android.os.Looper.loop(Looper.java:145)
        at android.app.ActivityThread.main(ActivityThread.java:5942)
        at java.lang.reflect.Method.invoke(Native Method)
        at java.lang.reflect.Method.invoke(Method.java:372)
        at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:1399)
        at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1194)
    Caused by: java.lang.ClassNotFoundException: Didn't find class "java.util.function.Consumer" on path: DexPathList[[zip file "/data/app/me.retrodaredevil.solarthing.android-1/base.apk"],nativeLibraryDirectories=[/vendor/lib, /system/lib]]
        at dalvik.system.BaseDexClassLoader.findClass(BaseDexClassLoader.java:56)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:511)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:469)
        at libcore.reflect.InternalNames.getClass(InternalNames.java:53)
        ... 59 more
        Suppressed: java.lang.ClassNotFoundException: java.util.function.Consumer
W/System.err:     at java.lang.Class.classForName(Native Method)
        at java.lang.BootClassLoader.findClass(ClassLoader.java:781)
        at java.lang.BootClassLoader.loadClass(ClassLoader.java:841)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:504)
            ... 61 more
        Caused by: java.lang.NoClassDefFoundError: Class not found using the boot class loader; no stack available

To Reproduce
Enable Android's desugaring and try to serialize something on an older phone (this error doesn't happen on newer phones)

Expected behavior
Enabling desugaring should not break Jackson serialization.

Versions
Kotlin: 1.3.72
Jackson-module-kotlin: 2.11.0, 2.11.1
Jackson-databind: 2.11.0, 2.11.1, 2.11.2-SNAPSHOT

Additional context
If I disable Android's desugaring, this error goes away. If you need more information, I'm happy to provide the class which I tried to serialize or the properties of the ObjectMapper. The objects I'm trying to serialize involve a few nested objects with generics, so posting the classes being serialized would involve posting a lot of classes. This is using Kotlin, but I don't believe that the Jackson-module-kotlin is causing this.

I don't really understand why Jackson feels the need to try and find the Consumer interface just because it might exist. I'd like to enable Android's new desugaring features but I don't really understand why this exception seems to be a side effect of doing that.

[cowtowncoder]

Jackson will try to resolve all super types to find all relevant annotations, both direct ones and mix-in annotations. There isn't really any way around that at the moment.
It seems odd that such interfaces are exposed, however; shouldn't those be removed by the desugaring processor?

I think it has been suggested that failure to introspect a type should be handled quietly, but I am not sure this would be easy to implement (mostly worried about hiding real issues), esp. since there seems to be legit Class reference and problem only manifests itself when trying to introspect method definitions.

Another possible approach would be to be able to somehow instruct introspector to completely ignore certain (types of) interface types -- but challenge here would be that of how identify them, and especially without having to maintain a big list of such types.
But facility that would allow even just use of such list (names of classes to skip, essentially; perhaps just interface types) could allow writing modules for this purpose, either by users, or maybe as 3rd party extensions for Android developers.

[retrodaredevil]

It seems odd that such interfaces are exposed, however; shouldn't those be removed by the desugaring processor?

Yeah, this is very odd. I don't actually know how they're getting exposed. I'd need to do some more debugging to figure out what's actually going on. I would think they should be removed by the desugaring processor, but I'm confused why enabling the core library desugaring caused this to happen in the first place (since these are Kotlin classes I'm serializing anyways). I was hoping there would be something to do without diving into every detail of the classes I'm serializing.

[cowtowncoder]

I think something similar has been reported in the past (cases where Android SDK ends up exposing a class that "is not there"), so it might not be isolated incident. Would be good to figure out how to avoid or alleviate the problem of course, just not sure how yet. Some sort of extensibility to make it at least possible to forcibly ignore introspection might be start, but for that would probably need some way of reproduction for unit test(s). But that might get tricky too... although now that I said that, I am thinking there was a test that did this somehow, compiling against jar but excluding from test runtime or such.

[cramptonism]

Also experiencing this issue after enabling coreLibraryDesugaringEnabled. Thanks for the detailed report.

[retrodaredevil]

Today I decided to see if updating dependencies would magically fix this problem. It didn't. I went down a rabbit hole of searching for solutions again, so here's some info for future reference.

It's worth noting that others are having this problem in a bunch of other random places:

• https://github.com/googlemaps/android-maps-utils/issues/787
• https://stackoverflow.com/questions/63688655/android-gradle-build-plugin-4-0-0-r8-desugaring-not-working-on-api-19 (This one is a different error, but worth noting)
• https://stackoverflow.com/questions/64969917 (Even more unrelated, but I'll still link it)

I also started changing different things to see if I could get different results. I changed Kotlin's jvmTarget to 1.6 which had no effect (my reasoning was that I thought Kotlin might try and throw java.util.function classes in a few places where it didn't need to be if it's on Java 8. I'm not sure if that's even a thing, but I tried it anyway.)

The error is also very inconvenience because it doesn't even tell me what class or method is causing this. I'm not even sure how to start debugging this. I'd probably have to start with a small class to serialize and try to find the error. If I knew what class was causing this, I'd probably try to figure out a way to work around it. It would be nice if we could wrap the exception thrown in ClassUtil.getClassMethods() and provide some info about what cls is.

[cowtowncoder]

@retrodaredevil great idea on exception wrapping for specific fail. Just to make sure, error you see is NoClassDefFoundError? If so, can catch+rethrow, and get this in 2.11.4 -- I probably need to solve one other problem, then release that version (now that 2.12.0 is out and about).

[retrodaredevil]

Yeah the exception is the same one that's shown in my original post. You can see it doesn't currently have any info about what cls is.

[cowtowncoder]

@retrodaredevil Added code to throw IllegalArgumentException with name of class on which failing call was made: change is in 2.11 branch (and later), pushed 2.11.4-SNAPSHOT if you can use that (or locally build).
Will be in 2.11.4 which I might release over the weekend, in couple of days, but would be good to know if handling helps.

[retrodaredevil]

@cowtowncoder I just tried out the change, and it's almost what I want, but it doesn't actually work because the call return contextClass.getDeclaredMethods(); is not wrapped in a try-catch block. As you can see in the original post, that's the call which is shown in the stacktrace.

So replacing it with something like this should work:

try {
    return contextClass.getDeclaredMethods();
} catch (NoClassDefFoundError exception) {
    return _failGetClassMethods(cls, exception);
}

I also don't think there's a reason to catch a Throwable. I'm not aware of other exceptions that are possible to pop up.

The line that is causing problems is: https://github.com/FasterXML/jackson-databind/blob/83d3ec2a5309c70776c5db745a73164f4916258d/src/main/java/com/fasterxml/jackson/databind/util/ClassUtil.java#L1172

[cowtowncoder]

lol. Well, that's why it is best to try out snapshots :)

I missed the obvious part where the follow-up catch-block obv. wouldn't apply to that part, so will need to add it.
Also curious as to the existing work-around, which really looks wrong at many levels, but unfortunately need to leave it in place for now (I suspect it can't be working very well but fight for another day).

[retrodaredevil]

Thank you. That's very helpful. Here's the error I got:

     Caused by: com.fasterxml.jackson.databind.JsonMappingException: Failed on call to `getDeclaredMethods()` on class `kotlin.collections.EmptyList`, problem
(java.lang.NoClassDefFoundError) java/util/function/Consumer (through reference chain: 
me.retrodaredevil.solarthing.android.prefs.saving.ProfileManagerData["profiles"]->java.util.ArrayList[0]->
me.retrodaredevil.solarthing.android.prefs.saving.ProfileData["profile"]->me.retrodaredevil.solarthing.android.prefs.SolarProfile["voltage_timer_nodes"])

I wonder if there's a way to not have EmptyList present in what I'm serializing. I'll report back if I can find a work around.

EDIT: I was able to serialize successfully by replacing some emptyList() calls with Java's Collections.emptyList(). Now that I've figured out this work around, I might start to look for an actual fix. I think this probably needs to be fixed in Android itself as I suspect that calling emptyList().javaClass.declaredMethods would reproduce this bug.

Here's an issue with basically the same problem: https://issuetracker.google.com/issues/161409148 (Although now I'm not sure why using Java's emptyList() method works and Kotlin's doesn't). Also now that I think about it why is getDeclaredMethods being called on EmptyList? It should be serialized as a JSON array so I don't know why getDeclaredMethods needs to be called.

[cowtowncoder]

@retrodaredevil On call to getDeclaredMethods -- Jackson typically introspects all details of types it handles first (including annotation access, fields, methods, for super types too), passing resulting BasicBeanDescription for code that decides which serializer/deserializer to use. Not all of that information is needed in all cases but it is difficult to reliably determine a priori what is and what is not needed. For example, @JsonCreator added on constructors and potential factory methods is needed for collection types.

One way that could avoid problems like this would be to make more of access lazy: I think I did prototype with something like that (for jackson 3.0, mostly), to pass something like Provider<BasicBeanDescription> instead of description. But my goal with that was to try to reduce cold-state startup time and it did not seem to have much effect there. Something like that (esp. more granular lazy access) could avoid problems like these.
But what would be useful would be reproducible cases to test that this would avoid need to look inside specific types.

Another possible work-around in this case could be to have some means of injecting sort of placeholder BasicBeanDescription for affected type (EmptyList). This is done in BasicClassIntrospector for a small number of core types (String, int, long, boolean I think) as optimization; but if there was a way for modules to do that it could possibly also avoid failure.
Still, it's lots of trial and error, and I have learned that some of these workarounds come back & bite me in the ass in the long run ("why does my mix-in on ImmutableList constructor not work any more in 2.11.3?").

[cramptonism]

Issue no longer reproduces for me on 2.11.4, thanks!

[cowtowncoder]

@cramptonism That is somewhat surprising, if quite positive, development. :)

Thank you for sharing; at least anyone who ends up here knows that specific upgrade may help.

[vn01971]

I'm still having the issue with com.fasterxml.jackson.core:jackson-databind:2.11.4 on Android API 21 devices with desugaring enabled.

com.applitools.eyes.android.common.exceptions.EyesException: Failed to connect to server 
     FATAL EXCEPTION: Thread-461
com.applitools.eyes.android.common.exceptions.EyesException: Failed to connect to server 
    at com.applitools.eyes.android.common.network.RestClient$1.run(RestClient.java:112)
    at java.lang.Thread.run(Thread.java:818)
Caused by: com.fasterxml.jackson.databind.JsonMappingException: Failed on call to `getDeclaredMethods()` on class `com.applitools.eyes.android.common.Properties`,
problem: (java.lang.NoClassDefFoundError) java.util.function.Consumer (through reference chain:
com.applitools.eyes.android.common.SessionStartInfoBody["startInfo"]->com.applitools.eyes.android.common.SessionStartInfo["properties"])
    at com.fasterxml.jackson.databind.SerializerProvider.reportMappingProblem(SerializerProvider.java:1309)
    at com.fasterxml.jackson.databind.SerializerProvider._createAndCacheUntypedSerializer(SerializerProvider.java:1447)
    at com.fasterxml.jackson.databind.SerializerProvider.findPrimaryPropertySerializer(SerializerProvider.java:652)
    at com.fasterxml.jackson.databind.ser.impl.PropertySerializerMap.findAndAddPrimarySerializer(PropertySerializerMap.java:72)
    at com.fasterxml.jackson.databind.ser.BeanPropertyWriter._findAndAddDynamic(BeanPropertyWriter.java:896)
    at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:706)
    at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:755)
    at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:178)
    at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:728)
    at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:755)
    at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:178)
    at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider._serialize(DefaultSerializerProvider.java:480)
    at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:319)
    at com.fasterxml.jackson.databind.ObjectMapper._writeValueAndClose(ObjectMapper.java:4409)
    at com.fasterxml.jackson.databind.ObjectMapper.writeValueAsString(ObjectMapper.java:3663)
    at com.applitools.eyes.android.common.network.RestClient$1.run(RestClient.java:97)
    ... 1 more

[cowtowncoder]

Right, at this point we can see which class has problematic reference, but that in itself will not resolve the problem, just makes it more visible.

[rontho]

Anyone having an update on this one ? I'm experiencing the same issue, Android 21 with desugaring enable and com.fasterxml.jackson.core:jackson-databind:2.12.0.

This is very problematic...

[cowtowncoder]

@rontho If someone can figure out something, I'm sure they'll add a note here.