Jackson-databind: Json serialization fails or a specific case that contains generics & static methods with generic parameters

[lhotari]

Reproducing the issue

I wasn't able to analyze the exact reason, however I have created a repository that reproduces the issue.
The problem seem to occur only with a certain structure and I wasn't yet able to isolate it further.

The test case that reproduces the issue
References CloudEvent, CloudEventImpl and AttributesImpl classes from CloudEvents Java SDK 1.3.0 .

to reproduce:

git clone https://github.com/lhotari/jackson-bug-2020-08-18
cd jackson-bug-2020-08-18
./gradlew test

fails with exception.

com.github.lhotari.jacksonbug.JacksonBugTest > reproduceSerializerBug() FAILED
    com.fasterxml.jackson.databind.JsonMappingException: Strange Map type java.util.Map: cannot determine type parameters (through reference chain: com.github.lhotari.jacksonbug.JacksonBugTest$MyValue["events"]->java.util.Collections$SingletonList[0]->io.cloudevents.v1.CloudEventImpl["attributes"])
        at com.fasterxml.jackson.databind.JsonMappingException.from(JsonMappingException.java:295)
        at com.fasterxml.jackson.databind.SerializerProvider.reportMappingProblem(SerializerProvider.java:1309)
        at com.fasterxml.jackson.databind.SerializerProvider._createAndCacheUntypedSerializer(SerializerProvider.java:1447)
        at com.fasterxml.jackson.databind.SerializerProvider.findValueSerializer(SerializerProvider.java:562)
        at com.fasterxml.jackson.databind.ser.impl.UnwrappingBeanPropertyWriter._findAndAddDynamic(UnwrappingBeanPropertyWriter.java:211)
        at com.fasterxml.jackson.databind.ser.impl.UnwrappingBeanPropertyWriter.serializeAsField(UnwrappingBeanPropertyWriter.java:102)
        at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:755)
        at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:178)
        at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serializeContents(IndexedListSerializer.java:119)
        at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:79)
        at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:18)
        at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:728)
        at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:755)
        at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:178)
        at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider._serialize(DefaultSerializerProvider.java:480)
        at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:319)
        at com.fasterxml.jackson.databind.ObjectMapper._writeValueAndClose(ObjectMapper.java:4407)
        at com.fasterxml.jackson.databind.ObjectMapper.writeValueAsString(ObjectMapper.java:3661)
        at com.github.lhotari.jacksonbug.JacksonBugTest.reproduceSerializerBug(JacksonBugTest.java:39)

        Caused by:
        java.lang.IllegalArgumentException: Strange Map type java.util.Map: cannot determine type parameters
            at com.fasterxml.jackson.databind.type.TypeFactory._mapType(TypeFactory.java:1178)
            at com.fasterxml.jackson.databind.type.TypeFactory._fromWellKnownClass(TypeFactory.java:1471)
            at com.fasterxml.jackson.databind.type.TypeFactory._fromClass(TypeFactory.java:1414)
            at com.fasterxml.jackson.databind.type.TypeFactory.constructType(TypeFactory.java:705)
            at com.fasterxml.jackson.databind.introspect.AnnotatedClass.resolveType(AnnotatedClass.java:229)
            at com.fasterxml.jackson.databind.introspect.AnnotatedMethod.getParameterType(AnnotatedMethod.java:143)
            at com.fasterxml.jackson.databind.introspect.AnnotatedWithParams.getParameter(AnnotatedWithParams.java:86)
            at com.fasterxml.jackson.databind.introspect.POJOPropertiesCollector._addCreators(POJOPropertiesCollector.java:500)
            at com.fasterxml.jackson.databind.introspect.POJOPropertiesCollector.collectAll(POJOPropertiesCollector.java:327)
            at com.fasterxml.jackson.databind.introspect.POJOPropertiesCollector.getJsonValueAccessor(POJOPropertiesCollector.java:203)
            at com.fasterxml.jackson.databind.introspect.BasicBeanDescription.findJsonValueAccessor(BasicBeanDescription.java:252)
            at com.fasterxml.jackson.databind.ser.BasicSerializerFactory.findSerializerByAnnotations(BasicSerializerFactory.java:396)
            at com.fasterxml.jackson.databind.ser.BeanSerializerFactory._createSerializer2(BeanSerializerFactory.java:216)
            at com.fasterxml.jackson.databind.ser.BeanSerializerFactory.createSerializer(BeanSerializerFactory.java:165)
            at com.fasterxml.jackson.databind.SerializerProvider._createUntypedSerializer(SerializerProvider.java:1474)
            at com.fasterxml.jackson.databind.SerializerProvider._createAndCacheUntypedSerializer(SerializerProvider.java:1442)
            ... 16 more

Comments

• Test passes with Jackson 2.11.1 version, but fails with 2.11.2 .
• Diff between Jackson 2.11.2 and 2.11.1

Changes in TypeFactory.constructType in 2.11.2 for https://github.com/FasterXML/jackson-databind/issues/2796 might have caused the change in behavior.
Looks like the problem that https://github.com/FasterXML/jackson-databind/commit/910edfb634f55cdb8d78ac7d9caf00d8133a11e6 fixes could have been similar.

By debugging it can be seen that resolving parameter types with the bindings in AnnotedClass doesn't produce the correct result.

Another observation here is that the method that is been processed by Jackson is produced by a lambda. The method name in the repro case is private static void io.cloudevents.v1.AttributesImpl.lambda$marshal$3(java.util.Map,java.time.ZonedDateTime)

The failure seems to happen when Jackson tries to resolve the parameters for this method generated by the lambda defined at this location:

https://github.com/cloudevents/sdk-java/blob/361a34cc639ddaa75b2a5080f117fc282be7625b/api/src/main/java/io/cloudevents/v1/AttributesImpl.java#L172-L173

[dariuszkuc]

I just hit the same error. My simple test case

@JsonIgnoreProperties(ignoreUnknown = true)
@JsonInclude(JsonInclude.Include.NON_NULL)
data class GenericResponse<T>(
    val data: T? = null,
    val extensions: Map<Any, Any>? = null
)

data class MyData(val foo: Int)

fun main() {
    val mapper = jacksonObjectMapper()
    val input =
        """{"data":{"foo":1}, "extensions":{"bar":2}}"""

    val response: GenericResponse<MyData> = mapper.readValue(input)
    println(response)
}

Above works fine in with Jackson 2.11.1 but fails in 2.11.2 with following exception

Exception in thread "main" com.fasterxml.jackson.databind.exc.InvalidDefinitionException: Strange Map type java.util.Map: cannot determine type parameters
 at [Source: (String)"{"data":{"foo":1}, "extensions":{"bar":2}}"; line: 1, column: 1]
    at com.fasterxml.jackson.databind.exc.InvalidDefinitionException.from(InvalidDefinitionException.java:62)
    at com.fasterxml.jackson.databind.deser.BeanDeserializerFactory.buildBeanDeserializer(BeanDeserializerFactory.java:227)
    at com.fasterxml.jackson.databind.deser.BeanDeserializerFactory.createBeanDeserializer(BeanDeserializerFactory.java:143)
    at com.fasterxml.jackson.databind.deser.DeserializerCache._createDeserializer2(DeserializerCache.java:414)
    at com.fasterxml.jackson.databind.deser.DeserializerCache._createDeserializer(DeserializerCache.java:349)
    at com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCache2(DeserializerCache.java:264)
    at com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCacheValueDeserializer(DeserializerCache.java:244)
    at com.fasterxml.jackson.databind.deser.DeserializerCache.findValueDeserializer(DeserializerCache.java:142)
    at com.fasterxml.jackson.databind.DeserializationContext.findRootValueDeserializer(DeserializationContext.java:491)
    at com.fasterxml.jackson.databind.ObjectMapper._findRootDeserializer(ObjectMapper.java:4711)
    at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4520)
    at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3466)
    at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3449)
    at example.ApplicationKt.main(Application.kt:25)
    at example.ApplicationKt.main(Application.kt)

[MartinTeeVarga]

Having similar issues. When I change @dariuszkuc 's example this way, it works:

data class GenericResponse<T>(
    val data: T? = null,
    val extensions: Map<Any, Any>? = null
)

data class MyData(val foo: Int)

fun main() {
    val typeRef = GenericResponse(MyData(1), emptyMap())::class.java

    val mapper = jacksonObjectMapper()
    val input =
        """{"data":{"foo":1}, "extensions":{"bar":2}}"""

    val response: GenericResponse<MyData> = mapper.readValue(input, typeRef)
    println(response)
}

I believe the problem might be somewhere in the TypeFactory refactoring.

[cowtowncoder]

Quick note: unfortunately Kotlin-based examples won't help a lot when working on databind (although they are obviously fine for Kotlin module), as I can not use them for unit tests.
It would be super helpful if one of simpler examples could be "javafied": I can follow the example but am afraid my translation might lose something (f.ex not sure whether Map<Any, Any> would exactly translate to Map<?, ?> or not -- sometimes differences matter).

But maybe I can simplify the original test which is Java only I think.

[cowtowncoder]

I hope to look into this soon -- I suspect that @lhotari is correct about issue related to (that is, fix that may have caused regression). I am guessing that things might have worked as a side-effect.

Was able to test against 2.12.0-SNAPSHOT with following def in build.gradle:

repositories {
    mavenCentral()
    maven {
        url "https://oss.sonatype.org/content/repositories/snapshots"
    }
}

and looks like the problem still persists (which is not entirely unexpected).

What I need is then just self-contained version to add as unit test; however, should also be able to import existing project into IDE for some investigation once I have time.

[MartinTeeVarga]

Breaking Java code similar to the above Kotlin example would be:

public class ObjectMapperGenericClassTest extends BaseTest {

    static final String JSON = "{ \"field\": { \"number\": 1 }, \"map\": { \"key\": \"value\" } }";

    static class GenericEntity<T> {
        T field;

        Map map;

        public void setField(T field) {
            this.field = field;
        }

        public T getField() {
            return field;
        }

        public Map getMap() {
            return map;
        }

        public void setMap(Map map) {
            this.map = map;
        }
    }

    static class SimpleEntity {
        Integer number;

        public void setNumber(Integer number) {
            this.number = number;
        }

        public Integer getNumber() {
            return number;
        }
    }

    public void test() throws Exception {
        ObjectMapper m = new ObjectMapper();
        GenericEntity<SimpleEntity> genericEntity = m.readValue(JSON, new TypeReference<GenericEntity<SimpleEntity>>() {});
    }
}

It works in 2.11.2 if I add any type parameter to the Map and it also works if I remove the type parameter <T> and the whole SimpleEntity and keep the Map without type parameter:

    static final String JSON = "{ \"map\": { \"key\": \"value\" } }";

    static class GenericEntity {

        Map map;

        public Map getMap() {
            return map;
        }

        public void setMap(Map map) {
            this.map = map;
        }
    }

    public void test throws Exception {
        ObjectMapper m = new ObjectMapper();
        GenericEntity genericEntity = m.readValue(JSON, new TypeReference<GenericEntity>() {});
    }

[MartinTeeVarga]

In my real case I am actually seeing:

java.lang.ClassCastException: java.util.LinkedHashMap cannot be cast to MyClassAsTypeParameterOfAnotherClass

I am still working on a minimal example for that.

[lhotari]

Thanks for the repros @dariuszkuc and @MartinTeeVarga . It seems that the issue with Kotlin and the example that @MartinTeeVarga provided are somewhat different than what I have. In my case, the issue is in serialization. I assume that the problems are related.
@cowtowncoder I now had some time to isolate the issue from the repository that I originally provided. This is the simplest case I was able to strip it down to. I made some observations that if I change on of the 3 things: "Entity -> Entity", "remove public static Attributes dummyMethod(Map attributes)" or "remove @JsonUnwrapped", the test will pass and JSON serialization won't fail. It seems that the JsonUnwrapped together with generics and the static "dummyMethod" (in the repro) trigger the issue that I have faced.

public class JacksonBugIsolatedTest {
    static final class Wrapper {
        // if Entity<?> -> Entity , the test passes
        private final List<Entity<?>> entities;

        @JsonCreator
        public Wrapper(List<Entity<?>> entities) {
            this.entities = entities;
        }

        public List<Entity<?>> getEntities() {
            return this.entities;
        }
    }

    public static class Entity<T> {
        @JsonIgnore
        private final Attributes attributes;

        private final T data;

        public Entity(Attributes attributes, T data) {
            this.attributes = attributes;
            this.data = data;
        }

        // if @JsonUnwrapped is removed, the test passes
        @JsonUnwrapped
        public Attributes getAttributes() {
            return attributes;
        }

        public T getData() {
            return data;
        }

        @JsonCreator
        public static <T> Entity<T> create(Attributes attributes, T data) {
            return new Entity<>(attributes, data);
        }
    }

    public static class Attributes {
        private final String id;

        public Attributes(String id) {
            this.id = id;
        }

        public String getId() {
            return id;
        }

        @JsonCreator
        public static Attributes create(String id) {
            return new Attributes(id);
        }

        // if this method is removed, the test passes
        public static Attributes dummyMethod(Map attributes) {
            return null;
        }
    }

    // this test passes with Jackson 2.11.1, but fails with Jackson 2.11.2
    @Test
    public void reproduceSerializerBug() throws JsonProcessingException {
        ObjectMapper objectMapper = new ObjectMapper().findAndRegisterModules();
        Entity<String> entity = new Entity<>(new Attributes("id"), "hello");
        Wrapper val = new Wrapper(Collections.singletonList(entity));
        // fails with com.fasterxml.jackson.databind.JsonMappingException: Strange Map type java.util.Map: cannot determine type parameters (through reference chain: com.github.lhotari.jacksonbug.JacksonBugIsolatedTest$Wrapper["entities"]->java.util.Collections$SingletonList[0]->com.github.lhotari.jacksonbug.JacksonBugIsolatedTest$Entity["attributes"])
        System.out.println(objectMapper.writeValueAsString(val));
    }
}

This test class is also available in the repro repo: https://github.com/lhotari/jackson-bug-2020-08-18/blob/master/src/test/java/com/github/lhotari/jacksonbug/JacksonBugIsolatedTest.java

Also tried to add the test to jackson-databind: https://github.com/lhotari/jackson-databind/commit/ecdfd6c73e02d83bdc8f03de8246a730dc585d80

[cowtowncoder]

Thank you everyone for help here! Yes, it is possible there are multiple issues and they might need separate fixes (or at least test cases). I'll first focus on @lhotari 's case.

[cowtowncoder]

Hmmh. findAndRegisterModules() is bit problematic as it could load anything -- looking at code, I am guessing it probably loads jackson-module-parameter-names since otherwise parameter names are not visible. But I should be able to add @JsonProperty annotations to remove that dependency.

[cowtowncoder]

Ok, no problem, I can reproduce this. What appears to be the problem is that type bindings passed are for enclosing class, not for type resolved; and number of parameters mismatches (it is all around wrong value but might "work" otherwise since no actual binding information is available just placeholder).

[cowtowncoder]

Ok, had to make a bit more fundamental change than what I was hoping for: basically clear out TypeBindings passed for static factory methods. This seems correct to me, although I am not quite sure why failure occurred (meaning that I suspect there is some other problem affecting resolution). So I don't think change is wrong, I just didn't think it should have affected anything.
No new test failures, although would not be surprised if some edge case somewhere might have changed.

Looking forward to other test failures, if @MartinTeeVarga or @dariuszkuc can provide one.

[cowtowncoder]

Will file a separate issue for @MartinTeeVarga 's issue: that is similar, but does not require static method to trigger.
It also covers deserialization side which is good, extending coverage a bit.

[cowtowncoder]

Follow-up issue: #2846.

[dariuszkuc]

@cowtowncoder any eta on the 2.11.3 release? I would like to test it out but would rather use released version

[cowtowncoder]

@dariuszkuc I need to balance a few things so no firm answer: hoping to get 2.12.0-rc1 out first, followed by 2.11.3. Release takes a while (couple of hours) and it has only been bit over month since 2.11.2.
At the same time I do realize that this is actually a significant issue for many users so it is high priority.

With all that, hopefully by end of September or first week of October.

[dariuszkuc]

just to close up the loop -> 2.11.3 does solve my issue and seems to be working fine.

[cowtowncoder]

@dariuszkuc thank you for confirming this.

[superdurszlak]

Hi @cowtowncoder ,
I am still able to reproduce com.fasterxml.jackson.databind.JsonMappingException: Strange Map type exception on io.cloudevents.v1.CloudEventImpl["attributes"] attribute when using ObjectWriter#writeValueAsBytes:

class CloudEventSerializer : Serializer<CloudEventImpl<Foo>> {
    override fun serialize(topic: String?, data: CloudEventImpl<Foo>?): ByteArray = ObjectMapper()
        .writerFor(object : TypeReference<CloudEventImpl<Foo>>() {})
        .writeValueAsBytes(data)
}

The problem persists for following versions:

• CloudEvents 1.3.0
• Kotlin 1.3.72 and 1.4.10 alike
• Jackson 2.11.1 through 2.11.3

Interestingly, when one skips ObjectMapper#writerFor and uses ObjectMapper#writeValueAsBytes instead, the problem is gone with newer Jackson releases as reported by others.

[cowtowncoder]

@superdurszlak I would need a full, stand-alone reproduction. If that is possible, please file a new issue and I can take a look.

The part about writerFor() vs writeValueAsBytes() is interesting, but not totally strange assuming writerFor() passes explicit type, whereas writeValueAsBytes() has to do with type-erased Class of value. So types, resolution are different.