Kotlinx.serialization: Polymorphic serializer, ignore unregistered objects when deserializing list of polymorphic objects

[luca992]

I have made a serializer class for the json:api specificaiton.

@Serializable
data class JsonApiResponseItem(
        override val jsonapi: JsonApiVersion,
        override val included: List<@Polymorphic JsonApiObject>? = null,
        @Polymorphic val data: JsonApiObject): JsonApiResponse() {

}

included can return many different types of objects which I am successfully using the polymorphic feature to deserialize. But this only works if all possible object types are registered in the scope of class JsonApiObject for polymorphic serialization.

If any types are not registered an error like is thrown

io.ktor.client.call.ReceivePipelineException: Fail to run receive pipeline: 
kotlinx.serialization.SerializationException: example-type is not registered for polymorphic 
serialization in the scope of class api.jsonapi.JsonApiObject

Is it possible to ignore deserializing polymorphic type objects which are not registered for polymorphic serialization without throwing an error? This would really be helpful because it would prevent deserialization from throwing errors if any new objects added to the included list... and it would be convenient if don't actually need to deserialize some of the included objects

[sandwwraith]

I think this feature is more about to __skip deserialization of element in the list if an error occured__. Because polymorphic serializer does not know whether it is in the list now or not, and it has to return something (e.g. if the same error occurs when deserializing the data field from your example, there is no meaningful way to recover). Also, this requires from each encoder the support of __skipping malformed element__, so the list deserializer could start deserializing next one.

[luca992]

Yeah, that sounds right.
Maybe another option would be to allow this in nonstrict mode by allowing a backup serializer to be registered for polymorphic classes in SerializersModule.

maybe something along the lines of:

@Serializable
data class EmptyJsonApiObject : JsonApiObject()

Json(context = SerializersModule {
    polymorphic(JsonApiObject::class) {
        JsonApiObjectUser::class with JsonApiObjectUser.serializer()
        JsonApiObjectEmployee::class with JsonApiObjectEmployee.serializer()
        addBackUpSerializer(EmptyJsonApiObject.serializer())
    }
},
        configuration = JsonConfiguration(
                strictMode = false,
        )
)

//addBackUpSerializer would require EmptyJsonApiObject to extend JsonApiObject

So in nonStrict mode at least, instead of throwing kotlinx.serialization.SerializationException: example-type is not registered for polymorphic serialization in the scope of class api.jsonapi.JsonApiObject, it would attempt to de-serialize using EmptyJsonApiObject's deserializer. Which I think should work fine, because nonStrict allows deserializing to an empty class.

My suggestion sounds pretty hacky though. But might be something to consider until support for skipping malformed element can be added.

[outadoc]

I think this feature is more about to skip deserialization of element in the list if an error occured.

Hi— do you know if that's something that is possible at the moment? Or a planned feature? It sounds like a good solution for my use case.

[suushiemaniac]

I'd like to +1 this issue / feature request, as I am running into a very similar situation.

In my particular use case, I am working with a common interchange format for a sports event hosted at the association's website.

There is a list of competitors and the association supplies basic data (name, nationality). There is an option to include extension fields, which contain any kind of data -- but they can be synced back to the central association API.

For example, one application may use this to store T-Shirt sizes. Another application may need to register whether the person is left- or right-handed etc. When they store that kind of information, it automatically becomes available in the general API for the competitors' list.

I want to be able to write and parse (hence the @Serializable feature) my own extensions about god knows what without having to parse T-Shirt sizes or left-handedness preferences from other applications.

[pdvrieze]

@suushiemaniac I would say that the best option for your case is to create your own version of the Json format. The format is perfectly able to see that polymorphic serialization is needed and that no data type is needed. More importantly it may be able to use its own private way of determining the data type in the first place.

[suushiemaniac]

@pdvrieze Thanks for the suggestion! Can you please specify what you mean by:

create your own version of the Json format

Unfortunately I don't have any control over the JSON specification used by the competition association website. The only API that I have access to yields a result like this (abridged):

{
    "eventName": "Super Serialize Competition",
    "competitors": [
        {
            "name": "John Doe",
            "nationality": "Nepal",
            "extensions": [
                {
                    "id": "polymorphicFoo",
                    "data": {
                        "yes": true,
                        "no": false
                    }
                },
                {
                    "id": "polymorphicBar",
                    "data": [
                        "one",
                        "two",
                        "three"
                    ]
                },
                {
                    "id": "polymorphicMyStuff",
                    "data": {
                        "favoriteMusic": "Jazz",
                        "likesLasagna": true
                    }
                }
            ]
        },
        {
            "name": "Jane Doe",
            "nationality": "Canada",
            "extensions": [...]
        }
    ]
}

Here, only polymorphicMyStuff was created and specified by me. The other two extension objects were created and uploaded by entirely different people but I see the output when querying the API as well.

I would really prefer if there was a skip unknown polymorphic entity feature (that could be enabled/disabled by config) as suggested above.

[pdvrieze]

What I meant with format is basically an encoder/decoder pair. Looking at your data, you might however be able to get away with using an alternative (de)serializer instead, either for the list or the elements.

[suushiemaniac]

Thanks for the explanation! I think I get your idea, but I believe it is a much larger effort than simply extending the library to allow for passing a skipUnknown flag somewhere.

Effectively right now I have to copy/paste your entire ListLikeSerializer source code (since it's all sealed within the library) except for one tiny modification as outlined above. This seems extremely overkill.

[pdvrieze]

The trouble (also with SerialModule) for your case is that PolymorphicSerializer does not behave the way you'd like it to. There are two ways around that:

• The format (encoder/decoder) remaps how polymorphic serialization actually works.
• You use a different serializer than PolymorphicSerializer (either specified through @SerializeWith or by having a custom serializer for the parent - I would go with @SerializeWith)

This custom serializer for elements could be given as much knowledge about the data as desired. You could also, in the serializer special case the json (or any) format and handle it differently - for example by first parsing the data to JsonObject; detecting the actual type to use from that; and then deserializing the object with the (de)serializer that you determined you need. In case you want to ignore something you can then just return a default value of some sort (maybe encapsulate the json).

When understanding the serialization library, it is key to remember that fundamentally it is format independent. Because of that it cannot assume much about formats. For example, for some (binary) formats it is not possible to "skip" an unknown element as there is no size or end marker.

The format (json in this case) could however have a mode where missing keys (or even missing polymorphic values) are ignored - or fed to a "handler" that allows the user to specify the behaviour.

[suushiemaniac]

I definitely see your point here about the library being format-agnostic, @pdvrieze. For any other user who stumbles upon this issue, I also found the conversation in https://github.com/Kotlin/kotlinx.serialization/pull/514 to be very insightful.

However, it _still_ seems like a desirable change to allow for custom (dynamic!) hooks being invoked via SerialModule context. Even with independent formats, the user would be given control as sort of a "last resort" before finally an exception is thrown. (see my linked https://github.com/Kotlin/kotlinx.serialization/issues/697 for details).

Ultimately, with the current state of the project it seems like the best option to write a custom serializer. While that does achieve my goal, I still insist that it feels wrong to (effectively) copy so much code with so few changes to get what I'm looking for. Thanks for the constructive discussion! :)

[werner77]

We solved this issue by adding a wrapper around polymorphic types:

/**
 * Serializable class which wraps a value which is optionally decoded (in case the client does not have support for
 * the corresponding value implementation).
 */
@Serializable(with = WrappedSerializer::class)
data class Wrapped<T : Any>(val value: T?) {
    fun unwrapped(default: T): T {
        return value ?: default
    }
}

using this serializer:

/**
 * Serializer for [Wrapped] values.
 */
class WrappedSerializer<T : Any>(private val valueSerializer: KSerializer<T?>) : KSerializer<Wrapped<T>> {
    override val descriptor: SerialDescriptor = valueSerializer.descriptor

    private val objectSerializer = JsonObject.serializer()

    override fun serialize(encoder: Encoder, value: Wrapped<T>) {
        valueSerializer.serialize(encoder, value.value)
    }

    override fun deserialize(decoder: Decoder): Wrapped<T> {
        val decoderProxy = DecoderProxy(decoder)
        return try {
            Wrapped(valueSerializer.deserialize(decoderProxy))
        } catch (ex: Exception) {
            // Consume the rest of the input if we are inside a structure
            decoderProxy.compositeDecoder?.let {
                decoderProxy.decodeSerializableValue(objectSerializer)
                it.endStructure(valueSerializer.descriptor)
            }
            Wrapped(null)
        }
    }
}

private class DecoderProxy(private val decoder: Decoder) : Decoder by decoder {

    var compositeDecoder: CompositeDecoder? = null

    override fun beginStructure(descriptor: SerialDescriptor, vararg typeParams: KSerializer<*>): CompositeDecoder {
        val compositeDecoder = decoder.beginStructure(descriptor, *typeParams)
        this.compositeDecoder = compositeDecoder
        return compositeDecoder
    }
}

and we defined wrapping/unwrapping extension functions for ease of use:

/**
 * Convenience function to unwrap a list of [Wrapped] values
 */
fun <T : Any> List<Wrapped<T>>.unwrapped(): List<T> {
    return this.mapNotNull { it.value }
}

/**
 * Convenience function to wrap a list of [Any] values
 */
fun <T : Any> List<T>.wrapped(): List<Wrapped<T>> {
    return this.map { Wrapped(it) }
}

This does work, all though of course it would be great if the library could perform this magic for us.