Kotlinx.serialization: Regression when serialize wrapped Any type property annotated with @Contextual

[rwsbillyang]

Describe the bug
Kotlin version: 1.4.0 + kotlinx-serialization-core:1.0.0-RC) + @Contextual val data: Any? reports:

Exception in thread "main" kotlinx.serialization.SerializationException: Serializer for class 'Any' is not found.

But when (Kotlin version: 1.3.70 + kotlinx-serialization-runtime:0.20.0), @ContextualSerialization val data: Any? works fine.

To Reproduce

@Serializable
abstract class Box(
        var code: String,
        var msg: String?
){
    constructor(): this(OK, null)
    companion object{
        const val OK = "OK"
    }
}

@Serializable
//data class DataBox(@ContextualSerialization val data: Any?) : Box(OK,null)  //kotlin 1.3.70
data class DataBox(@Contextual val data: Any?) : Box(OK,null) //kotlin 1.4.0
{
    constructor(): this(null)
}

fun test3(){
    val json = Json{}
    //val str = json.stringify(DataBox.serializer(),DataBox(data = Person("Tom"))) //kotlin 1.3.70, ouput: {"code":"OK","msg":null,"data":{"name":"Tom"}}
    val str = json.encodeToString(DataBox.serializer(),DataBox(data = Person("Tom")))// kotlin 1.4.0: Execption!!! 
    println(str)
}

reports error:

Exception in thread "main" kotlinx.serialization.SerializationException: Serializer for class 'Any' is not found.
Mark the class as @Serializable or provide the serializer explicitly.
at kotlinx.serialization.internal.Platform_commonKt.serializerNotRegistered(Platform.common.kt:127)
at kotlinx.serialization.ContextualSerializer.serialize(ContextualSerializer.kt:52)
at kotlinx.serialization.json.internal.StreamingJsonEncoder.encodeSerializableValue(StreamingJsonEncoder.kt:223)
at kotlinx.serialization.encoding.Encoder$DefaultImpls.encodeNullableSerializableValue(Encoding.kt:296)
at kotlinx.serialization.json.JsonEncoder$DefaultImpls.encodeNullableSerializableValue(JsonEncoder.kt)
at kotlinx.serialization.json.internal.StreamingJsonEncoder.encodeNullableSerializableValue(StreamingJsonEncoder.kt:15)
at kotlinx.serialization.encoding.AbstractEncoder.encodeNullableSerializableElement(AbstractEncoder.kt:94)
at DataBox.write$Self(main.kt)
at DataBox$$serializer.serialize(main.kt)
at DataBox$$serializer.serialize(main.kt:32)
at kotlinx.serialization.json.internal.StreamingJsonEncoder.encodeSerializableValue(StreamingJsonEncoder.kt:223)
at kotlinx.serialization.json.Json.encodeToString(Json.kt:73)
at MainKt.test3(main.kt:138)

Expected behavior
Expect the behavior of 1.4.0 + 1.0.0-RC same as one of 1.3.70 + 0.20.0

Environment

• Kotlin version: 1.4.0
  
  
  
  • Library version: 1.0.0-RC
  
  
  • Kotlin platforms: JVM
  
  
  • Gradle version: 6.3
  
  
  • IDE version: IntellijIDEA 2020.2.1
    
    Other relevant context : MacOS 10.12.6, JDK 1.8.0_261
  
  

[ZhouGongZaiShi]

I ran into the same issue.
Looking through Contextual serialization, I found that example-serializer-17.kt runs with the same exception.

[rwsbillyang]

I found in the latest version kotlin 1.4.0, it works after add contextual configuration as following:

    val json = Json{
            serializersModule = SerializersModule {
                  contextual(Person.serializer())
        }
}

It also works if use Polymorphic at similar way:

@Serializable
private data class Box6(
        var code: String = "OK",
        var msg: String? = null,
        @Polymorphic var data: Any? = null
)

@Serializable
private data class Person6(val name: String)

fun testBoxAnyWithPolymorphic(){
    val format = Json{
        serializersModule = SerializersModule {
            polymorphic(Any::class) {
                subclass(Person6::class) //register subclass for run-time
            }
        }
    }
        val box = Box6(data = Person6("Tom"))
        val str = format.encodeToString(box) 
        println("Polymorphic: $str") //Polymorphic: {"code":"OK","msg":null,"data":{"type":"Person6","name":"Tom"}}
}

[qwwdfsad]

Thanks for the self-contained report!

The previous version was working by accident, yet it's still can be considered as a regression.

Contextual serialization is supposed to work with serializer registered in the serializers module and with a fallback serializer that is associated with the type of the property. E.g. for @Contextual val int: Int fallback serializer is Int.serializer().

In your example, you have Any? as a type, but Any does not have any (pun intended) default serializers.
Previously it worked by introspecting an actual KClass of the serializable value, but it is neither correct nor safe: it didn't work with generics, it cannot have been deserialized back and it opened the whole can of worms with security issues.

I'll update the documentation to ContextualSerializer, but the current behaviour is here to stay

[kikothemaster]

I'm using Map for the feedback purposes where attributes could be any. But in the new version, I cannot use that anymore 👎

[sandwwraith]

@kikothemaster you can follow the suggested workaround with @Polymorphic Any

[kikothemaster]

@sandwwraith Yeah I've tried, but then I try to add subclass(String::class, String.serializer() it throws an exception java.lang.IllegalArgumentException: Serializer for String of kind STRING cannot be serialized polymorphically with class discriminator.

[kikothemaster]

@rwsbillyang Did you tried polymorphic with generic types (String, Int, Double, ...) ?

[rwsbillyang]

@rwsbillyang Did you tried polymorphic with generic types (String, Int, Double, ...) ?

I tried just now, it does not work, same error:

@Serializable
private data class Box6(
        var code: String = "OK",
        var msg: String? = null,
        @Polymorphic var data: Any? = null
)

@Serializable
private data class Person6(val name: String)

fun testBoxAnyWithPolymorphic() {
    val format = Json {
        serializersModule = SerializersModule {
            polymorphic(Any::class) {
                subclass(Person6::class) //works ok
                //subclass(String::class) //fail: Exception in thread "main" java.lang.IllegalArgumentException: Serializer for String of kind STRING cannot be serialized polymorphically with class discriminator.
                //subclass(Long::class) //fail: Exception in thread "main" java.lang.IllegalArgumentException: Serializer for Long of kind LONG cannot be serialized polymorphically with class discriminator.
            }
        }
    }
    val box = Box6(data = Person6("Tom"))

    println("Polymorphic: ${format.encodeToString(box)}") //Polymorphic: {"code":"OK","msg":null,"data":{"type":"Person6","name":"Tom"}}

    try {
        println("Polymorphic with default Json: ${Json.encodeToString(box)}")
    } catch (e: Exception) {
        //fail box polymorphic with default Json: Class 'Person6' is not registered for polymorphic serialization in the scope of 'Any'.Mark the base class as 'sealed' or register the serializer explicitly.
        println("fail box polymorphic with default Json: ${e.message}")
    }

    val box2 = Box6(data ="Test box String")
    println("Box String with Polymorphic: ${format.encodeToString(box2)}")
    //Exception in thread "main" kotlinx.serialization.SerializationException: Class 'String' is not registered for polymorphic serialization in the scope of 'Any'. Mark the base class as 'sealed' or register the serializer explicitly.



    val data: Long = 1L
    val box3 = Box5(data = data)
    println("Box primitive Long with Polymorphic: ${format.encodeToString(box3)}")
    //Exception in thread "main" kotlinx.serialization.SerializationException: Serializer for class 'Any' is not found. Mark the class as @Serializable or provide the serializer explicitly.
}

But it works with @Contextual:

@Serializable
data class Box5(
        var code: String = "OK",
        var msg: String? = null,
        @Contextual var data: Any? = null
)

@Serializable
data class Person5(val name: String)

fun testBoxAnyWithContextual() {
    val format = Json {
        serializersModule = SerializersModule {
            contextual(Person5.serializer())
            contextual(String.serializer())//Need add this line, "Any" can be String
            contextual(Long.serializer())//Need add this line, "Any" can be Long
        }
    }

    val box = Box5(data = Person5("Tom"))
    println("Contextual: ${format.encodeToString(box)}") // ok if add: contextual(Person5.serializer())
    //Contextual: {"code":"OK","msg":null,"data":{"name":"Tom"}}

    try {
        println("Contextual with default Json: : ${Json.encodeToString(box)}")
    } catch (e: Exception) {
        //fail with default Json: Serializer for class 'Any' is not found.Mark the class as @Serializable or provide the serializer explicitly.
        println("fail with default Json: ${e.message}")
    }

    val box2 = Box5(data ="Test box String")
    println("Box primitive String with Contextual: ${format.encodeToString(box2)}")//ok if add: contextual(String.serializer())
    //Box primitive String with Contextual: {"code":"OK","msg":null,"data":"Test box String"}


    val data: Long = 1L
    val box3 = Box5(data = data)
    println("Box primitive Long with Contextual: ${format.encodeToString(box3)}")//ok if add: contextual(Long.serializer())
    //Box primitive Long with Contextual: {"code":"OK","msg":null,"data":1}
}

[rwsbillyang]

I think using T is a good idea as following. What a pity call.respond in ktor doesn't support generic for now.
(https://github.com/ktorio/ktor/issues/2013)

@Serializable
data class Box3<T>(var code: String = "OK", var msg: String? = null, val data: T? = null)
@Serializable
data class Person3(val name: String)

/**
 * Test box generic
 *
 * https://github.com/Kotlin/kotlinx.serialization/blob/master/docs/basic-serialization.md#generic-classes
 * */
fun testGeneric(){
    // {"code":"OK","msg":null,"data":{"name":"Tom"}}
    println(Json.encodeToString(Box3(data = Person3("Tom"))) )

    //{"code":"OK","msg":null,"data":[{"name":"Tom"},{"name":"Tom2"}]}
    println(Json.encodeToString(Box3(data = listOf(Person3("Tom"),Person3("Tom2")))))

    //{"code":"OK","msg":null,"data":"box primitive String"}
    println(Json.encodeToString(Box3(data = "box primitive String")))

    //{"code":"OK","msg":null,"data":1.23}
    println(Json.encodeToString(Box3(data = 1.23F)))
}

[VincentJoshuaET]

How do I use either Contextual or Polymorphic for Map?

[kikothemaster]

@VincentJoshuaET

Map<String, @Contextual @RawValue Any>

val json= Json {
    serializersModule = SerializersModule {
        contextual(String.serializer())
        contextual(Int.serializer())
        contextual(Double.serializer())
    }
}

In my case, Any could be String, Int or Double.

[yusufceylan]

@kikothemaster @VincentJoshuaET

I am still get the same error, am I missing something?

Retrofit instance

    fun provideRetrofit(baseUrl : String, okHttpClient: OkHttpClient) : Retrofit {
        val contentType = "application/json".toMediaType()
        val json = Json {
            serializersModule = SerializersModule {
                contextual(String.serializer())
                contextual(Int.serializer())
                contextual(Double.serializer())
                contextual(Long.serializer())
                contextual(Boolean.serializer())
            }
        }
        return Retrofit.Builder()
            .baseUrl(baseUrl)
            .client(okHttpClient)
            .addConverterFactory(json.asConverterFactory(contentType))
            .build()
    }

Api interface

    @POST("post")
    suspend fun sendPost(@Body params : HashMap<String, @Contextual @RawValue Any>) : GenericResponse<Post>

Still get error

Unable to create @Body converter for java.util.HashMap<java.lang.String, java.lang.Object> (parameter #1)

[kikothemaster]

Try Map instead of HashMap.

[yusufceylan]

Try Map instead of HashMap.

Still same @kikothemaster

[mpiniarski]

Is there a way to make it work for Map<String, Any>?

[kikothemaster]

yes, this works:

Map<String, @Contextual @RawValue Any>

val json= Json {
    serializersModule = SerializersModule {
        contextual(String.serializer())
        contextual(Int.serializer())
        contextual(Double.serializer())
    }
}

In my case, Any could be String, Int or Double.

[mpiniarski]

How do get the RawValue annotation?

[kikothemaster]

import kotlinx.android.parcel.RawValue

[mpiniarski]

I see. I'm not working on an Android project, but a multiplatfrom one. Without the RawValue i'm getting
kotlinx.serialization.SerializationException: Serializer for class 'Any' is not found.
when deserializing a json string.

[VincentJoshuaET]

Is there a way to make it work for Map<String, Any>?

Can you do JsonObject? instead of Map, then get the JsonElement value and do the type checking?

[mpiniarski]

It would be much better for me to still use Any in data classes, but if there is no other way I will have to use JsonElement.

Is it possible to write a custom serializer for Any type?
I've tried, and it's easy to implement a serialize method, but in deserialize method I don't see any way to determine what's the type of deserialized json element.

class AnySerializer : KSerializer<Any> {
    override val descriptor: SerialDescriptor = ...

    override fun deserialize(decoder: Decoder): Any {
        // No way to decode as JsonElement and check what's the type
        return decoder.decodeString()
    }

    override fun serialize(encoder: Encoder, value: Any) {
        when (value) {
            is String -> encoder.encodeString(value)
            is Int -> encoder.encodeInt(value)
            // ...
        }
    }
}