Kotlinx.serialization: Strict JSON encountered unknown key

[rodrigohsb]

I tried some basic tests and I came across an unexpected scenario

This scenario works perfectly:

@Serializable
data class SimpleApiResponse (val id: String, val label: String)

```
{
"id": "123456789,
"label": "abc"
}

But when added another key into the json above, like this:

{
"id": "123456789,
"label": "abc",
"anotherKey": "def"
}

I got this error

```kotlin
kotlinx.serialization.SerializationException: Strict JSON encountered unknown key: anotherKey
                      at kotlinx.serialization.json.JSON$JsonInput.readElement(JSON.kt:294)
                      at co.stone.kyc.datasource.webservice.PendingQuestionsResponse$$serializer.load(Unknown Source:18)
                      at co.stone.kyc.datasource.webservice.PendingQuestionsResponse$$serializer.load(payload.kt:13)
                      at kotlinx.serialization.KInput.read(Serialization.kt:209)
                      at kotlinx.serialization.json.JSON.parse(JSON.kt:43)
                      at kotlinx.serialization.json.JSON$Companion.parse(JSON.kt:53)
                      at co.stone.kyc.datasource.KYCDefaultDataSource$loadData$1.onResponse(KYCDefaultDataSource.kt:132)
                      at okhttp3.RealCall$AsyncCall.execute(RealCall.java:153)
                      at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
                      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1162)
                      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:636)
                      at java.lang.Thread.run(Thread.java:764)

Is there any way to escape unmapped keys? Any annotation?

[JakeWharton]

JSON.nonstrict

[rodrigohsb]

Tks!
Any reason for nonstrict not be used as default?

[pdvrieze]

Because you easily noticed the problem that it was stricter than desired but the opposite you would most likely not have noticed.

[qwwdfsad]

Any reason for nonstrict not be used as default

Yes, by default, we opt-in for maximal safety, including type checks and runtime checks.
If frontend sends to backend slightly malformed JSON, it's better to know about it in advance, not when your system failed because field name has a typo and some value was ignored. In most cases, such check prevents errors on integration testing stage.

Though sometimes, you have to ignore missing fields, then you can use nonstrict mode

[raydenvoldeskine]

Could not find any documentation on how to use nonstrict mode if serializer is created with the @Serializable, not manually. Can someone help here?

[sandwwraith]

@fracturizer nonstrict is the property of Json format, not model. It should work with arbitrary serializer.

[raydenvoldeskine]

In other words, I should use Json.nonstrict.parse() instead of Json.parse(). Found it. Thanks.

[mochadwi]

Any reason for nonstrict not be used as default

Yes, by default, we opt-in for maximal safety, including type checks and runtime checks.
If frontend sends to backend _slightly_ malformed JSON, it's better to know about it in advance, not when your system failed because field name has a typo and some value was ignored. In most cases, such check prevents errors on integration testing stage.

Though sometimes, you have to ignore missing fields, then you can use nonstrict mode

How to use this for retrofit? Does this only apply for custom / manual JSON serialization?

@qwwdfsad @sandwwraith

EDITED
For those doesn't using custom Json de/serialization, use JsonConfiguration(strictMode = false) on retrofit converter factory

E.g:

// your retrofit builder
.addConverterFactory(
    Json(
        JsonConfiguration(strictMode = false)
    ).asConverterFactory(MediaType.get("application/json"))
)

[sandwwraith]

@mochadwi I think your question is more related to the repository of retrofit converter (https://github.com/JakeWharton/retrofit2-kotlinx-serialization-converter). But yes, I think your solution is correct

[ngortheone]

Is it possible to save all encountered unknown keys to some Map<String, JsonObj> value?
e.g.

data class MyData(val x: Int, val unknown: Map<String,JsonObj>)

[sandwwraith]

@ngortheone Currently this is not possible. You can deserialize json to JsonObject though which is merely a map.

[tellisnz-shift]

@sandwwraith - is there a recommended way of doing schema migration?

for example, say I add a field in a new version of a data class and send that to a client using the old version. I'd want clients to be able to deserialize, use the fields they know about, and then serialize the object to pass back to the server, without losing the new field if the server sent it. I guess sort of proxy the client data object to the JsonObject and serialize the JsonObject, if that makes sense?

because i guess if i just happily use non strict, on serialisation to send back to the server the new field would be dropped?

[sandwwraith]

@tellisnz-shift Well, if you try not to lose any fields client don't know about, then JsonObject is your way. However, I think it is pretty OK for a client to lose fields it doesn't know about and send back old JSON

[joaquini]

Latest versions of both Kotlin Serialization and JakeWharton's Converter should do:

Json {
    ignoreUnknownKeys = true
}.asConverterFactory(contentType)

[FerhatBahceci]

Using the Spring Webclient awaitBody() will trigger Kotlin serialization for the respective @Serializable receiver response class.

Unknown keys exception is thrown on the receiver side since some of the fields are missing.

Is it possible to set ignoreUnknownKeys = true somehow for these receiver classes other than implementing custom serializers?

SenderA(id,name,age) --> GET awaitBody<(id,name)>() throws unknown keys.

[FerhatBahceci]

val json = Json { ignoreUnknownKeys = true isLenient = true }

val strategies = ExchangeStrategies
    .builder()
    .codecs { clientDefaultCodecsConfigurer ->
        run {
            clientDefaultCodecsConfigurer.defaultCodecs()
                .kotlinSerializationJsonDecoder(KotlinSerializationJsonDecoder(json))
            clientDefaultCodecsConfigurer.defaultCodecs()
                .kotlinSerializationJsonEncoder(KotlinSerializationJsonEncoder(json))

        }
    }.build()

return WebClient
    .builder()
    .exchangeStrategies(strategies)
    .baseUrl(baseUrl!!)
    .build()