Kotlinx.serialization: Option to not write optional values

[czeidler]

Is it possible to add an option to not write optional values?

For example, either if in nonstrict mode or add an parameter to @Optional. Something like:

@Optional(dontwritewhen=2) val value: Int = 2

or

@Optional(writedefault=false) val value: Int = 2

[sandwwraith]

So, did I understand you correctly? Given some class

@Serializable
class Data(@Optional(writedefault=false) val value: Int = 2)

you expect JSON.stringify(Data(3)) be {value: 3} and JSON.stringify(Data(2)) be {}?
Basically, you try to save space in output stream by omitting values with defaults

[czeidler]

Yes that is correct. Maybe a better usecase is this one:

@Serializable
class(val value: Int, @Optional val optionalData: OptionalData? = null)

If optionalData is null you don't really want to have optionalData:null in your JSON output. It might even causes problems for other external JSON parsing code that expect a JSON object or nothing but not a null value.

Another usecase is that you may want to read optional values but never write them (e.g. because they are deprecated)

[elizarov]

I wonder if that shall be a feature of the serializable class or a feature of the format? Does it make sense to automatically support elision of default values when _both_ the class has _eligible_ defaults (I'm thinking about compile-time constants and defaults here) and the format supports it?

[czeidler]

What do you mean with format? json, protobuffer, cbor? Think it depends on the use-case so IMHO it should be configurable.

[elizarov]

Btw, this is a very important feature for ProtoBuf that (unlike JSON) does not support the notion of null value, so there is not way to write null value to ProtoBuf unless the feature of omitting default is supported. Consider this case:

Consider this case:

@Serializable class SampleMessage {
    @Optional @SerialId(1) var name: String? = null
}

fun main(args: Array<String>) {
    val message = SampleMessage()
    println(ProtoBuf.dumps(message))
}

It crashes with

Exception in thread "main" kotlinx.serialization.SerializationException: null is not supported
    at kotlinx.serialization.TaggedOutput.writeTaggedNull(Tagged.kt:38)
    at kotlinx.serialization.TaggedOutput.writeNullValue(Tagged.kt:80)
    at kotlinx.serialization.KOutput.writeNullableSerializableValue(Serialization.kt:151)
    at kotlinx.serialization.KOutput.writeNullableSerializableElementValue(Serialization.kt:198)
    at SampleMessage.write$Self(TestOneOf.kt)
    at SampleMessage$$serializer.save(TestOneOf.kt)
    at SampleMessage$$serializer.save(TestOneOf.kt:5)
    at kotlinx.serialization.KOutput.write(Serialization.kt:99)
    at kotlinx.serialization.protobuf.ProtoBuf.dump(ProtoBuf.kt:415)
    at TestOneOfKt.main(TestOneOf.kt:17)

[rocketraman]

I am commenting here so I can easily find this issue again in the future, since I can't do that by subscribing alone.

[abissell]

A major use-case for this is implementing the JSON-RPC spec. On the Request object, the spec says the params member "MAY be omitted." This would map most cleanly to a nullable Kotlin field, but without the ability to omit optional values it will write out params: null, which violates the spec.

Anyway, I know this is a feature under active development, just wanted to point out one more use case. :)

[pdvrieze]

My major question is what the intended semantics of optional are. Currently it means "may be missing in the input", but will always be written. It may be possible to just have a switch that specifies the semantics (eg. omit_if_null, omit_if_default_value, never_omit, omit_if_tag). That would really benefit from default value support in annotations.

omit_if_tag would allow a tag to be set on the session (like that it is in compat mode) that can then be checked for individual fields without requiring custom serialization.

Implementation is easy if this information is available in some form to the serializer.

[coder82]

I have to vote for this, saves bandwidth when we serialize!

[SapuSeven]

Any news on this feature?

[sandwwraith]

New framework design contains solution for this problem: https://github.com/Kotlin/KEEP/blob/serialization/proposals/extensions/serialization.md#optionality

[GoMino]

@sandwwraith can you provide an exemple on how we can use the CompositeEncoder to not write optional value ?

Can't find any docs on this :(

[sandwwraith]

@GoMino for now, workaround would be to write code like this: https://github.com/Kotlin/kotlinx.serialization/blob/eap13/runtime/common/src/test/kotlin/kotlinx/serialization/CustomSerializersTest.kt#L67

[GoMino]

@sandwwraith thanks, is it compatible with kotlin Native and JS ?

[sandwwraith]

Yes, these are multiplatform interfaces

[InsanusMokrassar]

@GoMino for now, workaround would be to write code like this: https://github.com/Kotlin/kotlinx.serialization/blob/eap13/runtime/common/src/test/kotlin/kotlinx/serialization/CustomSerializersTest.kt#L67

Hello

What if I need to skip. for example, nulls on serialization process in very huge count of data classes?

[pdvrieze]

@InsanusMokrassar Currently you are probably best of creating a custom encoder that does this in a way configured by that encoder (using annotations, a hardcoded list, initialisation of the encoder, etc.)

[InsanusMokrassar]

@pdvrieze ok, thank you:)

[davidbrazilparker]

So there's a possible fix for this that I found which is kind of hacky but will resolve this for Kotlin data classes.

Here's an example of a data class that I serialize which has an optional serializable parameter:

@Serializable
data class Result( @Optional @SerialName("status_id") var statusId: Int? = null)

Currently with how the Kotlinx serialization library is written we write to our JSON map the value of null as a Kotlin String "null".

So for the example above the resulting JSON object would look as follows:

{
    "status_id":null
}

Cool so this will mean I'm not going to change this value when I POST this payload right? NOPE

null is not a valid JSON object value for the API I'm sending POST requests to. Now I'll admit I don't have much experience with API's (I'm learning 🤓). But for my use case in particular this a huge issue. Now for what I think could be the solution.

In the following file:
https://github.com/Kotlin/kotlinx.serialization/blob/master/runtime/common/src/main/kotlin/kotlinx/serialization/json/JsonParser.kt#L23

Change this line:

internal const val NULL = "null"

to

internal const val NULL = ""

Why? JSON treats the empty string, "", value as a null value. What this means is that if we set a value as "" we won't be changing that value when we POST the JSON payload.

The goal of this is so our Data Class Serialized as JSON will look like this instead:

{
    "status_id":""
}

I'm not saying this is going to cover all use cases but for my purposes and the issues I've run into, I think this is a simple solution to a problem that's been causing me plenty of headaches.

[InsanusMokrassar]

@davidbrazilparker so, you offer to change output of JSON#stringify from

{"output": null}

to

{"output": ""}

Is it right? I don't think that this solve better in case of skipping some values (like null) at least for the reason that it is not obviously enough. If to talk conversation topic: it is absolutely opposite to the main idea: skip fields serialization in some cases.

[sandwwraith]

@davidbrazilparker No, an empty string is not equivalent to a JSON null literal. It may be possible in your particular API, but not in general case. Even absence of value in some cases is not equivalent to a null literal.

[davidbrazilparker]

Yeah my mistake sorry guys 😛

[raderio]

In Jackson you can use
@JsonInclude(Include.NON_NULL)
https://www.baeldung.com/jackson-ignore-null-fields

[InsanusMokrassar]

@raderio can I use this with built-in JSON (kotlinx.serialization.json.JSON)? As I understand it is not built-in opportunity. I have seen the link which you have included in your comment, but anyway Jackson is external library.

[raderio]

@InsanusMokrassar it is an example how this is done in Jackson, maybe will be good reference in implementation this here.

[InsanusMokrassar]

@raderio Excuse me, I see

[sandwwraith]

Starting from 0.10.0 and Kotlin 1.3.20, both library and plugin support this feature according to design described in KEEP.

You can create an instance of Json class with the corresponding setting: Json(encodeDefaults = false) to skip default values during serialization.

[Egorand]

Is there a similar solution for configuring ProtoBuf to skip optional values?

[jpink]

@sandwwraith I see this as a major ProtoBuf issue, because the protocol doesn't know NULL.

And you give Json(encodeDefaults = false) as a workaround where the issue isn't showstopper.

[sandwwraith]

@jpink There's a separate issue for it: #71

[valeriyo]

I'm sorry, but Json(encodeDefaults = false) is not sufficient. I have a bunch of hardcoded values in my request classes like this:

@Serializable
class DoSomethingRequest(
  @SerialName("data") val data: String?
) {
  @SerialName("source") val source = "android"
}

As soon as I enable encodeDefaults = false -- the source field will disappear from the JSON. I would like a more precise control of optionality, per field.