Kotlinx.serialization: Deserialize Protobuf list of objects

[webfrea-k]

I started working on custom serializer since this:
val protoSports = ProtoBuf.load(ProtoSports.serializer(), byteArrayRes)
only works correctly on Android (iOS throws NotImplementedException due to lack of reflection).

ProtoSports has only one property:
@SerialId(1) val sports: List<Sport>.

I've successfully deserialized 'simpler' objects with properties of type String, Long, etc. like this:

override fun deserialize(input: Decoder): Sport {
            val inp: CompositeDecoder = input.beginStructure(descriptor)
            var id: Long = 0
            lateinit var name: String
            loop@ while (true) {
                when (val i = inp.decodeElementIndex(descriptor)) {
                    CompositeDecoder.READ_DONE -> break@loop
                    0 -> id = inp.decodeLongElement(descriptor, i)
                    1 -> name = inp.decodeStringElement(descriptor, i)
                    else -> throw SerializationException("Unknown index $i")
                }
            }
            inp.endStructure(descriptor)
            return Sport(id, name)
        }

My question is, how can I deserialize list of objects e.g. Sport?
I've tried like this, but it doesn't work:

loop@ while (true) {
    when (val i = inp.decodeElementIndex(descriptor)) {
    0 -> {
        val sport = inp.decodeSerializableElement(descriptor, i, Sport.serializer())
        sports = sports.plus(sport)
    }
}

Thanks!

[webfrea-k]

The error is thrown at the first line of Sport deserialize method which looks like this:
val inp = input.beginStructure(Sport.descriptor)

[sandwwraith]

You can use inp.decodeSerializableElement(descriptor, i, ArrayListSerializer(Sport.serializer()))

[sandwwraith]

Btw, can you post stacktrace about the lack of reflection? I suppose ProtoBuf.load(ProtoSports.serializer(), byteArrayRes) should work because it have explicit serializer here. If it does not, this is probably a bug.

[webfrea-k]

I might be missing something obvious, but the code below fails with error: kotlinx.serialization.protobuf.ProtobufDecodingException: Expected wire type 2, but found 5

ProtoSports serializer:

@Serializable
data class ProtoSports(
        @SerialId(1) val sports: List<Sport>
)  {
    @Serializer(forClass = ProtoSports::class)
    companion object : KSerializer<ProtoSports> {

        override val descriptor: SerialDescriptor = object : SerialClassDescImpl("ProtoSports") {
            init {
                addElement("sports")
            }
        }
        override fun serialize(output: Encoder, obj: ProtoSports) {
            val elemOutput = output.beginStructure(descriptor)
            elemOutput.encodeSerializableElement(descriptor, 0, Sport.list, obj.sports)
            elemOutput.endStructure(descriptor)

        }
        override fun deserialize(input: Decoder): ProtoSports {
            val inp: CompositeDecoder = input.beginStructure(descriptor)
            var sports: List<Sport>  = listOf()
            loop@ while (true) {
                when (val i = inp.decodeElementIndex(descriptor)) {
                    CompositeDecoder.READ_DONE -> break@loop
                    0 -> {
                        val sport = inp.decodeSerializableElement(descriptor, i, ArrayListSerializer(Sport.serializer()))
                        sports = sports.plus(sport)
                    }
                    else -> throw SerializationException("Unknown index $i")
                }
            }
            inp.endStructure(descriptor)
            return  ProtoSports(sports)
        }
    }
}

Sport serializer:

@Serializable
data class Sport(
        @SerialId(1) var _id: Long,
        @SerialId(2) var name: String
) {
    @Serializer(forClass = Sport::class)
    companion object : KSerializer<Sport> {
        override val descriptor: SerialDescriptor = object : SerialClassDescImpl("Sport") {
            init {
                addElement("_id")
                addElement("name")
            }
        }
        override fun serialize(output: Encoder, obj: Sport) {
            val compositeOutput: CompositeEncoder = output.beginStructure(descriptor)
            compositeOutput.encodeLongElement(descriptor, 0, obj._id)
            compositeOutput.encodeStringElement(descriptor, 1, obj.name)
            compositeOutput.endStructure(descriptor)        }

        override fun deserialize(input: Decoder): Sport {
            val inp: CompositeDecoder = input.beginStructure(Sport.descriptor)
            var id: Long = 0
            lateinit var name: String
            loop@ while (true) {
                when (val i = inp.decodeElementIndex(descriptor)) {
                    CompositeDecoder.READ_DONE -> break@loop
                    0 -> id = inp.decodeLongElement(descriptor, 0)
                    1 -> name = inp.decodeStringElement(descriptor, 1)
                    else -> throw SerializationException("Unknown index $i")
                }
            }
            inp.endStructure(descriptor)
            return Sport(id, name)
        }
    }
}

Above code fails on line val inp: CompositeDecoder = input.beginStructure(Sport.descriptor) with the error stated at the top.

I've also tried calling
val sport = inp.decodeSerializableElement(descriptor, 0, ArrayListSerializer(Sport.serializer()))
directly in deserialize method of ProtoSports which returns empty array (and the deserialize metod of Sport is never called).

So, am I missing something obvious here? :)
Thanks.

[webfrea-k]

It might be worth mentioning that if I delete the custom serializers, and I just call val protoSports = ProtoBuf.load(ProtoSports.serializer(), responseByteArray)

The data is properly parsed and displayed on Android, while on iOS it throws Expected wire type 2, but found 5

[webfrea-k]

Test data:

val byteArray = byteArrayOf(10,16,8,1,18,6,83,111,99,99,101,114,24,-54,-62,-25,-121,2,10,16,8,5,18,6,84,101,110,110,105,115,24,-115,-61,-103,-90,2,10,20,8,4,18,10,73,99,101,32,72,111,99,107,101,121,24,-15,-42,-109,-114,6,10,20,8,2,18,10,66,97,115,107,101,116,98,97,108,108,24,-42,-40,-51,-1,3,10,18,8,6,18,8,72,97,110,100,98,97,108,108,24,-45,-46,-68,-121,5,10,20,8,23,18,10,86,111,108,108,101,121,98,97,108,108,24,-54,-31,-53,-71,4,10,17,8,3,18,8,66,97,115,101,98,97,108,108,24,-66,-88,-84,15,10,19,8,7,18,9,70,108,111,111,114,98,97,108,108,24,-119,-109,-7,-82,2,10,14,8,101,18,5,82,97,108,108,121,24,-48,-77,-55,102,10,19,8,100,18,9,83,117,112,101,114,98,105,107,101,24,-27,-23,-79,-59,7,10,19,8,40,18,9,70,111,114,109,117,108,97,32,49,24,-96,-72,-39,-31,5,10,15,8,70,18,6,78,97,115,99,97,114,24,-4,-46,-105,45,10,15,8,69,18,5,77,111,116,111,51,24,-27,-23,-79,-59,7,10,15,8,68,18,5,77,111,116,111,50,24,-27,-23,-79,-59,7,10,16,8,67,18,6,77,111,116,111,71,80,24,-27,-23,-79,-59,7,10,18,8,11,18,8,83,112,101,101,100,119,97,121,24,-64,-12,-127,-105,5,10,15,8,12,18,5,82,117,103,98,121,24,-107,-54,-118,-25,4,10,22,8,13,18,12,65,117,115,115,105,101,32,82,117,108,101,115,24,-11,-17,-20,-110,2,10,14,8,15,18,5,66,97,110,100,121,24,-53,-119,-1,68,10,27,8,16,18,17,65,109,101,114,105,99,97,110,32,70,111,111,116,98,97,108,108,24,-30,-69,-14,-53,5,10,16,8,19,18,7,83,110,111,111,107,101,114,24,-43,-93,-67,3,10,17,8,17,18,7,67,121,99,108,105,110,103,24,-78,-11,-64,-33,7,10,21,8,20,18,12,84,97,98,108,101,32,84,101,110,110,105,115,24,-101,-127,-4,105,10,15,8,22,18,5,68,97,114,116,115,24,-45,-98,-60,-128,3,10,22,8,24,18,12,70,105,101,108,100,32,104,111,99,107,101,121,24,-3,-94,-88,-48,4,10,19,8,26,18,9,87,97,116,101,114,112,111,108,111,24,-92,-114,-42,-126,3,10,16,8,29,18,6,70,117,116,115,97,108,24,-19,-28,-19,-88,4,10,19,8,31,18,9,66,97,100,109,105,110,116,111,110,24,-118,-84,-94,-18,7,10,22,8,34,18,12,66,101,97,99,104,32,86,111,108,108,101,121,24,-73,-62,-49,-66,6,10,16,8,37,18,6,83,113,117,97,115,104,24,-50,-20,-116,-35,7,10,24,8,109,18,14,67,111,117,110,116,101,114,45,83,116,114,105,107,101,24,-51,-10,-42,-87,6,10,27,8,110,18,17,76,101,97,103,117,101,32,111,102,32,76,101,103,101,110,100,115,24,-89,-78,-48,-29,5,10,16,8,111,18,6,68,111,116,97,32,50,24,-57,-125,-38,-90,2,10,19,8,61,18,9,80,101,115,97,112,97,108,108,111,24,-43,-124,-105,-79,4)

val protoSports = ProtoBuf.load(ProtoSports.serializer(), byteArray)

Classes:

@Serializable
data class ProtoSports(
        @SerialId(1) @Optional val sports: List<Sport>? = emptyList()
)
@Serializable
data class Sport(
        @SerialId(1) var _id: Long,
        @SerialId(2) var name: String
)

[sandwwraith]

This could be a problem with an underlying byte storage. I'll try to dig more into it, thanks for the examples

[webfrea-k]

Sure thing. The provided byte array deserializes into 34 Sport items on android (correct) while on iOS throws wire type error. The expected wire type (2 - Length delimited) is also correct, since sports is repeated field.

[sandwwraith]

I've finally figured this problem out, and it is not related to bytebuffer.

In the Native serialization plugin, there is no support for @SerialInfo annotations in the 1.3.11 versions, therefore protobuf ids were auto-assigned using property index: https://github.com/Kotlin/kotlinx.serialization/blob/master/runtime/native/src/main/kotlin/kotlinx/serialization/protobuf/ProtobufPlatform.kt#L22

Unfortunately, properties are enumerated from 0 and 0 is not a valid protobuf id, therefore externally produced bytes can't be correctly parsed (fun fact, if bytes were produced by kotlinx.serialization itself, parser does not complain).

1.3.20 will have support for custom annotations on Native, and this problem is fixed in dev: https://github.com/Kotlin/kotlinx.serialization/commit/e62e23b559b4943d2b61a13b1ceaea939937ff4e

[r3h0]

In the last hour, I got a Kotlin-JS client talking to my Kotlin-JVM server with Ktor and kotlinx.serialization. I used Protobuf as the transport, and I happened upon this bug before I got to the iOS (native) client implementation. It's so nice that I can just switch transports and keep using kotlinx.serialization pretty much as-is! This is an amazing library.

Sorry if this is off-topic... just chiming in to say, "thanks for the great work!"

[webfrea-k]

Great catch @sandwwraith ! I was worried that something might be wrong with our protobuf implementation, since deserialization was working OK if object was serialized using this library :)

Any idea about when can we expect next release (from current dev)? I'm having troubles configuring everything to work from dev branch..

p.s. Great work, love the project so far!

[sandwwraith]

Current dev requires a specific version of Kotlin directly from CI builds. I think things would be more stable around next EAP of 1.3.20, so it would be possible to release EAP version of library

[sandwwraith]

You can try 0.10.0-eap-1 alongside Kotlin 1.3.20-eap-52

[webfrea-k]

eap-52 is problematic with current Ktor version. I'll try to make it work and post the results here once it works.

p.s. Have a great 2019!

[sandwwraith]

This is also included in stable 0.10.0 release, which would be eventually compatible with all other libraries

[webfrea-k]

FYI: I've managed to get it working. Proto parsing now works as expected on both platforms.

Thanks for the great work! :)