Akka.net: Akka.Persistence - serialization oriented around event versioning

[Horusiath]

At the current state, our SQL-based journals use fully qualified type name for each event in order to be able to deserialize it later → correct serializer is obtained by the type name of an event included in the manifest column. This however seems to be bad for several reasons:

1. Most of serializers we use already include type name as part of payload in order to support polymorphism i.e. both our core serializers, Json.NET and Hyperion, work that way. This causes increased overall size of the payload and metadata.
2. It's not guaranteed, that we'll be able to deserialize event to type described in manifest (reason: pt. 1). This is especially nasty in event type name / assembly changes, since even change in manifest won't help the users, since the original type is still encoded into payload.

Yes, we still advice people not to use default serializers for persistence, using the ones with explicit schema instead (like proto-buf or Bond), but I guess most of them will do that anyway.

My proposition here was to stop relying on manifest column - instead we could retrieve serializers directly by their id (each serializer in akka has it's unique identifier). Those identifiers are simple integers (so a lot more compact in size), and at least some of them can have ID value configured through HOCON - and all of them could possibly have so in the future.

The only issue for me is backward compatibility with existing solutions. I think we could either:

1. Add nullable SerializerId column and make Manifest column nullable too. In SQL journal logic we'd first try to fetch serializer by ID (for new events), falling back to manifest (for old events).
2. At some point i.e. v1.5, remove Manifest column for the sake of SerializerId and provide a tool, that will automatically migrate old events into their new shape.

Any thoughts? /cc @akkadotnet/core

[Danthar]

Im all for to improve our persistence system for increased performance.

However I don't care about solving the scenario where people are stupid enough to change their event names or move them between assemblies and expect stuff to still work.
What i mean by that is that this limitation is a known issue for all eventsourced systems. And there are well known solutions to solve them (event adapters, converting the entire stream using an helper-app).
Its a known limitation I personally can live with.

As far as backwards compat goes. I'd vote for option 1. However eventually we would want to remove the manifest column + logic anyway. At which point users would have to convert their older data anyway. Either via an update statement or something more drastic :P

[Aaronontheweb]

However I don't care about solving the scenario where people are stupid enough to change their event names or move them between assemblies and expect stuff to still work. What i mean by that is that this limitation is a known issue for all eventsourced systems.

My thoughts exactly. The manifest should include both the serializer ID and whatever the actual type that's serailized is. Combination of both of those bits should be enough information for people using version tolerance or changing serializers or whatever.

[Horusiath]

Closing this issue - I think, we've already made a decision and applied it since v1.3.1 forward.