Marten: Event Store Overhaul for 2.0

[jeremydmiller]

// CC: @jarroda, @jeffdoolittle, @wastaz, @danielwertheim, @ericgreenmix, & @Silv3rcircl3

Just a single point to discuss the proposed changes to the Event Store for 2.0 to make this all actionable. My opinion is that anything that is a breaking change gets done for 2.0, but everything else could technically wait. So far, we might be getting away with no breaking API changes -- in which case I might vote to push some of this off until 2.1 so we can get the multi-tenancy out sooner.

1. Non-typed streams: #694. Might be nothing but new overloads on StartStream without the generic parameter. I'd prefer not to force existing usages to change though
2. Event Metadata #780.
3. Event Store Partitioning: #770.
4. Multi-Tenancy #754. No clue yet how we'll do this one.

There are some other issues under the "Event Store" tag that aren't all that big: https://github.com/JasperFx/marten/labels/Event%20Store

What else? What's important in this list to you? What's missing? And who'd be willing to take on any of this?

[ericgreenmix]

This PR https://github.com/JasperFx/marten/pull/756 is a breaking change. Would be great if it could make it into 2.0

[jeremydmiller]

Doh, sorry, didn't mean to forget that one.

[wastaz]

In no internal order, off the top of my head:

• Non-typed streams, yes please! AggregateType has just caused problems for me and I dont like it at all. I could see that marking streams with a type as metadata could be useful purely from an exploration perspective, but in that case I would probably want that type metadata to not have any relation to the C# type system but rather be pure metadata. Also hopefully this would solve #705 I guess? ;)

• On the topic of metadata, I like the idea. The main thing that I would like to use it for off the top of my head is reading partial streams (i.e. give me all events for the stream with id X that has the metadata tag "transaction", and then marten would just read up those events from the event store), or doing cross-stream queries (i.e. give me all events from all streams with some metadata set). As for the cross-stream one, I dont think I would be using that very much (or at all) but it could prove useful in some instances, the first one (partial streams) however would absolutely be something that I would want to use.

• While talking metadata, if we do end up with event metadata I think we should also consider putting stream metadata in as well. It's something that GES can do, and it seems like a simple thing to add that could provide some useful value.

• Snapshots. Some type of support for this would be useful (or at least some documentation on recommended strategies for it).

• Being able to use string stream id's.

• Nicer support for non-marten projections. Occasionally I want to use the projections in order to project to a normal relational model, or to some other non-marten thing. Currently Im solving this by implementing the IProjection interface. This however requires me to create a fake type to return from the Produces property which creates an empty table in my database. I also have to do some boilerplate code here for matching event types etc. Nothing huge, but it would be nice to have some small QoL changes here just to make the experience a bit easier.

• Clearer path for projection replay scenarios. How should we do replays for async and inline projections?

• On the topic of replays, this is a pure QoL suggestion and I have no real problem doing it myself by just querying the mt_events table directly. But occasionally I want to replay events by sending them on the bus (so not using a projection). In those cases it could be useful to be able to query marten for all events in global sequence order with some type of filtering (only these stream ids, only streams matching this stream metadata etc). As said, I can do this manually in sql so this is really low on my priority list.

• The issues currently in the 2.0 milestone marked docs for eventstore would be _really_ nice to have.

• It would be nice to explore the possibility of getting rid of the need of calling StoreOptions.Events.AddEventTypes and providing all event types upfront. Im currently solving this by having my projections implement an IAggregate interface and doing assembly scanning and reading all argument types for all Apply methods and registering them at application start. Im thinking it might also be possible to do any bootstrapping for these on the fly caching the result? Not a huge deal, but another small QoL thing. Maybe this is too much work to be worth it though.

I am probably also able to help out by contributing some of these features. Probably not all of them though ;)
I may come up with more ideas or things I forgot later on.

[wastaz]

Also, maybe #686 could be useful to have if you feel like this is something that marten should support? I can certainly see how it could be useful in some cases, but I can also see that it might be not so useful in other cases so... any comments on this would be appreciated :)

[jeremydmiller]

@wastaz Thanks for all the input. This one: "Clearer path for projection replay scenarios. How should we do replays for async and inline projections?" -- you can use the async daemon as is today to replay any type of projection.

[wastaz]

@jeremydmiller You can? I might just have totally missed that.
The main reason why I said "clearer path" was because I suspected that there might be some functionality like this that I might just not have found yet. Ive been manually deleting rows from mt_event_progression and restarting the async daemon which has felt a bit...like a workaround.

In that case I'd say that just putting some docs up about this would be good enough, especially the async projection daemon is very lacking in the docs area. But basically I'd say that in that case maybe just putting a section about "replaying projections" and documenting the basic cases would be quite enough. :)

[ericgreenmix]

Out of all these issues, I don't see anything that is holding us back and would be okay with having them all move to a 2.1 release. None of these features are really mission critical for us, but this would be my list of things we could use.

• #596
• Event Store Partitioning
• Some version of rolling snapshots on projections

[jeremydmiller]

// CC: @jarroda, @jeffdoolittle, @wastaz, @danielwertheim, @ericgreenmix, & @Silv3rcircl3

So for a status update, we now have:

• Non-typed streams
• Opt into using strings as the stream id's
• StartStream() will throw an exception if the stream already exists
• StreamState has a Created property now
• The IProjection interface was changed a bit to make it easier to do custom projections
• and a couple other minor fixes: https://github.com/JasperFx/marten/issues?q=is%3Aissue+label%3A%22Event+Store%22+is%3Aclosed+sort%3Aupdated-desc

If we can reconvene this list again, what's absolutely got to be in 2.0? As in, Jeremy is getting burned out on this and I wanna be done soon;)

My thinking for what absolutely has to be in 2.0 is:

• Change the event store so that you don't have to pre-register event types
• Do the multi-tenancy support for the event store #754 (dunno what this means exactly)
• More docs on the event store. I could absolutely use some help here, especially on the async daemon (hint, hint).

Other things y'all have talked about that I don't think generate breaking changes and hence don't have to be in 2.0:

• Event metadata #780. I think it's a good idea in theory, but I don't have a strong concept for what the API's would look like or what you'd actually store (dictionary? duplicated fields?). What would the Linq support additions look like here?
• Rolling snapshots of projections. I decided not to do that way back in 1.0 thinking they just weren't completely necessary. I think this is a place where I'd be asking for help from y'all. Either way, I think it can be in 2.1

[ericgreenmix]

Great stuff. I am in agreement with everything. I will try to write some docs on the async daemon later this week when I have time.

Hate to keep being a pain about this, but I would still love it if #756 could get pulled in.

[jeffdoolittle]

I too agree with the above. I think the metadata can be held off until we have more clarity on actual use cases for it. Regarding rolling snapshots, I don't see why it couldn't wait until 2.1.

[marcpiechura]

My main concern was the Opt into using strings as the stream id's and since that's already fixed I don't have any opinions for the other parts, but sounds reasonable overall.

[wastaz]

I have one or two use cases that event metadata would be a perfect fit for that I'm holding off on implementing manually in the hopes that event metadata will make it into 2.0. I could take that off your hands and send you a PR if you are starting to feel burned out - however I'm quite swamped myself right now so I'm not sure if I could send you that PR before end of this month which may be a bit late if you want to get 2.0 out soon?

As for snapshots, they're purely on the "nice to have"-list, but not in the least bit vital for me in 2.0.

Edit: Also, 👏 👏 👏 👍 for all the great work done so far for 2.0 @jeremydmiller!

[jeremydmiller]

@wastaz Ah, thanks. For the metadata, from what you were saying earlier about how you'd want to query directly against the event table, that kind of pushes me to thinking that the metadata should be in separate fields. I'm still not sure how I'd want to deal with exposing that information within the Linq support. Maybe something like:

Where(x => x.HasMetadata("key", value))

I did do some changes to the Events.Append() and Events.StartStream() so that you get the EventStream object back. My thought is that you'd add metadata to some kind of new dictionary property of that object. For events, we could do the same thing on the Event<T> class. For now, I'm going to claim that we can add this later w/o breaking API changes, so it slides to 2.1;-)

[jeremydmiller]

I'm starting the work on not needing to preregister event types. Once that's done, I'm closing this issue.