Eventstore: Read all events without including internal ($) streams

[carlos-sarmiento]

I've checked the documentation for EventStore and I have not been able to find a way to read ALL events from a database that does not include events from internal ($) streams. This is particularly important when reading through a network as there is a lot of data wasted.

I think that a flag in the ReadAllEventsForwardAsync (and Backwards) methods could very easily achieve this.

[jen20]

There is no built in way to do this and currently no server side filtering. You _can_ write a custom projection that effectively gives you your own stream of just user events, though I'd recommend not doing this as reads go via an optimised route when reading the $all stream directly.

[jen20]

OTOH it might not be a bad addition to filter user events server side. It would need careful thought for protocol backwards compatibility though.

[jen20]

@gregoryyoung @pgermishuys any thoughts?

[pgermishuys]

This would be a nice addition.

[gregoryyoung]

Is the "network savings" really worth it? The whole point $all exists is
that its network stupid complexity simple.

On Mon, Oct 5, 2015 at 5:54 PM, Pieter Germishuys [email protected]
wrote:

This would be a nice addition.

—
Reply to this email directly or view it on GitHub
https://github.com/EventStore/EventStore/issues/718#issuecomment-145597533
.

Studying for the Turing test

[carlos-sarmiento]

Well, right now I'm working with a system that hosts the database in Azure. But some systems are local, so reading all events take an ungodly amount of time.

I agree that in most cases client-side filtering is enough, but with these kinds of connection, filtering would be a blessing.

If the development cost is small, I would say it is a worthy addition

[gregoryyoung]

"Well, right now I'm working with a system that hosts the database in
Azure. But some systems are local, so reading all events take an ungodly
amount of time. "

Why?

What system events are you getting? If $statistics and you aren't using
them actively and have a slow writing db then just increase the $statistics
time.

On Mon, Oct 5, 2015 at 7:12 PM, Carlos Gustavo Sarmiento <
[email protected]> wrote:

Well, right now I'm working with a system that hosts the database in
Azure. But some systems are local, so reading all events take an ungodly
amount of time.

I agree that in most cases client-side filtering is enough, but with these
kinds of connection, filtering would be a blessing.

If the development cost is small, I would say it is a worthy addition

—
Reply to this email directly or view it on GitHub
https://github.com/EventStore/EventStore/issues/718#issuecomment-145619648
.

Studying for the Turing test

[gregoryyoung]

I should add or set the $maxCount/$maxAge on the $statistics stream and
scavenge occasionally which will get rid of all the old ones.

On Mon, Oct 5, 2015 at 7:15 PM, Greg Young [email protected] wrote:

"Well, right now I'm working with a system that hosts the database in
Azure. But some systems are local, so reading all events take an ungodly
amount of time. "

Why?

What system events are you getting? If $statistics and you aren't using
them actively and have a slow writing db then just increase the $statistics
time.

On Mon, Oct 5, 2015 at 7:12 PM, Carlos Gustavo Sarmiento <
[email protected]> wrote:

Well, right now I'm working with a system that hosts the database in
Azure. But some systems are local, so reading all events take an ungodly
amount of time.

I agree that in most cases client-side filtering is enough, but with
these kinds of connection, filtering would be a blessing.

If the development cost is small, I would say it is a worthy addition

—
Reply to this email directly or view it on GitHub
https://github.com/EventStore/EventStore/issues/718#issuecomment-145619648
.

Studying for the Turing test

Studying for the Turing test

[carlos-sarmiento]

As you mentioned @gregoryyoung most of the events are $stats. I guess we could increase the time from 30 to something higher, but if my understanding is correct, we would lose granularity on stats right?

Regarding the time taken, the connection we are using locally is 10MB down, so you can imagine the cost of many unnecessary events.

Removing old statistics events would be a better option since we could at least keep a lot more information on recent performance.

[gregoryyoung]

Just set $maxage on stats stream and run scavenge

On Mon, Oct 5, 2015 at 7:53 PM, Carlos Gustavo Sarmiento <
[email protected]> wrote:

As you mentioned @gregoryyoung https://github.com/gregoryyoung most of
the events are $stats. I guess we could increase the time from 30 to
something higher, but if my understanding is correct, we would lose
granularity on stats right?

Regarding the time taken, the connection we are using locally is 10MB
down, so you can imagine the cost of many unnecessary events.

Removing old statistics events would be a better option since we could at
least keep a lot more information on recent performance.

—
Reply to this email directly or view it on GitHub
https://github.com/EventStore/EventStore/issues/718#issuecomment-145630880
.

Studying for the Turing test

[pgermishuys]

Is this still something we want to look at? e.g. server side filtering?

[dnauck]

Currently i'm using the $all stream for client read models as projections a beta or just add extra complexity (i still need all my events excluding the systems internal ones).

I don't know if this is really needed but would remove the client side system events filter.

[kspeakman]

The first time I tried to build read projections, it was quite unexpected that the $all stream produced internal events. I guess internal events are visible to me in case I want to write my own stats dashboard? Seems like those should be in a separate $stats or $system stream as I have never used them and just have to filter them out.

Reducing stats granularity is not an ideal solution for operations. I'd definitely set $maxage to reduce space usage.

[gregoryyoung]

$all reads all events. It is optimized in a different way for reading all
events (does not require index hits)

On Fri, Sep 9, 2016 at 7:01 PM, Kasey Speakman [email protected]
wrote:

The first time I tried to build read projections, it was quite unexpected
that the $all stream produced internal events. I guess internal events
are visible to me in case I want to write my own stats dashboard? Seems
like those should be in a separate $stats or $system stream as my API has
never been concerned with them and just has to filter them out.

Reducing stats granularity is not an ideal solution for operations. I'd
definitely set $maxage to reduce space usage.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/EventStore/EventStore/issues/718#issuecomment-245991336,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAXRWlKEZk9qDEAfuzE65Zx9ac015-d_ks5qoZ8CgaJpZM4GJId8
.

Studying for the Turing test

[kspeakman]

Right, I get why it's implemented that way. I was just curious if there was a reason for it being _exposed_ that way. Is it a common use case for people to process GES events? If it's uncommon, don't you see value in an $allNonSystem (or something) stream? Less to send across the wire.

An index hit shouldn't be needed if you filter out system events by name just before sending across the wire. You aren't saving any read IO since the events are still all being loaded, but there's also no index hit. It's no different from what consuming programs currently do, except the consuming programs filter after they've gone across the wire.

[sorry, prone to edit]

[bklooste]

The big thing for me is if you try to get 4096 record you need to keep re fetching since 99% of events on a cluster are $ events and you end up getting 1 record so then you need more complex logic to get the records. The client cant talk direct since it may be running over 3G or some other crap link..which would take forever with all the $ events in a cluster ( ok for a server) ... Currently this takes so long due to system events that to get 4096 non system records that the gateway times out which i shouldn't be dealing with.

[gregoryyoung]

if you have 99% of events being internal events you must have a really low
workload cluster. Try this.

fromAll();
when(
$any : function(s,e) {
if(!e.name.startsWith('$') linkTo('mystream')
);

Then use a subscription to mystream which will contain all your events.

On Mon, Jul 24, 2017 at 8:08 AM, bklooste [e-mail] wrote:

The big thing for me is if you try to get 4096 record you need to keep re
fetching since 99% of events on a cluster are $ events and you end up
getting 1 record so then you need more complex logic to get the records.
The client cant talk direct since it may be running over 3G or some other
crap link..which would take forever with all the $ events in a cluster ( ok
for a server)

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/EventStore/EventStore/issues/718#issuecomment-317337368,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAXRWjwwG3jDSygJVj1N2KkrMer8rSgUks5sRELqgaJpZM4GJId8
.

--
Studying for the Turing test

[RytisLT]

and here's same code but it actually compiles:

fromAll().
   when({
        $any : function(s,e) {
            if (e.eventType && !e.eventType.startsWith('$'))
                linkTo('allNonSystemEvents', e);
   }});

[kspeakman]

Deleted previous comment. I get why now...

(educated guess) The $all stream reads bytes off disk and shoves them straight out over TCP. In order to filter out system events, the bytes would have to be examined (possibly deserialized) first. Projections already provide this functionality.

This is probably why most classic databases arrived an an arrangement of separating system data (like stats) from user data. I.e. control plane vs data plane. This also jives with the concept of "schemas" (or just sub-containers) within a database.

[Salgat]

@kspeakman Unless I'm mistaken doesn't https://github.com/EventStore/EventStore/blob/daddeca886f43e40489efd7d9f45cb6ddd4aed8a/src/EventStore.Core/Services/SubscriptionsService.cs#L272 have access to EventType through the EventRecord evnt parameter?

[kspeakman]

I was just guessing and didn't dig thru the source code, so it's very possible. The only other reservation I can think of is prepare/commit position gaps of the returned stream. But again, a guess. Otherwise, I can't explain why this would be a bad thing. I find that people usually have reservations for a good reason, though. (But still glad this issue remains open.)

[Salgat]

It looks like you could do a

!subscription.IgnoreSystemEvents || !evnt.EventType.StartsWith("$")

Check to that handler to filter them out but I'm sure theres more to it.

[gregoryyoung]

This causes write amplification. its far more efficient to use the $all
reads and filter on read. $all goes down a different code path and is
optimized for reading all events. Its more efficient to read $all and
filter on the client side in almost every case (a store that basically
never gets any user events might be an exception here but its not a very
useful one :))

On Fri, Apr 6, 2018 at 1:26 AM, Austin Salgat [e-mail]
wrote:

It looks like you could do a

!IgnoreSystemEvents || !evnt.EventType.StartsWith("$")

Check to that handler to filter them out but I'm sure theres more to it.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/EventStore/EventStore/issues/718#issuecomment-379032411,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAXRWv5uXu5LczXJcHD5wnQTYoFGjzvzks5tlmG5gaJpZM4GJId8
.

--
Studying for the Turing test

[kspeakman]

Write amplification on a read operation? Why, stats?

[gregoryyoung]

I meant if you use a projection to create a custom my_own_all it causes
write amplification sorry if not clear.

On Fri, Apr 6, 2018 at 9:32 AM, Kasey Speakman [e-mail]
wrote:

Write amplification on a read operation? Why, stats?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/EventStore/EventStore/issues/718#issuecomment-379131699,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAXRWkHsptLdWCjvmEeetkacZUnIvxTgks5tltPPgaJpZM4GJId8
.

--
Studying for the Turing test

[kspeakman]

So what's the issue with a built-in $data_all stream that uses $all internally and !subscription.IgnoreSystemEvents || !evnt.EventType.StartsWith("$") to avoid sending system events? Not a huge deal, but it is a surprise to new users and sends unnecessary data.

[gregoryyoung]

I am referring to the pattern of writing a projection to make a stream of
linkTos for all as opposed to just reading all eg

fromAll().
when({
$any : function(s,e) { if(!e.name.startsWith($)) linkTo("myall",
e)}
})

On Fri, Apr 6, 2018 at 9:41 AM, Kasey Speakman [e-mail]
wrote:

So what's the issue with a $data_all stream that uses $all internally and
!subscription.IgnoreSystemEvents || !evnt.EventType.StartsWith("$") to
avoid sending system events?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/EventStore/EventStore/issues/718#issuecomment-379132783,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAXRWj0s8sV3Ecdv5IuPiWjtXzMQPop5ks5tltXGgaJpZM4GJId8
.

--
Studying for the Turing test

[kspeakman]

Ah yes, I understood. I was circling back to the original question of a built-in stream like $all but without the system events. :)

[Salgat]

@gregoryyoung I think he's asking specifically why we don't have an optional simple EventType filter in the server-side subscription code (see the code I linked).

[gregoryyoung]

Why bother with server side filtering? system events are usually sparse and
are very cheap to send. It could be implemented trivially I just don't see
much value in it. A well written subscriber can consumer 10k/s of
events/second.Are the 1% that are system events that important?

$all is also primarily meant to be a replication channel thus you can
listen to $all on one node and create an identical node

On Fri, Apr 6, 2018 at 10:02 AM, Kasey Speakman [e-mail]
wrote:

Ah yes, I understood. I was circling back to the original question of a
built-in stream like $all but without the system events. :)

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/EventStore/EventStore/issues/718#issuecomment-379135439,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAXRWv9fgOb4jfEkkB9_6L8bVOuVzd8Dks5tltrOgaJpZM4GJId8
.

--
Studying for the Turing test

[kspeakman]

It's not that important, no. It's just principle of least surprise. One less gotcha to remember. (The first time I subscribed, I was greeted with a crash b/c I was taking the type and using it to deserialize. But it was a system event... totally unexpected.)

I probably won't be writing the code to replicate and make an identical node tho. :)

[Salgat]

This caught my attention because we have started to take advantage of all stream catchup subscriptions and even having a couple system projections enabled like $by_event_type and $by_category can double/triple the number of events being sent through the subscription ("$>" event types). It'd be nice to avoid this unnecessary overhead (since I'm receiving the actual event anyways and don't need the duplicate unresolved linked event).

[mat-mcloughlin]

closed by #2009