Eventstore: Concept of "Database"

[nbarnwell]

I know the current recommended approach to separating data within an ES instance is to prefix stream names with something. Perhaps the application name.

However there aren't really any good ways to manage/backup/restore/query using this approach, as it's only a convention. I'd love a way to delete all the data for a given app (say, for testing scenarios) from ES, without affecting data used by other systems on the same instance.

I know we could "just" (I hate that word) set-up more instances, so we have an instance per application/environment/clusternode, but that is more setup and headache from a sysadmin point of view (that is a LOT of ports to reason about), and somehow feels unnecessary.

Imagine being able to stop the instance, delete the database directory and then start it up, with the other databases intact.

At the moment, putting data into ES in a non-production, testing environment feels a bit like throwing a bucket of sand on a beach in the hope you can get it back later. You'd better have made a note of the stream IDs at the time...

[urbanhusky]

Multi-tenancy would be an interesting feature

[Narvalex]

Is there any update on this? Is the recommended aproach to just have multiple instances of EventStore on the same machine in order to achieve partition on a single machine? @gregoryyoung, @pgermishuys , @hayley-jean , @shaan1337 ?

[eesprit]

:+1: that would be really nice for multi-tenants applications, where you want to be able to backup/restore at the tenant level, or deleting all data relying to one tenant.

[wooderz]

I know I'm late to the party, but a plus 1 from me. I'm desperate for this feature

[kaancfidan]

This is also a problem for integration testing against EventStoreDB.

Since we can't setup and tear down a separate database instance per test case, we have to take extra care with tests that involve catch up or persistent subscriptions to projections. This "extra care" can sometimes mean we need to limit some scenarios as they would definitely interfere with others.

[StevenBlair123]

This is also a problem for integration testing against EventStoreDB.

Since we can't setup and tear down a separate database instance per test case, we have to take extra care with tests that involve catch up or persistent subscriptions to projections. This "extra care" can sometimes mean we need to limit some scenarios as they would definitely interfere with others.

We use docker for spinning up a new eventstore for each test. It adds a little overheard on each test, but generally, it's been fine for us (and we run 100's test)

[kaancfidan]

This is also a problem for integration testing against EventStoreDB.
Since we can't setup and tear down a separate database instance per test case, we have to take extra care with tests that involve catch up or persistent subscriptions to projections. This "extra care" can sometimes mean we need to limit some scenarios as they would definitely interfere with others.

We use docker for spinning up a new eventstore for each test. It adds a little overheard on each test, but generally, it's been fine for us (and we run 100's test)

We're using Gitlab CI as our CI/CD solution and specifying a single EventStoreDB service in docker-compose style is much more convenient than calling docker commands in test code. Also your solution adds explicit dependency to docker whereas just specifying a connection string from environment is much more portable (some developers might opt to just install EventStoreDB on their host machine for whatever reason).

My point is; yes there are workarounds but none of them are as convenient as just creating/deleting a separate database on a running instance.

[alexeyzimarev]

Yeah, I do the same with GitLab, use services in the pipeline. Transient (test) data is easy to handle when using the short-lived infrastructure.

[kaancfidan]

Yeah, I do the same with GitLab, use services in the pipeline. Transient (test) data is easy to handle when using the short-lived infrastructure.

How do you make sure the test cases in the same test session do not affect each other then?

When there are services to be tested that have subscriptions to projections, the test cases start affecting each other as the projections also include events from other cases.

[alexeyzimarev]

We always use something like Autofixture and all our stream names are randomly generated.

[kaancfidan]

We always use something like Autofixture and all our stream names are randomly generated.

Thanks for the hint.

We assign names to streams related to corresponding aggregates which is the standard way of doing things as I understand (e.g. User-79d435a3-c0c2-4f5f-baf7-00b6d2be6a38 for a particular user stream). We actually keep these names in service configurations and pass from environment but it never occurred to me that we could also use these names to separate test case data.

So if I understand what you mean correctly, we could change the example from above to something like this for each test case: User(some random text)-79d435a3-c0c2-4f5f-baf7-00b6d2be6a38.

Being able to separate databases would be much simpler but this is a workaround that we can use.

[kaancfidan]

We now have a use-case where we can't use the above workaround anymore. We have a service that subscribes to an event type projection (e.g. $et-SomethingHappened) and we can't change the event type names per test case.

So having a "database" instance per test would be a great solution for our current problem.

[Eshva]

Namaste! What's up to it? Do you plan to add the concept of database to EventStoreDB?