Dapr: Multi PubSub 'PubSubName' Abstraction

[alex-doe]

In what area(s)?

/area runtime

Ask your question here

In the current Multi PubSub implementation the PubSubName is transported via the CloudEvent envelope.
We encountered problems of abstraction in two example scenarios - Here a simplified description:

A service (A) emits a message to the topic "somethingHappend" which is subscribed by two other services (B,C)
These services are, except for the abstracted PubSub channel, completely independent of each other.
But now all services must name their PubSub component identically, otherwise the messages on the topic cannot be consumed.
This creates a service coupling "by naming conventions"

Service (A) receives messages from a PubSub subscription which is not powered by a dapr hosted service (B)
Service B must now know under which PubSubName Service A has included the PubSub.
Because he additionally has to serialize the PubSubName into the CloudEvent envelope as an extension.

I hope this describes the problem to some degree.

shouldn't the PubSubName be noted in the context of the specific pubSub subscription and then be used for further resolution when the event occurs in the context of a message?

[youngbupark]

cc/ @yaron2 @mchmarny @orizohar

[mchmarny]

I may not get your specific use-case but, the topic name is only unique in the context of a specific component (e.g. comp1/orders or comp2/orders). For two services to exchange events they must be using the same, fully qualified topic. So yes, all services subscribing to the same topic must name their PubSub component identically.

We are in a process of enabling subscriptions declaratively, which may to some degree allow you to "de-couple" these subscribers. Would that help?

[alex-doe]

And this is exactly where we asked ourselves the, admittedly heretical, question whether it is necessary/correct that all services must share the same name for a pubsub component. This leads to a (technical) coupling between producer and consumer.

Maybe a more detailed description of one of our use-cases can give a better insight:

We have an external, onpremise [windows service ugh], system which emits events in Azure ServiceBus.
Now we consume these events via a dapr-enabled service in the cloud for further processing. If we emit the events onPremise we have to add the component name in the CloudEvent envelope according to the current dapr implementation.
If we now must change this component name, it is necessary to update all onpremise systems. This means that a technical implementation detail is dragged across the boundaries of an abstract pubsub component.

IMHO producer and consumer should not have a coupling.

Dapr Documentation:
https://github.com/dapr/docs/tree/master/howto/consume-topic#use-pubsub-to-consume-messages-from-topics

Pub/Sub is a very common pattern in a distributed system with many services that want to utilize decoupled, asynchronous messaging. Using Pub/Sub, you can enable scenarios where event consumers are decoupled from event producers.

MS Architecture Docs:
https://docs.microsoft.com/en-us/azure/architecture/guide/architecture-styles/event-driven

• Producers and consumers are decoupled.
• No point-to-point integrations. It's easy to add new consumers to the system.

Events, Data Points, and Messages by Clemens Vasters:

https://azure.microsoft.com/de-de/blog/events-data-points-and-messages-choosing-the-right-azure-messaging-service-for-your-data/

The core of the sales support application isn’t telling those two subscribers what to do and isn’t even aware of them. They are authorized consumers of events published by the source application, but the coupling is very loose, and removing these consumers doesn’t impact the source application’s functional integrity.

Please don't misunderstand, I would like to discuss if the current implementation does not break with the core idea of technological decoupling. Which, in my opinion, will make dapr a gigantic enrichment in software development.

But maybe I haven't quite understood the Dapr PubSub concept yet :-D
Here my understanding:

When starting the dapr Runtime all configured "PubSub" components are loaded.
Then for each component a subscription in the specific pubSub component implementation is ensured, and a new message callback is passed to the concrete component subscription. Because a subscription is created for each component, the name of the component could also be stored here and transferred for further routing when a new event/message occurs.

This would mean that the event producer does not need to have any knowledge about its consumers.
But maybe I understand it also completely wrong

[mchmarny]

Thanks for the additional clarity, @alex-doe. I think understand it now. Let me look into this and get back to you.

[mchmarny]

Looks like this was a side effect of the multiple pub/sub support we added in v.0.10. Fix is already in on schedule for v0.11 here