Nservicebus: Message routing polymorphism inconsistencies

[timbussmann]

Related to https://github.com/Particular/V6Launch/issues/68

Note: Moved the TL;DR; up for convenience of new readers. See the full configuration details and samples after the TL;DR; section

TL;DR;

Current state:

Messages types were a bit simplified to make the hierarchies easier to follow.

• V5:
  
  
  
  • Subscribing
  
  
  • Subscribing to Animal event with a route for a Cat event: :white_check_mark:
  
  
  • Subscribing to Cat event with a route for the Animal event: :no_entry_sign:
  
  
  • Sending
  
  
  • Sending a Animal message with a route for a Cat message: :white_check_mark:
  
  
  • Sending a Cat message with a route for the Animal message: :no_entry_sign:
  
  
• V6 MEM:
  
  
  
  • Subscribing
  
  
  • Subscribing to Animal event with a route for a Cat event :white_check_mark:
  
  
  • Subscribing to Cat event with a route for the Animal event :no_entry_sign:
  
  
  • Sending
  
  
  • Sending a Animal message with a route for a Cat message :no_entry_sign:
  
  
  • Sending a Cat message with a route for the Animal message :white_check_mark:
  
  
• V6 Code-First:
  
  
  
  • Subscribing
  
  
  • Subscribing to Animal event with a route for a Cat event :no_entry_sign:
  
  
  • Subscribing to Cat event with a route for the Animal event :no_entry_sign:
  
  
  • Sending
  
  
  • Sending a Animal message with a route for a Cat message :no_entry_sign:
  
  
  • Sending a Cat message with a route for the Animal message :white_check_mark:
  
  

where :white_check_mark: means the message is sent and :no_entry_sign: means an exception ("no destination found")

We discussed three potential solutions:

• Follow v5 (inheriting from children)
• Inherit from parent (~ OO inheritance)
• Require 1:1 mapping <---- We have chosen this one

Task-force Proposal:

While the first intention was to follow v5, we repeatedly felt this approach is just plain wrong since we were not able to come up with any scenario where this makes sense. When talking about inheriting settings, following the regular OO approach makes more sense (so Cat and Dog inherit from Animal, instead of Animal receiving both Cat and Dog routes).

After further discussions, we were questioning inheriting routes altogether and came up with the proposal that _every sent message must have matching route for that exact type_, no routes are inherited (you can still easily register all messages of a hierarchy using assembly overloads for the routing). In other words:

for commands, if you send X, you should have a mapping for X. And we should throw otherwise. Nothing else makes sense. And my gut is that for events, interpreted as "where should I send the subscription request" should work the exact same way. So if you have a handler for X, you should only send a subscription request if you have a configured publisher endpoint for X.

therefore on the comparison, this is the new proposal:

• V6:
  
  
  
  • Subscribing
  
  
  • Subscribing to Animal event with a route for a Cat event :no_entry_sign:
  
  
  • Subscribing to Cat event with a route for the Animal event :no_entry_sign:
  
  
  • Sending
  
  
  • Sending a Animal message with a route for a Cat message :no_entry_sign:
  
  
  • Sending a Cat message with a route for the Animal message :no_entry_sign:
  
  

(routing API and Message Endpoint Mapping should behave the same, therefore collapsed the specific options)

To be clear, this proposal is a breaking change in behavior from V5:

• For Commands:
  
  
  
  • In V5, if you sent a ShipProducts command but only had an endpoint mapping for ShipRefrigeratedProducts, it would work and the message would be delivered to that endpoint.
  
  
  • This doesn't make any sense, because what if you also have a ShipFrozenProducts? What then?
  
  
  • In this proposal for V6, trying to send ShipProducts without a mapping specifically for ShipProducts would not work and would throw an exception. You would need to create a mapping specifically for ShipProducts.
  
  
• For Subscription Requests (message-based pub/sub only)
  
  
  
  • In V5, if you had a handler for ProductsShipped (only) but only an endpoint mapping for RefrigeratedProductsShipped, the subscription request would be sent. When Refrigerated was published, you'd get a ProductsShipped and be able to deal with it on that basis only.
  
  
  • This doesn't make sense either, because what if there's FrozenProductsShipped?
  
  
  • In this proposal for V6, no subscription request would be sent and you would receive no message.
  
  
• The inverse cases are unaffected, i.e. if you sent ShipRefrigeratedProducts and only had a mapping for ShipProducts, you obviously can't send a message. True in V5 and will continue to be so in V6.
• At its core, this proposal boils down to say what you mean and mean what you say.

Please remember, this is about sending commands and subscription messages (if needed), not for publishing and does also not affect the way we handle polymorphic messages at the handling endpoint. If ShipRefrigeratedProducts arrives and you have handlers for both that and ShipProducts, both will be invoked within the same unit of work.

Issue:

We have major inconsistencies between on the behavior of routing for polymorphic messages and events between events and commands and v5 and v6. Let me summarize the current behaviors:

the messages used:

public class MyMessage : ICommand
{
}

public class MySubMessage : MyMessage
{
}

public class MyEvent : IEvent
{
}

public class MySubEvent : MyEvent
{
}

Version5:

Subscribing

bus.Subscribe(typeof(MyEvent));

+

  <UnicastBusConfig>
    <MessageEndpointMappings>
      <add Assembly="v5-assembly"
           Type="MySubEvent"
           Endpoint="..." />
    </MessageEndpointMappings>
  </UnicastBusConfig>

this works, using the endpoint configured for the sub class. (although MyEvent is not assignable to MySubEvent)

bus.Subscribe(typeof(MySubEvent));

+

  <UnicastBusConfig>
    <MessageEndpointMappings>
      <add Assembly="v5-assembly"
           Type="MyEvent"
           Endpoint="..." />
    </MessageEndpointMappings>
  </UnicastBusConfig>

this does not work and throws an exception because no route was found for MySubEvent (although MySubEvent is assignable to MyEvent).

Sending

bus.Send(new MyMessage());

+

  <UnicastBusConfig>
    <MessageEndpointMappings>
      <add Assembly="v5-assembly"
           Type="MySubMessage"
           Endpoint="..." />
    </MessageEndpointMappings>
  </UnicastBusConfig>

this works and is consistent with the subscription behavior. Same for the inverse case:

bus.Send(new MySubMessage());

+

  <UnicastBusConfig>
    <MessageEndpointMappings>
      <add Assembly="v5-assembly"
           Type="MyMessage"
           Endpoint="..." />
    </MessageEndpointMappings>
  </UnicastBusConfig>

This won't work (again, consistent with subscribe)

V6

Subscribing

endpoint.Subscribe(typeof(MyEvent));

+

  <UnicastBusConfig>
    <MessageEndpointMappings>
      <add Assembly="v6-assembly"
           Type="MySubEvent"
           Endpoint="..." />
    </MessageEndpointMappings>
  </UnicastBusConfig>

This works without any exceptions, same as in v5. Whereas

routingConfig.RegisterPublisherForType("...", typeof(MySubEvent));

doesn't work and throws an exception.

The inverse scenario:

endpoint.Subscribe(typeof(MySubEvent));

+

  <UnicastBusConfig>
    <MessageEndpointMappings>
      <add Assembly="v6-assembly"
           Type="MyEvent"
           Endpoint="..." />
    </MessageEndpointMappings>
  </UnicastBusConfig>

or

routingConfig.RegisterPublisherForType("...", typeof(MyEvent));

both don't work.

Sending

endpoint.Send(new MyMessage());

+

  <UnicastBusConfig>
    <MessageEndpointMappings>
      <add Assembly="v5-assembly"
           Type="MySubMessage"
           Endpoint="..." />
    </MessageEndpointMappings>
  </UnicastBusConfig>

This doesn't work (whereas it does in v5). Same for the code first API:

routingConfig.RouteToEndpoint(typeof(MyMessage), "...");

This also throws an exception because no route was found.

But the following configurations work in v6:

endpoint.Send(new MySubMessage());

+

  <UnicastBusConfig>
    <MessageEndpointMappings>
      <add Assembly="v5-assembly"
           Type="MyMessage"
           Endpoint="..." />
    </MessageEndpointMappings>
  </UnicastBusConfig>

or

routingConfig.RouteToEndpoint(typeof(MySubMessage), "...");

[danielmarbach]

I think we should align with V5 and change to the following

• V6:
  
  
  
  • Subscribing
  
  
  • Subscribing to base event with a route for a subclassed event
    
    
    
    
    
    • MessageEndpointMappings: :white_check_mark:
    
    
    
    • Code first routing API: :white_check_mark:
    
    
    
  
  
  • Subscribing to subclassed event with a route for the base event
    
    
    
    
    
    • MessageEndpointMappings: :no_entry_sign:
    
    
    
    • Code first routing API: :no_entry_sign:
    
    
    
  
  
  • Sending
  
  
  • Sending a base message with a route for a subclassed message
    
    
    
    
    
    • MessageEndpointMappings: :white_check_mark:
    
    
    
    • Code first routing API: :white_check_mark:
    
    
    
  
  
  • Sending a subclassed message with a route for the base message
    
    
    
    
    
    • MessageEndpointMappings: :no_entry_sign:
    
    
    
    • Code first routing API: :no_entry_sign:
    
    
    
  
  

Unless the task force is able to outline all the implications of making more scenarios possible I'd vote at this stage align with V5

[DavidBoike]

I agree with V5 and with sending being consistent with subscription behavior. Agree with @danielmarbach that we should align with V5 unless we have a real good reason not to.

[timbussmann]

let's include this on the Particular/V6Launch#68 PoA then?

[danielmarbach]

:+1:

[timbussmann]

since we currently decided to fix this for v6, this may have strange behavior when using multiple sub classes. E.g. you use SubClass1 and SubClass2 deriving from BaseClass, you need to configure a route for both SubClass1 and SubClass2, since they can't inherit the routing from the BaseClass. In case you now wan't to send a BaseClass message, it's not even clear what the behavior would be (since currently it would send 2 messages if SubClass1/2 are configured to be handled at different locations. If we fix that, one route will win over the other).

inheriting routes from the base message seems more natural to OO design rather than "leaking" routes from child to base messages. Is this something we should address/discuss further (maybe a pdev issue)?

[DavidBoike]

Hmm, that's neat.

In V5, if I have a base type and 2 child types, and I try to send a message specifically for each one...

If my config looks like this:

<MessageEndpointMappings>
  <add Messages="WeirdRoutingTricks.BaseMsg, WeirdRoutingTricks" Endpoint="WRTBase" />
  <add Messages="WeirdRoutingTricks.Child1, WeirdRoutingTricks" Endpoint="WRTChild1" />
  <add Messages="WeirdRoutingTricks.Child2, WeirdRoutingTricks" Endpoint="WRTChild2" />
</MessageEndpointMappings>

Then WRTBase receives no messages, and WRTChild2 receives the Base + Child2 messages.

OTOH, if I have this config:

<MessageEndpointMappings>
  <add Messages="WeirdRoutingTricks.Child1, WeirdRoutingTricks" Endpoint="WRTChild1" />
  <add Messages="WeirdRoutingTricks.Child2, WeirdRoutingTricks" Endpoint="WRTChild2" />
  <add Messages="WeirdRoutingTricks.BaseMsg, WeirdRoutingTricks" Endpoint="WRTBase" />
</MessageEndpointMappings>

Then as you might expect, each endpoint gets the message destined for it.

So that's definitely an odd, last one wins scenario.

But I think you need to disabuse yourself from "more natural to OO design". This isn't classes and polymorphism exactly. This is more like sending letters to ACME Corp, and to the Sales/Accounting departments within ACME. ACME has a building, the Sales Department is on the 3rd floor, and Accounting is on the 7th floor.

Just because you know the floors for Sales and Accounting don't mean you can send a message addressed to ACME to either of those floors. You need to know which floor the general mail room is on.

And, just because you know the address for ACME at large (let's pretend the mail room won't forward messages to the correct floors) you can't send a message for the Sales department to the front door.

I almost think that if you are going to Bus.Send() a message of type T, you should perhaps be required to have a mapping specifically for exactly type T?

It's OK to send a Child (which inherits from base) and not have a mapping for Base, however, as long as you're not trying to specifically send a Base. Because the whole point is for the receiver to be able to run handlers for both Base and Child.

Does that sufficiently confuse the issue?

[bording]

@DavidBoike For me this actually clarifies things a good bit. The inheritance chain of the command should only matter once the message is delivered, and the receiver has to decide which handlers to execute.

From the sending side, each type should have its own separate mapping, because there should be one destination for each command. Trying to inherit the mappings here just confuses the one-to-one nature of commands.

[bording]

Even if you want to relax the "each type should have its own separate mapping" restriction then that could work too if the logic is to check the routing table for the actual type, and then walk up the inheritance hierarchy, and stop once you've found a single match. That would let you have a single base entry if you wanted, but if you also added derived entries, those would work too, and order of the registrations wouldn't matter.

There would need to be some different logic for events, but if you follow the same basic logic, but walk the entire hierarchy instead of stopping at the first match, that should get you the entire list of subscribed destinations.

[timbussmann]

Even if you want to relax the "each type should have its own separate mapping" restriction then that could work too if the logic is to check the routing table for the actual type, and then walk up the inheritance hierarchy, and stop once you've found a single match.

that would be the OO approach, essentially finding the closest match? From a technical perspective we probably don't want to walk hierarchies at runtime, but let's keep that out of the converation for now, that's solvable.

Just because you know the floors for Sales and Accounting don't mean you can send a message addressed to ACME to either of those floors. You need to know which floor the general mail room is on.

That analogy doesn't really hold up? Usually you don't need to know where the mail room is, that's handled internally? If all departments are in the same building, you just send it to that address and trust in the company being able to dispatch the message internally (essentially the handlers?). The scenario gets more to what we're talking about if these departments are in different buildings, having different addresses. If you send a base message, you send it to the "headquarter" not to the sales department in the other building (and this is what we currently are doing).

I'm not sure I get the "desired" behavior from your comment @DavidBoike

But again, I think this conversation maybe belongs to PDev?

[Scooletz]

@timbussmann Just a note. EventStore uses a simple MessageHierarchy caching all the hierarchical information on the start.

[timbussmann]

@Scooletz we're having the MessageMetadataRegistry which also caches the hierarchy information. But we need to figure out how to make better use of it belongs to the SerializationFeature

[danielmarbach]

Not sure it belongs to SerializationFeature. It just happens to be created as part of that feature. I think the boundaries there are all wrong. I think everything related to message metadata, messages actually belongs to routing. Serialization can get info about messages from routing when needed but not the other way around.

[Scooletz]

Yes @timbussmann I've just found it. Scratch my previous comment.

[johnsimons]

Was this tested only with MSMQ ? Rabbit also supports it, how does Rabbit behave ?

[bording]

Was this tested only with MSMQ ? Rabbit also supports it, how does Rabbit behave ?

Rabbit doesn't have to send subscription messages, so that part isn't relevant for Rabbit.

For sending commands, the behavior would be the same, since you use the same APIs to register the logical sending destinations.

[johnsimons]

Rabbit doesn't have to send subscription messages, so that part isn't relevant for Rabbit.

I know, hence the reason I am asking.
IMO it should be consistent across transports

[johnsimons]

:+1: to throw on sending an Animal to a Cat mapping, in v6

Also consider patching v5, since this can result in message being sent to an endpoint that can't consume it, hence message would be send to error.

[mikeminutillo]

• In V5, if you had a handler for ProductsShipped (only) but only an endpoint mapping for RefrigeratedProductsShipped, the subscription request would be sent. When Refrigerated was published, you'd get a ProductsShipped and be able to deal with it on that basis only.
• This doesn't make sense either, because what if there's FrozenProductsShipped?
• In this proposal for V6, no subscription request would be sent and you would receive no message.

What if RefrigeratedProductsShipped and FrozenProductsShipped come from different endpoints. I should be able to set up a generic subscriber and have it get messages from both sources shouldn't I?

[DavidBoike]

@mikeminutillo So you would then have a handler for ProductsShipped, but mappings for Frozen/Refrigerated, but no mapping for the base?

I think you may be on to something here.

[timbussmann]

@mikeminutillo @DavidBoike I think this is a similar scenario to the v1/v2 messages scenario we talked about? So in essence: If these events are sent from entirely different endpoints, they probably shouldn't use any inheritence in the first place. Otherwise it should still work as long as you manually subscribe to RefrigeratedProductsShipped and FrozenProductsShipped (because autoscubscription won't do that without matching handlers) and provide a publisher route for each event (talking about message driven pub/sub transports obviously)?

[danielmarbach]

@DavidBoike the scenario outlined by @mikeminutillo is a pretty common one. Especially when you split your service into business components and the components into individual autonomous components. But If I understand the proposal correctly the scenario @mikeminutillo described would work due to the polymorphic dispatch on the endpoint.

When you have

class Handler : IHandleMessages<ProductsShipped>
{
    void Handle();
}

and routes for Frozen/Refrigerated then the above handler would be invoked due to polymorphic dispatch.

What confuses me though a little bit in the discussion is the following:

Please remember, this is about sending commands and subscription messages (if needed), not for publishing and does also not affect the way we handle polymorphic messages at the handling endpoint

and the messages outlined here sound more like events (that would be published) instead of sent. I think for this conversation it is crucial to not mix up the two.

[danielmarbach]

@udidahan I think this issue requires your involvement as a specialist!

[danielmarbach]

Btw. I disagree with @johnsimons about backporting to v5 unless we have desk cases and data that shows it is a real problem for our customers.

[danielmarbach]

@timbussmann

they probably shouldn't use any inheritence in the first place.

I think that statement doesn't hold up entirely. See https://github.com/Particular/NServiceBus/issues/3911#issuecomment-234179127

[udidahan]

For this scenario:

What if RefrigeratedProductsShipped and FrozenProductsShipped come from different endpoints. I should be able to set up a generic subscriber and have it get messages from both sources shouldn't I?

As @danielmarbach stated, that would occur when a publisher decides to split itself into two business components.

The thing is that in that case it would be the responsibility of the publisher to make that work - meaning having shared "subscription storage" between the two new publishing endpoints. There should be no requirement to change subscriber configuration.

[timbussmann]

and the messages outlined here sound more like events (that would be published) instead of sent. I think for this conversation it is crucial to not mix up the two.

well the reason we're talking about events is mainly because of the subscription messages. The proposal seems to hold up for commands so far? The danger here is the autosubscribe feature which may behave differently resulting in "missing" events.
If you have an endpoint publishing FrozenProductsShipped AND ProductsShipped the ProductsShipped wouldn't arrive at the subscriber if the subscriber only registers a publisher for FrozenProductsShipped although the subscriber has a handler for both messages because (because no subscription message for ProductsShipped will go out because of no configured destination. I think this would behave differently in rabbit/broker based transports because autosubscribe doesn't require a publisher address for subscriptions and therefore can subscribe to both events if I understood that correctly. Also that would currently work in v5 because we find the Cat(FrozenProductsShipped) route when trying to subscribe to Animal (ProductsShipped) and therefore two subscriptions go out.

This gets problematic at the point where you have different endpoints for FrozenProductsShipped and RefrigeratedProductsShipped where both endpoints also publish a ProductsShipped because you can't configure more than one publisher for ProductsShipped (unless they use a shared subscription storage as mentioned by Udi).

[ramonsmits]

We have sending and publishing. Assumption one, sending a _command_ always has one logical destination when not explicitly setting a destination.

Commands are defined by the target. We could have a situation where the target wants to stay compatible with a previous version of a command and want to share a large chunk of the processing code base. If a handler doesn't exist (anymore) for a base class command type that the endpoint previously has owned then it rightfully gets forwarded to the error queue. Deprecating that command should result in empty handler making that fact explicit or should have some kind of null handler.

Probably the only question that needs to be asked is if we have Animal, could any of its children be allowed to be send to a different logical endpoint? If that is no, then why would we need to explicit routes for any descendants? Needing to map those explicitly doesn't make sense and the same applies to subscriptions. If a yes, then you will have to make weird routing decisions.

Any inheritance chain of classes or interfaces should never result in having multiple logical endpoints thus owners / roots (which potentially can happen as interfaces allow multiple inheritance).

Second, how often will people define routes based on types? Wouldn't it make more sense to based that on assemblies but here the more interesting thing would be _what when the base class is in a different assembly?_

All of this is based when looking at it from the behavioral and configuration perspective. Not specific to the current implementation or behavior of either V5 or V6.

I would state:

• Only configure inheritance roots as routes
• If we detect no routes on the root(s) throw an error
• If we detect multiple routes with different logical destinations via the routes throw an error
• If we detect multiple routes with the same logical destination raise a warning stating only the root(s) need to be mapped

Any other mapping would trigger my _why would you do that_ sensor. Maybe there are use cases and if these exist should then be part of this issue as a requirement.

[timbussmann]

Second, how often will people define routes based on types? Wouldn't it make more sense to based that on assemblies but here the more interesting thing would be what when the base class is in a different assembly?

I think that's an important point. As soon as you use assembly based routings, you will have routes for all messages in the hierarchy, which may also lead the autosubscription to a different behavior compared to specific type routes when requiring explicit mappings.

[DavidBoike]

From @mikeminutillo:

What if RefrigeratedProductsShipped and FrozenProductsShipped come from different endpoints. I should be able to set up a generic subscriber and have it get messages from both sources shouldn't I?

You had me going for a bit, but the comments from Daniel and Udi about BCs solidified my thinking that this proposal is correct.

Those two events can't "come from" different endpoints. In fact, events don't "come from" endpoints at all. When a message-driven pubsub transport publishes an event, it queries subscription storage by Type and returns a collection of destination endpoints.

That's why what I said before is so important:

Please remember, this is about sending commands and subscription messages (if needed), not for publishing...

In Mike's situation, and as Udi described, it's up to the publisher to make this work. One physical endpoint should be responsible for the subscription requests for (and share subscription storage with) all endpoints involved.

And to @ramonsmits's point, this does get a lot easier in aggregate when you based routings on assemblies, because you naturally send subscription requests for related events to the same address based on assembly. But we have to break it down to the per-type logic because that's what it boils down to when making a decision for each individual message type.

So back to Mike's situation. Those 2 messages can't "come from" different endpoints. You would have a handler for ProductsShipped and configure the publisher to be the endpoint designated to handle subscription requests for ProductsShipped and all of its inherited forms.

[Scooletz]

Does implementing multiple interfaces like

public interface IFrozenProductsShipped : IProductShipped, IOperationExecuted{}
public interface IProductShipped : IEvent {}
public interface IOperationExecuted : IEvent {}

should be taken into consideration as well?

[timbussmann]

I'll try to summarize some of the points from the discussion and the behavior of the proposal here, I hope it makes sense:

(Assuming that publishing events respects subscription hierarchies, e.g. when publishing a Dog message an Animal subscriber will receive the message if there is a subscription for Animal)

The proposed rule is: routings can not be inherited or passed on

This means:

• If you send Dog, you need a route for Dog otherwise it will throw
• If you subscribe to Cat you need a route for Cat otherwise it will throw

Regarding subscriptions (this is no change from the current implementation afaik):

• If you want to receive Dog, you need to subscribe to Dog
• If you want to receive Animal, you need to subscribe to Animal

Note the emphasis on receive. If you have a handler for Animal, that still gets invoked when handling a Dog.

Whereas Autosubscription automatically tries to subscribe to every event it finds a handler for.

ALL transports need to explicitly subscribe to Animal if they want to receive Animal (I think this is not a new rule and actually is not really part of the proposal?). Autosubscription usually already takes care of this but you have finer grained control when using manual subscriptions. For non-brokered transports, the proposed change is, that this also requires an explicit route for Animal, even when there already is a route for Dog. As long as you're using assembly based registrations, this is no change and behaves as for example RabbitMQ. If you explicitly map types to addresses, you would now have to add a new entry for Animal (probably to the same destination). We probably would have to ensure/verify that we don't throw in case autosubscribe tries to subscribe to Animal but finds no publisher route for it.

So when we compare autosubscriptions between MSMQ and Rabbit, there may be a difference:

• Rabbit will always subscribe to Animal
• MSMQ will only subscribe to Animal if it can find a route for it (or should it throw an just require a route?)

manual subscriptions behave the same. If you're using manual subscription on endpoint X which has handlers for Animal and Dog but you only subscribe to Dog, you won't receive the Animal event from publisher Y without subscribing to Animal explicitly (whereas you now need to have a route when using MSMQ for that). This should be true for all transports?

@DavidBoike @bording did I get something wrong?

[SzymonPobiega]

@timbussmann so on MSMQ, when I have a know who's publishing Animal and I have a handler for Dog, would it work? I think that the _single logical publisher_ rule implies that if Dog is a specialization of Animal, they have to have the same publisher.

[timbussmann]

@timbussmann so on MSMQ, when I have a know who's publishing Animal and I have a handler for Dog, would it work? I think that the single logical publisher rule implies that if Dog is a specialization of Animal, they have to have the same publisher.

so you have Endpoint X with a handler for Dog and a registered route for Animal to Endpoint Y and Endpoint Y publishes Animal?

In that case, endpoint X would not subscribe at all to Y (unless you also have a handler for Animal. So publishing Animal or Dog would not arrive at X.

I think that the single logical publisher rule implies that if Dog is a specialization of Animal, they have to have the same publisher.

I think the goal of the proposal is to make the concepts and the code easier by eliminating implications.

Just because we support polymorphic messages and handling of polymorphic messages shouldn't force us to also implement polymorphic routing. let's keep it simple (again, when using assembly based routing and autosubscription, this scenario would work).

[DavidBoike]

@bording and I discussed this together.

@timbussmann: If I understand correctly, you're restating the original thesis of the proposal, but we're missing the "aha" moment that I realize now we only discussed in Slack. We need to handle the situation relating to this comment from Udi on a very old issue: https://github.com/Particular/NServiceBus/issues/1583#issuecomment-25155437 Or maybe you are stating this, but we're not properly understanding you.

If Dog and Cat are published from (really, subscriptions are managed from) different endpoints (against our guidance) then you would need to do:

// Consider this pseudo-code, can't remember the current API as it's in flux anyway.
// Also, this config is for an endpoint that ONLY has a handler for Animal
cfg.RegisterPublisher(typeof(Animal), "EndpointPublishingDog");
cfg.RegisterPublisher(typeof(Animal), "EndpointPublishingCat");

As a result of this config, subscription messages would need to be sent to both EndpointPublishingDog and EndpointPublishingCat.

This would take care of Mike's situation above in https://github.com/Particular/NServiceBus/issues/3911#issuecomment-234150159, would address what Udi says in https://github.com/Particular/NServiceBus/issues/1583#issuecomment-25155437, and align with how Rabbit (or the new topology for Azure) would work as well.

[timbussmann]

If I understand correctly, you're restating the original thesis of the proposal

yes

but we're missing the "aha" moment

I just tried to restate the "current" porposal after the discussions and to separate some of the concerns (publishing/registering/handling/routing) to show where we really want to make a change and where not, because we're always talking about the the point where events are published, not about the subscriptions routing.

your sample will work with our proposal under the following conditions:

• You are using auto-subscription and also have a handler for Animal (while you also can have handlers for Dog and Cat, but we would need to make sure autosubscribe doesn't throw because it can't find routes for Dog/Cat). You already stated in the sample that this would be the case.
• OR you are manually subscribing to Animal (without the need of having an Animal handler).

These situations work in MSMQ because

• the subscriber subscribes for Animal on both registered endpoints (because of autosubscribe or a manual subscription).
• when publishing, as stated in my comment before, subscriptions to base events are also considered. When publishing Dog it will also send that message to all Animal subscribers.
• At the receiver, the message is still of type Dog and through polymorphic handles, both Dog and Animal handlers will be invoked, although we only subscribed to Animal.

I think the conditions stated above also apply for RabbitMQ. But this scenario is not affected by our proposal at all. We don't change anything on that behavior compared to before, since the publisher registrations are already using Animal directly (so no inheritance used in the first place). We would change behavior in this slightly different scenario:

// Endpoint with autosubscribe and only a handler for `Animal`
cfg.RegisterPublisher(typeof(Dog), "EndpointPublishingDog");
cfg.RegisterPublisher(typeof(Cat), "EndpointPublishingCat");

this scenario won't work, because autosubscribe will try to subscribe to Animal (because we don't have handlers for Dog/Cat) and it can't find a route because it will no longer "inherit" the route from Dog/Cat as it did previously. Again, I think this is valid behavior. If you want to subscribe to Animal, have a route for Animal, simple rule. The other breaking change is the scenario described by @SzymonPobiega on this comment (very similar to david's example, but no handler or explicit subscription for Animal)which I think we can live with for the sake of a cleaner mental model (still complicated though).

[timbussmann]

the task-force decided on a call that we're going to continue with the proposal since there were no major objections while we are able to provide consistency across the transports, simplify the rules and nearly full backward compatibility. The scenarios which are affected should be very rare and there are known ways how to make them work even with the new approach.

[SzymonPobiega]

Can somebody involved summarize the decision made on that call? I specifically don't follow the idea that a command should require specific routing i.e. if I have SalesCommand base class and all my sales endpoints inherit from it, I see value in configuring routing for the SalesCommand.

[timbussmann]

Can somebody involved summarize the decision made on that call?

@SzymonPobiega the decision made is stated above. There were no new arguments other than nobody felt involved enough for a week to actively chime in and challenge the proposal during the RFC process.

i.e. if I have SalesCommand base class and all my sales endpoints inherit from it, I see value in configuring routing for the SalesCommand.

we were not able to come up with a scenario where you end up with that kind of inheritance and configuration. Message inheritance shouldn't be a mechanism to require less routing configurations since it may conflict with the behavior when handling/subscribing messages. In your scenario, can still use the assembly based overloads to quickly configure a route for all messages within an assembly/namespace without relying on inheritance or you can configure your own routing rule which inspects message hierarchies.

[ramonsmits]

By having all these routes explicit would it then make sense to serialize a different message by route type?

If a Dog is published and we have a route for Animal, would it be possible to only serialize Animal for that route?

Sending the whole Dog message currently potentially leaks data. Currently, you would need to split these in two types if you do not want to disclose data. By not inheriting routes and making each route explicit on type this seems to potentially solve that problem if the step that does the serialization is aware of this route destination meta data.

This has been a security concern for some time and I remember talking with @jdrat2000 about this.

[timbussmann]

If a Dog is published and we have a route for Animal, would it be possible to only serialize Animal for that route?

Careful. Publishing doesn't use routes. Publishing queries subscribers. We don't change anything on that part, that's not in scope. We're not changing anything on that side so no changes regarding that scenario.

Sending the whole Dog message currently potentially leaks data.

That sounds like an entirely different issue to me which is not in scope for routing itself.

[ramonsmits]

Careful. Publishing doesn't use routes. Publishing queries subscribers.

Publishing doesn't use a route directly but is querying the subscribers on type where the result is a route isn't it?

That sounds like an entirely different issue to me which is not in scope for routing itself.

I'm not saying we should change the message (de)serializering behavior within the scope of this issue. I was just wondering that if we have routes explicit on type because each type in the inheritance graph will now have an explicit route if it could be used for that.

[timbussmann]

Publishing doesn't use a route directly but is querying the subscribers on type where the result is a route isn't it?

no, the result is an address (we differentiate this from a route within routing) provided when subscribing. The query does check for subscribers on a base class though, but we're not touching the inheritance on that part. That's more something related to polymorphic handlers than route inheritance (and to behave the same way broker based transports do).

[SzymonPobiega]

I'd like to revisit the decision made here. We still haven't updated the upgrade guide on the MEM interpretation changes. I think we should try to stick to the old behavior for MEM and implement the new no-inheritence policy only for the code-first APIs.

I think it is more important to make the upgrade easier than to make MEM and code-first consistent since we want to drop MEMs eventually.

Another question is the handling of overlapping registrations. V5 in MEM uses two different approaches. For the command routes it overrides less-specific routes with more specific (type > namespace > assembly in terms of route specificity). For the event routes, it summarizes the route specifications so if, for a given type, an assembly, namespace and type routes match, it will have *_three_ registered publishers.

I belive we should keep that (arguably awkward) behavior in V6 when it comes to MEM but I believe we should not allow it in code-based APIs where we should actually use the overriding rules similar to the ones we use for commands. If somebody really needs to have two publishers, he can get low-level and use one of the different PublisherAddress factories.

@timbussmann @DavidBoike @bording what do you think?

[DavidBoike]

I don't have strong feelings about the MEM either way, since we're phasing them out, and since every case that would be affected is kind of an edge case anyway.

[bording]

@SzymonPobiega Overall, I'd prefer if everything worked in the way we described in this issue, even if that does mean changing the MEM behavior. Having to document that we have two different ways to configure routing and that they work differently (and have the code for these paths work differently) seems more trouble than documenting the behavior changes in the upgrade guide.

But, if everyone else prefers to have the MEM work like V5, I can live with that.

Regarding the publisher stuff, I think that the RegisterPublisher API should allow for multiple publishers and not override them. Otherwise behavior we described above, which matches how brokers like the RabbitMQ transport already behave, can't work.

[timbussmann]

I'm with @bording, Having a different behavior on each API is probably more trouble than emulating the same behavior as in v5 for edge cases. I think we should have a section in the upgrade guide, pointing to users what kind of MEM may change behavior with the migration and how you would need to configure them in v6.

I don't have a very strong opinion about it though. If we feel that the chances are very high that we introduce a major issues while upgrading, that might change my opinion.

[SzymonPobiega]

There is no major risk for commands because if V6 fails to find a route, it will throw and the user will contact us.

Events are worse. If you upgrade everything will continue to work because the subscription stores are already populated. But once you reset them (which might happen long after the deployment of upgraded version), the subscriber might be missing some routes and will not re-subscribe for some events. There will be no exception, messages will just stop to flow. I think we should fix it because it is hard for me to imagine how we would explain that situation to a user in a desk case.

[SzymonPobiega]

@timbussmann @bording I think I know why V5 behaves it does with regards to pub/sub routing.

The following config:

UnicastBusConfig>
    <MessageEndpointMappings>
      <add Assembly="Subscriber" Type="Subscriber.Dog" Endpoint="Dog" />
      <add Assembly="Subscriber" Type="Subscriber.Cat" Endpoint="Cat" />
    </MessageEndpointMappings>
  </UnicastBusConfig>

makes sense if you think about it in terms of _message ownership_ (which was the way MEMs were documented in V5). In that particular system there is no owner of the Animal event. Concrete animals Dog and Cat however do have owners. It does make sense from the logical perspective if you look at the MEM as centralized registry of which endpoints own which concrete messages rather than a local configuration for routing.

What I think is broken is that in V5 it does not work the opposite direction i.e. if there is a single known owner of Animal event I should be able to subscribe to Dog and Cat without any additional config.

With regards to commands I would argue that the original V6 behavior was correct. If there is an owner (meaning somebody who handles) a given base command Animal, that information should be enough to route derived commands Dog and Cat. That said I don't have a super strong opinion we should bring that behavior back but I do have a strong opinion we should process events the same way V5 did.

[timbussmann]

so based on the discussion here it looks we have a "major issues while upgrading" which means that we need to align the MEM behavior to the v5 behavior?

[timbussmann]

I like @SimonCropp's idea here https://github.com/Particular/NServiceBus/issues/1942#issuecomment-240718088 but I'm not sure I can grasp the full impact of such a change (regarding the behavior and the total effort) instantly.

[bording]

I'm still of the opinion we can make the MEM work like was proposed, and including throwing if we don't have a route for a handler, as mentioned by @SimonCropp, shouldn't that actually get us to where we wouldn't be losing messages?

As mentioned here https://github.com/Particular/NServiceBus/issues/1942#issuecomment-34580529, I definitely prefer treating the handlers as the intent to subscribe, as it is for broker transports. If we do that, then we should be able to throw if we don't have a configured publisher for a handled type.

[SzymonPobiega]

It makes sense to throw when there is no route but I still believe that MEM semantics are different than code API semantics and, while I can understand why in code APIs we don't want inheritence support, I do believe that in MEMs the inheritence support is OK.

[SzymonPobiega]

@bording Throwing when there is no route sounds good but this is also a breaking behavior. Endpoints that did work in V5 would not start, throwing exceptions. Because MEMs differ from environment to environment there is chance the exception will be only observed on production. This is still way better then lost messages. If you and @andreasohlund are fine with this behavior, I am fine too and we can finally implement this last piece of routing puzzle.

[SzymonPobiega]

After talking to @andreasohlund we concluded the best option to bite the bullet is to support the MEMs the same wasy as V5 does it. It saves us upgrade issues. "Fixing" the usage of MEM (by adding additional checks) seems to be not worth the effort of us and users since we plan to obsolete them in the next major.

[bording]

So that means we'd have the code APIs act like we proposed originally here, and the MEMs work like they do in V5? If can actually maintain separate behaviors for both, then I guess that's something I can live with since we're getting rid of MEMs soon.

Even if we have MEMs working as they did in V5, don't we still need to throw if we don't have a route for the handler?

[andreasohlund]

don't we still need to throw if we don't have a route for the handler

I think its ok for users to no longer have routes for some handlers/messages temporarily while evolving parts of their systems. Us throwing would just cause pain in the form of false negatives.

We should IMO make sure that:

• We throw if a message is received but there is no handlers (or a callback), this already works
• Throw a good error if you try to send a message with no routing. Needs to be verified (already covered by https://github.com/Particular/NServiceBus/issues/2931#issuecomment-241734326

[timbussmann]

what's the status on this? I assume we can close this? ( @SzymonPobiega can you maybe adjust the initial text to summarize the current behavior correctly?)

[SzymonPobiega]

The end result is V6 tries to handle MEM routes the same way as in V5 (respecting inheritence). It does not respect inheritence in the code-first APIs. I raised an issue to document the route inheritence mechanism.