Nservicebus: Independent instances

[SzymonPobiega]

taskforce: @SzymonPobiega @tmasternak

Problem

If we adopt the container/compartment/endpoint identity idea we will always have an endpoint ID so we need to use an explicit API to decide if endpoint is going to use it to create individualized queues
ASB V5 has an option to use individual queues so events are delivered to each instance. This is important both for backwards compat and for data distribution scenarios

The solution has been rejected after the discussion and workaround has been proposed. The workaround needs a POC to be finally accepted.

Possible solution

Three modes of operation with regards to definition of the instance:

• Sharing
• Competing
• Independent

Default is "sharing". Callbacks require at least "Competing".

Sharing

All instances of the endpoint connected to the same broker node compete receiving from the same queue. They all share the same address. That single shared address is used as subscriber address. From outside a cluster is indistingiushable from a single-instance endpoint.

Competing

All instances of the endpoint connected to the same broker node compete receiving from the same queue. In addition to that, each instance has its own address based on its unique identity. The shared address is used as subscriber address to ensure each event is delivered to a single instance of a scaled-out competing endpoint.

Independent

Each instance is logically independent of others and has its own address. No shared queue. The individual address is used as subscriber address to ensure each instance gets a copy of each event. Commands, by default, are routed round-robin.

Rationale

There seems to be value in "independent" mode that is not limited to ASB backwards compat scenarios. Imagine an endpoint which represents a type of IoT device (a light switch). It makes sense to treat all instances of light switch manager as a single endpoint yet each instance is independent and should get a copy of an event if it is subscribed to one.

Decision made

The decision has been made to not solve the problem of data distribution at this point (V6) and come back to it later, possibly in form of a backplane (proposed by @mauroservienti ). Instead, a workaround will be proposed for Azure customers who want described data distribution / cache invalidation feature. The workaround is based on usage of an additional endpoint instance co-hosted with the primary one. The additional instance would have a name composed of a base name and role ID. This will force the NSB routing to treat each of these as an independent logical endpoint (not instances of same endpoint) and events will be routed to each one.

[SzymonPobiega]

This is a _RFC_. The original discussion happened on Slack cc @yvesgoeleven @andreasohlund @tmasternak

[SzymonPobiega]

@Particular/engineers @udidahan please comment.

[mauroservienti]

Is this to re-introduce the QueuePerInstance thing that we just decided to drop?

[SzymonPobiega]

@mauroservienti it turned out QueuePerInstance has value in some scenarios. The question is if we are going to hack it (e.g. include Azure role instance ID in endpoint name) or support it in the core.

[udidahan]

Data distribution is currently not a requirement.

What exactly is the backwards compatibility issues that would be caused by the "endpoint ID / routing instance ID" proposal?

[yvesgoeleven]

@udidahan in the new implementation (core v6), when instance identification is on, there will be a receive infrastructure for both the instance level and the shared level. (this is different from v5) This will cause the 'old' ASB topology to subscribe to events both on the shared and on the individual level, leading to message duplication for all events.

[yvesgoeleven]

@udidahan The data distribution scenario is quite common in azure though, people use nsb that way. As the caching for webrole instances has had a bumpy history (of many unreliable solutions), some people rely on regular machine memory as a cache and use invalidation events to ensure all machines update their memory state

[mauroservienti]

Isn't this something that already has a solution? backplanes are the solution to distribute data among instances of the same endpoint.

[yvesgoeleven]

@mauroservienti explain? what backplanes? (let's move this to slack)

[SzymonPobiega]

My approach (after processing what @yvesgoeleven explained to me) is that these instances are more like independent endpoints that happen to share to codebase. Their behavior is more similar to a stand-alone non-scaled out endpoint.

[mauroservienti]

@yvesgoeleven when using for example SignalR, but any other IoT thing behaves like a SignalR WebSocket connection, the connected party is physically connected to an instance, so far so good.

Obviously in a scaled-out environment, using competing consumers, the endpoint instance receiving the message may not be the one where the client (IoT or SignaleR or ...) is connected.

A backplane is the suggested approach in such scenarios, a backplane behaves like a bus connecting only the scaled out instances, so what happens is that the receiving endpoint instance will deliver to all the scaled out instance a backplane message notifying all the instances that a message of a certain type has been received by instance XYZ, but that instance does not know what to do with the incoming message, or may be knows but also knows that others may be are interested as well.

[yvesgoeleven]

All 3 modes seem valuable to me outside of the ASB scope. the shared (typical cloud worker) & independent mode (thinking of Adam Jones' farming vehicules), and I see competing as a solution for callbacks during scaleout.

[yvesgoeleven]

@mauroservienti That's the pattern, but there is no 'solution' for it, signalr has no guaranteed delivery over websockets and there is also no server side interception point that could be used to clear the webrole cache either, so it cannot be used for this use case. which reminds me, nservicebus itself is used as a backplane for signalr https://github.com/roycornelissen/SignalR.NServiceBus which makes it imo clear that people at least mentally map nservicebus to this instance level independence concept that szymon is proposing here.

[mauroservienti]

@yvesgoeleven as you said the backplane is the pattern, however a backplane does not change at all the behavior of the scaled out instances, that is what is proposed here.

To me scale out is always competing consumers, we have a clear requirement for callbacks and we have a solution for that, we may have a requirement as well in the IoT space and a backplane is the solution, so let's build a backplane feature and not change the semantics of scale out.

https://github.com/Particular/PlatformDevelopment/issues/263

[udidahan]

Again, we're getting back to data distribution.

While I agree that this is a problem that developers struggle with, I don't think we need to try to solve it with "NServiceBus". We can look at creating additional capabilities that plug into and extend the platform for that.

[udidahan]

re: message duplication - @yvesgoeleven isn't this something that is already possible with an end-to-end v5 solution?

[yvesgoeleven]

@udidahan no it wouldn't, in v5 we would start 1 receive infrastructure... with a endpoint-name or a endpoint-instanceid-name, now we start both

[SzymonPobiega]

@udidahan this is not necessarily about data distribution. @mauroservienti to me the notion of instance does not imply scale out. An instance is deployed and running copy of an endpoint. All the instances (which constitute an endpoint) might have slightly different behavior depending on the needs:

• Sharing endpoints are purely about scaling out
• Competing endpoints can be used for scaling out but they have other capabilities e.g. you can address a single node. They are not "transparent".

These concepts do exist in the wild but are usually hidden. I've seen people deploying the same codebase twice and using it for processing messages of different priorities. You can call these two different endpoints but I'd rather think about such scenario as two independent instances of the same logical endpoint.

[mauroservienti]

the issue, IMO, @SzymonPobiega is that targeting a specific instance is very dangerous and very unlikely to be a solution to anything. Let me outline a scenario that may affect a SignalR and/or IoT environment.

• the client (SignalR/IoT) is connected to a specific instance
• the system wants to send a message to the connected client

if we support targeting a specific instance this is supposed to work, but it is not going to work all the times, and even if we could say the same for callbacks we all know that callbacks are volatile and there to support legacy systems migration.

In the above sample two common things may happen:

Scenario A

• The connected client loses connection while the message destined to it is in flight
• The client reconnects and obviously may reconnect to a different instance

Scenario B

• The instance the client is connected to crashes while the message destined to it is in flight
• The client is disconnected and obviously reconnects to a different instance

In both cases the message is basically lost, a backplane will solve both issues without altering the competing consumer scenario.

Using the same approach we could use a backplane to solve the callback issue as well without requiring any additional instance queue or whatever specific to callbacks, however we already have a ca;;back feature we need to be backward compatible with, this is another story :smile:

[SeanFeldman]

I might be misunderstanding something, but after discussions it comes to:
Azure transports had data distribution for quite some time, implemented with queue-per-instance. But data distribution was delivering both commands _and_ events.

With the new implementation of callbacks, same queue-per-instance concept is utilized, but it's intended to be used solely with commands only.

Sounds like re-use/abuse scenario. Either callbacks or data distribution capability. To support both, the 3rd option is needed.

[andreasohlund]

I still don't see the need for 3 modes. I propose 2 modes and a "flag".

Ie:

Modes

Competing - All instances can compete for messages
Independent - Every instance is independent

Flag

UniquelyAdressable - This instance must be uniquely addressable on demand.

Features like Callbacks etc will require this flag to be true on the endpoint level since it has to be known at install time. When those features send/reply/etc messages that need to hit a unique instance they will request the specific outgoing operation to honor this.

If in Competing mode: The transport has to provide the unique address ability in some way or throw a NotSupportedException.

If in Independent: Nothing needs to be done since its supported by definition.

This solves the confusion around the difference btw the Competing and Sharing modes proposed.

Thoughts?

[SzymonPobiega]

@andreasohlund I like that. So in ASB case it would be:

Competing + ~UniquelyAdressable -> singe input queue, subscription based on endpoint name
Competing + UniquelyAdressable -> two queues, subscriptions based on endpoint name (not supported in legacy topology)
Independent -> single queue, subscriptions based on instance ID

I still have a tiny hunch that a bool flag is a smell and we might miss a proper name for Competing + UniquelyAdressable

[yvesgoeleven]

What would independent mean with UniquelyAdressable to false? A flag + 2 states == 4 states, while we need 3 imo. It smells...

[SzymonPobiega]

@yvesgoeleven that's true. Mr Evans would not be proud of us using a bool flag ;-)

[mauroservienti]

Seems that we cannot find an agreement on when to meet to discuss business requirements related to this.

Proposals on how to move forward?

[mikeminutillo]

Thinking out loud.

Uniquely Addressable is not a flag, it is a feature and it is a pre-requisite dependency for Independent and for Callbacks.

Uniquely Addressable means that we can route a message to a specific physical instance of a logical endpoint. In order for that to work, the running instance needs to be able to receive messages at an address based on physical instance ID.

When we have Callbacks enabled, we need to know the physical address of this instance and use it override the outgoing return address on Sends with callbacks.

Independent works similarly. When you create a subscription, it needs to override the logical endpoint address with the physical instance address in the subscription binding.

With that definition in mind, would it make sense to be able to use this feature (or not) on each subscription. i.e. config.HandleOnEveryInstance<TEvent>() with a global config.HandleAllEventsOnEveryInstance() for backwards compat with ASB.

Alternatively this could be enabled with a SubscriptionOptions setting and we could have a configuration option to set it or not when we the AutoSubscribe feature runs?

[abombss]

@mikeminutillo makes the most sense. Talk about it in terms of logical and physical endpoints. Just require an address mapping for each. Subscriptions always use the logical address mapping and callbacks always use the physical address mapping.

Competing + Addressable

Logical: sales.orderservice => sales/orderservice
Physical: sales/orderservice.[instanceid]

Competing + Distributed

Logical: sales.orderservice => sales/orderservice
Physical: sales/orderservice

Independent

Logical: sales.orderservice => sales/orderservice.[nstanceid]
Physical: sales/orderservice.[instanceid]

[SzymonPobiega]

OK. I think we agree that we need something like this. Now the question is:

• Should we have three modes of operation (Sharing, Competing, Independent)
• Should we have two modes of operation (Competing, Indepdent) where in Competing mode you additionally have a bool flag that says if individual addressing is enabled

I am still a fan of the first approach although I don't necessarily love the names of the modes.

[mauroservienti]

I think we agree that we need something like this

I don't :smile: , but seems to be the minority.

[SzymonPobiega]

@mauroservienti In the long run, I agree we might use something like the backplane. What I am saying is that in the V6 time frame, in order to support data distribution behaviors built into Azure transport, we need something like this.

[yvesgoeleven]

I'm with @SzymonPobiega, it's clear and intent revealing model (not keen on the names either)

[SzymonPobiega]

@andreasohlund @mauroservienti what should we do for the V6 release?

[mauroservienti]

I go on thinking that this should not be supported and encouraged in any way by NServiceBus, uniquely addressable instances are a solution in search of a problem, IMO. We couldn't come up with valid business cases, the only one is a generic data distribution scenario that:

• We said in Bangkok we don't want to support via NServiceBus
• Has a well known architectural solution called backplane, that is not uniquely addressable instances

As said this is not going to work in a real production environment.

As for https://github.com/Particular/NServiceBus/issues/3406, where seems that a business scenario is still missing, if we can find a solid business scenario I'd be more than happy to support it, otherwise I think we have more important things to do with greater customer impact.

[bording]

@mauroservienti Uniquely addressable instances (or per-instance queues) are needed for callbacks, but that is already working in the current callbacks implementation, where a shared queue and a per-instance queue are used at the same time.The new aspect being talking about in this issue is adding the "independent" mode where the subscriptions are routed to the per-instance queue instead of the shared queue.

From what I understand, the primary motivator for this issue, is that the current callbacks implementation breaks some aspects of the ASB transport, because it did previously have a data distribution mode. They are currently getting duplicate messages because both the shared queue and the per-instance queues are getting copies of published messages.

So, either we need to introduce the "independent" mode to maintain backward compatibility or decide that we want a breaking change in the ASB transport.

[johnsimons]

So does this problem affect v6 release? And if so, is it a show stopper for the release ? If yes to all then add the label to it

[mauroservienti]

@bording yeah, the background is clear. Still there is no business requirement. Basically my fear is that we end up introducing something now to support something we did wrong previously and we'll end up deprecating it in the next major. We have a similar issue with the SQL Transport multi-instance feature.

[mauroservienti]

Again, I strongly disagree with this one, but seems to be the only one, so even if I think it is completely wrong if the majority thinks we should do it, let's do it.

[boblangley]

I do not think this should be a core behavior. We should be pointing those
customers to a better way to handle distributed caching.

Forgive my ignorance, but if we decide we have to support this, is there a
way to do so outside of the core and make it an ASB specific feature?

On Wed, Mar 16, 2016, 11:10 PM Mauro Servienti [email protected]
wrote:

@bording https://github.com/bording yeah, the background is clear.
Still there is no business requirement. Basically my fear is that we end up
introducing something now to support something we did wrong previously and
we'll end up deprecating it in the next major. We have a similar issue with
the SQL Transport multi-instance feature.

—
You are receiving this because you are on a team that was mentioned.
Reply to this email directly or view it on GitHub
https://github.com/Particular/NServiceBus/issues/3495#issuecomment-197723375

[mikeminutillo]

@mauroservienti I think I stand in your minority as well. I wonder if we can do this as an external plugin and not support it out of the box as a part of Core V6.

Jinx @boblangley

[mauroservienti]

Agree, we should introduce this, along with backplane support, as a separate feature.

[SzymonPobiega]

The only hack I can think of is to expose, from the core perspective, both listen addresses (shared and per-instance) to the transport. Transport then decides if it is going to bind the subscriptions to the shared one or to the individual one.

We can then have something like:

config.UseTransport<AzureServiceBus>().DeliverEventsToEachInstance();

and if there is no instance ID provided, that call would blow up.

[boblangley]

Could we just have them start up a second endpoint for the data distribution scenario, overwriting the endpoint name with an instance identifier? Is the cost too high for doing this?

This would encourage them to separate this concern from the normal messaging.

[SzymonPobiega]

@boblangley that would be a bigger hack since in that scenario each instance of such _data distribution endpoint_ would have to have a unique _endpoint name_ so you would have to do something like this:

var uniqueEndpointName = "CacheInvalidator" + GetMyRoleInstanceID();
var config = new EndpointConfiguration(uniqueEndpointName);

we thought about similar approach to callbacks (just spin another instance) but we abandoned the idea in favor of continuing to support callbacks as first-class citizens. On the other hand, the different here is that @udidahan says callbacks are _on_ mission while data-distribution is (currently) _off_ mission.

[SzymonPobiega]

Marking it as V6 because this is a showstopper and we _need_ to make a decision here.

[johnsimons]

I still think for us to be production safe we need to randomize the instanceid on every run

[andreasohlund]

I still think for us to be production safe we need to randomize the instanceid on every run

That's going to create an ever increasing amount of orphan queues? That would cause the OPS people to come after us with pitch forks?

[boblangley]

@SzymonPobiega Yes, that is exactly what I am suggesting. Essentially once you move to this queue per instance, at least for these subscribed events, they are each acting as discrete logical endpoints. This idea that they are both a participating as physical endpoints in a single logical endpoint AND acting as discrete logical endpoints is what makes it contentious.

As such I wouldn't consider that a hack. They are just setting up so each host can participate in both logical endpoints (the shared and the individual).

[andreasohlund]

@SzymonPobiega how about we schedule a call early next week including @udidahan and make a call on this one since it relates to the discussion with him we had in Bangkok?

[SzymonPobiega]

@andreasohlund I am not a big fan of calls since they require to agree on same time slot. If we can make a decision in the async way it will always be more inclusive and we will always have more opinions present.

[andreasohlund]

I am not a big fan of calls since they require to agree on same time slot.

Agreed but we seems to be going in circles and this is a key decision for v6 ?

[SzymonPobiega]

Currently I believe the best option is the one proposed by @boblangley . It seems to be acceptable by @mauroservienti and @WilliamBZA (no new data distribution APIs) and by myself. Let's see if @yvesgoeleven considers it good-enough given his insight into Azure customers' needs. If we can't proceed with that proposal, let's schedule a call and make an arbitrary decision on that call.

[andreasohlund]

Currently I believe the best option is the one proposed by @boblangley

Wouldn't that result in callbacks no longer working when scaled out? (something that @udidahan said should work in Bangkok)

Not against it either just making sure that we are aware

[SzymonPobiega]

we are talking here about a very specific Azure case where you want events to be delivered to every NSB instance. Here's what you do:

• You use a regular instance for processing commands

var regular = new EndpointConfiguration("MyEndpoint");

• If you also need callbacks, you enable unique addressing for that instance

regular.EndpointInstanceId(GetAzureRoleId());

• If you also need to get events delivered to all NSB instances of your regular endpoint, you also spin up a "subsidiary" endpoint

var dataDistributionEndpoint = new EndpointConfiguration("MyEndpointDataDistribution" + GetAzureRoleId());

so the end result might be:

var regularEndpointConfig = new EndpointConfiguration("MyEndpoint");
regularEndpointConfig.EndpointInstanceId(GetAzureRoleId());

var dataDistributionEndpointConfig = new EndpointConfiguration("MyEndpointDataDistribution" + GetAzureRoleId());

ConfigureSharedStuff(regularEndpointConfig);
ConfigureSharedStuff(dataDistributionEndpointConfig);

var regularEndpoint = await Endpoint.Start(regularEndpointConfig);
var dataDistributionEndpointConfig = await Endpoint.Start(dataDistributionEndpointConfig);

[udidahan]

Just to clarify this point:

@udidahan says callbacks are on mission while data-distribution is (currently) off mission

Our mission is to make developers better - so both are on-mission. When it comes to vision, though, callback are part of the short-term vision - data distribution is not.

[johnsimons]

That's going to create an ever increasing amount of orphan queues? That would cause the OPS people to come after us with pitch forks?

@andreasohlund if we don't do it, then it is not safe.

Why is not safe:

1. Users will hardcode the instanceid, I can nearly guarantee it;
2. On IIS AppPool recycles start AppPool side by side, so we have competing consumers on the same instanceid queue, and the inmemory dictionary won't pick it up;

I wish there was a different solution, but I cannot see how to avoid it without randomized queues, can u?

Since really the only usage of the Callbacks is in IIS, I see point 2 as a dealer breaker.
All this said, currently the production version works exactly like this (in other words, it is also broken), so we can just document this as a known issue and just deliver exactly what we have today .

[SimonCropp]

Users will hardcode the instanceid, I can nearly guarantee it;

pity cecil isnt a viable option since u could detect if a hard coded string is being passed in

[johnsimons]

IMO, to move this forward, I would code exactly the same solution we currently have for callbacks and would put data distribution in the back burner.

[SzymonPobiega]

We're not talking here how this instance ID is going to be generated. That's probably going to be covered by guidance. We're trying to decide if current data distribution feature of ASB can be achieved by adding another endpoint which names contains a unique component (e.g. role ID).

[SzymonPobiega]

@yvesgoeleven @SeanFeldman please check if the double instance approach @boblangley proposed would solve the problem of data distribution.

[SeanFeldman]

@SzymonPobiega this would achieve the goal. I don't know if there are drawbacks for the existing customers that are already using data distribution. @yvesgoeleven thoughts on this?

[yvesgoeleven]

This could work as we can host multiple endpoints in one process now, right? Are these appdomain isolated? Would any of their settings potentially conflict running 2 endpoints in different modes in the same process?

[SzymonPobiega]

@yvesgoeleven there is no strict need for appdomain isolation (at least in the core there is no static stuff apart from the logging config) so it should be possible to just create two IEndpointInstance objects side-by-side.

The NServiceBus.Host does not support having more than one instance though.

[yvesgoeleven]

@SzymonPobiega don't think that is a problem, it usually involves web front ends anyway, they are selfhosted

[tmasternak]

@yvesgoeleven @SzymonPobiega Can we go with the second endpoint approach than? Or do we want to validate any of this with PoC?

[yvesgoeleven]

@tmasternak I want to do PoC as I'm not yet convinced that configuration settings and assembly scanning will not get into the way if there are no appdomains isolating the endpoints

[SeanFeldman]

@yvesgoeleven assembly scanning will need to take place to separate between endpoints. That's a known "issue" with multi-hosting in the same appdomain.

[tmasternak]

@yvesgoeleven @SeanFeldman so assuming that we can achieve proper separation only with spearate AppDomains is it still a valid approach?

[yvesgoeleven]

@tmasternak I'll tell you after the poc :p

[andreasohlund]

@tmasternak can you update the issue with the taskforce and put the POC into the POA?

[tmasternak]

@yvesgoeleven I think it makes sense for you to be part of TF as well. At least until the PoC is done.

Additionally do we want to track progress on PoC here or as a separate issue?

[SzymonPobiega]

POC issue here: https://github.com/Particular/NServiceBus/issues/3638

[ramonsmits]

@SzymonPobiega @tmasternak @andreasohlund Does the issue description still reflect the discussion here? Its almost impossible to read.

It also misses why the issue is closed.

Can the description contain a TL;DR?

[andreasohlund]

Agreed, can you update @SzymonPobiega ?

[SzymonPobiega]

@andreasohlund @ramonsmits updated the description.

[ramonsmits]

Seems that this desk case also needs this model:

• https://nservicebus.desk.com/agent/case/17882

[SzymonPobiega]

@ramonsmits you mean, if for endpoint X you want to have instance-per-tenant then these instances would be _independent_? I am not sure if that would solve their problem. Because if there is another endpoint Y which also needs to be instance-per-tenant then with independent instance model if Y publishes an event, it will be routed to all instances of X (instead of being routed to a single instance which belongs to the same tenant)