Masstransit: Reliable messaging

[voroninp]

As far as I understand MT in comparison to NServiceBus does not support reliable messaging out of the box. (correct me, if I'm wrong, I'd be very happy to be mistaken).

Yet I see an option of using the underlying RabbitMQ for that purpose.
Here is the workflow:

• do some biz rules
• generate special correlation id
• store this id in db
• publish a message
• commit transaction

But instead of going to consumers message first goes to a special 'loopback queue', where it is processed by the sender/publisher part or by some dedicated service which has a task of looking inside DB for a corresponding correlation id, and in case of success putting message in correct exchanges.

[phatboyg]

I'm not sure I understand what you're asking about, as I'm unfamiliar with this particular aspect of NServiceBus. The tools are there to do what you seem to be suggesting, but as MT doesn't do anything with .NET transactions, it's unlikely that what you need is already baked into the framework.

[petedavis]

http://looselycoupledlabs.com/2014/11/masstransit-versus-nservicebus-fight/

[alexeyzimarev]

This feature of NServiceBus is called Outbox. It is basically the implementation of the outbox pattern. MT supports in-memory outbox, which means that all messages that are send from some consumer, will only be sent at the very last moment before the message that is being consumed, is ACKed to the bus. This, in its turn, means that if your "something" like a database transaction inside the same consumer, will fail, the messages will also not be sent out. This covers _a lot_. Really.

As per the blog post, I as a person who worked with both NSB and MT for years, can say it is rather bogus. Many aspects are not touched in this "comparison".

[voroninp]

@alexeyzimarev And what happens, if we publish new message not as reaction to consumed message?

[alexeyzimarev]

The NSB outbox solves the issue of having DTC when you have multiple infrastructure pieces in one transaction, such as a database transaction and message publishing. Generally message publishing is not considered as an operation that tends to fail. Most failures come from databases (constraint violation, locking, etc).

[voroninp]

@alexeyzimarev So should we perform publishing before committing a database transaction when consuming another MT message, and publish after commit when processing was not initiated within consumer? Unfortunately, there is nothing in docs concerning MT's in-memory outbox.

[mehdime]

Generally message publishing is not considered as an operation that tends to fail. Most failures come from databases (constraint violation, locking, etc).

I guess that this is the crux of the issue here and what @voroninp is getting at.

When it comes to distributed systems at scale, "it usually works" isn't good enough. Yes, in the immense majority of cases, if something fails during the processing of a message it will be the database transaction (or the code prior to committing the DB transaction). The message publishing done after the DB transaction has been committed generally doesn't fail.

Except when it does. You could be very unlucky and your bus goes down or the whole server crashes at this precise moment. And yes, these are extremely unlikely events. But at scale, they will happen.

The end-result is a trickle of potentially impossible to reproduce and extremely hard to troubleshoot weird issues. I.e. the end-result is a flaky system. Which really is your worst nightmare.

I'd actually be interested to hear about how others have addressed this issue. Or about how others have decided that it wasn't worth addressing (and what the rationale was).

Possible solutions are the DTC (if supported by all components involved, which is not the case for RabbitMQ) or the solution proposed by Udi, which involves storing the incoming Message ID, all domain data and the pending queue messages in an ACID database within a single transaction.

Both solutions feel expensive (and restrictive).

Another solution, which we've tried for our last project was to mandate the implementation of all command handlers to be idempotent but to always publish any subsequent messages even if it detects that the command has already been processed (e.g. the CreateUserCommand handler will always publish the UserCreatedEvent even if it finds out that the requested user had already been created). There are many obvious flaws with this approach as well.

Yet another solution is to just say that we'll take the flakiness for those very rare edge cases in order to keep the system simple and as scalable as possible.

Would love to hear about other people's opinions on this.

[alexeyzimarev]

As far as I remember, the Outbox feature in NServiceBus specifically targeted the distributed transaction issue, when one logical operation must be performed across multiple infrastructural components, i.e. updating the database and publishing to the bus. This was done as a response to the lack of DTC support on most popular message brokers.

I feel that vast majority of these issues can be solved by changing the system design. For example, in your example of publishing domain events, we use two different systems.

In event-sourced systems all events are saved to the EventStore and publishing to the bus goes from there. So we have to distinct operations to commit changes to the database, and to publish them to the message bus. Updating read models via projections also reads events directly from the event store.

In state-only systems (traditional), we use MT ImMemoryOutbox. If database fails - we retry. If database succeeds - we publish an event. If the event fails to publish - the command is not ACKed anyway, so we will process it again. Here we indeed come to the requirement to have idempotent commands and handlers since there is a risk of processing the same command more than once. But again, this comes to the system design. In general, message brokers already require us to do this since, for example, RabbitMQ guarantees _at least once_ delivery.

As per the reliability, we run RabbitMQ in production for many years without a single failure. This is probably the most reliable piece of the whole system.

[oshvartz]

I would be glad to have the InMemoryOutbox not only as part of consuming a message but as part of transaction

[wokket]

We have a custom component built internally that serialises the outgoing messages to the database alongside the business data with transactional support, and a background thread sending those messages from the database... I believe this is close to what NServiceBus provides out of the box.

We're effectively trading message latency for stronger delivery guarantees to ensure a message will _eventually_ be sent for those messages we deem important enough and are unable/unwilling to resolve in other means (eg through forced user retries etc). Note that these are messages at the 'head' of the system (eg User has placed a new Order through the website) so we don't have an existing consumer/retry semantic available.

[danilobreda]

I'm opening again this discussion because I recently had the same problem in production. And so far I have not really understood what happened. The truth is, we will never reach perfection, they are two separate, distinct systems, database and messaging.
For me the most 100% way would be for example postgresql to integrate with rabbitmq, which for what I know does not exist, at this moment.
Depending on what each service accomplishes, there are ways to 'fix' those flaws on the service scope. As mentioned by @mehdime
On my case, i use a microservice architecture on a ERP. If any infrastructure problems occur, we'll be fried.
I believe that Masstransit and Saga help a lot with event management so that if something bad happens, everything will be solved, or an alarm will ring. For human intervention.
So I do not think there is a solution at masstransit level @phatboyg

[alexeyzimarev]

My last two cents in this issue.

NServiceBus reliable messaging (Outbox) only works when you use the same database and access method (framework) for your usual database operations performed from inside the message consumers and for the outbox persistence itself. This database also needs to support transactions that span across several operations. It means that NSB supports this feature only for NHibernate, SQL (ADO.NET), RavenDb and ServiceFabric, using the synchronized storage.

I actually think it can be done with MassTransit by creating a persisted outbox filter. The database connection and transaction can be put into the context so they can be used inside a consumer.

This is the NSB implementation, it really looks like a filter:

[joshmouch]

I'm comparing CAP to MassTransit, and this will be a deciding factor for me. Can you provide a quick code example of how this Outbox pattern would be implemented with Azure ServiceBus and SQL Server? I'm not familiar enough with Mass Transit to take your awesome diagram and put it into practice.

[F0b0s]

@alexeyzimarev

I actually think it can be done with MassTransit by creating a persisted outbox filter.

Do you have any plans to implement this?

[phatboyg]

No, in fact, I wrote about why here: https://masstransit-project.com/2020/02/08/in-memory-outbox/

[shuk]

No, in fact, I wrote about why here: https://masstransit-project.com/2020/02/08/in-memory-outbox/

Unfortunately the link doesn't work. Could you actualize it?

[phatboyg]

Updated: https://masstransit-project.com/articles/outbox.html