Masstransit: Clear Send Endpoint Cache on Broker Reconnect

[phatboyg]

If the broker connection is severed, where the model context in RabbitMQ is no longer valid, any cached send endpoints on that model should be removed from the cache so that they are not mistakingly used after being disconnected. The model may indeed be dead already, but the endpoint still attempts to send using the context, which causes a strange failure.

• [ ] Send Endpoint - Ensure Model Context is being monitored for shutdown
• [ ] Send Endpoint Cache - Ensure removed from cache if model is shutdown
• [ ] Publish Endpoint - Ensure Model Context ...

• [ ] Publish Endpoint Cache - Ensure removed from cache ...

• [ ] Verify that participant/scope tasks are properly handled so that shutdown is unaffected by disconnect/reconnects.

[mrapavy]

We can easily reproduce this problem using MessageRequestClient in MassTransit 3.5.5 in isolated test application that has no other dependencies.

It's sufficient to send a MessageRequestClient request, then wait for 25 hours without any activity and after that period of time all subsequent requests will come back with TaskCancelled exception which originates in TaskSupervisor.cs:99:
if (_stoppingToken.IsCancellationRequested) throw new OperationCanceledException("The supervisor is stopping, no additional scopes can be created");

If it helps, I can provide our test program.

[phatboyg]

Considering the endpoint cache defaults to 24 hours in 3.5.5 that aligns right with the timeout and eviction. I'll try to reproduce it by cranking down the cache lifespan.

[mrapavy]

Some additional info after more testing was done: On my local rabbitmq instance, MT entered this state just after 6h 20min, stacktrace showed method in lambda atTaskSupervisorExtensions.cs:32 being called with stopEvent.Reason == "Closed by owner".

[phatboyg]

I'm hopeful this commit will resolve the issue, but I'm not positive. I think the long disconnect isn't occurring until the model is used, and then it fails, so this wouldn't fix that but it's a try.

[mrapavy]

Yes. The model (MessageRequestClient) needs to be used first in order to become unavailable after 25 hrs of inactivity.

Martin

On Mar 8, 2017, at 17:49, Chris Patterson [e-mail] wrote:

I'm hopeful this commit will resolve the issue, but I'm not positive. I think the long disconnect isn't occurring until the model is used, and then it fails, so this wouldn't fix that but it's a try.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.

[EAbychkov]

Faced with the same problem "The supervisor is stopping, no additional scopes can be created" any plans to resolve this? Or I should recreate MessageRequestClient every time I need to use it?

[phatboyg]

If you're in a message consumer, you should create it every time. If you're just sending a request from the bus, I'm not sure creating it each time would change the behavior.

[phatboyg]

I hope to have this resolved and released by the weekend. I just need to reproduce it.

[EAbychkov]

I'm developing asp web api application and I need to use MessageRequestClient in my Controller POST method, so if I initialize and Start() my IBusControl in Application_Start it works only for the first request.
If I make second request I got "The supervisor is stopping, no additional scopes can be created"
What is the proper working model with MessageRequestClient object in ASP.NET WEB API ?

[phatboyg]

Okay, so this is entirely because of how the MessageRequestClient caches the ISendEndpoint, instead of caching the address itself. Send endpoints should not be held forever, as they get cleaned up. In fact, they should be created each time they are used, from the queue upon which they are created. I'm going to clean this oversight up, but anyone keeping an ISendEndpoint beyond the lifecycle will have this same issue.

[phatboyg]

So I created a new way to get at the request client, still working on it, but added a RequestClientFactory - which is used to get the client for the inbound message context so that the conversationId is passed around properly, as well as the source address.

[GokGokalp]

What is the last situation @phatboyg ?
Are you still working on?

[phatboyg]

Oh this was changed. I'm not sure it's in master yet though I'd have to check.

[phatboyg]

Yep in 3.5.6

[GokGokalp]

@phatboyg great! I'll check it.