Masstransit: Infinite increase channels count on RabbitMq with many clients

[ChipsetSV]

Hello.

I discovered the bug.
Client use ClientRequest for request data from a server. Server receives request, creates SendEndpoint and creates channel on connection to RabbitMq for send response. Client receives response and it's Ok.
But each client have unique temporary queue for receiving responses, and server for each client creates new channel on connection. After closing the client and it's connection, channels from server to RabbitMq stay alive until close server connection.
This leads to an inifinite increase channels count on RabbitMq and memory leak on server (server stores in memory all these endpoints).

Expected behavior: endpoints with them's channels must be closed automatically, for example, by time interval without activity.

I tested it on https://github.com/MassTransit/Sample-RequestResponse with MT 3.5.7 and MT 5.1.2. Each start of Client (and request to server) initiates new channel from RequestService to RabbitMq.
I use RabbitMq v.3.7.5 and Erlang v.20.3.

[phatboyg]

It shouldn't be infinite, after 1 minute those channels should start to close and cleanup. The minimum lifetime for a cached send endpoint is 1 minutes, up to a maximum of 10,000 endpoints.

Those numbers might need to be adjusted, but that was the baseline chosen.

Also, why is your client creating a new response queue for every request? The bus endpoint should be used to handle the responses. It's significantly faster than connecting a new receive endpoint for every request.

[ChipsetSV]

It shouldn't be infinite, after 1 minute those channels should start to close and cleanup. The minimum lifetime for a cached send endpoint is 1 minutes, up to a maximum of 10,000 endpoints.

Yes, I looked at the code and saw the use GreenCache with parameters: capacity 10000, min time 1 min, max time 24h. And I decided to check it. I expected that channels should start to close after reaching a quantity limit (10000), but to my regret, this didn't happen. I was able to create 20000 channels and they are still alive.

I attach screenshots:
Connetion with many channels on RabbitMq (after idle more than 12h)
Snapshot in DotMemory of RequestService (after idle more than 12h)

Also, why is your client creating a new response queue for every request? The bus endpoint should be used to handle the responses. It's significantly faster than connecting a new receive endpoint for every request.

No, I don't create new response queue for every request, I use bus endpoint for each client. But each new client creates new bus endpoint for handle responses, and server creates new send endpoint for handle requests and send reponses. After closing all clients send endpoints on server still alive.

[phatboyg]

Are you creating a new bus per request? Again, you’re wasting system resources by not sharing a single bus and leaving it connected.

Yes, GreenCache will exceed the limit if the age of a cached item hasn’t passed the minimum. It’s by design.

I’d suggest you stop creating a new bus for each request. Create one and start it at process startup, then use it for all the requests.

[ChipsetSV]

I'm afraid that you misunderstood me.

No, I don't create a new bus per request. I use one bus per client-side application and leaving it connected. But my application is client-server application, I have one server-side application and many client-side applications for many people. Each client-side application can do requests to server. And as a result I have many channels on one connection at server-side, them still alive after closing all clients.

Yes, GreenCache will exceed the limit if the age of a cached item hasn’t passed the minimum. It’s by design.

Yes, I understand it. And I saw that in code minimum is a minute. But channels count has not changed in a minute, nor in 12 hours.

[ChipsetSV]

What can you say about this situation? It's easy to reproduce with Sample-RequestResponse. Our project already working in production but we met with this bug only after the update of RabbitMq (in latest versions of RabbitMq parameter max_channels by default is 2047, in previous versions it is 65535).

Can I hope that this bug will be fixed in the future?

[yashbajra]

I’d suggest you stop creating a new bus for each request. Create one and start it at process startup, then use it for all the requests.

@phatboyg I am also getting too many channels created. However, I am creating new bus for each request. In my scenario, I cannot create a single bus and share it in subsequent connections, since my code is injected to an external application, and I do not have control over it.
What should I do in such scenario?
Details are present at https://groups.google.com/forum/#!topic/masstransit-discuss/6gsS8Xuv7zE

[GreenKn1ght]

I ran into exactly same problem.

[BoOsEc]

Hi, I have the same problem. How can I solve that? It's urgent, I'm in production environment

[phatboyg]

If it's urgent, you need to change your application so that it doesn't create a new temporary queue for every request - use the same bus instance for each request and start/stop it with the application lifecycle.

[GreenKn1ght]

What can we do to help fix the issue?

[phatboyg]

@GreenKn1ght

If it's urgent, you need to change your application so that it doesn't create a new temporary queue for every request - use the same bus instance for each request and start/stop it with the application lifecycle.

[GreenKn1ght]

We use single bus per entire client application and stop the bus at application closing.

[phatboyg]

@GreenKn1ght then how are you running out of channels?

[GreenKn1ght]

I have no idea and thats the problem.
Let's start from scratch. As @ChipsetSV said, we able to reproduce this issue on https://github.com/MassTransit/Sample-RequestResponse.
If you run this example and start send requests to service, you'll see that number of channels in Channels tab was incresed. If you wait for one minute, number of connections will decrease to initial value even if bus was not stop. This is expected behavior and there is no issue.
But if you update MassTransit from 3.5.0 to 3.5.7 (same with 5.1.5) and repeat steps above, number of channels will not decrease after a while and even if you stop the bus. For now we can clear the channels only by closing server application.

[GreenKn1ght]

I found the commit that brought this behavior: 789bffe234b460af986b906dbc5af34797521ee2

[phatboyg]

So I finally took the time to reproduce the issue (with 5.1.5) and have it fixed. But it requires an update to GreenPipes in addition to changes I have in my local repository for MassTransit, which means I need to update a few other projects as well. I may try to get to it this weekend. But I did resolve the issue to where channels are discarded after they are no longer being used.

[GreenKn1ght]

Nice to hear it, thanks!

[ChipsetSV]

Great news, thanks!

[BoOsEc]

thanks! in advance for ur help

[ChipsetSV]

@phatboyg, hello.
When we can see a new version with this bugfix?

[phatboyg]

By the end of the year it should be released. You can test with a develop version to see if it resolves your issue and help verify the fix.

[ChipsetSV]

That's great )

[ChipsetSV]

I tested this case in test app. It's ok, channels are closing.

[thomastomanek]

Any update on when this'll hit master?

[phatboyg]

It's been in 5.2 and 5.2.1 since it was released last week...

[thomastomanek]

I'm still seeing this on 5.2.3, unless what I'm seeing is different to the bug described. Each new client causes MT to create a new channel per response that never gets closed.

See gist reproducing. The new client requirement is simulated by creating a new bus per request. In reality though this is new GUI sessions querying the servers.

Playing around on develop and your 'Serious Transport Refactoring' commit seems to fix it.....

https://gist.github.com/thomastomanek/76f6e3c301cabc104c159aced49461cb

[phatboyg]

Well, 5.3.1 was just released yesterday and has all of the latest bits.

And realize, the channels don't release immediately, they are released after a minute when a new channel is requested. The soft limit is 1000, after which they should hover around that value as the channel usage shifts in response to new requests.

[nick-randal]

This is an issue in 5.5.3 and 5.5.4. I don't know if it has regressed? I tested the .net FW and .net core versions with the same results.

My issue is exactly like those described. I have a long running host with individual clients. Every app uses a single bus instance.

Each new client instance causes one more new channel on the host when a Request/Response happens. We are approaching 2000 channels, well above the suggested 1000 limit. The channels are not expiring. The host had been running for several days, accumulating channels.

Using RabbitMQ 3.7.15 Erlang 22.0 on Windows Server.

[phatboyg]

When you're approaching 2000 channels, how old he is the old channel at that point? Because the lowest age, I believe, is 1 minute. Not positive, but pretty sure.

[nick-randal]

Days old. Each new channel is getting created by an app that runs every 10
minutes.

On Wed, Aug 7, 2019, 8:09 PM Chris Patterson [e-mail]
wrote:

When you're approaching 2000 channels, how old he is the old channel at
that point? Because the lowest age, I believe, is 1 minute. Not positive,
but pretty sure.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/MassTransit/MassTransit/issues/1165?email_source=notifications&email_token=ABOGMU6IWPUGPYH663KEQSDQDNP3FA5CNFSM4FKS54DKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD32BKHI#issuecomment-519312669,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABOGMUZZGCTNNDCT4BQBAZ3QDNP3FANCNFSM4FKS54DA
.

[nick-randal]

https://gist.github.com/nick-randal/6baeb89d07854cc5866203c54975fc9d

I named the files to simulate what our production system is doing. Host is a long running process which uses a single bus instance.

Client is an application that starts up periodically, uses a single bus instance but does do a Request / Response. Each time the client runs a channel gets orphaned in the host connection. These channels are never expiring and are going past 1000. We actually incurred an outage because we hit an internal limit in rabbit of 2047 channels.

[phatboyg]

I made a comment on your Gist, I don't think it's MT.

Also, on the other issue, I posted a Gist that I used to try to reproduce the issue, without success.
https://gist.github.com/phatboyg/eff85c26d5da1f4f56b6f729847dd3ad

[nick-randal]

Yes, I see the comment but that doesn't explain the exact same behavior in production where there is not interaction with the console. The console interaction is only in Linqpad.

[phatboyg]

How is the code hosted in production?

[nick-randal]

Windows Service is the host, on service start it opens a connection to the bus and maintains that single connection.

We have different worker applications that start up, do some work and exit. Each worker for the life of the application maintains a single connection to the bus. All other Publish / Consume methods are working as expected. This is specifically related to Request / Response.

We see that at the first request / response cycle a fresh channel is created on the host to respond to the request. However, at the point of the client application closing and stopping its bus. The host channel that was used to send is not closing. At seems that the host send channel is unaware that its consumer has closed its connections so I believe RabbitMQ's channel.Close() is not begin called.

Only in Linqpad do we see the 1000 channel limit but to me it begs the question, why aren't the channels closed when the consumer is no longer there?

On our production server we are seeing the channel creep exceeding 1000 and going all the way to the limit of 2047, at which point clients start getting exceptions like

operation none caused a connection exception not_allowed: "number of channels opened (22) has reached the negotiated channel_max (22)
as noted in https://www.rabbitmq.com/channels.html#lifecycle.

You have referenced a 60 second expiration on channels. I don't see any expiration or cleanup in production or linqpad in relation to channels. We are only seeing a 1000 channel cap when using linqpad but no cap in prod. I see Queues expiring at 60 seconds, if that is what you mean.

[phatboyg]

The host channel for the send endpoint that is used to deliver the response to the client bus instance is a channel on the host connection. It has no correlation whatsoever to the client channel on the client connection. The connections and channels are between the broker and the host or client, the client and host are not in any way directly connected. So there isn't a way for the client closing the channel to signal the host that the channel for that send endpoint should be closed.

The channels are closed when cached send endpoints are removed from the cache, which happens after one minute -- if the cached endpoint limit (1000) is reached, If it's under the limit, those channels may not be closed.

If you're still seeing channels increase, your host is either running a seriously old version of MassTransit that may have had the bug, or something else is blocking the host from closing those cached send endpoints. Like maybe something in your system is still using that endpoint, or holding a reference to it.

For real, there is nothing I can do if I can't reproduce the issue. And so far, everything works as expected on Mac or Windows, .NET Core or .NET Framework, RabbitMQ 3.6 or 3.7.

[phatboyg]

As for the cache lifetime, it's set here:
https://github.com/MassTransit/MassTransit/blob/develop/src/MassTransit/Transports/SendEndpointCache.cs#L29

And it's actually 10 seconds now, instead of 60.

    static SendEndpointCacheDefaults()
        {
            Capacity = 1000;
            MinAge = TimeSpan.FromSeconds(10);
            MaxAge = TimeSpan.FromHours(24);
        }

[nick-randal]

We are using 5.5.4.

I understand that all connections are to Rabbit and not to each other. The channels for request / response are not created until the first time that a Request / Response is started. Just as the channels are inflated from that initiation, my expectation would be along the lines of sending a, I'm leaving message so that the host can explicitly close that channel out instead of have the blind 1000 pool.

Couldn't this be an explicit call by choice to send a close out or just make client IDisposable and send a "Close it" command based on the type?

Otherwise the only way I can see to do this cleanly, is to go back to a basic RPC pattern with Publish on both ends.
Something like
https://gist.github.com/nick-randal/00eebecfc34d363a33fbaa85bd21ffcc

Send() seems like the preferred way to do this instead of having the Publish() fanout behavior, but I don't see a way to determine a URI at runtime for a connecting client to be able to call
GetSendEndpoint()

[phatboyg]

You really should figure out why your instance isn't behaving like it should, because honestly, every single combination of runtime and OS I've tried works as expected. Something in your code is preventing the endpoints from being cleaned up.

[nick-randal]

Since we have the same setup and I have tried both a linux and windows instance of rabbit, and various .net frameworks, it occurred to me that there is one distinct difference. Your example works for both you and us, however the timing is really the biggest difference. Blasting channels over a short period of time does hover at 1000. However, our request / response leak is happening once every ten minutes. So it takes days to reach that mark but then it doesn't stop at 1000. I think there may be an issue in the cache logic when the leaks are creeping along.

[phatboyg]

So how often is the host receiving a request from a new endpoint address?

[nick-randal]

It depends on the endpoint. At a minimum 1 request is made on a new endpoints startup but it might make more requests.

Based on your suggestion of using direct exchanges with routing keys, we have a working solution I am happy with. There are no channel leaks or accumulation. The only only has 1 channel per unique request / response type.

It is working but do you see anything that we might have done incorrectly or could have improved?

https://gist.github.com/nick-randal/0bf53d5659bd4a365aac18a042be2270

[phatboyg]

I'll look at it again shortly, but I think you can still use the request client on the client side, but publish from the server side with the routing key. I'll work up an example after a bit, once i get a chance to sit down and think about it.

[phatboyg]

FYI, with v6.2 (actually I think 6.1 maybe) there is no longer a channel increase with RabbitMQ using request/response. Responses are now sent on the same channel as the inbound message receiver.

[canertosuner]

I was having the same issue on v6.0 running on asp.net core v2.2 and it wasn't working as expected in terms of the number of tcpConn&channel. After upgrading it to latest version (v7.0.4) it disappeared.