Mqttnet: ConnectAsync fails when many Tasks are active

[grobmiller]

Describe the bug

For load tests we created an application that contains of a lot of Tasks (via Task.Run) which publish MQTT messages. Unfortunately ConnectAsync does not succeed to have a stable connection when a lot of Tasks are active. Note that the CPU load was very low since the Tasks didn't do very much.

This may seem like a "toy code" problem. But our production application has the same problems, too. As soon as a few dozen Tasks are running an MQTT connection is not possible anymore.

Which project is your bug related to?

• Client (3.0.11.0 .NET 4.5.2)

To Reproduce

Steps to reproduce the behavior:

1. Create 100 Tasks
2. Connect to an MQTT broker
3. Connect won't succeed

Expected behavior

ConnectAsync would be successful

### Code example
 Please provide full code examples below where possible to make it easier for the developers to check your issues.

```csharp
            // LOGGING
            MqttNetGlobalLogger.LogMessagePublished += (s, e) =>
            {
                var trace = $">> [{e.TraceMessage.Timestamp:O}] [{e.TraceMessage.ThreadId}] [{e.TraceMessage.Source}] [{e.TraceMessage.Level}]: {e.TraceMessage.Message}";
                if (e.TraceMessage.Exception != null)
                {
                    trace += Environment.NewLine + e.TraceMessage.Exception.ToString();
                }

                Console.WriteLine(trace);
            };

            // CREATE TASKS (which do almost nothing)
            bool run = true;

            for (int i = 0; i < 100; i++)
            {
                Task.Run(() =>
                {
                    while (run)
                    {
                        Thread.Sleep(100);
                    }
                });
            }

            Thread.Sleep(10000);

            // CONNECTING
            var options = new MqttClientOptionsBuilder()
                .WithClientId(Guid.NewGuid().ToString())
                .WithTcpServer("localhost",1883)
                .WithCleanSession()
                .Build();

            var result = mqttClient.ConnectAsync(options).Result;  // THIS WILL NOT SUCCEED

            Thread.Sleep(10000);
            run = false;

[OidaTiftla]

As far as I see, you are simulating 100 long running Tasks. Because of the Thread.Sleep(...) they will block a thread on the thread pool of the TPL. Which is the same as a long running Task will do.

If the code reaches ConnectAsync(...), the TPL will reach a point, where it needs a thread of the thread pool, but they are very busy with other tasks (tasks, which will not finish in this example, as long as the ConnectAsync(...) hasn't finished, maybe kind of deadlock).

As far as I know, the TPL will create more threads in the thread pool, if there is a lot of work to do, but I don't know if there is an upper limit on how much threads will be created.

Did you test it with less concurrent tasks? Maybe 10 or 20?

Or maybe 2 or 3 tasks, this should definitely work. (But I have only experience with short running tasks yet.)

[OidaTiftla]

I found your issue very interesting, so I did a little research and found the following, maybe this helps you: The managed thread pool

If you have short tasks that require background processing, the managed thread pool is an easy way to take advantage of multiple threads.

and

You can control the maximum number of threads by using the ThreadPool.GetMaxThreads and ThreadPool.SetMaxThreads methods.

and

When not to use thread pool threads: You have tasks that cause the thread to block for long periods of time. The thread pool has a maximum number of threads, so a large number of blocked thread pool threads might prevent tasks from starting.

In TaskScheduler Class it says:

Long-running tasks: You may want to explicitly prevent a task from being put on a local queue. For example, you may know that a particular work item will run for a relatively long time and is likely to block all other work items on the local queue. In this case, you can specify the System.Threading.Tasks.TaskCreationOptions option, which provides a hint to the scheduler that an additional thread might be required for the task so that it does not block the forward progress of other threads or work items on the local queue. By using this option you avoid the thread pool completely, including the global and local queues.

or you may create your own TaskScheduler like in this example.

[grobmiller]

Ok, thank you very much for the detailed answers and your investigations. And yes, I tried it with 10-20 threads and then it worked. That means the connect (with 100 tasks) really didn't just work because the thread pool was exhausted.

So it's not a problem in the MQTTnet library (which means this issue can be closed) but it's a problem in my understanding of things.

• Task.Run(...) shall only be used for short-running stuff
• Long running processes shall always have their own threads
• Thread.Sleep(...) does return computing resources to the CPU (so that another thread can be processed), but it does NOT return a thread to the thread pool (which is quite clear... in hindsight).

[OidaTiftla]

I'm glad I could help.

[SeppPenner]

Good to know that this is solved. Thank you @OidaTiftla. By the way, cool user name 😅