Nservicebus: New value for default maximum concurrency level in V6

[ramonsmits]

The current default maximum concurrency level is 100 for V6. The RavenDB persistence has issues with concurrency level set higher than the number of actual cores and using MSDTC and can cause serious throughput issues.

Maybe it's better to set the default to 1 to be compatible with V5 or use the number of cores to still reap the benefits of improved default throughput but not have the RavenDB issues.

[mauroservienti]

use the number of cores

:+1: In my RavenDB tests running with 32 as concurrency level caused tests to fail with CancelledTaskException, I haven't yet investigated the root cause.

[andreasohlund]

Maybe it's better to set the default to 1 to be compatible with V5

:-1: it was a conscious decision since we want to expose users for concurrency right away and it also make sense to have a value that will perform "well" as the default

or use the number of cores to still reap the benefits of improved default throughput but not have the RavenDB issues.

:+1: to pursue this, we only picked 100 since that was a good value for the sql connection pool. Perhaps max(2,number of cores) is a good start?

[MarcinHoppe]

@andreasohlund At first glance this seems misguided wrt SQL. If message processing lasts anything more than approx 0ms :) and you have more than 100 msgs in the input queue, then the connection pool gets exhausted immediately and user starts to get exceptions. Using NHibernate persistence (as is often the case with SQL transport) only makes it worse.

Anyone has similar experience?

[MarcinHoppe]

Here's an issue that is caused by this high default concurrency:

https://github.com/Particular/NServiceBus.SqlServer/issues/286

There's also an associated Desk case.

[andreasohlund]

How about we dial it back to max(2,number of cores) for now?

[andreasohlund]

@ramonsmits would 2 on a single core machine still screw up ravendb?

[mauroservienti]

No. I'm my 8 cores machine I'm observing small issues with 32 as concurrency level.

[mauroservienti]

Where by small I mean that given 50.000 messages it might happen that 20 generate an error and are retried.

[MarcinHoppe]

2 x number of cores should be more practical than 100 but still not very scientific :).

[andreasohlund]

@mauroservienti so numcores*4 would be an option then? lets go with that?

[andreasohlund]

Talking to @SimonCropp and @SzymonPobiega we concluded that max(2,number of cores) would be conservative but pretty ok since

• All users would be exposed to concurrency in development
• Performance tests would be pretty ok with default settings since they would likely be done on more powerfull machines
• Users can always change the setting to a higher value if needed

I'm going ahead with the change

[andreasohlund]

Please review https://github.com/Particular/NServiceBus/pull/4092

[andreasohlund]

Fixed in https://github.com/Particular/NServiceBus/pull/4092