Hystrix: How to resolve Thread pool rejections in Hystrix

[murthyneelam]

I have to support 15000 requests per minute, Could you please suggest me the Thread pool configurations

We used default Hystrix configurations other than these below things

Time out = 3000
circuitBreaker.requestVolumeThreshold = 100
CoreSize = 32
maxQueueSize = -1
queueSizeRejectionThreshold =5

with this configuration, we are able to see successful responses for 3400 req/min

When we try with 10000 req/min, more than 40% are getting rejected from thread pool with below configurations

Time out = 3000
circuitBreaker.requestVolumeThreshold = 100
CoreSizw = 55
maxQueueSize = -1
queueSizeRejectionThreshold =5

Could you please help us to get out of this issue and we are using 1.41 verssion of Hystrix

[mattrjacobs]

The premise behind bounded thread pools is that it's preferable to keep the worst case manageable and fail-fast in some cases than to allow unbounded executions.

From your descriptions, when you did 3400 req/min (= 57 req/sec), a thread pool of 32 was able to keep up. When you did 6000 req/min (=100 req/sec), a thread pool of 55 was able to keep up. This seems like a linear relationship.

Your choices are:
1) Keep expanding the threadpool. I'm not sure of your architecture, but the biggest threadpool in the internal system I operate at Netflix has 16 threads.
2) Break apart the call you're making. It's hard to tell from the stats you reported, but I will make the simplifying assumption that 0% rejection at 32 threads and 57 rps uses 16 threads on average. Note the true number is between 0-32 and I can't determine this from the data you posted. In this case, using Little's Law, the mean latency of your call is 16/57 sec = 280 ms. This is a pretty expensive operation. It may be worth dividing it up into smaller calls or spending time to reduce the latency.
3) Switch to nonblocking I/O. The problem with blocking I/O is that it consumes a scarce resource (a thread). If you used nonblocking I/O, you're just taking up space on a queue and you have a lot more room for concurrent connections. If you go down this route, you'll want to use HystrixObservableCommand to wrap the nonblocking call. This won't use threads, and you should be able to support a much higher degree of concurrency.

In general, Hystrix is not a silver bullet that allows you to support arbitrary concurrency. You have to figure out how much your architecture can support without Hystrix and then understand how to leverage Hystrix to make this architecture more resilient.

[mattrjacobs]

Closing due to inactivity. Please re-open if there's more to discuss