Hystrix: What's the relationship between thread pool properties and queue properties?

[asarkar]

I'm trying to understand the relationship among the following properties. The Wiki doesn't talk about these, other than just listing these properties. I'm also not clear on what's the queue for.

coreSize, maximumSize and maxQueueSize

The following thread pool settings allows for 3 concurrent requests. I don't see any change in behavior based on the value of maxQueueSize.

HystrixThreadPoolProperties.Setter()
    .withAllowMaximumSizeToDivergeFromCoreSize(true)
    .withCoreSize(3)
    .withMaxQueueSize(2)

[richardhsu-dev]

Here are my two cents after reading the code in-depth:

In short,

• coreSize: minimum number of worker threads that the ExecutorService keeps alive
• maximumSize: maximum number of worker threads that the ExecutorService can use. This is bounded by the system capacity, which if breached, causes the system to go crash.
• maxQueueSize: maximum number of Runnable task the ExecutorService can take in.

Ideally, coreSize <= maximumSize. However, allowMaximumSizeToDivergeFromCoreSize() allows you to override that, so even if you have coreSize > maximumSize, it'll just return whichever is biggest. So yes, it will allow for 3 concurrent requests. maximumSize is different from maxQueueSize, which is the number of Runnables submitted to the same threadpool.

Perhaps you actually meant .withMaximumSize(2)?

[asarkar]

@richardhsu-dev

Perhaps you actually meant .withMaximumSize(2)?

No, I meant withMaxQueueSize indeed. maximumSize default is 10 and I didn't see any need to change it.
I'll ask my question again based on what you said. In Java, a Thread executes a Runnable and the relationship is 1:1. Thus, the following are basically the same thing, because for executing every Runnable, there's a Thread required.

maximumSize: maximum number of worker threads that the ExecutorService can use. This is bounded by the system capacity, which if breached, causes the system to go crash.

maxQueueSize: maximum number of Runnable task the ExecutorService can take in.

Can you provide an example where different values of maximumSize and maxQueueSize have any significance?

[mattrjacobs]

I'll work on making the Wiki contain more useful information.

There are 2 data structures involved in a thread pool: the set of tasks (the actual pool), and a queue in front of the thread pool.

The thread pool contains all tasks which are currently running. Configuring this pool sets up the number of threads that may execute concurrently. Relevant config:

• coreSize: number of threads to keep in the pool at all times
• maximumSize: maximum number of threads in the pool
• allowMaximumSizeToDivergeFromCoreSize: allow the prior config values to differ (previously only coreSize was exposed, and maximumSize was forced to that value)

The queue is a different data structure. It contains tasks which have not yet begun to run. Tasks first get placed in this queue, and the thread pool picks them up from there. Relevant config:

• maxQueueSize: size of the queue (static - can only get set at startup)
• queueRejectionThreshold: some number below the maxQueueSize that controls maximum number of tasks in the queue. If queueRejectionThreshold is 5, and there are 5 tasks in the queue, the queue will reject the next task submitted.

Does that help?

[asarkar]

Thanks @mattrjacobs, that is very helpful. I've some follow up questions:

1. How long will a task wait in the queue before picked up for execution? Is there a timeout setting?
2. What happens if a SynchronousQueue is used maxQueueSize = -1, coreSize = maximumSize = 10 and I attempt to run 3 concurrent tasks?
3. What happens if a LinkedBlockingQueue is used maxQueueSize = 2, coreSize = maximumSize = 10 and I attempt to run 3 concurrent tasks?

Also, perhaps coreSize should be renamed to minimumSize; along with maximumSize, that'd help clarify the intent.

[mattrjacobs]

The names follow the JDK ThreadPoolExecutor convention: https://docs.oracle.com/javase/7/docs/api/java/util/concurrent/ThreadPoolExecutor.html

I think your questions should all be answered by the documentation there - let me know if you have more questions after reading that.

[asarkar]

I'll try to answer my own questions based on the Javadoc you linked to.

How long will a task wait in the queue before picked up for execution? Is there a timeout setting?

No timeout. Tasks wait in the queue until picked up for execution.

What happens if a SynchronousQueue is used maxQueueSize = -1, coreSize = maximumSize = 10 and I attempt to run 3 concurrent tasks?

Based on the following statement, this results in a new thread being created and the task is executed without queued.

If a request cannot be queued, a new thread is created unless this would exceed maximumPoolSize, in which case, the task will be rejected.

What happens if a LinkedBlockingQueue is used maxQueueSize = 2, coreSize = maximumSize = 10 and I attempt to run 3 concurrent tasks?

Same as using SynchronousQueue

I think a URL to the ThreadPoolExecutor will be very useful in the Wiki.

[bltb]

@asarkar , @mattrjacobs ... and just for laughs... :smile: ... it is possible to implement your own https://github.com/Netflix/Hystrix/blob/master/hystrix-core/src/main/java/com/netflix/hystrix/strategy/concurrency/HystrixConcurrencyStrategy.java with an alternative ThreadPoolExecutor implementation, e.g.
https://tomcat.apache.org/tomcat-8.0-doc/api/org/apache/tomcat/util/threads/package-summary.html

[asarkar]

@bltb That's good to know but I failed to see the relevance to my question. Can you elaborate?

[mattrjacobs]

@asarkar https://github.com/Netflix/Hystrix/wiki/Configuration#threadpool-properties

[asarkar]

@mattrjacobs Thanks for that update. My suggestions is to start the section ThreadPool Properties with a note "The behavior and naming conventions for the thread pool closely follow the ThreadPoolExecutor Javadoc", instead of putting somewhere else that may not strike the user as important.
Also, are my answers above correct (for completeness sake)?

[mattrjacobs]

Sure, thanks for the suggestion. I updated the Wiki as suggested.

Your answers are correct. One caveat - in the first case, an unsubscribe to a Hystrix command does remove the task from a thread pool queue, so even though there is not a timeout mechanism, the task doesn't sit there forever.

[asarkar]

@mattrjacobs Thank you for updating the Wiki. I didn't quite get what you said about unsubscribing from the Hystrix command. If it's sitting in the queue waiting for a thread, who is subscribing and unsubscribing?

[mattrjacobs]

Consider the following code:

Future<String> f = cmd.queue()
f.cancel()

If the command takes 1000ms, the work isn't stuck in the queue for 1000ms, the cancellation takes it out.

[asarkar]

Ah, makes sense. They're not unsubscribing from the command per se, but from the Future or Observable received as a result of submission. Thanks Matt, I'm closing this issue.

[asarkar]

OK @mattrjacobs , I lied, I've a follow up question. Say a task is submitted and sitting in the pool. The client put a timeout on the Future or Observable but doesn't explicitly cancel it. In that case, even if the task is never executed, is it taken out from the queue once the timeout expires?

[mattrjacobs]

Yes, there's an implicit unsubscription at every command completion

[ShikhaSingh1]

@mattrjacobs I came across this - "If fewer than corePoolSize threads are running, a new thread is created to handle the request" in various docs
After reading the above discussion, I got a question, when one says corePoolSize is the minimum(atleaset) number of threads that has to sit in the thread pool, how can we ever encounter a situation where we will have fewer threads than the corePoolSize as mentioned.

[ysheikh245]

Folks, if I want to limit my application to only have 5 transactions at any time, how can I leverage the coreSize, maximumSize and maxQueueSize() parameters?

• I tried coreSize=5, maximumSize=5, maxQueueSize= 5 as values respectively and I see 10 requests were processed when I sent 30 requests using JMeter
• I tried coreSize=5,, maximumSize=5, maxQueueSize= 1 as values respectively and I still saw 10 requests being processed when I sent 30 requests using JMeter
• I tried coreSize=1, maximumSize=5, maxQueueSize= 5 as values respectively and I still saw 10 requests being processed when I sent 30 requests using JMeter

I am not sure if I am understanding those parameters correctly. Any help will be appreciated.