Hystrix-javanica synchronous command creates thread

[andrewe123]

I am testing some timing out hystrix synchronous commands https://github.com/Netflix/Hystrix/tree/master/hystrix-contrib/hystrix-javanica#synchronous-execution

The thread pool runs out on the 10 test by default because something is creating threads (the fallback procedure?) Is this expected behavour? How can I close the threads from the synchronous command to prevent Hystrix becoming overloaded after 10 failures?

Stacktrace:

Stacktrace was: com.netflix.hystrix.exception.HystrixRuntimeException: getUserById could not be queued for execution and fallback failed.
...
Caused by: java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@b36ec557 rejected from java.util.concurrent.ThreadPoolExecutor@e9a1174c[Running, pool size = 10, active threads = 10, queued tasks = 0, completed tasks = 0]

[mattrjacobs]

@andrewe123 Whenever a HystrixCommand executes and uses thread-isolation, it consumes a Hystrix thread. If the threadpool runs out of threads due to more concurrency than threads, then Hystrix returns the fallback.

The threads are only used by Hystrix, and only when executing a command. Are you seeing thread-pool exhaustion after commands have finished, or during periods of high concurrent execution?

[dmgcodevil]

@mattrjacobs, hystrix uses only threads from internal thread pool per command group, if thread pool is fixed size then how thread overflow possible ? Or problem is different ?

[mattrjacobs]

In the console above, the threadpool has a fixed capacity of 10, and 10 active threads. When the 11th task is submitted against the threadpool (by executing the command), while those 10 are are still in flight, the command gets properly rejected.

Unless I'm missing something, this is all working as expected.

[andrewe123]

My unit tests look something like the following

@Test
Public void test_getUser() {
expectException(HystrixRuntimeException.class);
expectCause(TimeoutException.class);
service.getUser();
}

I was expecting the command would be finished when the exception is evaluated in the test, but it doesn't seem to be the case. If I have more than 10 unit tests in a class I get the rejected new thread exception instead of the expected timeout. I also tried another test with a single loop to 20 caughting the Hystrixruntimeexception exception inside, once it gets to the 11th run the exception cause is no longer a timeout but a rejection because the active thread limit is hit. I also tried sleeping for 500ms after each iteration of the loop with the same results.

Is this behaving as expected? If the threads are not released after 500ms after the exception is caught like in my test wouldn't hystrix always be rejecting new commands? I'm sure I'm missing something, but I don't know what it is.

[mattrjacobs]

One point here is that there's nothing Hystrix (or any other code) can do to release a thread. Hystrix can issue a timeout, which delivers a value to the caller, but it can't forcible terminate the work on the other thread. By default, it attempts to interrupt the thread on a timeout, but that doesn't have to be respected. Can you try checking for that InterruptedException in your simulated work?

If that doesn't pan out, can you share some of your code so I can take a deeper look?

[mattrjacobs]

Closing due to inactivity. Please re-open if there's more to discuss

[dmitrykvsap]

We're observing the error mentioned some time earlier in this thread a couple of years ago. Was the root cause identified? Looks like the issue was just closed.
Caused by: java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@b36ec557 rejected from java.util.concurrent.ThreadPoolExecutor@e9a1174c[Running, pool size = 10, active threads = 10, queued tasks = 0, completed tasks = 0

[mattrjacobs]

The issue was closed ~10 months ago.

Thread pool rejection is absolutely expected to some degree when using Hystrix. By design, Hystrix provides bounded thread pools that may fill up and reject work if they get full.

What's the precise nature of your situation?

[fehmicansaglam]

@mattrjacobs Could you help me clarify this? As in the issue description, when the log says fallback failed., does it still mean it was run? I am confused if there are no threads left for the command itself how the fallback can be run?

[mattrjacobs]

There are many failure modes, and Hystrix attempts to treat them all equivalently by running the user-supplied fallback:

• failure - fallback is run on the Hystrix thread
• threadpool full - fallback is run on the caller thread
• semaphore full - fallback is run on the caller thread
• short-circuit - fallback is run on the caller thread
• timeout - fallback is run on the thread which triggers the timeout (HystrixTimer)

[sahilchdhary]

I am having the similar issue. I am getting below error:

Task java.util.concurrent.FutureTask@6c15a847 rejected from java.util.concurrent.ThreadPoolExecutor@29172ba1[Running, pool size = 10, active threads = 10, queued tasks = 0, completed tasks = 0]
    at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047) ~[?:1.8.0_131]

Even though, there should not be any concurrency issue, as i am submitting only one request. So, ideally only one task should be submitted to the threadpool. But somehow, hystrix seems to be spanning out threads for the same task, until the pool gets exhausted, and the last thread just goes to fallback method, due to the above error.

Could someone help here ?
I am using hystrix-javanica:1.5.12 and spring 4.3.3-RELEASE