HystrixRuntimeException: TestCommand fallback execution rejected

[KaliappanS]

I am using Hystrix 1.5.5 version. When I do load testing of bigger load like 1000 thread/second, all the requests are going through fallback method. Meanwhile, I am getting below exception too. Why do I get this below exception. TestCommand is my custom Hystrix class.

Caused by: com.netflix.hystrix.exception.HystrixRuntimeException: TestCommand fallback execution rejected.
at com.netflix.hystrix.AbstractCommand.handleFallbackRejectionByEmittingError(AbstractCommand.java:1026)
at com.netflix.hystrix.AbstractCommand.getFallbackOrThrowException(AbstractCommand.java:858)
at com.netflix.hystrix.AbstractCommand.handleThreadPoolRejectionViaFallback(AbstractCommand.java:976)
at com.netflix.hystrix.AbstractCommand.access$400(AbstractCommand.java:59)
at com.netflix.hystrix.AbstractCommand$12.call(AbstractCommand.java:593)
at com.netflix.hystrix.AbstractCommand$12.call(AbstractCommand.java:587)
at rx.internal.operators.OperatorOnErrorResumeNextViaFunction$4.onError(OperatorOnErrorResumeNextViaFunction.java:140)
at rx.internal.operators.OperatorDoOnEach$1.onError(OperatorDoOnEach.java:72)
at rx.internal.operators.OperatorDoOnEach$1.onError(OperatorDoOnEach.java:72)
at com.netflix.hystrix.AbstractCommand$HystrixObservableTimeoutOperator$3.onError(AbstractCommand.java:1173)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:54)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)

[mukteshkrmishra]

What did you define in your fallback?

[KaliappanS]

In my fallback method, I am hitting another url using REST call and returning a xml response

[mattrjacobs]

The key line in the logs is:

Caused by: com.netflix.hystrix.exception.HystrixRuntimeException: TestCommand fallback execution rejected.

This means that too many fallbacks were executing concurrently. Since the goal of Hystrix is to protect application threads, and fallbacks may run on the caller thread, Hystrix needs to limit the amount of concurrent fallbacks. Whenever this situation occurs, the fallback is not run, and an exception is returned to the caller.

In general, the purpose of a fallback is to provide substitute work that is less costly than the original work. Here are 2 options for proceeding:

1) Use a non-network fallback. That will be fast and no fallback concurrency limits will be hit

2) Wrap the fallback in its own Hystrix command. In this case, you're still getting protection when the original run() fails and the fallback path gets entered. You will still need to defined a fallback for this new command, and it should do little work to avoid problems as well.

[KaliappanS]

I did set these two properties of my TestCommand. It got fixed. This issue is due to when more number of requests go to fallback method concurrently. Hystrix has certain limit for the concurrent request.

.withFallbackIsolationSemaphoreMaxConcurrentRequests(Integer.MAX_VALUE)
.withExecutionIsolationSemaphoreMaxConcurrentRequests(Integer.MAX_VALUE)

[mattrjacobs]

By setting those properties, you're throwing away the ability to use a concurrency limit on semaphore-isolated commands and on fallbacks.

If your system starts timing out, fallbacks will begin to run on your application threads, and the setting you made above (fallbackIsolation = Integer.MAX_VALUE) will mean that all application threads may get consumed by your fallbacks.

I strongly advise against doing this - the whole point of using Hystrix is to set up these sort of guardrails such that your system cannot become unhealthy. As I mentioned above, when you build a fallback, you should aim to do less work in the fallback than in the normal execution path. That means that failing-fast will actually relieve load on your system.

[mukteshkrmishra]

I agree with @mattrjacobs , fallback should be mean of alternate work at a less cost. Generally, I use a chain of fallbacks (wrapped in Hystrix commands) to make sure we have a defined fallback pattern with each having less cost of work.

[mattrjacobs]

Closing due to inactivity. Please re-open if there's more to discuss.

[prasadram]

Hi ,
we are getting below exception for few hystrix commands , please check below for our hystrix configuration
trace=com.netflix.hystrix.exception.HystrixRuntimeException: testCommand fallback execution rejected.
at com.netflix.hystrix.AbstractCommand.handleFallbackRejectionByEmittingError(AbstractCommand.java:1043)
at com.netflix.hystrix.AbstractCommand.getFallbackOrThrowException(AbstractCommand.java:875)
at com.netflix.hystrix.AbstractCommand.handleThreadPoolRejectionViaFallback(AbstractCommand.java:993)
at com.netflix.hystrix.AbstractCommand.access$400(AbstractCommand.java:60)
at com.netflix.hystrix.AbstractCommand$12.call(AbstractCommand.java:607)
at com.netflix.hystrix.AbstractCommand$12.call(AbstractCommand.java:601)
at rx.internal.operators.OperatorOnErrorResumeNextViaFunction$4.onError(OperatorOnErrorResumeNextViaFunction.java:140)
at rx.internal.operators.OnSubscribeDoOnEach$DoOnEachSubscriber.onError(OnSubscribeDoOnEach.java:87)
at rx.internal.operators.OnSubscribeDoOnEach$DoOnEachSubscriber.onError(OnSubscribeDoOnEach.java:87)
at com.netflix.hystrix.AbstractCommand$HystrixObservableTimeoutOperator$3.onError(AbstractCommand.java:1194)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:54)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
at rx.Observable.unsafeSubscribe(Observable.java:10211)
at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41)
at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30)
at rx.Observable.unsafeSubscribe(Observable.java:10211)
at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41)
at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
at rx.Observable.unsafeSubscribe(Observable.java:10211)
at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41)
at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30)
at rx.Observable.unsafeSubscribe(Observable.java:10211)

hystrix:
metrics.enabled: false
stream.queue.enabled: false
command:
default:
execution.timeout.enabled: true
execution.isolation.thread.timeoutInMilliseconds: 9000
testCommand:
fallback.enabled: true
execution.isolation.thread.timeoutInMilliseconds: 6000
circuitBreaker.requestVolumeThreshold: 50
circuitBreaker.sleepWindowInMilliseconds: 10000
metrics.rollingStats.timeInMilliseconds: 10000
circuitBreaker.errorThresholdPercentage : 40
execution.timeout.enabled: true
threadpool:
testCommandThreadPool:
coreSize: 20
maximumSize: 100
allowMaximumSizeToDivergeFromCoreSize: true

Our fallback is static response there is no network call

i have few questions can any one please help me

Few times we saw fall back rejected due to que size , check below few lines of stack trace
Caused by: java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@3bd484c4 rejected from java.util.concurrent.ThreadPoolExecutor@1fd3e875[Running, pool size = 100, active threads = 100, queued tasks = 0, completed tasks = 1177163]
at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047)

-> In above we didn't see any log which says that queue size is reached maximum level, but still it rejected
-> can any one please explain the thread pool queue is for actual call or the same thread pool also will be used for fall back as well ?
because if thread pool reaches maximum why it should reject rather than serving response from fallback
-> So the actual call and fallback will be executed in the same thread ?
-> i am checking different properties to avoid this exception , can below property will help for resolving the issue ?
fallback.isolation.semaphore.maxConcurrentRequests