Hystrix Timeout Not Getting Honoured

[pratikkumar19]

I am using hystrix to make a Network call . The execution isolation strategy is THREAD.
The thread timeout is set as 20 ms.
For 95 percentile requests the timeout is getting honoured, but for 98 and 99 percentile requests the time taken for getting response from Network Call is around 50-100 ms.

[mattrjacobs]

When you mention "time taken for getting response from Network call", what are you using to measure this? Is it the Hystrix latency metrics?

[pratikkumar19]

Yeah as seen from circuit breaker metrics the 99 percentile latency > 50 ms

[mattrjacobs]

Got it, thanks. In that case, my guess would be that there's some (small) number of executions that are getting unlucky. Timeouts occur via a HystrixTimer thread ticking and emitting a HystrixTimeoutException. If that thread doesn't execute precisely when asked for (GC, other system overhead), then the timeout will not fire precisely when you ask it to.

I'll write up a unit test that collects latencies on a short-timeout call, and see if I can replicate your finding.

[pratikkumar19]

Also to give you a full context I have 3 threadPools:
ThreadPool A (Timeout 10ms)- Working well
ThreadPool B(Timeout 20ms) - 99 percentile breach
ThreadPool C(Timeout 200ms) - Working well

Also when I was not using ThreadPool C , all threads in ThreadPool B used to timeout within 35ms.(even 99 percentile)

[mattrjacobs]

When I run the test at: https://github.com/mattrjacobs/Hystrix/blob/8ae1f0501040f0d059198c30bffebb92b1cdba84/hystrix-core/src/test/java/com/netflix/hystrix/HystrixCommandTest.java#L3122

I get the following results with 4 threads each executing 1000 commands:
(timeout = 30ms):
Number of samples : 4000
Min : 30
Mean : 30.931
Median : 31
90p : 32
95p : 32
99p : 34
Max : 59

(timeout = 20ms):
Number of samples : 4000
Min : 20
Mean : 20.882
Median : 21
90p : 22
95p : 22
99p : 24
Max : 43

(timeout = 10ms):
Number of samples : 4000
Min : 10
Mean : 10.92375
Median : 11
90p : 12
95p : 12
99p : 15
Max : 40

[pratikkumar19]

Sorry for not informing you earlier but the 3 threadpools I mentioned in the above comment are all operating with 40 threads each.

And what are the parameters in :
HystrixCommand cmd = getCommand(ExecutionIsolationStrategy.THREAD, AbstractTestHystrixCommand.ExecutionResult.SUCCESS, (TIMEOUT_IN_MS * 2) + 10, AbstractTestHystrixCommand.FallbackResult.SUCCESS,* TIMEOUT_IN_MS*);

[mattrjacobs]

That code is just ensuring that the latency of the command is greater than the timeout. That makes sure that, when run, the command always times out.

[pratikkumar19]

So what about fallback latency?? And the threadpool being null in case of gather metrics test??

[mattrjacobs]

Yes, the latency of the fallback is included in the overall latency metric. Is the fallback more complex/latent for the command you're seeing latency outliers?

[pratikkumar19]

Actually I have no fallback implemented. But I suspect that the multiple threadpools might be causing this breach.
And will the threapool timeout be considered for the fallback also?

[pratikkumar19]

Also currently it is deteriorating to 330 ms breaches in 99 percentile

[mattrjacobs]

Without knowing more about your system, it's really hard to say what the root cause may be. One experiment that may be useful is to change the number of HystrixTimer threads used. If too much work is happening on them, then raising the number may alleviate that pressure.

[pratikkumar19]

So how do I change number of Hystrix Timer Threads? Is the setting exposed to library user?
Also all my network calls are synchronous(using execute()).

[mattrjacobs]

The code is at com.netflix.hystrix.HystrixTimerThreadPoolProperties. The config value is: hystrix.timer.threadpool.default.coreSize=N. The default value is the number of processors that the JVM thinks you have.

[mattrjacobs]

Closing due to inactivity. Please re-open if there's more to discuss