Generator-jhipster: Setting up Ribbon's Connection timeout in microservice environment

[jvence]

I'm trying to configure the gateway's zuul connection timeout and have added the ConnectTimeout parameter to the gateway's application.yml The ConnectTimeout timeout seems to be ignored. We need a way to specify this parameter.

ribbon:
    eureka:
        enabled: true
    ConnectTimeout: 30000
    ReadTimeout: 30000

One of my microservices is slow to return a response (and that's ok) so I'm seeing lots of error in the gateway log:

Caused by: com.netflix.hystrix.exception.HystrixRuntimeException: hpwebservices timed-out and no fallback available.
    at com.netflix.hystrix.AbstractCommand$16.call(AbstractCommand.java:806)
    at com.netflix.hystrix.AbstractCommand$16.call(AbstractCommand.java:790)
    at rx.internal.operators.OperatorOnErrorResumeNextViaFunction$1.onError(OperatorOnErrorResumeNextViaFunction.java:99)
    at rx.internal.operators.OperatorDoOnEach$1.onError(OperatorDoOnEach.java:70)
    at rx.internal.operators.OperatorDoOnEach$1.onError(OperatorDoOnEach.java:70)
    at rx.internal.operators.OperatorDoOnEach$1.onError(OperatorDoOnEach.java:70)
    at com.netflix.hystrix.AbstractCommand$DeprecatedOnFallbackHookApplication$1.onError(AbstractCommand.java:1521)
    at com.netflix.hystrix.AbstractCommand$FallbackHookApplication$1.onError(AbstractCommand.java:1411)
    at com.netflix.hystrix.HystrixCommand$2.call(HystrixCommand.java:314)
    at com.netflix.hystrix.HystrixCommand$2.call(HystrixCommand.java:306)

[gmarziou]

Please use stackoverflow for this kind of question as this is not a jhipster issue, it's a spring cloud configuration issue.

I did solve this at work on Thursday but l aready forgot :), your ribbon config is not at the right place I think it should be under hystrix, if you don't find I will tell you on stackoverflow on Monday.

Great thing is that these global timeout values can be customized per service name.

[gmarziou]

Here is what I read to make it work:

https://github.com/spring-cloud/spring-cloud-netflix/issues/321

[gmarziou]

Here is my config, maybe we should consider adding to generated yml files.

zuul:
    host:
        connect-timeout-millis: 5000
        socket-timeout-millis: 10000

hystrix:
    command:
        default:
            execution:
                isolation:
                    thread:
                        timeoutInMilliseconds: 10000

[PierreBesson]

@gmarziou, if you found them to be useful in preventing issues, I would trust your judgment here, please submit those properties as PR.

More generally, It would be useful to set gateway's properties to their default values, so that people can easily find them.

For example, these were very important to tweak during our stress tests.
zuul.host.max-total-connections: 200
zuul.host.max-per-route-connections: 20

[gmarziou]

I'm not suggesting to use my values (they are specific to my project).

Like your zuul settings, I'm suggesting that we put default values and some comments so that users know what to adjust when they get timeout exceptions.

[pkrishnath]

@gmarziou that has saved my time

[pm0202]

this is worked for me ...
https://github.com/spring-cloud/spring-cloud-netflix/issues/321

[vspdontukurthi]

Just add below snippet at application.yml where located at gateway-ms that works perfect

~yml
hystrix:
command:
default:
execution:
isolation:
thread:
timeoutInMilliseconds: 30000
ribbon:
ReadTimeout: 60000
connection-timeout: 3000
eureka:
enabled: true
zuul:
host:
connect-timeout-millis: 5000
max-per-route-connections: 10000
max-total-connections: 5000
socket-timeout-millis: 60000
semaphore:
max-semaphores: 500
~

[cristianobombazar]

@vspdontukurthi in my case, ribbon.ReadTimeout solve the problem. Thanks!

[jsm174]

We had this issue too. One of our microservices was taking ~3 seconds to complete, so 90% of the time we would get the timeout error even though the service completed successfully.

Out of the box, Ribbon's ReadTimeout, and ConnectTimeout are both 1000. These values are found in RibbonClientConfiguration.java

The Ribbon timeout is calculated in AbstractRibbonCommand.java:

ribbonTimeout = (ribbonReadTimeout + ribbonConnectTimeout) * (maxAutoRetries + 1) * (maxAutoRetriesNextServer + 1);

maxAutoRetries and maxAutoRetriesNextServer values are found in DefaultClientConfigImpl.java

We updated our gateway's application.yml and added:

ribbon:
    ReadTimeout: 4000

This keeps the ribbonTimeout calculation at 10000 which matches the value of hystrix.command.default.execution.isolation.thread.timeoutInMilliseconds.

(4000 + 1000) * (0 + 1) * (1 + 1)

This avoids receiving errors like this:

gateway-app_1 | 2018-11-14 15:54:36.072 WARN 1 --- [ XNIO-2 task-16] o.s.c.n.z.f.r.s.AbstractRibbonCommand : The Hystrix timeout of 10000ms for the command microservice is set lower than the combination of the Ribbon read and connect timeout, 12000ms.

Finally, the Ribbon property names are case sensitive, ie ReadTimeout, not read-timeout, or readTimeout.

[PierreBesson]

@jsm174 You are welcome to contribute this information to the docs, it would be very useful to document the Spring Cloud stack defaults. Please open PR to https://github.com/jhipster/jhipster.github.io if you can.

[jsm174]

@PierreBesson I can do that. Where would the best place be to put this?

[PierreBesson]

https://github.com/jhipster/jhipster.github.io/blob/master/pages/microservices-in-production.md

[lakith]

@vspdontukurthi Your solution worked perfectly. Thank you

[cmyker]

@vspdontukurthi

Just add below snippet at application.yml where located at gateway-ms that works perfect

hystrix: 
  command: 
    default: 
      execution: 
        isolation: 
          thread: 
            timeoutInMilliseconds: 30000
ribbon: 
  ReadTimeout: 60000
  connection-timeout: 3000
  eureka: 
    enabled: true
zuul: 
  host: 
    connect-timeout-millis: 5000
    max-per-route-connections: 10000
    max-total-connections: 5000
    socket-timeout-millis: 60000
  semaphore: 
    max-semaphores: 500

Error here, has to be

ribbon: 
  ReadTimeout: 60000
  ConnectTimeout: 3000