Resilience4j: Circuit breaker not opening when @CircuitBreaker and @TimeLimiter used together

[georgeharley]

Resilience4j version:
1.3.1

Java version:
1.8.0_241

Spring Boot version:
2.2.5.RELEASE

If a method is annotated with @CircuitBreaker and @TimeLimiter annotations then the circuit breaker no longer seems to transition to OPEN when the failure rate is greater than the defined failureRateThreshold. If the @TimeLimiter annotation is removed from the method then the circuit breaker behaves as expected.

The below example code is based on this demo code. The annotated method is just a simple Spring Web HTTP GET handler in a @RestController class. Here the 100% failure fate is intentional.

    // @Bulkhead(name = "mybulkhead", type = Type.THREADPOOL)
    // @TimeLimiter(name = "mytimelimiter", fallbackMethod = "getFallback")
    @CircuitBreaker(name = "mycircuitbreaker", fallbackMethod = "getFallback")
    @GetMapping("/song")
    public CompletableFuture<String> getHandler() {
        return CompletableFuture.completedFuture(getSomeStringValue());
    }

    private String getSomeStringValue() {
        throw new RuntimeException("boo!");
    }

    private CompletableFuture<String> getFallback(HttpHeaders ignore, Throwable timeout) {
        return CompletableFuture.completedFuture("fallback!");
    }

With the bulk head and time limiter annotations left commented out, when the Spring Boot Web app is started and some light load is applied from the command line (just using curl from inside a while loop) then, as expected, after a few seconds the circuit opens. Evidence of that comes courtesy of a logging event consumer added in a Boot @Configuration class. It logs the event State transition from CLOSED to OPEN.

@Configuration
public class MyConfiguration {

    private static final Logger LOGGER = LoggerFactory.getLogger(ClientConfiguration.class);

    private final CircuitBreakerRegistry circuitBreakerRegistry;

    public ClientConfiguration(CircuitBreakerRegistry circuitBreakerRegistry) {
        this.circuitBreakerRegistry = circuitBreakerRegistry;
    }

    . . .

    @PostConstruct
    public void init() {
        CircuitBreaker circuitBreaker = circuitBreakerRegistry.circuitBreaker("mycircuitbreaker");
        circuitBreaker.getEventPublisher()
            .onStateTransition(event -> {
                CircuitBreaker.StateTransition transition = event.getStateTransition();
                LOGGER.info("Circuit breaker 'mycircuitbreaker' state change: {} !!!", transition);
            });
    }
}

If the application is run again with the bulk head and time limiter annotations uncommented and the same light request load applied then no circuit breaker state transition events are observed in the application log.

Here's an excerpt from the application.yml file containing configuration of the circuit breaker, time limiter, and bulk head.

resilience4j:
  circuitbreaker:
    configs:
      default:
        slidingWindowSize: 10
        slidingWindowType: TIME_BASED
        failureRateThreshold: 50
        waitDurationInOpenState: 5000
        minimumNumberOfCalls: 10
        permittedNumberOfCallsInHalfOpenState: 1
    instances:
      mycircuitbreaker:
        baseConfig: default
        failureRateThreshold: 33

  timelimiter:
    configs:
      default:
        timeoutDuration: 2000
    instances:
      mytimelimiter:
        baseConfig: default
        timeoutDuration: 5000
        cancelRunningFuture: true

  bulkhead:
    instances:
       mybulkhead:
         maxConcurrentCalls: 10
         maxWaitDuration: 100ms

I'm very new to Resilience4J so may well be making a very basic error. Just can't spot it.

[RobWin]

Hi,
its confusing that the CircuitBreaker works.
In order for the Spring AOP proxies to be created at runtime your class should either have an interface or else you should configure cglib.

https://docs.spring.io/spring/docs/current/spring-framework-reference/core.html#aop-introduction-proxies

[RobWin]

Could you add an interface to your class and check if it solves your issue?

[georgeharley]

@RobWin Adding an interface to the class made no difference to the reported behaviour.

I tried a few other things and stumbled on something that did make a difference. If the @CircuitBreaker and @TimeLimiter annotations are both present and they both reference the same fallback method in the class then the circuit breaker never opens when its failure rate threshold is exceeded (i.e. the behaviour seen yesterday). However, if the @CircuitBreaker and @TimeLimiter annotations reference _separate_ fallback methods then everything seems to work as hoped.

In a bit more detail...

Circuit breaker only
-

    @CircuitBreaker(name = "mycircuitbreaker", fallbackMethod = "getFallback")
    @GetMapping("/song")
    public CompletableFuture<String> getHandler() {
        return CompletableFuture.completedFuture(getSomeStringValue());
    }

    private String getSomeStringValue() {
        throw new RuntimeException("boo!");
    }

    private CompletableFuture<String> getFallback(Throwable throwable) {
        return CompletableFuture.completedFuture("fallback!");
    }

Intentionally arrange for 100% of requests to fail. When light request load is applied the circuit soon transitions from CLOSED to OPEN. Stays in either OPEN or HALF_OPEN from that point on. All as expected.

Circuit breaker and Time Limiter (same fallback method)
-

    @Bulkhead(name = "mybulkhead", type = Type.THREADPOOL)
    @TimeLimiter(name = "mytimelimiter", fallbackMethod = "getFallback")
    @CircuitBreaker(name = "mycircuitbreaker", fallbackMethod = "getFallback")
    @GetMapping("/song")
    public CompletableFuture<String> getHandler() {
        return CompletableFuture.completedFuture(getSomeStringValue());
    }

    private String getSomeStringValue() {
        throw new RuntimeException("boo!");
    }

    private CompletableFuture<String> getFallback(Throwable throwable) {
        return CompletableFuture.completedFuture("fallback!");
    }

Intentionally arrange for 100% of requests to fail. When light request load is applied the circuit stays CLOSED throughout. Unexpected behaviour.

Circuit breaker and Time Limiter (different fallback methods)
-

    @Bulkhead(name = "mybulkhead", type = Type.THREADPOOL)
    @TimeLimiter(name = "mytimelimiter", fallbackMethod = "getTimeoutFallback")
    @CircuitBreaker(name = "mycircuitbreaker", fallbackMethod = "getFallback")
    @GetMapping("/song")
    public CompletableFuture<String> getHandler() {
        return CompletableFuture.completedFuture(getSomeStringValue());
    }

    private String getSomeStringValue() {
        throw new RuntimeException("boo!");
    }

    private CompletableFuture<String> getFallback(Throwable throwable) {
        return CompletableFuture.completedFuture("fallback!");
    }

    private CompletableFuture<String> getTimeoutFallback(TimeoutException timeout) {
        return CompletableFuture.completedFuture("timeout fallback!");
    }

The @TimeLimiter references a different fallback method to the @CircuitBreaker. The new method expects a TimeoutException. Intentionally arrange for 100% of requests to fail. When light request load is applied the circuit soon transitions from CLOSED to OPEN and then stays in either OPEN or HALF_OPEN from that point on. The expected behaviour.

It's very pleasing to see things now working as hoped but I'm a little confused as to why the choice of fallback methods is making the difference.

[RobWin]

Thanks, now it's clear to me. That's the expected behavior.
Each decorator (CircuitBreaker, TimeLimiter) can have it's own fallback method. You can think of as an onion and the chain of responsibility pattern.

The decorator order is as follows:
Retry ( CircuitBreaker ( RateLimiter ( TimeLimiter ( Bulkhead ( YourMethod ) ) ) ) )

Exceptions are propagated from your method to the outside. The first fallback method which is responsible will handle the exception.
When your TimeLimiter and CircuitBreaker have the same fallback method, the TimeLimiter will handle the exception and convert it into a future which is completed successfully. Which means the CircuitBreaker won't count the calls as a failure.
When only your CircuitBreaker has a fallback method, it will count the call as a failure and then invoke the fallback.
I really think in your case it doesn't make sense to add a fallback method to the TimeLimiter since you want to count TimeoutExceptions as a failure.