Polly: Transition Open->HalfOpen only happens when the circuit is probed (call placed or CircuitState checked)

[shagilt]

Sometime, my UT is failing because state is not reset to HalfOpen from Open. We are emitting metrics and logs during this phase and i wants to validate this in my UT.
For my UT, I have configured durationOfBreak as 1 sec. Is there a different approach to test this? Do you think 1 sec is too low(though it is UT)?

[reisenberger]

@shagilt It is hard to tell what may be happening without seeing some code: please post sample code of the UT, if my below comments do not help.

The circuit-breaker does not transition [*] from Open to HalfOpen state until some action is made on the circuit-breaker (whether that action is placing another call through the circuit-breaker, or reading the CircuitState property). If you have a unit-test just waiting for time to pass, and then expecting the onHalfOpen delegate to be fired after the time has passed, the onHalfOpen delegate will not yet fire. Try reading the CircuitState property in the test, to trigger the circuit state transition to HalfOpen.

This [*] behaviour (not transitioning to HalfOpen until some other action on the circuit-breaker) is intentional: it avoids the perf cost of the circuit-breaker allocating an internal Timer and callback just to transition the circuit-state.

You can see (if of interest) the extensive tests Polly already has on circuit-breaker state transitions and further tests around the halfopen state and onHalfOpen delegate. This line demonstrates the [*] behaviour:: that the onHalfOpen delegate is not triggered until some other action is made on the policy.

[shagilt]

Thank you @reisenberger. Reading CircuitState property solved my UT issue. Do you recommend the same in production code so that state transition is guaranteed from Open to HalfOpen after durationOfBreak?

As a side note, this behavior is not documented in the wiki https://github.com/App-vNext/Polly/wiki/Circuit-Breaker#circuit-state-for-health-reporting-polly-v40 [ from wiki "_The circuit remains open for the configured durationOfBreak. After that timespan, but before the next action, the circuit transitions to: Half-Open_"].

[reisenberger]

Glad that sorted :+1: . Thanks for the wiki feedback: have made the part of the circuit-breaker wiki you quoted more precise. The timing that this is triggered is also documented later in the same wiki:

onHalfOpen: The delegate is executed immediately after the circuit transitions to half-open. _Note:_ the delegate does not execute automatically after the automated break timespan has expired. It executes when state is next queried - for example, at the next attempt to execute an action, or when the state is next queried manually.

Re the question:

Do you recommend the same in production code so that state transition is guaranteed from Open to HalfOpen after durationOfBreak?

It depends whether the timing of emitting/logging that action is critical to you. Consider also how high your circuit throughput is (/is likely to be) in relation to the durationOfBreak. In any high throughput system you will 'pretty soon' get another call through the circuit anyway that will trigger the same transition 'pretty soon' after the durationOfBreak. For example, if durationOfBreak is 10 seconds but you typically get 20, 50, 100s calls per second through the circuit, you will typically very soon after the 10 seconds get another call which will trigger the open->halfOpen transition. So (for these kinds of circuit) it hardly seems worth implementing your own timing trigger in order to make the onHalfOpen timing marginally more precise.

This is the design choice we made for Polly: Polly is geared to perf for those high-throughput systems, so we choose _not_ to allocate a Timer and trigger a callback within Polly to transition open->halfOpen at 'precisely' the designated time (same design choice as Hystrix IIRC). The trade-off of that design choice though is that the timing of the open->halfOpen transition is not precise in slower throughput systems.

[shagilt]

Ok, got it. In our scenario, we dont perform any action until circuit become HalfOpen or Closed. I will add CircuitState 'read' so that state transition is guaranteed.

And thanks for updating the wiki!

[reisenberger]

Cool. I've renamed the issue title to aid discoverability around the way half-open behaves. Closing as I think we're done here, but let us know (here, or fresh issue) if you need anything else! Thanks for your feedback and contribution to the Polly community!