Polly: Suggested Implementation of Circuit Breaker In AWS Lambdas

[RichieRunner]

Summary: What are you wanting to achieve?
I want to be able to track Circuit Breaker state across AWS Lambda invocations (by updating and reading from an Environmental Variable) that all instances of the Lambda will share.

Every runtime, the Circuit Breaker's state of how many exceptions have already occurred will always be at 0. Therefore, I would need to be keeping this count outside of my program in an Environmental Variable.

However, I'm stuck on how I would set the state of the Circuit Breaker itself at every runtime to how many exceptions have already occurred (by tracking my Environmental Variable) so that the handledEventsAllowedBeforeBreaking configured property knows when it should trip the breaker.

[reisenberger]

@RichieRunner Apologies for delayed response - been on travel 5 days.

To run through some candidates for maintaining state across function executions in AWS lambda:

( _My functions experience is closer to Azure. Anyone with deeper AWS Lambda experience able to confirm/deny/shed deeper light on the q or observations below?_ )

Environment variables: Is it possible (from within an AWS Lambda) to update the state of an environment variable, such that the new state will be picked up by subsequent and parallel function invocations? From a quick read of AWS doco/blogs, it sounds as if environment variables may work similarly to Azure, typically set at deployment time or (for dev) in the config of your dev environment.

AWS Lambda execution context: AWS suggest (in that doco and these best practice docs) to use this for sharing state across invocations. However, there are no guarantees: the execution context _may_ be re-used across invocations. My shallow reading is this approach is similar to using statics within Azure functions. It probably works (shares the state) when the same warm 'function app' (or similar execution environment) is re-used, but not when a new instance is cold-started; eg after an idle period; or when your cloud provider scales the function app out. That might be an ok compromise for some use cases.

AWS Lambda step functions: Theoretically it might be possible to construct a circuit-breaker using step functions, with states, in particular the task state to execute another Lambda. However, I can't say if it is really viable. This looks analogous to using Durable functions in Azure. The problem with using _durable orchestration functions_ for a circuit-breaker in Azure was that eternal orchestrations are contra-indicated - the state history builds up - and similar constraints affect AWS step functions. The 'workround' for that in Azure was to "continue as new execution" (AWS has similar), but (in the Azure approach) there were limitations with missing requests while the durable function was restarting-as-new.

Durable Entity Functions: The Azure functions teams are solving the general case (persisting state across function invocations) at a deeper architectural level, through Durable Entity functions, which are like Distributed 'Actors' with persistent state. We plan shortly to demonstrate a functions-native Distributed Circuit Breaker using Azure Durable Entity Functions. Does AWS have any feature similar to Durable Entity Functions?

---

Did you - or does anyone - have other ideas for the circuit-breaker native-in-AWS-lambda case?

[aprooks]

Afaik Lambdas don't provide any shared context and suggested approach was to use dynamodb as distributed state. This should work both for curcuit breakers and bullheads.

[reisenberger]

Yes, I forgot to state the obvious 🙂 - as @aprooks says, you can always use any distributed store like DynamoDb, Redis or similar.

(That is also what we intend for an unrelated-to-functions distributed circuit-breaker, as discussed eg in #287. My thinking/comments were around any possibilities native to AWS functions, because in Azure Functions we now/soon have Durable Entity functions, which means we can do persistent distributed state from within functions without having to reach out to an external state store like Redis or DynamoDb/CosmosDb ... it's pretty cool).

[RichieRunner]

Yes, storing state in an external store would be the tradeoff we'd have to make in this case. Thank you for your attention to this inquiry.