Dapr: Dapr side-car should not need a restart when a component/config is applied

[amanbha]

Currently in k8s, Dapr needs a pod restart to pick new configuration after they have been applied.
This will cause inconsistency in scenarios in which few pods of an app are restarted by platform. Restarted pods will use new config whereas pods which are not restarted will continue to use old config.

[msfussell]

There is a discussion here on the overlap between dynamic updating of components/config by dapr and when the K8s operator manages the rollout of a new application. We need to get options from both sides for this proposal.

[amanbha]

Adding notes from maintainers meeting:
Pros and cons of dynamic updating the components::

1. dynamic updating of components will solve a "split-brain" problem due to app scale out, failover.
2. dynamic updating of components can also lead to issues as application code might not be ready for the change.

Suggested approaches:

1. Keep the current behavior and document best practise that after applying the new component yaml, user should do upgrade of the application (or do pod restarts) to pick up updated components.
2. Do dynamic upload but give user an option to opt out as an annotation.
3. Add a dynamic upload property to each component, dapr sidecar will do "hot-reload" of the component when it receives the notification from operator only when this property is set to true.

[AaronCrawfis]

Proposal

Given the above what it we:

1. Keep current behavior as default (pods do not pickup changes to components until restarted) to address Pro/Con #1
2. Write documentation to address Approach #1
3. Add metadata to components and configuration yaml files to opt-in to hot reloading for all pods/applications to pick up this component instantly.
   
   
   
   • This will allow customers to rollout hot-patches or errors to applications without restarting entire application
   
   
4. (optional) Add ability for applications/pod specs to be annotated/flagged to always pick up components instantly upon change, no matter what above metadata says

This will allow 3 cases:

Default Behavior

1. Application running locally or kubernetes
2. New configuration applied
3. Application doesn't pick up new configuration until restarted/scaled

Apply Hot Reload to Component/Configuration

1. Application running locally or kubernetes
2. New configuration/component applied with metadata set to opt into hot reload
3. All applications pick up new configuration/component

Apply Hot Reload to Application/Pod

1. Application running locally or kubernetes with flag or annotation set to hot reload
2. New configuration/component applied (with or without metadata for hot reload, doesn't matter)
3. Application picks up new configuration/component

We could brand this as a new feature that users opt into if they want to.

[RadoslavGatev]

It seems the code for the hot reload is already in place. According to the operator logs, sidecars are successfully establishing connections to it and component changes are being successfully detected. However, for some reason sidecars are not getting notified about the modified component.

[yaron2]

@RadoslavGatev The operator side is mostly in place, however the sidecar part is currently (intentionally) ignoring updates.

We (maintainers) have spoken about this issue today and I just triaged this into the 0.12.0 milestone.