Dapr: Sidecar returns 500 when underlying component is restarted

[christian-be]

In what area(s)?

/area runtime

What version of Dapr?

0.8.0

Expected Behavior

A service in kubernetes minikube cluster publishes messages via the injected sidecar to a message-broker (rabbit-mq). The broker is exposed as a service in the cluster.

Actual Behavior

Everything works as expected, when the broker is running when the pod containing the publisher is starting up. Messages get published to the broker without any issues. The problem appears, when the broker is restarted. Until the pod containing the publisher is restarted as well, the dapr sidecar returns status 500 and this error message:

{"errorCode":"ERR_PUBSUB_PUBLISH_MESSAGE","message":"Exception (504) Reason: "channel/connection is not open""}

Is this expected behaviour or should the dapr sidecar attempt to reconnect, when the broker restarts or is offline for a short time?

Release Note

RELEASE NOTE: IMPROVED RabbitMQ's PubSub implementation to automatically reconnect on 'channel/connection is not open'

[youngbupark]

Thanks for reporting this issue. I think component implementation needs to have a logic to check the connectivity and then reconnect the broker.

@yaron2 @amanbha @orizohar @msfussell / I think this is good issue we must review carefully. Typically, Redis client reconnects redis server if it is disconnected. I assume that not all client implementations would retry to connect to the server. We may need to review all components' behaviors.

[artursouza]

Issue is component specific. This one is in RabbitMQ component.