Prefect: Reconnect Docker Agent on daemon restart

[joshmeek]

Description

When a Docker Agent is running and connected to a docker daemon it will raise a broken pipe error when the daemon restarts and a new flow run is found.

Expected Behavior

Docker Agent either gracefully reconnects to the daemon or exits when the pipe is broken. I'm leaning on the side of reconnecting as all it takes is reinstantiating the Docker low level API client

[cicdw]

Yea I like the idea of pinging with each query and then attempting to reconnect if a ping fails; if the reconnect fails I think the Agent should actually exit and / or "deregister" itself so that it will be clear this Agent is not functional.

[wagoodman]

I think that deregistration would be a good next step, for the meantime the PR only implements a shutdown to prevent pulling work that will not be done.

[joshmeek]

@zdhughes Is currently working on the deregistration as part of another PR

[zdhughes]

To be clear, that's related to deregistering from Prefect Cloud. But thank you-- forgot to push the Core-side PR, and will do so now.