Compose: Restart/Reconnect containers connected via 'network_mode: service' automatically when main service is restarted

Created on 1 Apr 2019  路  7Comments  路  Source: docker/compose

Is your feature request related to a problem? Please describe.
When running the following docker-compose.yml:

version: "3.7"

services:

  mother:
    image: alpine
    command: "sleep 999999"
    restart: always

  child:
    image: alpine
    command: "sleep 888888"
    network_mode: "service:mother"

If the mother container is restarted for any reason (crash / manual restart), the child container loses its network forever.

$ docker-compose restart mother
$ docker-compose exec child ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever

The child container is fully disconnected from the world. It will not reattach to mother's network. It will be unable to communicate with other containers and the internet. Does it make any sense at all to continue running the child container in this state?

Describe the solution you'd like
Whenever a service is restarted which has other services connected to it via 'network_mode: service', then reconnect those other services or restart them if reconnecting is technically unfeasible.

Describe alternatives you've considered
A workaround using a healthcheck and autoheal is described here: https://github.com/docker/compose/issues/6329#issuecomment-452874796

In discussions of other issues related to 'network_mode: service' it is suggested to use a user defined network instead. But as far as i know there are container compositions which require 'network_mode: service', for example when putting multiple containers behind a vpn. Please correct me if I'm wrong.

kinquestion statu0-triage

Most helpful comment

@ndeloof

when you use "service" network_mode (i.e. sharing network namespace between containers), loosing connectivity on restart is really the expected behaviour.

Intuitively i would not call this expected behaviour. I give you some real world examples: In my home network if my network connection is dependent on a cable being plugged into my machine and i plug this cable out and then back in, I expect my machine to reconnect. Or if my network connection is dependent on some other machine, i.e. my router, and i restart that machine, i expect my network to be back up again after restarting that machine

Comparable to using "host" network and getting the node shut down and service restarted elsewhere on cluster.

I agree to this comparison as it demonstrates how useless such kind of behaviour is. This is why you would never configure your cluster in a way to run a service without a vital resource being present. And therefore i think it would be a good idea to also stop doing that in docker compose. When using "service" network_mode, the services are highly coupled, so that one cannot live without the other one. In a mother child configuration the child is strongly dependent on mother and it never makes sense to have the child running without a mother. There is no single good reason why you would not also stop the child if mother is gone / or cannot reunite with the child after being restarted.

For this purpose, use your compose file to define an explicit network connecting services together.

As i already stated in the original issue, there are some container configurations where creating an explicit network is not sufficient and instead you have to share the network adapter itself. For example forcing any kind of container to connect via a VPN container. Therefore your suggestion doesn't solve the problem.

All 7 comments

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

AFAIK this is still an issue.

This issue has been automatically marked as not stale anymore due to the recent activity.

when you use "service" network_mode (i.e. sharing network namespace between containers), loosing connectivity on restart is really the expected behaviour. Comparable to using "host" network and getting the node shut down and service restarted elsewhere on cluster.

Such usage only makes sense for highly coupled containers (typically: containers in a kubernetes Pod) but not for services communicating together in a reliable way. Automatically restarting the dependent service would help you hide the networking constraints of your architecture but this is just cheating, better get your architecture to embrace the risk for dependent service being restarted or sacled up/down. For this purpose, use your compose file to define an explicit network connecting services together.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@ndeloof

when you use "service" network_mode (i.e. sharing network namespace between containers), loosing connectivity on restart is really the expected behaviour.

Intuitively i would not call this expected behaviour. I give you some real world examples: In my home network if my network connection is dependent on a cable being plugged into my machine and i plug this cable out and then back in, I expect my machine to reconnect. Or if my network connection is dependent on some other machine, i.e. my router, and i restart that machine, i expect my network to be back up again after restarting that machine

Comparable to using "host" network and getting the node shut down and service restarted elsewhere on cluster.

I agree to this comparison as it demonstrates how useless such kind of behaviour is. This is why you would never configure your cluster in a way to run a service without a vital resource being present. And therefore i think it would be a good idea to also stop doing that in docker compose. When using "service" network_mode, the services are highly coupled, so that one cannot live without the other one. In a mother child configuration the child is strongly dependent on mother and it never makes sense to have the child running without a mother. There is no single good reason why you would not also stop the child if mother is gone / or cannot reunite with the child after being restarted.

For this purpose, use your compose file to define an explicit network connecting services together.

As i already stated in the original issue, there are some container configurations where creating an explicit network is not sufficient and instead you have to share the network adapter itself. For example forcing any kind of container to connect via a VPN container. Therefore your suggestion doesn't solve the problem.

This issue has been automatically marked as not stale anymore due to the recent activity.

Was this page helpful?
0 / 5 - 0 ratings