Salt: docker_container.running disconnects container from the network if hostname: and network_mode: used simultaneously.

Created on 2 Aug 2019 · 16Comments · Source: saltstack/salt

Description of Issue

docker_container.running disconnects container from the network after second state.apply if hostname: container-hostname and network_mode: custom-network used simultaneously in state.

Setup

salt/test-container/init.sls:

test-container:
  docker_container.running:
    - image: ubuntu:18.04
    - hostname: any-name-1
    - network_mode: custom-network
    - command: infinity
    - entrypoint: sleep
    - restart_policy: always

custom-network is pre created by other state.

Steps to Reproduce Issue

Just salt 'minion-id' state.apply test-container twice on same minion.
On first salt will create test-container on minion, on second run it will disconnect test-container from custom-network.
First run output:

----------
          ID: test-container
    Function: docker_container.running
      Result: True
     Comment: Created container 'test-container'
     Started: 15:30:31.079152
    Duration: 1897.652 ms
     Changes:
              ----------
              container_id:
                  ----------
                  added:
                      09d2a02bc845f36347b6a91f36a990beca2366f3bc6496bf743d7c59e5d5b6fd
              state:
                  ----------
                  new:
                      running
                  old:
                      None

Second run output:

----------
          ID: test-container
    Function: docker_container.running
      Result: True
     Comment: Container 'test-container' is already configured as specified. Disconnected from network 'custom-network'.
     Started: 15:31:15.897764
    Duration: 1521.696 ms
     Changes:
              ----------
              container:
                  ----------
                  Networks:
                      ----------
                      custom-network:
                          ----------
                          Aliases:
                              ----------
                              new:
                                  None
                              old:
                                  - 09d2a02bc845

Versions Report

Master:

Salt Version:
           Salt: 2018.3.2

Dependency Versions:
           cffi: 1.5.2
       cherrypy: unknown
       dateutil: 2.8.0
      docker-py: Not Installed
          gitdb: 0.6.4
      gitpython: 1.0.1
          ioflo: Not Installed
         Jinja2: 2.8
        libgit2: 0.24.0
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: 1.0.3
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: Not Installed
      pycparser: 2.14
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: 0.24.0
         Python: 2.7.12 (default, Dec  4 2017, 14:50:18)
   python-gnupg: 0.3.8
         PyYAML: 3.11
          PyZMQ: 15.2.0
           RAET: Not Installed
          smmap: 0.9.0
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.1.4

System Versions:
           dist: Ubuntu 16.04 xenial
         locale: UTF-8
        machine: x86_64
        release: 4.4.0-116-generic
         system: Linux
        version: Ubuntu 16.04 xenial

Minion:

Salt Version:
           Salt: 2018.3.2

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 2.6.1
      docker-py: 3.7.2
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.10
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: 1.0.7
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.15rc1 (default, Nov 12 2018, 14:31:15)
   python-gnupg: 0.4.1
         PyYAML: 3.12
          PyZMQ: 16.0.2
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.2.5

System Versions:
           dist: Ubuntu 18.04 bionic
         locale: ISO-8859-1
        machine: x86_64
        release: 4.18.0-20-generic
         system: Linux
        version: Ubuntu 18.04 bionic

Bug P2 severity-medium

Source

omegarus

All 16 comments

@omegarus Thanks for the report. I can confirm this is still happening in the 2019.2.x branch as well.

garethgreenaway on 2 Aug 2019

@omegarus I need to learn to read comments. Anyways, investigating this now. :+1:

xeacott on 12 Aug 2019

@terminalmage Just seeing if there is anything that jumps out at you. It seems like there are many attempts to reconnect the network in any disconnect event...

xeacott on 12 Aug 2019

@omegarus Would you be able to provide the custom-network state?

xeacott on 13 Aug 2019

This looks like it may be a case of incorrect documentation. The docs for the network_mode option only differ from the docker-py docs in that there is one extra option for this argument:

    - Any name that identifies an existing network that might be created
      with ``docker.network_present``.

But given that Salt never does anything with network_mode other than passing it directly through to docker-py, it seems like this documentation was from earlier in the process of overhauling Salt's docker networking support and just wasn't cleaned up.

@omegarus My guess is that you want the container to be connected to the custom-network docker network. If that is the case, you likely want to use the networks config option (as documented here).

terminalmage on 13 Aug 2019

@terminalmage Thank you for a detailed response (and for exhaustive docker documentation). @omegarus see if that can point you in the right direction for the state file. If not, then I will get back to investigating this.

xeacott on 13 Aug 2019

Hey @omegarus could you let me know if this did work for you? I'll close the issue in a week or so if it did.

xeacott on 9 Sep 2019

@xeacott @terminalmage Sorry for late response.
If I remember right, when I used networks config option container ends up connected to two networks default and custom-network. Also I saw a bug about this behaviour somewhere, but i can't find it right now.
If i use network_mode container will be connected only to network specified in network_mode config option.

omegarus on 10 Sep 2019

Found a bug: #46708

omegarus on 10 Sep 2019

@omegarus So if I'm interpreting this correctly...

Previously, the behavior that terminalmage addressed in the issue you referenced above is that when using the networks param, the docker container created would persist in having the default network attached _and_ the custom network described by the networks param. To mitigate this juggling of multiple networks, the PR put in solved this and ensured that the only network the container was connected to was specified by networks.

That was merged, but are you experiencing the incorrect behavior now with using the custom_networks param? Or, is the behavior addressed by the PR not in 2018.3.2?

Just trying to recall exactly what needs to be addressed in this ticket. 😃

xeacott on 14 Sep 2019

@xeacott I'am experiencing same behaviour as described in #46708 issue, but in version 2018.3.2. Looks like behaviour addressed by the PR is not in 2018.3.2. Also I checked docker_container.py file and changes introduced in PR is not there. Is it expected ?
Work around for this(I tested it today) is to set networks and network_mode params simultaneously, and it works. But if you try to set hostname parameter you will get behaviour described in my issue.

omegarus on 16 Sep 2019

Gotcha! Thanks for the update, and thanks for letting me know the code isn't there in that branch. I haven't poked around with this specific issue since we commented last, but with the updated info that should help a lot!

xeacott on 16 Sep 2019

👍1

Is this fixed soon (2019.2.2 ?)

Thank you

dynek on 15 Oct 2019

👀2

It should also be noted that watch_action: SIGHUP no longer works when hostname and network_mode are used simultaneously.

supfors on 16 Dec 2019

any news on this issue? I manage a lot of containers with this and every time I run a state.apply the state triggers a change (with no change at all) and therefore triggers a lot of other states...