Salt: Napalm proxy-minion throwing exception while configured to work with multi master. [please reopen]

[omegarus]

Description of Issue/Question

I'am trying to run napalm proxy-minion with multi master configuration with no luck.
Proxy minion is trowing this exception and starting initialization again:

[ERROR   ] Cannot execute "get_facts" on [unspecified hostname] as . Reason: not connected!
[ERROR   ] Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/salt/utils/napalm.py", line 158, in call
    raise Exception('not connected')
Exception: not connected

[INFO    ] nxos proxy __virtual__() called...
[INFO    ] ssh_sample proxy __virtual__() called...
[INFO    ] Connected (version 2.0, client OpenSSH_7.2)
[INFO    ] Authentication (password) successful!
[INFO    ] initialized: session-id=34760 | server_capabilities=<dictionary-keyiterator object at 0x7f6203829aa0>
[INFO    ] Although 'dmidecode' was found in path, the current user cannot execute it. Grains output might not be accurate.
[INFO    ] Requesting 'ExecuteRpc'
[INFO    ] Requesting 'ExecuteRpc'
[INFO    ] Requesting 'ExecuteRpc'
[INFO    ] Requesting 'ExecuteRpc'
[INFO    ] Requesting 'ExecuteRpc'
[INFO    ] Requesting 'ExecuteRpc'
[INFO    ] Requesting 'ExecuteRpc'
[INFO    ] Requesting 'ExecuteRpc'
[INFO    ] Although 'dmidecode' was found in path, the current user cannot execute it. Grains output might not be accurate.
[INFO    ] Updating job settings for scheduled job: __mine_interval
[INFO    ] Running scheduled job: __proxy_keepalive (runtime missed by 18 seconds)
[ERROR   ] Exception argument of type 'bool' is not iterable occurred in scheduled job
[CRITICAL] Unexpected error while connecting to 192.168.1.2
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/salt/minion.py", line 997, in _connect_minion
    yield minion.connect_master(failed=failed)
  File "/usr/lib/python2.7/dist-packages/tornado/gen.py", line 870, in run
    value = future.result()
  File "/usr/lib/python2.7/dist-packages/tornado/concurrent.py", line 215, in result
    raise_exc_info(self._exc_info)
  File "/usr/lib/python2.7/dist-packages/tornado/gen.py", line 876, in run
    yielded = self.gen.throw(*exc_info)
  File "/usr/lib/python2.7/dist-packages/salt/minion.py", line 1188, in connect_master
    yield self._post_master_init(master)
  File "/usr/lib/python2.7/dist-packages/tornado/gen.py", line 870, in run
    value = future.result()
  File "/usr/lib/python2.7/dist-packages/tornado/concurrent.py", line 215, in result
    raise_exc_info(self._exc_info)
  File "/usr/lib/python2.7/dist-packages/tornado/gen.py", line 879, in run
    yielded = self.gen.send(value)
  File "/usr/lib/python2.7/dist-packages/salt/minion.py", line 3628, in _post_master_init
    }, persist=True)
  File "/usr/lib/python2.7/dist-packages/salt/utils/schedule.py", line 333, in add_job
    self.persist()
  File "/usr/lib/python2.7/dist-packages/salt/utils/schedule.py", line 222, in persist
    remove_hidden=True)
  File "/usr/lib/python2.7/dist-packages/salt/utils/schedule.py", line 158, in _get_schedule
    for item in _schedule[job]:
TypeError: 'bool' object is not iterable

If proxy-minion is configured to use only one of the masters(first or second but not both) everything is working good.

Setup

proxy comnfig file:

master:
  - 192.168.1.1
  - 192.168.1.2
saltenv: network/base
pillarenv_from_saltenv: True
#multiprocessing: False

pillar for this proxy:

proxy:
    ----------
    proxytype:
        napalm
    driver:
        junos
    host:
        192.168.1.254
    username:
        superuser
    password:
        megapassword
    optional_args:
        ----------
        port:
            22

Steps to Reproduce Issue

Start napalm proxy minion with multi master in configuration:

master:
  - 192.168.1.1
  - 192.168.1.2

Versions Report

salt-proxy --version
salt-proxy 2018.3.2 (Oxygen)

root@salt-master-1:/tmp# salt --versions-report
Salt Version:
           Salt: 2018.3.2

Dependency Versions:
           cffi: 1.5.2
       cherrypy: unknown
       dateutil: 2.4.2
      docker-py: Not Installed
          gitdb: 0.6.4
      gitpython: 1.0.1
          ioflo: Not Installed
         Jinja2: 2.8
        libgit2: 0.24.0
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: 1.0.3
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: Not Installed
      pycparser: 2.14
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: 0.24.0
         Python: 2.7.12 (default, Dec  4 2017, 14:50:18)
   python-gnupg: 0.3.8
         PyYAML: 3.11
          PyZMQ: 15.2.0
           RAET: Not Installed
          smmap: 0.9.0
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.1.4

System Versions:
           dist: Ubuntu 16.04 xenial
         locale: UTF-8
        machine: x86_64
        release: 4.4.0-116-generic
         system: Linux
        version: Ubuntu 16.04 xenial

[Ch3LL]

ping @saltstack/team-network any ideas here?

[omegarus]

Any suggestions ? :)

[rallytime]

@mirceaulinic ?

[mirceaulinic]

No idea, I've never use multi master myself in this context.
The error looks similar to the one from https://github.com/saltstack/salt/issues/47695 so try upgrading to 2018.3.3 to check if that's true and fixed.

[omegarus]

@mirceaulinic Then 2018.3.3 will be released ?

[rallytime]

We are working on getting 2018.3.3 released right now, just working out some final bugs.

[omegarus]

I will check @mirceaulinic suggestion after 2018.3.3 is released.
Thanks.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

[omegarus]

Hi Sorry for delay.
I tried 2018.3.3 and getting same error:

[INFO    ] Updating job settings for scheduled job: __mine_interval
[INFO    ] Running scheduled job: __proxy_keepalive (runtime missed by 11 seconds)
[CRITICAL] Unexpected error while connecting to 192.168.1.2
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/salt/minion.py", line 997, in _connect_minion
    yield minion.connect_master(failed=failed)
  File "/usr/local/lib/python2.7/dist-packages/tornado/gen.py", line 870, in run
    value = future.result()
  File "/usr/local/lib/python2.7/dist-packages/tornado/concurrent.py", line 215, in result
    raise_exc_info(self._exc_info)
  File "/usr/local/lib/python2.7/dist-packages/tornado/gen.py", line 876, in run
    yielded = self.gen.throw(*exc_info)
  File "/usr/lib/python2.7/dist-packages/salt/minion.py", line 1189, in connect_master
    yield self._post_master_init(master)
  File "/usr/local/lib/python2.7/dist-packages/tornado/gen.py", line 870, in run
    value = future.result()
  File "/usr/local/lib/python2.7/dist-packages/tornado/concurrent.py", line 215, in result
    raise_exc_info(self._exc_info)
  File "/usr/local/lib/python2.7/dist-packages/tornado/gen.py", line 879, in run
    yielded = self.gen.send(value)
  File "/usr/lib/python2.7/dist-packages/salt/minion.py", line 3631, in _post_master_init
    }, persist=True)
  File "/usr/lib/python2.7/dist-packages/salt/utils/schedule.py", line 333, in add_job
    self.persist()
  File "/usr/lib/python2.7/dist-packages/salt/utils/schedule.py", line 222, in persist
    remove_hidden=True)
  File "/usr/lib/python2.7/dist-packages/salt/utils/schedule.py", line 158, in _get_schedule
    for item in _schedule[job]:
TypeError: 'bool' object is not iterable

Proxy minion only connects to first master and can receive tasks from it.

[omegarus]

@rallytime Can you please reopen the issue ?
Thanks.

[omegarus]

@Ch3LL Can you reopen this issue please ?
It still happens in 2018.3.3 version.
Thanks.

[stale[bot]]

Thank you for updating this issue. It is no longer marked as stale.

[mattlan]

I'm observing the same behavior using 2019.2.0.