Salt: Setting `ping_interval` in salt-minion's config (version 2019.2.1) prevents it from starting

[javierbertoli]

Description of Issue

Due to errors in the the minion's function remove_periodic_callback(), when setting the parameter ping_interval in the minion configuration (ie, to check against a DDNS master, as described here), the minion fails to start and floods the logs with

[CRITICAL] Unexpected error while connecting to salt
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/minion.py", line 1027, in _connect_minion
    minion.tune_in(start=False)
  File "/usr/lib/python3/dist-packages/salt/minion.py", line 2733, in tune_in
    self.remove_periodic_callbback('ping', ping_master)
AttributeError: 'Minion' object has no attribute 'remove_periodic_callbback'

At first it seems to be caused by a typo in remove_periodic_callbback() (note the 2 bb) but when fixing it, the error changes to

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/minion.py", line 1035, in _connect_minion
    minion.tune_in(start=False)
  File "/usr/lib/python3/dist-packages/salt/minion.py", line 2741, in tune_in
    self.remove_periodic_callback('ping', ping_master)

and does not start either.

Steps to Reproduce Issue

1. Install the salt-minion v2019.2.1 packages
2. Add the following config parameters

cat > /etc/salt/minion.d/test.conf << EOF
auth_tries: 2
master_tries: -1
auth_safemode: false
ping_interval: 5
EOF

1. Run salt-minion -l debug to get the error.
2. Remove the ping_interval parameter, repeat step 3 and it will run OK.

Versions Report

Salt Version:
           Salt: 2019.2.1

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 2.7.3
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.10
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 3.7.3 (default, Apr  3 2019, 05:39:12)
   python-gnupg: Not Installed
         PyYAML: 3.13
          PyZMQ: 17.1.2
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.3.1

System Versions:
           dist: debian 10.1 
         locale: utf-8
        machine: x86_64
        release: 5.3.0-rc5-amd64
         system: Linux
        version: debian 10.1 

[cmcmarrow]

I can confirm that this happens as on Ubuntu as well.

[morsik]

Same on CentOS 7.6 with minions deployed via Salt Cloud.

[0x647262]

Master:

Salt Version:
           Salt: 2019.2.1

Dependency Versions:
           cffi: Not Installed
       cherrypy: unknown
       dateutil: 2.6.1
      docker-py: Not Installed
          gitdb: 2.0.3
      gitpython: 2.1.8
          ioflo: Not Installed
         Jinja2: 2.10
        libgit2: 0.26.0
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: 0.26.2
         Python: 3.6.8 (default, Aug 20 2019, 17:12:48)
   python-gnupg: 0.4.1
         PyYAML: 3.12
          PyZMQ: 16.0.2
           RAET: Not Installed
          smmap: 2.0.3
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.2.5

System Versions:
           dist: Ubuntu 18.04 bionic
         locale: UTF-8
        machine: x86_64
        release: 4.15.0-1048-aws
         system: Linux
        version: Ubuntu 18.04 bionic

Minion:

Salt Version:
           Salt: 2019.2.1

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 2.6.1
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.10
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 3.6.8 (default, Aug 20 2019, 17:12:48)
   python-gnupg: 0.4.1
         PyYAML: 3.12
          PyZMQ: 16.0.2
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.2.5

System Versions:
           dist: Ubuntu 18.04 bionic
         locale: UTF-8
        machine: x86_64
        release: 4.15.0-64-generic
         system: Linux
        version: Ubuntu 18.04 bionic

As mentioned in the original report, downgrading from to 2019.2.0 or removing the ping_interval configuration option resolves the issue

[sastorsl]

Same on Red Hat Enterprise Linux Server 7 / RHEL7

# salt-call -V
Salt Version:
           Salt: 2019.2.1

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 1.5
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.7.2
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: 0.31.0
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.5 (default, Jun 11 2019, 14:33:56)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 15.3.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.1.4

System Versions:
           dist: redhat 7.7 Maipo
         locale: UTF-8
        machine: x86_64
        release: 3.10.0-1062.1.1.el7.x86_64
         system: Linux
        version: Red Hat Enterprise Linux Server 7.7 Maipo

Errors in /var/log/salt/minion

2019-10-01 09:15:18,923 [salt.minion      :2313][ERROR   ][2611] This Minion is already running. Not running Minion.tune_in()
2019-10-01 09:15:18,946 [salt.minion      :2692][INFO    ][2611] Minion is ready to receive requests!
2019-10-01 09:15:18,947 [salt.minion      :1049][CRITICAL][2611] Unexpected error while connecting to 10.246.104.52
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/salt/minion.py", line 1027, in _connect_minion
    minion.tune_in(start=False)
  File "/usr/lib/python2.7/site-packages/salt/minion.py", line 2733, in tune_in
    self.remove_periodic_callbback('ping', ping_master)
AttributeError: 'Minion' object has no attribute 'remove_periodic_callbback'

NOT having this problem with salt-2019.2.0 on an identical server with redhat 7.6 Maipo running 3.10.0-957.12.2.el7.x86_64.

[hedrickbt]

I have created a support ticket for this:
https://help.saltstack.com/hc/en-us/requests/4183

[Xopherus]

Has there been a version released with this fix? If not, when can we expect that?

[hedrickbt]

@Xopherus ,

The following pull request is being tested. https://github.com/saltstack/salt/pull/54777

The plan is for the fix to be included in 2019.2.2. The last update I have from support is 10 days ago. I asked for an update today.

A notice has been put out to let people know to avoid going to 2019.2.1 at https://repo.saltstack.com/

[hedrickbt]

Be on the look out for a possible release mid-next-week.

[cachedout]

Now that https://github.com/saltstack/salt/pull/54777 is merged, shouldn't this be closed?

[javierbertoli]

Applied the patch when I submitted #54777 and it has been working for me since then.

[hedrickbt]

2019.2.2 has been released with the patch.