Salt: master logs full of WARNINGS due to zmq / tornado ioloop.py

Created on 31 Oct 2018 · 22Comments · Source: saltstack/salt

Description of Issue/Question

My master logs are full of following messages:

2018-10-31 10:15:03,887 [py.warnings      :156 ][WARNING ][67202] /usr/lib/python2.7/dist-packages/zmq/eventloop/ioloop.py:211: RuntimeWarning: IOLoop.current expected instance of <class 'zmq.eventloop.ioloop.ZMQIOLoop'>, got <tornado.platform.epoll.EPollIOLoop object at 0x7fde0b1c3fd0>
  ioloop.IOLoop.instance() is IOLoop.instance(), "tornado IOLoop already initialized"

I think I get these messages each time I restart my master (cannot restart right now).

I have a multi-master configuration and I am using reactors and orchestrators + a gitfs backend.

Versions Report

root@cisaltmaster:~/un-peu-de-sel/scripts# salt --versions-report
Salt Version:
           Salt: 2018.3.2

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 2.5.3
      docker-py: Not Installed
          gitdb: 2.0.0
      gitpython: 2.1.1
          ioflo: Not Installed
         Jinja2: 2.9.4
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: 0.24.0
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.8
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.13 (default, Sep 26 2018, 18:42:22)
   python-gnupg: Not Installed
         PyYAML: 3.12
          PyZMQ: 16.0.2
           RAET: Not Installed
          smmap: 2.0.1
        timelib: Not Installed
        Tornado: 4.4.3
            ZMQ: 4.2.1

System Versions:
           dist: debian 9.0
         locale: ANSI_X3.4-1968
        machine: x86_64
        release: 4.4.98-6-pve
         system: Linux
        version: debian 9.0

Aluminium Bug CS-R2 CS-S3 Confirmed Core Transport ZD ZMQ phase-plan status-to-do

Source

remyd1

👍2

All 22 comments

@DmitryKuzmenko Thoughts on this? This might be a duplicate of an existing issue.

garethgreenaway on 1 Nov 2018

@garethgreenaway hm... not sure. This could mean we have to change the way we're getting the current ioloop somewhere else from tornado.ioloop.IOLoop.current() to salt.utils.zeromq.ZMQDefaultLoop.current(). But first we have to understand where. =)

For now I can put it to my backlog.

DmitryKuzmenko on 7 Nov 2018

I have exactly the same issue:

2018-11-14 09:04:43,436 [py.warnings      :156 ][WARNING ][10755] /usr/lib/python2.7/dist-packages/zmq/eventloop/ioloop.py:211: RuntimeWarning: IOLoop.current expected instance of <class 'zmq.eventloop.ioloop.ZMQIOLoop'>, got <tornado.platform.epoll.EPollIOLoop object at 0x7f2d7ab31a50>
  ioloop.IOLoop.instance() is IOLoop.instance(), "tornado IOLoop already initialized"

Salt Version:
           Salt: 2018.3.2

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 2.6.1
      docker-py: Not Installed
          gitdb: 2.0.3
      gitpython: 2.1.8
          ioflo: Not Installed
         Jinja2: 2.10
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: 0.27.0
           Mako: 1.0.7
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.15rc1 (default, Apr 15 2018, 21:51:34)
   python-gnupg: 0.4.1
         PyYAML: 3.12
          PyZMQ: 16.0.2
           RAET: Not Installed
          smmap: 2.0.3
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.2.5

System Versions:
           dist: Ubuntu 18.04 bionic
         locale: UTF-8
        machine: x86_64
        release: 4.15.0-1025-aws
         system: Linux
        version: Ubuntu 18.04 bionic

I'm using transport tcp if this change something.

dawidmalina on 14 Nov 2018

👍1

Looks like problem with ZMQ is a bit bigger: https://github.com/saltstack/salt/issues/46905

dawidmalina on 29 Nov 2018

Same problem here (after upgrade from Ubuntu 16.04 to 18.04):

Salt Version:
           Salt: 2018.3.3

Dependency Versions:
           cffi: Not Installed
       cherrypy: unknown
       dateutil: 2.6.1
      docker-py: Not Installed
          gitdb: 2.0.3
      gitpython: 2.1.8
          ioflo: Not Installed
         Jinja2: 2.10
        libgit2: 0.26.0
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: 1.3.10
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: 0.26.2
         Python: 3.6.7 (default, Oct 22 2018, 11:32:17)
   python-gnupg: 0.4.1
         PyYAML: 3.12
          PyZMQ: 16.0.2
           RAET: Not Installed
          smmap: 2.0.3
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.2.5

System Versions:
           dist: Ubuntu 18.04 bionic
         locale: UTF-8
        machine: x86_64
        release: 4.15.0-42-generic
         system: Linux
        version: Ubuntu 18.04 bionic

2018-12-13 07:38:06,701 [py.warnings      :99  ][WARNING ][14815] /usr/lib/python3/dist-packages/zmq/eventloop/ioloop.py:211: RuntimeWarning: IOLoop.current expected instance of <class 'zmq.eventloop.ioloop.ZMQIOLoop'>, got <tornado.platform.epoll.EPollIOLoop object at 0x7fdd4ed81710>
  ioloop.IOLoop.instance() is IOLoop.instance(), "tornado IOLoop already initialized"

2018-12-13 07:38:06,758 [py.warnings      :99  ][WARNING ][14811] /usr/lib/python3/dist-packages/zmq/eventloop/ioloop.py:211: RuntimeWarning: IOLoop.current expected instance of <class 'zmq.eventloop.ioloop.ZMQIOLoop'>, got <tornado.platform.epoll.EPollIOLoop object at 0x7fdd4ed81710>
  ioloop.IOLoop.instance() is IOLoop.instance(), "tornado IOLoop already initialized"

2018-12-13 07:38:06,816 [py.warnings      :99  ][WARNING ][14803] /usr/lib/python3/dist-packages/zmq/eventloop/ioloop.py:211: RuntimeWarning: IOLoop.current expected instance of <class 'zmq.eventloop.ioloop.ZMQIOLoop'>, got <tornado.platform.epoll.EPollIOLoop object at 0x7fdd4ed81710>
  ioloop.IOLoop.instance() is IOLoop.instance(), "tornado IOLoop already initialized"

2018-12-13 07:38:06,919 [py.warnings      :99  ][WARNING ][14812] /usr/lib/python3/dist-packages/zmq/eventloop/ioloop.py:211: RuntimeWarning: IOLoop.current expected instance of <class 'zmq.eventloop.ioloop.ZMQIOLoop'>, got <tornado.platform.epoll.EPollIOLoop object at 0x7fdd4ed81710>
  ioloop.IOLoop.instance() is IOLoop.instance(), "tornado IOLoop already initialized"

Using default transport.

akrus on 13 Dec 2018

Was there any update / progress on this one? I've got a fresh 18.04.3 Py3 Salt Master box with the same issues.

TheVakman on 5 Nov 2019

👍1

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

stale[bot] on 7 Jan 2020

This issue is still a problem

skorpy2009 on 14 Jan 2020

Agreed, this is still a problem and should be re-opened.

TheVakman on 10 Feb 2020

Thank you for updating this issue. It is no longer marked as stale.

stale[bot] on 11 Feb 2020

ZD-4623

doesitblend on 11 Feb 2020

👍1

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

stale[bot] on 12 Mar 2020

It's still the issue for salt 3000

Chupaka on 12 Mar 2020

😕2

Thank you for updating this issue. It is no longer marked as stale.

stale[bot] on 12 Mar 2020

@Chupaka could you please provide the full output of your salt --versions-report? And any details about the configuration and reproduction steps please.

DmitryKuzmenko on 14 Apr 2020

This is not first time I'm trying to reproduce this. I've tried different libraries and salt versions packs I have in my arsenal. Didn't helped.

DmitryKuzmenko on 14 Apr 2020

HI,

I got this issue too with salt version 2018.3.5 and 2019.2.4 in a python script that uses the salt-module to run some states.

@DmitryKuzmenko After some debugging and testing I could reproduce this issue by running the following simple script:

import salt.config, salt.wheel, salt.client
wheel=salt.wheel.WheelClient(salt.config.master_config('/etc/salt/master'))
wheel.cmd('key.list', ['accepted'])
print(salt.client.Caller().cmd('test.version'))

When wheel.cmd is removed the script runs successfully and prints the version but when the wheel-call is initiated before salt.client the version command end with the stacktrace due to the IOLoop issue:

/usr/lib/python2.7/dist-packages/salt/minion.py:804: RuntimeWarning: IOLoop.current expected instance of <class 'zmq.eventloop.ioloop.ZMQIOLoop'>, got <tornado.platform.epoll.EPollIOLoop object at 0x7f48e2531410>
  io_loop = ZMQDefaultLoop.current()
Traceback (most recent call last):
  File "bla2", line 5, in <module>
    print(salt.client.Caller().cmd('test.version'))
  File "/usr/lib/python2.7/dist-packages/salt/client/__init__.py", line 2114, in __init__
    self.sminion = salt.minion.SMinion(self.opts)
  File "/usr/lib/python2.7/dist-packages/salt/minion.py", line 806, in __init__
    lambda: self.eval_master(self.opts, failed=True)
  File "/usr/lib/python2.7/dist-packages/tornado/ioloop.py", line 453, in run_sync
    self.start()
  File "/usr/lib/python2.7/dist-packages/tornado/ioloop.py", line 802, in start
    ncallbacks = len(self._callbacks)
TypeError: object of type 'NoneType' has no len()

Interestingly when you change the code like the following (force Caller() intialization before wheel) it runs as intended:

import salt.config, salt.wheel, salt.client

salt.client.Caller()
wheel = salt.wheel.WheelClient(salt.config.master_config('/etc/salt/master'))
wheel.cmd('key.list', ['accepted'])
print(salt.client.Caller().cmd('test.version'))

I hope this helps in identifying the real issue.
I've changed my code to initialize Caller() first for now.

Regards,
Andreas

root360-AndreasUlm on 13 May 2020

@root360-AndreasUlm thank you for analyze and example. I'm getting the same stack trace but without the IOLoop warning. Could you please also provide your salt --versions-info? Probably it depends on ZMQ/Tornado version.

DmitryKuzmenko on 13 May 2020

@DmitryKuzmenko

# salt --versions-report
Salt Version:
           Salt: 2019.2.4

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 2.8.0
      docker-py: Not Installed
          gitdb: 2.0.3
      gitpython: 2.1.8
          ioflo: Not Installed
         Jinja2: 2.10
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: 1.0.7
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.17 (default, Apr 15 2020, 17:20:14)
   python-gnupg: 0.4.1
         PyYAML: 3.12
          PyZMQ: 16.0.2
           RAET: Not Installed
          smmap: 2.0.3
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.2.5

System Versions:
           dist: Ubuntu 18.04 bionic
         locale: UTF-8
        machine: x86_64
        release: 4.15.0-1041-aws
         system: Linux
        version: Ubuntu 18.04 bionic

The package is installed from official saltstack repositories (cached via our s3 bucket):

# apt policy salt-master 
salt-master:
  Installed: 2019.2.4+ds-1
  Candidate: 2019.2.4+ds-1
  Version table:
 *** 2019.2.4+ds-1 1001
        500 http://repo-ubuntu-eu-central-1-b.root360.cloud/repo.saltstack.com/apt/ubuntu/18.04/amd64/2019.2 bionic/main amd64 Packages
        100 /var/lib/dpkg/status
     2017.7.4+dfsg1-1ubuntu18.04.1 500
        500 http://repo-ubuntu-eu-central-1-b.root360.cloud/archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages
     2017.7.4+dfsg1-1 500
        500 http://repo-ubuntu-eu-central-1-b.root360.cloud/archive.ubuntu.com/ubuntu bionic/universe amd64 Packages

root360-AndreasUlm on 13 May 2020

👍1

@remyd1 I've reproduced it with pyzmq==16.0.2. This message disappears after upgrading to pyzmq>=17.0.0. The same is for Py3.7 on 3000.1 and master branch.
Thank you for details!

DmitryKuzmenko on 13 May 2020

Unfortunately, Ubuntu 18.04 has only python3-zmq 16.0.2-2build2 in its repos.

Chupaka on 13 May 2020

@Chupaka yes, I see this. I've put this to the pipeline, we'll discuss what's the better way to fix it.

DmitryKuzmenko on 13 May 2020

Was this page helpful?

0 / 5 - 0 ratings