Salt: spurious warnings from event.wait when event is not fired

Created on 10 May 2015 · 25Comments · Source: saltstack/salt

The following works dandily. Various parts of the state tree install listen bits.

HTTP_ipt_update_notification:
  event.wait:
    - name: resin/iptables/seed-02.mysql.proxymesh.020.lab.rackspace.ord/HTTP/update
    - data:
      access:
        - 192.237.167.224
        - 10.208.136.119
      services:
        - sshd:
          address: 0.0.0.0
          port: 22
          proto: tcp
        - nginx:
          address: 0.0.0.0
          port: 443
          proto: tcp

When a change triggers mod_watch:

----------
          ID: HTTP_ipt_update_notification
    Function: event.wait
        Name: resin/iptables/seed-02.mysql.proxymesh.020.lab.rackspace.ord/HTTP/update
      Result: True
     Comment: Event fired
     Started: 01:23:40.598193
    Duration: 32.448 ms
     Changes:
              ----------
              data:
                  None
              tag:
                  resin/iptables/seed-02.mysql.proxymesh.020.lab.rackspace.ord/HTTP/update

And the event:
(are the accompanying reqs bits supposed to be tagging along?)

Tag: resin/iptables/seed-02.mysql.proxymesh.020.lab.rackspace.ord/HTTPS/update
Data:
{ 'cmd': '_minion_event',
  'data': { '__reqs__': { 'onchanges': [],
                          'onfail': [],
                          'prereq': [],
                          'prerequired': [],
                          'require': [],
                          'watch': [ { '__env__': 'base',
                                       '__id__': 'HTTPS_iptables_tcp_ipv6',
                                       '__sls__': 'services.https',
                                       'chain': 'HTTPS',
                                       'comment': 'https - saltstack 2015.5.0',
                                       'ctstate': 'NEW',
                                       'family': 'ipv6',
                                       'fun': 'insert',
                                       'jump': 'ACCEPT',
                                       'listen_in': [...],
                                       'match': [...],
                                       'name': 'HTTPS_iptables_tcp_ipv6',
                                       'order': 10051,
                                       'position': 1,
                                       'proto': 'tcp',
                                       'require': [...],
                                       'require_in': [...],
                                       'source': '::1',
                                       'state': 'iptables',
                                       'table': 'filter',
                                       'watch_in': [...]},
                                     { '__env__': 'base',
                                       '__id__': 'HTTPS_iptables_tcp_ipv4',
                                      [lots more of this],
                                     }],
            'access': ['192.237.167.224', '10.208.136.119'],
            'services': [ { 'address': '0.0.0.0',
                            'port': 22,
                            'proto': 'tcp',
                            'sshd': None},
                          { 'address': '0.0.0.0',
                            'nginx': None,
                            'port': 443,
                            'proto': 'tcp'}],
            'sfun': 'wait'},
  'id': 'seed-02.mysql.proxymesh.020.lab.rackspace.ord',
  'pretag': None,

Subsequent runs of this state tree elicit the following.

----------
          ID: HTTP_ipt_update_notification
    Function: event.wait
        Name: resin/iptables/seed-02.mysql.proxymesh.020.lab.rackspace.ord/HTTP/update
      Result: True
     Comment:
     Started: 01:21:27.775560
    Duration: 1.714 ms
     Changes:
    Warnings: 'access', 'services' and 'data' are invalid keyword arguments for
              'event.wait'. If you were trying to pass additional data to be
              used in a template context, please populate 'context' with 'key:
              value' pairs. Your approach will work until Salt Carbon is out.
              Please update your state files.

This does not occur with event.send.

Aluminium Bug Confirmed Core phase-execute severity-medium status-to-do

Source

gladiatr72

👍4

Most helpful comment

I got the same issue, with Salt 2016.11

danilochilene on 6 Dec 2016

👍3

All 25 comments

@gladiatr72, what is the output of salt --versions-report?

jfindlay on 11 May 2015

salt-master (cent7):

           Salt: 2015.5.0
         Python: 2.7.5 (default, Jun 17 2014, 18:11:42)
         Jinja2: 2.7.2
       M2Crypto: 0.21.1
 msgpack-python: 0.4.6
   msgpack-pure: Not Installed
       pycrypto: 2.6.1
        libnacl: 1.4.2
         PyYAML: 3.10
          ioflo: 1.2.1
          PyZMQ: 14.3.1
           RAET: 0.6.3
            ZMQ: 3.2.5
           Mako: Not Installed

(ubuntu 14.04)

               Salt: 2015.5.0
             Python: 2.7.6 (default, Mar 22 2014, 22:59:56)
             Jinja2: 2.7.2
           M2Crypto: 0.21.1
     msgpack-python: 0.3.0
       msgpack-pure: Not Installed
           pycrypto: 2.6.1
            libnacl: Not Installed
             PyYAML: 3.10
              ioflo: Not Installed
              PyZMQ: 14.5.0
               RAET: Not Installed
                ZMQ: 4.0.5
               Mako: Not Installed

gladiatr72 on 12 May 2015

I'm getting the warning, but the event is indeed firing (and the data keyword is indeed being handled properly). The documentation states that data is a proper keyword here. https://docs.saltstack.com/en/latest/ref/states/all/salt.states.event.html#salt.states.event.wait

Output from applying event.wait state with data keyword:

          ID: org/openssh-key-signed/host.cluster
    Function: event.wait
      Result: True
     Comment:
     Started: 11:00:29.559249
    Duration: 2.386 ms
     Changes:
    Warnings: 'data' is an invalid keyword argument for 'event.wait'. If you
              were trying to pass additional data to be used in a template
              context, please populate 'context' with 'key: value' pairs. Your
              approach will work until Salt Carbon is out. Please update your
              state files.

versions report:

Salt Version:
           Salt: 2015.8.8.2

Dependency Versions:
         Jinja2: 2.2.1
       M2Crypto: 0.20.2
           Mako: Not Installed
         PyYAML: 3.11
          PyZMQ: 14.5.0
         Python: 2.6.6 (r266:84292, Jan 22 2014, 09:42:36)
           RAET: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.0.5
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: Not Installed
          gitdb: 0.5.4
      gitpython: Not Installed
          ioflo: Not Installed
        libgit2: Not Installed
        libnacl: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
         pygit2: Not Installed
   python-gnupg: Not Installed
          smmap: 0.8.1
        timelib: Not Installed

System Versions:
           dist: centos 6.6 Final
        machine: x86_64
        release: 2.6.32-504.30.3.el6.x86_64
         system: CentOS 6.6 Final

timcharper on 6 Jun 2016

I got the same issue, with Salt 2016.11

danilochilene on 6 Dec 2016

👍3

This warning message create unnecessary confusion; need fix. @jfindlay

avdhoot on 12 Feb 2017

This is still happening with 2016.11.3.

I've followed the documentation for event.wait and have the following states:

apache2:
  service.running:
    - reload: True

api-checks:
  event.wait:
    - name: iland/monitor/api
    - data:
        nodename: {{ grains['nodename'] }}
        ip: {{ grains['ipv4'] | first() }}
    - watch:
      - service: apache2

When there are changes applied during the highstate and the apache2 service is reloaded then I see this on the event bus:

'event_|-api-checks_|-iland/monitor/api_|-wait': {
  '__id__': 'api-checks',
  '__run_num__': 104,
  'changes': {
    'data': {
      'ip': '10.144.252.102',
      'nodename': 'api-proxy01.dev'
    },
    'tag': 'iland/monitor/api'
  },
  'comment': 'Event fired',
  'duration': 31.379,
  'name': 'iland/monitor/api',
  'result': True,
  'start_time': '17:45:33.678152'

However, when the apache2 state does not need to be reloaded and therefore the api-checks state is not run, I receive a warning:

'event_|-api-checks_|-iland/monitor/api_|-wait': {
  '__id__': 'api-checks',
  '__run_num__': 102,
  'changes': {},
  'comment': '',
  'duration': 0.731,
  'name': 'iland/monitor/api',
  'result': True,
  'start_time': '17:43:57.976954',
  'warnings': ["'data' is an invalid keyword argument for 'event.wait'.  If you were \
                trying to pass additional data to be used in a template context, \
                please populate 'context' with 'key: value' pairs. Your approach \
                will work until Salt Oxygen is out. Please update your state files."]
}

corywright on 30 Mar 2017

👍2

This warning still exists in 2017.7 / Nitrogen.

maltris on 4 Aug 2017

Still in 2017.7.1.

Is this just a simple matter of changing the wait() function arguments in the event state module to accept **kwargs? Or is there something more required?

boltronics on 7 Sep 2017

Not clear, still works for me on 2017.7.1.

danilochilene on 7 Sep 2017

Just ran into this as well on 2017.7.2. Is this a matter of adding the same arguments to the wait function that the send function has? @jfindlay

If so, could it looks like this:

def wait(name,
         data=None,
         preload=None,
         with_env=False,
         with_grains=False,
         with_pillar=False,
         **kwargs):

anitakrueger on 13 Dec 2017

Still an issue in 2017.7.4. It would be really nice running Salt without all these red warnings everywhere I use event.wait at some point. :wink:

boltronics on 8 Mar 2018

Is there anyone still working on this? I am getting this warning but the event is fired which confuses me.

abh23 on 7 Aug 2018

It works but the warning is annoying.

danilochilene on 7 Aug 2018

@bicofino : exactly my point! :D

abh23 on 7 Aug 2018

With the new salt-minion version(salt-2019.2.0-1) it broke.

Passed invalid arguments: 'data' is an invalid keyword argument for 'event.wait'.

The docs are not clear https://docs.saltstack.com/en/latest/ref/states/all/salt.states.event.html

danilochilene on 26 Feb 2019

👍1

I switched to event.send with unless directive.

danilochilene on 27 Feb 2019

As @bicofino says the event.wait function is broken in 2019.2.0. It stops running the state on minion and returns:

Passed invalid arguments: 'data' is an invalid keyword argument for 'event.wait'.

This is critical for us, we have a lot of installations using event.wait as event.send can't handle the watch function.

omolto on 22 Mar 2019

👍1

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

stale[bot] on 8 Jan 2020

Doesn't look stale...

anitakrueger on 8 Jan 2020

👍2

Thank you for updating this issue. It is no longer marked as stale.

stale[bot] on 8 Jan 2020

As @danilochilene has said, this is now broken in the 2019.2.x line of releases.

$ salt -N api-proxy state.highstate
apipxy01.domain:
    ERROR executing 'state.highstate': 'data' is an invalid keyword argument for 'event.wait'
ERROR: Minions returned with non-zero exit code

The state that is triggering the error:

reactor-trigger-api-proxy-apache-checks:
  event.wait:
    - name: ilanddev/monitoring/check-api-proxy-apache
    - data:
        host: {{ grains['ipv4'] | first() }}
        stack_name: {{ salt.ecs.conf('stack_name') }}
    - watch:
      - service: service-apache2

Version info:

$ salt --versions-report
Salt Version:
           Salt: 2019.2.3

Dependency Versions:
           cffi: 1.11.5
       cherrypy: unknown
       dateutil: 2.7.3
      docker-py: Not Installed
          gitdb: 2.0.3
      gitpython: 2.1.8
          ioflo: Not Installed
         Jinja2: 2.10
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: 1.0.7
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
      pycparser: 2.18
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.17 (default, Nov  7 2019, 10:07:09)
   python-gnupg: 0.4.1
         PyYAML: 3.12
          PyZMQ: 16.0.2
           RAET: Not Installed
          smmap: 2.0.3
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.2.5

System Versions:
           dist: Ubuntu 18.04 bionic
         locale: UTF-8
        machine: x86_64
        release: 4.15.0-88-generic
         system: Linux
        version: Ubuntu 18.04 bionic

corywright on 19 Mar 2020

The examples in the event.wait documentation for 2019.2 and 3000 both indicate that data is a valid keyword.

corywright on 23 Mar 2020

Doing some research on this:

wait was created in this commit
it was further refined in this commit, where it stay basically unchanged. The comment on the return was "Noop . The state system will call the mod_watch function instead". The documentation also indicated that supplying arbitrary data was A-OK.
the behavior was changed in Carbon (2016.11.0) to fail where arguments were unsupported. But it doesn't appear that this was ever intended to be the case here.

Additionally, I can send this:

salt-call event.send 'bad/warning' arbitrary=data totally=allowed sfun=wait

Which results in this event:

bad/warning     {
    "_stamp": "2020-03-26T20:30:41.228118",
    "cmd": "_minion_event",
    "data": {
        "__pub_fun": "event.send",
        "__pub_jid": "20200326203041214471",
        "__pub_pid": 890,
        "__pub_tgt": "salt-call",
        "arbitrary": "data",
        "sfun": "wait",
        "totally": "allowed"
    },
    "id": "kevin",
    "tag": "bad/warning"
}

If I add a **kwargs parameter to states/event.py wait and then use this state:

funtimes:
  test.succeed_with_changes

Some Event:
  event.wait:
    - name: bad/warning
    - data:
        arbitrary: data
        totally: allowed
    - watch:
      - test: funtimes

Running state.apply results in this event:

bad/warning     {
    "_stamp": "2020-03-26T20:30:41.228118",
    "cmd": "_minion_event",
    "data": {
        "__pub_fun": "event.send",
        "__pub_jid": "20200326203041214471",
        "__pub_pid": 890,
        "__pub_tgt": "salt-call",
        "arbitrary": "data",
        "sfun": "wait",
        "totally": "allowed"
    },
    "id": "kevin",
    "tag": "bad/warning"
}

If you remove the extra bits, data contains the exact same information.

Based on this, the correct fix is to add **kwargs to states.event.wait.

For a current workaround, I would recommend doing something like:

`mkdir -p /srv/salt/_states && cp $(python3 -c 'import salt.states.event; print(salt.states.event.__file__)') /srv/salt/_states/event.py sed -i 's/sfun=None)/sfun=None, **kwargs)/` /srv/salt/_states/event.py salt \* saltutil.sync_all
After that it you should no longer get this error. There are some upcoming changes to event.py in future versions of Salt, so you should make sure to keep this file up to date until we get the fix in a released version.

HTH!

waynew on 26 Mar 2020

👍1

Thanks @waynew.

I've patched event.py by adding **kwargs to the wait function prototype, and added the patched event.py file to our _states directory (not in /srv/salt/). After running a saltutil.sync_all, we are no longer getting errors in states that call event.wait.

I appreciate you investigating this, and I look forward to the fix being applied upstream.

corywright on 26 Mar 2020

In preparation for https://github.com/saltstack/community/blob/master/doc/Community-Message.pdf I'm getting all of our hosts running the 2019.2.x series of Salt and I just hit this. Ouch! Now I remember why I couldn't upgrade...

Thanks @waynew and @corywright for both investigating and confirming that **kwargs really was all that was required.

boltronics on 27 Apr 2020

👍1

Was this page helpful?

0 / 5 - 0 ratings