Salt: Salt's presence detection

[ghostsquad]

http://docs.saltstack.com/en/latest/ref/runners/all/salt.runners.manage.html#salt.runners.manage.alived

I couldn't find any documentation on what "presence detection" actually does. Can this be explained?

[rallytime]

Thanks for the request/question @ghostsquad. There are some docs here that describe presence events. Does that help answer you question?

[ghostsquad]

That is an extremely brief explanation.

1. There should be a link to the presence events doc in the runners.manage documentation.
2. My interpretation of the documentation leads me to believe that runners.manage.alived will return only minions that are not in the lost state. Is this correct?
3. With default settings, presence_events: False, what is expected from runners.managed.alived?

[rallytime]

Yes, I agree that the explantation could use some more love. I am not particularly familiar with presence events and will default to either @cachedout or @basepi to help answer your questions more completely.

Either way, we should get some more information in the docs, for sure.

[brad-alexander]

bump https://docs.saltstack.com/en/latest/ref/runners/all/salt.runners.manage.html#module-salt.runners.manage also appears to have docstrings copied/pasted between functions and doesn't accurately differentiate the purpose of each function. These docs need some <3

[rallytime]

Thanks for the bump, @brad-alexander. I've applied some more labels here to hopefully get some more eyes on this.

[flowstate]

+1 can we get some more info here?

[rallytime]

Hi @ghostsquad, @brad-alexander, and @flowstate (thanks for the bump) I have looked at the code for presence_events on the master and I think the answers to the questions above are as follows:

There should be a link to the presence events doc in the runners.manage documentation.

Agreed. This is simple to add.

My interpretation of the documentation leads me to believe that runners.manage.alived will return only minions that are not in the lost state. Is this correct?

First, let's back up and look at what presence detection on the master is actually doing when we set presence_events: True. It looks to me like there are actually two presence events fired, depending on the state of the minions. When the handle_presence function is called, we will _always_ fire the event that reports the list of currently connected minions with the present key in the event data. If there are minions that are lost or new, they will get their own event fired, which results in a new or lost key in the data sent back by the event. These events look like this in the logs:

[DEBUG   ] Sending event - data = {'new': ['test-minion1'], '_stamp': '2016-08-31T23:24:06.841639', 'lost': []}
[DEBUG   ] Sending event - data = {'_stamp': '2016-08-31T23:24:06.854819', 'present': ['test-minion1']}
...
[DEBUG   ] Gathering reactors for tag salt/presence/change
...
[DEBUG   ] Gathering reactors for tag salt/presence/present
...
[DEBUG   ] Sending event - data = {'new': [test-minion2'], '_stamp': '2016-08-31T23:25:11.680873', 'lost': []}
[DEBUG   ] Sending event - data = {'_stamp': '2016-08-31T23:25:11.682804', 'present': ['test-minion1', 'test-minion2']}
...

Now, after the looking at the code for the salt-run manage.alived function, it looks like the alived function is _only_ going to return minions that are currently connected. It does not list lost or new minions. I think the presence functionality from a runner was updated for the RAET transport, but for the other transports, we're only checking for our currently connected minions.

With default settings, presence_events: False, what is expected from runners.managed.alived?

That is a good question. The docs do indicate that you need presence_events set to True, for this function to work. However, that doesn't appear that is the case. I set presence_events: False in my master config and restarted my master. While I do not see the presence events firing as mentioned above in my debug logs, I still get the list of minions that are currently connected to my salt-master:

root@rallytime:~# cat /etc/salt/master
# /etc/salt/master
presence_events: False
root@rallytime:~# salt-run manage.alived
- test-minion1
- test-minion2
root@rallytime:~# salt-key
Accepted Keys:
test-minion1
test-minion2
Denied Keys:
Unaccepted Keys:
Rejected Keys:

This result makes sense with the code, however it does seem like it might be confusing according to the docs. Rather than check for the presence_events setting and changing the behavior of the runner function, I am more inclined to just fix the docs on that particular issue.

I am happy to submit a pull request to update the docs to explain all of this. Before doing so, does this make sense? Am I missing anything?

[brad-alexander]

The only other thing I would add is that there are a lot of functions with the same docstring, so if you don't inherently know what reaped, joined, allowed, etc mean in context it doesn't really do much to explain the functions.

Specifically these functions:

'''Print a list of all minions that are up according to Salt's presence detection (no commands will be sent to minions)'''

• salt.runners.manage.allowed(subset=None, show_ipv4=False)
• salt.runners.manage.alived(subset=None, show_ipv4=False)
• salt.runners.manage.joined(subset=None, show_ipv4=False)
• salt.runners.manage.list_state(subset=None, show_ipv4=False, state=None)
• salt.runners.manage.present(subset=None, show_ipv4=False)
• salt.runners.manage.reaped(subset=None, show_ipv4=False)

'''Print a list of all minions that are NOT up according to Salt's presence detection (no commands will be sent to minions)'''

• salt.runners.manage.list_not_state(subset=None, show_ipv4=False, state=None)
• salt.runners.manage.not_alived(subset=None, show_ipv4=False)
• salt.runners.manage.not_allowed(subset=None, show_ipv4=False)
• salt.runners.manage.not_joined(subset=None, show_ipv4=False)
• salt.runners.manage.not_present(subset=None, show_ipv4=False)
• salt.runners.manage.not_reaped(subset=None, show_ipv4=False)

[flowstate]

Need to read and digest, but … thank you! This is (to my mind) the single
largest black box in saltstack. I will provide further feedback shortly.

On Wed, Aug 31, 2016 at 6:57 PM, Nicole Thomas [email protected]
wrote:

Hi @ghostsquad https://github.com/ghostsquad, @brad-alexander
https://github.com/brad-alexander, and @flowstate
https://github.com/flowstate (thanks for the bump) I have looked at the
code for presence_events on the master and I think the answers to the questions
above
https://github.com/saltstack/salt/issues/22386#issuecomment-90225951
are as follows:

There should be a link to the presence events doc in the runners.manage
documentation.

Agreed. This is simple to add.

My interpretation of the documentation leads me to believe that
runners.manage.alived will return only minions that are not in the lost
state. Is this correct?

First, let's back up and look at what presence detection on the master is
actually doing when we set presence_events: True. It looks to me like
there are actually two presence events fired, depending on the state of the
minions. When the handle_presence
https://github.com/saltstack/salt/blob/3057f804489677e4f6d3cf96464dcf4d6661d1b1/salt/master.py#L347
function is called, we will _always_ fire the event that reports the list
of currently connected minions with the present key in the event data.
_If_ there are minions that are lost or new, they will get their own
event fired, which results in a new or lost key in the data sent back by
the event. These events look like this in the logs:

[DEBUG ] Sending event - data = {'new': ['test-minion1'], '_stamp': '2016-08-31T23:24:06.841639', 'lost': []}
[DEBUG ] Sending event - data = {'_stamp': '2016-08-31T23:24:06.854819', 'present': ['test-minion1']}
...
[DEBUG ] Gathering reactors for tag salt/presence/change
...
[DEBUG ] Gathering reactors for tag salt/presence/present
...
[DEBUG ] Sending event - data = {'new': [test-minion2'], '_stamp': '2016-08-31T23:25:11.680873', 'lost': []}
[DEBUG ] Sending event - data = {'_stamp': '2016-08-31T23:25:11.682804', 'present': ['test-minion1', 'test-minion2']}
...

Now, after the looking at the code for the salt-run manage.alived
function, it looks like the alived function is _only_ going to return
minions that are currently connected. It does not list lost or new minions.
I think the presence functionality from a runner was updated for the RAET
transport, but for the other transports, we're only checking for our
currently connected minions.

With default settings, presence_events: False, what is expected from
runners.managed.alived?

That is a good question. The docs do indicate that you need
presence_events set to True, for this function to work. However, that
doesn't appear that is the case. I set presence_events: False in my
master config and restarted my master. While I do not see the presence
events firing as mentioned above in my debug logs, I still get the list of
minions that are currently connected to my salt-master:

root@rallytime:~# cat /etc/salt/master

/etc/salt/master

presence_events: False
root@rallytime:~# salt-run manage.alived

• test-minion1
• test-minion2
  root@rallytime:~# salt-key
  Accepted Keys:
  test-minion1
  test-minion2
  Denied Keys:
  Unaccepted Keys:
  Rejected Keys:

This result makes sense with the code, however it does seem like it might
be confusing according to the docs. Rather than check for the
presence_events setting and changing the behavior of the runner function,
I am more inclined to just fix the docs on that particular issue.

I am happy to submit a pull request to update the docs to explain all of
this. Before doing so, does this make sense? Am I missing anything?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/saltstack/salt/issues/22386#issuecomment-243937716,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAYEMNbDmrw1ZheTNn4n1a4-ca54Ydquks5qlhTtgaJpZM4D6_LY
.

[grinapo]

Does it even work under zmq? I switched presence_events: True in master and the event contains "present":[]
I have tried restarting some minions, no change, zero in lists, except up and down.

[rallytime]

@grinapo Can you please post what version of salt you're using? And are the minions up and responding to the master?

[grinapo]

Yes, all the minions are just fine, and they are all present in the "up" list.
(As a sidenote: is there a way to list which IP addresses do the minions use to communicate with the master?)

Salt Version:
Salt: 2016.3.2

Dependency Versions:
cffi: Not Installed
cherrypy: Not Installed
dateutil: 2.4.2
gitdb: 0.6.4
gitpython: 2.0.5
ioflo: Not Installed
Jinja2: 2.8
libgit2: Not Installed
libnacl: Not Installed
M2Crypto: Not Installed
Mako: 1.0.4
msgpack-pure: Not Installed
msgpack-python: 0.4.8
mysql-python: Not Installed
pycparser: Not Installed
pycrypto: 2.6.1
pygit2: Not Installed
Python: 2.7.12+ (default, Sep 1 2016, 20:27:38)
python-gnupg: Not Installed
PyYAML: 3.12
PyZMQ: 15.2.0
RAET: Not Installed
smmap: 0.9.0
timelib: Not Installed
Tornado: 4.4.1
ZMQ: 4.1.5

System Versions:
dist: debian stretch/sid
machine: x86_64
release: 4.4.16-1-pve
system: Linux
version: debian stretch/sid

[jimmykarily]

I'm also getting inconsistent results from manage.present and manage.up commands. I have one minions connected to the master:

~ $ salt-run manage.up
- 4c0669a7dcfa

I have presence events enabled in /etc/salt/master:

presence_events: True

I can see the presence event (which shows no present minions):

~ $ salt-run state.event pretty=True
salt/presence/present   {
    "_stamp": "2017-01-02T14:18:21.309058", 
    "present": []
}

and salt-run manage.present returns nothing.

Here are my versions:

~ $ salt --versions
Salt Version:
           Salt: 2016.3.4

Dependency Versions:
           cffi: 1.6.0
       cherrypy: 3.6.0
       dateutil: 2.6.0
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.8
        libgit2: 0.24.0
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: 1.0.4
   msgpack-pure: Not Installed
 msgpack-python: 0.4.8
   mysql-python: 1.2.5
      pycparser: 2.14
       pycrypto: 2.6.1
         pygit2: 0.24.0
         Python: 2.7.12 (default, Jun 28 2016, 06:57:42) [GCC]
   python-gnupg: 0.3.8
         PyYAML: 3.12
          PyZMQ: 15.2.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.4.2
            ZMQ: 4.2.0

System Versions:
           dist:   
        machine: x86_64
        release: 4.8.12-1-default
         system: Linux
        version: Not Installed

and on the minion:

bash-4.4# salt-minion --versions
Salt Version:
           Salt: 2016.3.4

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.8
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.8
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
         pygit2: Not Installed
         Python: 2.7.12 (default, Jun 28 2016, 06:57:42) [GCC]
   python-gnupg: Not Installed
         PyYAML: 3.12
          PyZMQ: 15.2.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.4.2
            ZMQ: 4.2.0

System Versions:
           dist:   
        machine: x86_64
        release: 4.8.13-1-ARCH
         system: Linux
        version: Not Installed

I don't know if this is relevant but I'm running the minion as a docker container.

According to the docs I should be seeing a presence/change event when the minion connects but I never do. These are the events I see when the minion starts:

salt/auth   {
    "_stamp": "2017-01-02T14:24:09.181373", 
    "act": "accept", 
    "id": "4c0669a7dcfa", 
    "pub": "-----BEGIN PUBLIC KEY-----\nMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAsYS1TTwiLh6f+g1IfLw9\nKMYBE2uGZgYK7aHbIuELs5ywJtu72uwoNDLQYhQGef0MSvmkMKWAuzHI8gKdk6tN\nWGf1mUWr1SPNh5XcD4yqUFIO2oPvD1v3IvTgoD4YC+GO9Vdns/x50O7HeP6gx6hU\nun+b0uVS1xmesm7Mg68OVWW3bjuXrbiE8PtKNzO923GlGulvpzBpjk1xKZhQOHrI\nHsUka54ZBI8d9E9ZyRm7nlKR62uGTO5zM7mvun7m8nOifx2BQkCEXmD0twwpmF7Z\nt1IQprHmmBvr20X0EerlPp6bm37H1b0iz1jvV90smQsU+Os56Y0O03/WA6K7TZ0P\naQIDAQAB\n-----END PUBLIC KEY-----", 
    "result": true
}
minion_start    {
    "_stamp": "2017-01-02T14:24:09.782269", 
    "cmd": "_minion_event", 
    "data": "Minion 4c0669a7dcfa started at Mon Jan  2 14:24:09 2017", 
    "id": "4c0669a7dcfa", 
    "pretag": null, 
    "tag": "minion_start"
}
salt/minion/4c0669a7dcfa/start  {
    "_stamp": "2017-01-02T14:24:09.799579", 
    "cmd": "_minion_event", 
    "data": "Minion 4c0669a7dcfa started at Mon Jan  2 14:24:09 2017", 
    "id": "4c0669a7dcfa", 
    "pretag": null, 
    "tag": "salt/minion/4c0669a7dcfa/start"
}
salt/presence/present   {
    "_stamp": "2017-01-02T14:24:24.202840", 
    "present": []

[jimmykarily]

I just tried setting up a minion on the host (no docker container) and it works. I get a presence/change event:

salt/presence/change    {
    "_stamp": "2017-01-02T14:36:30.094589", 
    "lost": [], 
    "new": [
        "Lomasi"
    ]
}
salt/presence/present   {
    "_stamp": "2017-01-02T14:36:30.095259", 
    "present": [
        "Lomasi"
    ]
}
salt/run/20170102163641407387/new   {
    "_stamp": "2017-01-02T14:36:41.721758", 
    "fun": "runner.manage.present", 
    "fun_args": [], 
    "jid": "20170102163641407387", 
    "user": "sudo_dimitris"
}
salt/run/20170102163641407387/ret   {
    "_stamp": "2017-01-02T14:36:41.801643", 
    "fun": "runner.manage.present", 
    "fun_args": [], 
    "jid": "20170102163641407387", 
    "return": [
        "Lomasi"
    ], 
    "success": true, 
    "user": "sudo_dimitris"
}
salt/presence/present   {
    "_stamp": "2017-01-02T14:37:30.545285", 
    "present": [
        "Lomasi"
    ]
}

and

~ $ sudo salt-run manage.present
- Lomasi

the versions on the working minion:

[dimitris@Lomasi ~]$ sudo salt-minion --versions
Salt Version:
           Salt: 2016.11.1

Dependency Versions:
           cffi: 1.9.1
       cherrypy: Not Installed
       dateutil: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.8
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: 0.24.0
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.8
   mysql-python: Not Installed
      pycparser: 2.17
       pycrypto: 2.6.1
         pygit2: Not Installed
         Python: 2.7.13 (default, Dec 21 2016, 07:16:46)
   python-gnupg: Not Installed
         PyYAML: 3.12
          PyZMQ: 16.0.2
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.4.2
            ZMQ: 4.2.0

System Versions:
           dist:   
        machine: x86_64
        release: 4.8.13-1-ARCH
         system: Linux
        version: Not Installed

I have a later version of salt on this minion. Maybe an already fixed bug?

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.