Salt: Add extra tag info for Beacons that produce multiple status messages

[erwindon]

Is your feature request related to a problem? Please describe.
The Beacon facility of Salt publishes events when conditions are met (or just periodically). Some beacon providers generate at most a single message every time they are doing their work. But some beacon providers monitor multiple objects and can generate multiple messages in each interval. One example is the diskusage, which potentially generates a message per disk. Most beacon providers add additional information to the beacon message tag so that the information maintains its context. However, some beacon providers do not add this information, e.g. the same diskusage. This causes generic monitoring software to treat these events as updates of previous information of the same object, while it is actually about different objects.

Describe the solution you'd like
Extend some of the beacon tags with additional information. At least for the following providers:

• diskusage: TAG based on MOUNT should be added
• haproxy: TAG based on SERVER should be added
• network_info: TAG based on INTERFACE should be added
• pkg: TAG based on PKG should be added
• ps: TAG based on PROCESS should be added

The other providers do not need action because they produce:

1. only single messages (aix_account, avahi_announce, bonjour_announce, cert_info, glx_info, load, memusage, proxy_example, telegram_bot_msg, twilio_txt_msg); or
2. properly extend the tag (adb, inotify, journald, log_beacon, napalm_beacon, network_settings, sense_hat, service, sh, smartos_imgadm, status); or
3. are producing discrete events (btmp, wtmp, watchdog).

Additionally,"event" type beacons should be identifiable as such. to distinguish them from "status" type beacons. for "event" beacons, every instance may be important. for "status" beacons, typically only the latest one is important. the standard ones are btmp, wtmp and watchdog (note: the watchdog beacon uses its own package name to add to the tag, barely useful). for events, I propose a new payload field beacon-type: event. The opposite would be: beacon-type: status, but that can be the default. The field name is unique enough to not conflict/overlap with any field in any beacon provider.

Additional context
The beacon messages may be handled by a generic message processor. For display, or to keep the 'latest' state.
In a generic message processor you do not want to have the knowledge of the beacon message structure, specifically about which field is a "key" field for each beacon type. The uniform way is to append that knowledge to the beacon tag as already done by several beacon providers. But in these cases that is not done.
And for the "standard" beacons in the current release, it is just the 5 ones mentioned. But any newly introduced beacon, whether standard or custom, may have the same problem again.

e.g. a beacon tag like salt/beacon//network_info/wlp4s0 is then expected.
It can be realised easily because the beacon framework already supports extending the tag.

[DmitryKuzmenko]

@erwindon could you please clarify your case for me. I've configured a beacon like this:

beacons:
  network_info:
    - interfaces:
        wlp4s0:
          type: greater
          bytes_recv: 0
        tap0:
          type: greater
          bytes_recv: 0
        lo:
          type: greater
          bytes_recv: 0

And I'm seeing this events in master log:

[DEBUG   ] Sending event: tag = salt/beacon/alpha/network_info/; data = {'interface': 'wlp4s0', 'network_info': {'bytes_sent': 124732545, 'bytes_recv': 1164847263, 'packets_sent': 648340, 'packets_recv': 2317795, 'errin': 0, 'errout': 0, 'dropin': 0, 'dropout': 0}, 'id': 'alpha', '_stamp': '2020-05-14T00:13:57.957564'}
[DEBUG   ] Sending event: tag = salt/beacon/alpha/network_info/; data = {'interface': 'tap0', 'network_info': {'bytes_sent': 144413, 'bytes_recv': 30832769, 'packets_sent': 731, 'packets_recv': 164656, 'errin': 0, 'errout': 0, 'dropin': 357, 'dropout': 0}, 'id': 'alpha', '_stamp': '2020-05-14T00:13:57.958089'}
[DEBUG   ] Sending event: tag = salt/beacon/alpha/network_info/; data = {'interface': 'lo', 'network_info': {'bytes_sent': 70909518, 'bytes_recv': 70909518, 'packets_sent': 787539, 'packets_recv': 787539, 'errin': 0, 'errout': 0, 'dropin': 0, 'dropout': 0}, 'id': 'alpha', '_stamp': '2020-05-14T00:13:57.958535'}

So for each event we can't determine the interface from the event id. But we see it in event['interface'] field. Do you want to be able to filter these events by event id? Say salt/beacon/<minion_id>/network_info/wlp4s0?

[erwindon]

@DmitryKuzmenko
The beacon messages may be handled by a generic message processor. For display, or to keep the 'latest' state.
In a generic message processor you do not want to have the knowledge of the beacon message structure, specifically about which field is a "key" field for each beacon type. The uniform way is to append that knowledge to the beacon tag as already done by several beacon providers. But in these cases that is not done.
And for the "standard" beacons in the current release, it is just the 5 ones mentioned. But any newly introduced beacon, whether standard or custom, may have the same problem again.

Your example salt/beacon/<minion_id>/network_info/wlp4s0 is exactly the solution I had in mind.
And it can be realised easily because the beacon framework already supports extending the tag.

[DmitryKuzmenko]

@erwindon thank you for explanation! It's a good idea!

[erwindon]

@DmitryKuzmenko the next thing would be to make the "event" type beacons identifiable as such. to distinguish them from "status" type beacons. for "event" beacons, every instance may be important. for "status" beacons, typically only the latest one is important. the standard ones are btmp, wtmp and watchdog (note: the watchdog beacon uses its own package name to add to the tag, barely useful). for events, I propose a new payload field beacon-type: event. The opposite would be: beacon-type: status, but that can be the default. The field name is unique enough to not conflict/overlap with any field in any beacon provider.

[DmitryKuzmenko]

@erwindon looks reasonable for me.
@team-core I want someone else to take a look at these ideas.

[erwindon]

when applicable, I'm volunteering to create a PR for both code updates and doc/topics/beacons/index.rst update.

[DmitryKuzmenko]

@erwindon could you please put your comment https://github.com/saltstack/salt/issues/57174#issuecomment-628460030 to the issue text to not lose it.

[erwindon]

not impatient, but may be I should mention: "done"