Netdata: Alerta - Clear event error on default settings doesn't work.

[GitStoph]

Bug report summary

sudo /usr/libexec/netdata/plugins.d/alarm-notify.sh test yields:

# SENDING TEST WARNING ALARM TO ROLE: sysadmin
2019-01-10 09:34:48: alarm-notify.sh: INFO: sent alerta notification for: netdata test.chart.test_alarm is WARNING to 'Netdata'
# OK

# SENDING TEST CRITICAL ALARM TO ROLE: sysadmin
2019-01-10 09:34:48: alarm-notify.sh: INFO: sent alerta notification for: netdata test.chart.test_alarm is CRITICAL to 'Netdata'
# OK

# SENDING TEST CLEAR ALARM TO ROLE: sysadmin
2019-01-10 09:34:48: alarm-notify.sh: ERROR: failed to send alerta notification for: netdata test.chart.test_alarm is CLEAR to 'Netdata', with HTTP error code 500.
# FAILED

OS / Environment

Ubuntu 16.04 Linux netdata 4.4.0-141-generic #167-Ubuntu SMP Wed Dec 5 10:40:15 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Noteworthy part of alerta config: (via /etc/alertad.conf)

SEVERITY_MAP = {
    'fatal': 0,
    'critical': 1,
    'warning': 4,
    'indeterminate': 5,
    'ok': 5,
    'unknown': 9
}
COLOR_MAP = {
    'severity': {
        'fatal': 'blue',
        'critical': 'red',
        'warning': '#1E90FF',
        'indeterminate': 'lightblue',
        'ok': '#00CC00',
        'unknown': 'silver'
    }
}

Component Name

plugins.d/alarm-notify.sh

Steps To Reproduce

Spin up Alerta server via their default tutorial. https://docs.alerta.io/en/latest/gettingstarted/tutorial-1-deploy-alerta.html#tutorial-1
Configure the plugin for hitting alerta.
Run the above command to send a test alert.

Expected behavior

# SENDING TEST WARNING ALARM TO ROLE: sysadmin
2019-01-10 09:40:21: alarm-notify.sh: INFO: sent alerta notification for: netdata test.chart.test_alarm is WARNING to 'Netdata'
# OK

# SENDING TEST CRITICAL ALARM TO ROLE: sysadmin
2019-01-10 09:40:21: alarm-notify.sh: INFO: sent alerta notification for: netdata test.chart.test_alarm is CRITICAL to 'Netdata'
# OK

# SENDING TEST CLEAR ALARM TO ROLE: sysadmin
2019-01-10 09:40:21: alarm-notify.sh: INFO: sent alerta notification for: netdata test.chart.test_alarm is CLEAR to 'Netdata'
# OK

Workaround or suggested fix:

https://github.com/netdata/netdata/blob/6f73c3be9a5c5b2efa2fc741dd6666812356504c/health/notifications/alarm-notify.sh.in#L1396 Change this line to say: CLEAR) severity="ok" ;;

It's not necessarily a "BUG" per se, but it won't work with their documented default config from their tutorial. If you'd like to leave the script how it is, I totally understand. Maybe someone out there googling this issue in the future will find this post and save a little time.
Thanks for the good work!

[cakrit]

Awesome bug report @GitStoph , we really appreciate the time it took to provide such clear information. The PR to fix it has been created.

[cakrit]

"cleared" is a valid severity as per https://media.readthedocs.org/pdf/alerta/latest/alerta.pdf page 10, so it's just a matter of what one leaves in the alerta config file.

[GitStoph]

Yup! Perhaps the better solution may be to update the README (https://docs.netdata.cloud/health/notifications/alerta/) to reflect what's on page 10 there? I didn't mean to infer that the code was wrong, just that someone looking to integrate it the first time would likely run into this issue. A quick snippet to say they should ensure their /etc/alertad.conf file includes:

SEVERITY_MAP = {
    'fatal': 0,
    'critical': 1,
    'warning': 4,
    'indeterminate': 5,
    'ok': 5,
    'cleared': 5,
    'unknown': 9
}
COLOR_MAP = {
    'severity': {
        'fatal': 'blue',
        'critical': 'red',
        'warning': '#1E90FF',
        'indeterminate': 'lightblue',
        'ok': '#00CC00',
        'cleared': '#00CC00',
        'unknown': 'silver'
    }
}

Thanks for looking. Mostly was just hoping to save someone else the time if they ran into this.