Awx: Slack integration fails with invalid_auth for notifications

[atlekberg]

ISSUE TYPE

• Bug Report

SUMMARY

Slack integration fails with invalid_auth for notifications.

ENVIRONMENT

• AWX version: 9.3.0
• AWX install method: docker on linux
• Ansible version: 2.9.5
• Operating System: Ubuntu 18.04.4
• Web Browser: Google Chrome

STEPS TO REPRODUCE

Slack credentials created by adding new configuration to the "Incoming WebHooks" Slack App. Verified that these work by posting to channel with Curl.

Create a new notification of type Slack. Add token (xxx/xxxxx/xxxxxxx format) and destination channel. Save and click on test bell.

EXPECTED RESULTS

Message posted to the Slack channel specified.

ACTUAL RESULTS

Error message in GUI:

Slack notifications: Norification failed.
Slack Notification unable to send drift: Tower Notification Test 2 https://someserver.somedomain.com (invalid_auth)

ADDITIONAL INFORMATION

Log output below.

2020-04-01 09:21:36,106 DEBUG    awx.main.notifications.slack_backend {'ok': False, 'error': 'invalid_auth', 'headers': {'Content-Type': 'application/json; charset=utf-8', 'Content-Length': '55', 'Connection': 'keep-alive', 'Date': 'Wed, 01 Apr 2020 09:21:36 GMT', 'Server': 'Apache', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains; preload', 'Access-Control-Allow-Headers': 'slack-route, x-slack-version-ts, x-b3-traceid, x-b3-spanid, x-b3-parentspanid, x-b3-sampled, x-b3-flags', 'Referrer-Policy': 'no-referrer', 'X-Slack-Backend': 'h', 'Access-Control-Expose-Headers': 'x-slack-req-id, retry-after', 'Vary': 'Accept-Encoding', 'X-Content-Type-Options': 'nosniff', 'X-Slack-Req-Id': 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', 'X-XSS-Protection': '0', 'Content-Encoding': 'gzip', 'Access-Control-Allow-Origin': '*', 'X-Via': 'haproxy-www-cy4d', 'X-Cache': 'Miss from cloudfront', 'Via': '1.1 bcd5dadccb0831729969c938747ff79a.cloudfront.net (CloudFront)', 'X-Amz-Cf-Pop': 'OSL50-C1', 'X-Amz-Cf-Id': 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'}}
2020-04-01 09:21:36,107 ERROR    awx.main.notifications.slack_backend Exception sending messages: Slack Notification unable to send drift: Tower Notification Test 2 https://someserver.somedomain.com (invalid_auth)
2020-04-01 09:21:36,109 ERROR    awx.main.tasks Send Notification Failed Slack Notification unable to send drift: Tower Notification Test 2 https://someserver.somedomain.com (invalid_auth)
Traceback (most recent call last):
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/main/tasks.py", line 324, in send_notifications
    sent = notification.notification_template.send(notification.subject, notification.body)
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/main/models/notifications.py", line 196, in send
    return backend_obj.send_messages([notification_obj])
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/main/notifications/slack_backend.py", line 56, in send_messages
    raise RuntimeError("Slack Notification unable to send {}: {} ({})".format(r, m.subject, ret['error']))
RuntimeError: Slack Notification unable to send drift: Tower Notification Test 2 https://someserver.somedomain.com (invalid_auth)

[ryanpetrello]

@atlekberg

I'm having trouble reproducing this with my Slack account and token:

The token I've got is in a format like xxx-xxxxxxxxx-xxxxxxxxxxxxxxxxxxxxx (not slashes)

invalid_auth from Slack's HTTP response makes it sound like maybe something's wrong with your token?

[atlekberg]

So I finally figured it out. What threw me off was that when I tested my slash-based-token it worked with both Curl and the Ansible Slack module. Replacing the slashes with hyphens did not work by the way.

Searching the official Slack documentation I had a hard time finding anything about bot integration tokens and that's why I initially followed the Ansible Slack module documentation. Continuing the search I found the part about legacy tokens: https://api.slack.com/legacy/custom-integrations/legacy-tokens

Using this method I can generate a token that works with Tower notifications - so yay!

However; it seems this way of creating tokens are discouraged and will be deprecated soon:

You're reading this because you're looking for info on legacy custom integrations - an outdated way for teams to integrate with Slack. These integrations lack newer features and they will be deprecated and possibly removed in the future. We do not recommend their use.

Legacy token generator
This tool will not allow creation of new tokens beginning May 5th, 2020. Learn more about why you shouldn't use it anymore.

In light of this, and provided I haven't misunderstood something, it might be time to rework the Slack integration in AWX/Tower 😅

[ryanpetrello]

Thanks for reporting this, @atlekberg. I was unaware of this upcoming change to Slack's API.

I've filed this issue here to track it: https://github.com/ansible/awx/issues/6610

[ryanpetrello]

Hey @atlekberg,

It looks to me like AWX is already equipped to handle this. Give this comment a look:

https://github.com/ansible/awx/issues/6610#issuecomment-613035465

[atlekberg]

@ryanpetrello thanks for the tip! I can confirm that this works. There's one additional step that needs to be done at the end - and that is to add the Bot/App to the channel.