Sentry: Slack Global Integration: oauth error (WST apps deprecated)

[anthonyangel]

I'm trying to enable the new glopal Slack integration on a fresh install of Sentry 9 (installed as per the instructions for pip / venv)

I'm getting an error when trying to connect to a Slack workspace, OAuth Error: Invalid client_id parameter - I'm assuming that this is expecting to find a connection from the hosted version of Sentry, rather than a self-hosted version.

The redirect_uri parameter in the URL is the URL for our self-hosted sentry, and the client_id is blank

[ryanpineo]

I encountered this issue myself. I've managed to get it partially working where the integration will send messages to slack but the interactive buttons on the slack page don't work.

Rough steps:

1. Create your own slack workspace app called Sentry: https://api.slack.com/apps/new_app_token
2. Get your client id and client secret from the slack app page and put it in your config.yml under slack.client-id and slack.client-secret.
3. In the slack app page go to the "Oauth & permissions" page and add your domain to the redirect urls. e.g. https://sentry.mycompany.com

Then restart sentry and you should be able to link the app from within the sentry UI with no more invalid client id error.

Im still trying to figure out how to get the buttons from within the slack messages to work.

[lipsumar]

You need to do a bit more: https://forum.sentry.io/t/how-to-configure-slack-in-your-on-prem-sentry/3463

However the actions don't seem to work for us either.

[EvanPurkhiser]

Would anyone who's having the error with slack actions be able to verify that slack is making API requests to sentry by checking their service logs? Slack should be talking to the endpoint /extensions/slack/action/. It sounds like sentry is having a problem handling the action. There should be some relevant information in the logs hopefully.

One thing to verify is that this is setup correctly in your app config:

[jonasvp]

I ran into the same problem. The link posted by @lipsumar was extremely helpful but the OAuth URL threw an error on Sentry's side. From the URL parameters I figured out that under "OAuth & Permissions" in Slack's app config, you need to add more Scopes:

• channels:read
• groups:read
• links:read
• links:write
• users:read
• team:read

However, the permission chat:write required by Sentry does not exist anymore, Slack only offers chat:write:bot. I assume that's why the OAuth process keeps failing for me.

[jonasvp]

Exchanging chat:write to chat:write:bot here gets me another step further. It now returns a Sentry error page under the url /extensions/slack/setup/ saying "Failed to retrieve token from the upstream service."

Any help would be appreciated.

[EvanPurkhiser]

It looks like slack is making some changes.

As of today, the creation of new workspace apps will be disabled. Apps that use workspace tokens will continue to run for the next calendar year until October 10, 2019. In this interim period, documentation for workspace apps will still be available, but we will not be investing substantial resources in bug fixes or new feature development.

A little bit of a backstory here, we built Slack ontop of the developer preview for workspace tokens, since it was looking like it was going to be the next generation way to build slack apps, but seems like Slack has decided to pivot from that direction and actually disable creation of workspace token apps.

I've been looking around a bit to see if there's a good quick solution we can hack in, but I'm not sure if there is one. This seems to imply that you can still create workspace apps but it doesn't look like you can.

As always, very open to contributions, we've started Q4 with some new work, but I'll see what I can do to make it back around to fix this soon.

[dcramer]

I encourage y’all to explain to slack how much you’re enjoying this. We don’t have the resources to rewrite this integration every 6 months because they decide they want to deprecate another API.

[andrejbaran]

@dcrammer @EvanPurkhiser would you be willing to take PR that fixes this?

It is fairly simple to fix the sentry side. I have just done that for our onpremise installation.

[dcramer]

@andrejbaran yes but there’s certainly complexity in that we need to make sure it will still work for existing users (both saas and self hosted). I’ll let Evan comment but I think they were talking to someone at Slack about this very issue.

[andrejbaran]

@dcramer I was thinking the same since we are self hosted we don't have access to all functionality. Is this area covered by tests?

From what I can tell the fix consists of changing the permission (as it was mentioned earlier) and changing the access token exchange url in slack oauth provider/backend plus one need to create a slack app with bot and set correct slack credentials in sentry config.

[andrejbaran]

One thing that comes to my mind is that the fix is for 9.0 version and I know the integration changed since, so it would be good to test against what is in master

[dcramer]

What we could likely do is build a fixture that mocks an account that'd be setup using the current scheme and ensures it still functions identically, and then allow new accounts to use a new scheme.

[andrejbaran]

@dcramer would you be willing to do it? I could afterwards submit my PR and we could move forward with the fix.

[jiyda-stellic]

@andrejbaran How did you fix the issue on premise?

• changing the permission from chat:write to chat:write:bot (Done)
• slack oauth provider/backend (How to do that ?)
• set correct slack credentials in sentry config (which credentials exactly?)

@dcramer I was thinking the same since we are self hosted we don't have access to all functionality. Is this area covered by tests?

From what I can tell the fix consists of changing the permission (as it was mentioned earlier) and changing the access token exchange url in slack oauth provider/backend plus one need to create a slack app with bot and set correct slack credentials in sentry config.

[dcramer]

@EvanPurkhiser maybe you can comment on existing test coverage as im not familiar w/ it

[bsergean]

I'm happy to test a manual server side fix ; I can manually hijack our sentry 9 python code if needed. However I'm not very slack integrations savy, so if a blog post or git snippet can explain that with screenshots etc.. that would be really sweet.

[harrisjoseph]

Is un-deprecating the previous slack integration an option here? It was _far_ easier to set up on-prem, and the Sentry Notifications slack application still instructs you to pip-install the older integration:
https://github.com/getsentry/sentry-slack

Unless I'm mistaken, the existing set-up instructions now point to something that doesn't work, so should probably come with visible warnings:
https://docs.sentry.io/integrations/slack/
https://forum.sentry.io/t/how-to-configure-slack-in-your-on-prem-sentry/3463

[dcramer]

The old slack integration did very different things -- all it did was send a notification. The new one provides unfurls, actions, and more complex notifications. We (being Sentry) are also certainly not going to maintain two separate integrations for Slack, but you can certainly still install the old integration.

[andrejbaran]

@jiyda-stellic @dcramer sorry for the late response.

2) adjust slack integration/provider

• provider: https://github.com/ftshp/sentry/blob/9.0.0/src/sentry/identity/slack/provider.py:

14c14
<     oauth_access_token_url = 'https://slack.com/api/oauth.access'
---
>     oauth_access_token_url = 'https://slack.com/api/oauth.token'

• integration: https://github.com/ftshp/sentry/blob/9.0.0/src/sentry/integrations/slack/integration.py

47c47
<         'chat:write:bot',
---
>         'chat:write',
103c103
<                 'external_id': data['user_id'],
---
>                 'external_id': data['authorizing_user_id'],

3) In config.yml you need to set

slack.client-id: 'you'r custom apps client id'
slack.client-secret: 'your custom app's client secret'
slack.verification-token: 'your custom app's token'

[BoLaMN]

can confirm the changes from @andrejbaran works

diff --git a/src/sentry/identity/slack/provider.py b/src/sentry/identity/slack/provider.py
index c989ae9..8d31eab 100644
--- a/src/sentry/identity/slack/provider.py
+++ b/src/sentry/identity/slack/provider.py
@@ -11,7 +11,7 @@ class SlackIdentityProvider(OAuth2Provider):
     # TODO(epurkhiser): This identity provider is actually used for authorizing
     # the Slack application through their Workspace Token OAuth flow, not a
     # standard user access token flow.
-    oauth_access_token_url = 'https://slack.com/api/oauth.token'
+    oauth_access_token_url = 'https://slack.com/api/oauth.access'
     oauth_authorize_url = 'https://slack.com/oauth/authorize'

     oauth_scopes = (
diff --git a/src/sentry/integrations/slack/integration.py b/src/sentry/integrations/slack/integration.py
index 7646c81..3491fab 100644
--- a/src/sentry/integrations/slack/integration.py
+++ b/src/sentry/integrations/slack/integration.py
@@ -72,7 +72,7 @@ class SlackIntegrationProvider(IntegrationProvider):
         'channels:read',
         'groups:read',
         'users:read',
-        'chat:write',
+        'chat:write:bot',
         'links:read',
         'links:write',
         'team:read',
@@ -128,7 +128,7 @@ class SlackIntegrationProvider(IntegrationProvider):
             },
             'user_identity': {
                 'type': 'slack',
-                'external_id': data['authorizing_user_id'],
+                'external_id': data['user_id'],
                 'scopes': [],
                 'data': {},
             },

[andrejbaran]

@dcramer @EvanPurkhiser what are the next steps?

[dcramer]

They haven’t changed. I wasn’t saying it’s not impossible to fix we just don’t have time right now. If someone can submit a PR that doesn’t break existing integrations (instanced) but switches to the new (old?) api for new integrations I think we could accept that. We’d need to QA and make sure all of the features still work though.

[BoLaMN]

@dcramer Im not sure i follow "submit a PR that doesn’t break existing integrations" as atm the slack integration is just plain broken without the patch

[dcramer]

@bolamn it is not broken for our saas users at all

[EvanPurkhiser]

@BoLaMN et al:

We have to be careful here because sentry is not just an open source piece of software, but also the core codebase that powers https://sentry.io/, the business behind sentry that allow us to keep making the open source project great for you guys!

The problem is, on sentry.io we have thousands of organizations using the Slack Integration which is powered by _workspace tokens_. As slack has said in their blog post, the workspace app will continue to work for the next year:

Apps that use workspace tokens will continue to run for the next calendar year until October 10, 2019.

However, as you and others have pointed out, unfortunately for new users of the open source Sentry software, slack has disabled creation of new workspace token apps:

As of today, the creation of new workspace apps will be disabled.

This leaves us (and you all!) in a hard place. We can't just merge in a change to switch the slack app over to something that is working, because the working way using bot-tokens does not simply work with the current workspace app that thousands of our paying customers are using.

Again, unfortunately it doesn't seem like slack has laid out a happy-path for migrating off workspace apps and back to these old style bot tokens (that as far as I can tell, have the same problems as they always have had such as the app being removed when the installing user is removed from slack), other than the following sentence in their blog post:

If you were testing the workspace token (xoxa), we recommend that you move toward user and bot token compatibility so you can take advantage of these forthcoming improvements.

---

That said, I can say that we are actively working with slack (be it, it is not a high priority right now due to the unexpected nature of this problem) to understand what the proper migration path will be for our live sentry.io Slack app.

Re test coverage:

I can say we don't have deep test coverage specific to slack apps and their permissions, it covers more of just the functionality of the app. Merging something will definitely involving having to understand what has changed between workspace apps and whatever slack has deemed we should be migrating too.

Unfortunately we don't have more to share than that right now.

[BoLaMN]

Unfortunately this makes it nearly impossible for us to test if it will break the older workspace tokens as were unable to create workspace tokens to see if the change will affect it, thou you'll be forced to make the change as some point within the next year

[DevJackSmith]

TL;DR: Older slack plugin still works. (Thanks @EvanPurkhiser)

so not sure how long this will last or if this is related, but I was able to install this plugin in Slack and follow it's included instructions on our Sentry on-premise to configure the project settings, and it works. https://slack.com/apps/A0F814BEV-sentry-notifications

Sentry On-Premise version: 9.0.0
Didn't do anything else. Only this.
Does not use any tokens, works through webhooks.

[EvanPurkhiser]

@DevJackSmith That is the older slack plugin, which is missing any new features from the slack integration :)

[rebz]

Stuck on the chat:write as well. When attempting to use the old Slack Plugin (Sentry Notifications), after setup and then clicking _Test Plugin_ I receive the error, "There was an error with this Plugin."

At this time I have zero Slack notifications in my self-hosted Sentry. Is there a work-around?

Edit: Just re-read andrejbaran's comment. I'll try that out this weekend.

[webmozart]

I can confirm this issue - stuck on chat:write. The old integration works for me.

[Roach]

Jason (@roach) from Slack here,

We know this has been a bit of a hassle and we apologize for the Workspace Token rollback. I'm working with the team to see if there's a simple solution which works across both types (Workspace and Bot).

The problem is that in order to call chat.postMessage with a Workspace Token App, the app needs the chat:write permission scope, whereas an app using Bot tokens requires bot scope.

https://api.slack.com/methods/chat.postMessage

Going forward, it would simply be a matter of replacing all of the scopes in the identity_oauth_scopes array:

    identity_oauth_scopes = frozenset([
        'channels:read',
        'groups:read',
        'users:read',
        'chat:write',
        'links:read',
        'links:write',
        'team:read',
    ])

with the bot scope: https://api.slack.com/bot-users#methods

The problem there, is that you'd have to add a config flag to specify the app type, which is doable but I'm not sure where in Sentry's configs that would live.

[Tenzer]

Just a thought: Do the different tokens from Slack not have different prefixes? Perhaps the prefix of the token could be checked and the scopes array could be changed based on that?

If the scopes are the only difference between how the two tokens/app types work, that might be an easy way to support both.

[Roach]

@Tenzer You won't know what type of token it is until it's issued from the OAuth handshake. 😞

I've been brainstorming with the team and I there isn't a globally accepted scope for both token types. The best solution we can offer is a feature flag to set WTA vs Bot app... I'll play around with the a couple options and see what makes sense. I'll post a PR for review once I have a proof-of-concept.

[EvanPurkhiser]

@Roach will slack be providing documentation / transitionary steps to move apps back off of Workspace tokens?

Based on the blog post, it sounds like we'll have until the near end of this year to make some kind of transition (back to bot apps, to something else?). I would be good with a flag to switch between WTA and a bot app as long as at some point we're able to unify them into one thing and we can transition the sentry.io workspace app without all of our users (both on-premise and SASS) having to go through a difficult flow.

[Roach]

@EvanPurkhiser We'll provide some info on migration. Can you point me to the Slack setup portion of the docs? I haven't been able to find anything outside of the Integration setup panel in Sentry's settings 🤔

Based on the blog post, it sounds like we'll have until the near end of this year to make some kind of transition (back to bot apps, to something else?). I would be good with a flag to switch between WTA and a bot app as long as at some point we're able to unify them into one thing and we can transition the sentry.io workspace app without all of our users (both on-premise and SASS) having to go through a difficult flow.

Bot tokens (xoxb) will continue to be the canonical token type, which the bot token being deconstructed into it's component access scopes (e.g. chat:write:bot). I've tested the integration locally and it works as intended (from what I can tell with limited Sentry data) with a bot token.

[EvanPurkhiser]

There's of course this: https://docs.sentry.io/workflow/integrations/global-integrations/slack/

Then there are the old workspace app setup docs https://forum.sentry.io/t/deprecated-by-slack-how-to-configure-slack-in-your-on-prem-sentry/3463/10

[Roach]

I'm going to spend some time this week adding a SLACK_WORKSPACE_TOKEN (name tbd) flag into the Slack config for the app so they you'll be able to define what token type the app should expect on startup.

This flag will allow the app to know what set of scopes the app requires (bot vs the array outlined above) and construct the OAuth URL appropriately. This slightly changes _what_ the app's bot user has access to, but since the app is open source, your users can inspect and see what the bot is referencing on Slack. More info on the bot user scope may be found on our docs site: https://api.slack.com/bot-users

I'm going to default that flag to false, as you're unable to create new Workspace Token apps at this time. Is Sentry uses this codebase for the production/cloud version of the Sentry Slack connector, you'll need to override that flag to false. This functionality doesn't prevent new users from authing the cloud version of the service, and allows new self-hosted users to supply their Bot (xoxb) tokens and auth as expected.

[EvanPurkhiser]

Is Sentry uses this codebase for the production/cloud version of the Sentry Slack connector, you'll need to override that flag to false.

Yep, I can make that happen when we merge your work in. Thank you vey much @Roach !

[EvanPurkhiser]

13184 Implements a settings switch to use the bot apps instead of workspace apps. This will resolve this issue 🎉 🎉 🎉

[karlanke]

@EvanPurkhiser would you be willing to set up a docker tag with this fix for folks who want to set up new on premise instances? I tried to get one working based on your branch, but it looks like the slack client-id is missing (presumably secret?)

[bogdan-calapod]

Also a short explanation on how to upgrade / rebuild dockers in order to implement the settings switch would be great 🙏🏼

[vldanch]

@EvanPurkhiser Good afternoon, I have a problem with the integration with the watch in the docker, currently in the watch docker version: 9.1.1-onbuild, during the integration I get the following error: User tokens do not support scope: chat:write, I added these weak values ​​in my config.yml and slack.client-id, slack.client-secret, slack.verification-token
How to solve this problem? thank you in advance

[bogdan-calapod]

@vldanch If the error you are receiving from slack is regarding the chat scope, the fix is in a PR by @EvanPurkhiser. I can't help you with how to update the code in Docker to include this fix, but from what I can gather, we would need a Docker tag and rebuild the containers based on that tag.

[EvanPurkhiser]

We have to do some work to bring together a 9.1.2 release, which we don't yet have a timeline for, but will happen.

sentry's master branch right now relies on newer infrastructure

[vldanch]

Нам нужно проделать определенную работу, чтобы собрать релиз 9.1.2, для которого у нас пока нет графика, но это произойдет.

главный филиал часового прямо сейчас опирается на более новую инфраструктуру

good day @EvanPurkhiser why is your link to a blog not active: https://blog.sentry.io/2019/05/16/introducing-snuba-sentrys-new-search-infrastructure/

[bogdan-calapod]

We have to do some work to bring together a 9.1.2 release, which we don't yet have a timeline for, but will happen.

So we can't expect this fix until the 9.1.2 release out, is that correct ?

Just asking so we know how to plan our upgrades 😅

[analytik]

Not to be rude, but is there any reason why major infrastructure changes are done in bugfix versions? Seeing cases like this, I don't have to explain how useful following semver is.

[EvanPurkhiser]

So we can't expect this fix until the 9.1.2 release out, is that correct ?

Yes correct!

Not to be rude, but is there any reason why major infrastructure changes are done in bugfix versions? Seeing cases like this, I don't have to explain how useful following semver is.

There are no infrastructure changes being made in 9.1.2. We do follow semver.

sentry's master branch right now relies on newer infrastructure

I am just saying we have to backport the slack fix into a 9.1.2 release because master has various infrastructure changes.

[analytik]

Aaaah that makes sense. Sorry, I misunderstood.

[iamgbr]

Sorry to bother you, do you have an estimate timeline for 9.1.2?

[EvanPurkhiser]

@BYK do we have a public timeline for 9.1.2?

[BYK]

We are aiming to make it available by Monday, July 15th. No promises yet tho as we don't want to ship something broken, especially in a patch release. If anyone is courageous and have time, feel free to try out this PR/branch: https://github.com/getsentry/sentry/pull/13902

[jussirantala]

I'm getting the same error with 9.1.2

[BYK]

@EvanPurkhiser - is there any manual intervention needed for this? Generating a new token or something?

[DHoogland]

I've tried adding the parameters to config.yml, only my client_id stays empty. What do I have to do, to get the parameters working in Sentry?

slack.client-id: 'xxxx'
slack.client-secret: 'xxxx'
slack.verification-token: 'xxxx'

[karlanke]

I've been setting them as environment variables - SENTRY_SLACK_CLIENT_ID, SENTRY_SLACK_CLIENT_SECRET, and SENTRY_SLACK_VERIFICATION_TOKEN

That seems to be working for me.

[DHoogland]

I've been setting them as environment variables - SENTRY_SLACK_CLIENT_ID, SENTRY_SLACK_CLIENT_SECRET, and SENTRY_SLACK_VERIFICATION_TOKEN

That seems to be working for me.

Awesome! That seems to work, now I ran into a "Invalid request." message.
We'll figure it out. Thanks for the tips!

Edit: It was the root url, changed it via the ‘sentry config’ command. system.url-prefix is the option name for that. (I'm using a nginx proxy for SSL)

[EvanPurkhiser]

Oh strange, the config.yml should work just fine as well. You're restarting your server after configuring these correct?