Onpremise: Error when trying to access the github integration settings
An error happens when trying to open the github integration settings and it wont work. I just noticed this today and it was working before, so I don't know when it started happening.
Here are more details regarding the internal error: /settings/0soft/integrations/github/1/
It's worth noting that I'm using SENTRY_PYTHON3=1 for that deployment, which might point to a python3 problem (but may be related to something else).
I noticed that I was receiving since last week some emails saying that "sentry was unable to fetch commits" from some deploys that I was making. That might be also related to the issue.
bellini666
All 11 comments
Hi @bellini666 and thanks for reporting this! Can you please provide your docker-compose logs as requested in the issue template as without them all we can do is speculate.
BYK
on 21 Oct 2020
@BYK sure, here it is the whole log of when I opened the page itself:
nginx_1 | 192.168.64.8 - - [21/Oct/2020:17:32:57 +0000] "GET /settings/blbventures/integrations/github/1/ HTTP/1.1" 200 5221 "https://sentry.blb.ventures/organizations/blbventures/issues/?project=4&project=2" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.75 Safari/537.36" "201.95.167.119"
nginx_1 | 192.168.64.8 - - [21/Oct/2020:17:32:58 +0000] "GET /api/0/assistant/?v2 HTTP/1.1" 200 108 "https://sentry.blb.ventures/settings/blbventures/integrations/github/1/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.75 Safari/537.36" "201.95.167.119"
nginx_1 | 192.168.64.8 - - [21/Oct/2020:17:32:58 +0000] "GET /api/0/organizations/?member=1 HTTP/1.1" 200 729 "https://sentry.blb.ventures/settings/blbventures/integrations/github/1/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.75 Safari/537.36" "201.95.167.119"
nginx_1 | 192.168.64.8 - - [21/Oct/2020:17:32:58 +0000] "GET /api/0/internal/health/ HTTP/1.1" 200 0 "https://sentry.blb.ventures/settings/blbventures/integrations/github/1/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.75 Safari/537.36" "201.95.167.119"
nginx_1 | 192.168.64.8 - - [21/Oct/2020:17:32:58 +0000] "GET /_static/1603224574/sentry/images/favicon.png HTTP/1.1" 200 998 "https://sentry.blb.ventures/settings/blbventures/integrations/github/1/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.75 Safari/537.36" "201.95.167.119"
nginx_1 | 192.168.64.8 - - [21/Oct/2020:17:32:58 +0000] "POST /api/1/envelope/?sentry_key=752ca61777084ee580c575f1ff80910c&sentry_version=7 HTTP/1.1" 200 2 "https://sentry.blb.ventures/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.75 Safari/537.36" "201.95.167.119"
nginx_1 | 192.168.64.8 - - [21/Oct/2020:17:32:58 +0000] "GET /api/0/organizations/blbventures/?detailed=1 HTTP/1.1" 200 12996 "https://sentry.blb.ventures/settings/blbventures/integrations/github/1/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.75 Safari/537.36" "201.95.167.119"
nginx_1 | 192.168.64.8 - - [21/Oct/2020:17:32:58 +0000] "GET /api/0/organizations/blbventures/broadcasts/ HTTP/1.1" 200 2 "https://sentry.blb.ventures/settings/blbventures/integrations/github/1/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.75 Safari/537.36" "201.95.167.119"
nginx_1 | 192.168.64.8 - - [21/Oct/2020:17:32:58 +0000] "GET /api/0/organizations/blbventures/integrations/1/ HTTP/1.1" 200 364 "https://sentry.blb.ventures/settings/blbventures/integrations/github/1/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.75 Safari/537.36" "201.95.167.119"
nginx_1 | 192.168.64.8 - - [21/Oct/2020:17:32:58 +0000] "GET /api/0/organizations/blbventures/config/integrations/ HTTP/1.1" 200 17642 "https://sentry.blb.ventures/settings/blbventures/integrations/github/1/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.75 Safari/537.36" "201.95.167.119"
nginx_1 | 192.168.64.8 - - [21/Oct/2020:17:32:59 +0000] "GET /api/0/organizations/blbventures/issues/?limit=25&project=2&query=is%3Aunresolved&shortIdLookup=1&statsPeriod=14d&cursor=1603299441000:0:1 HTTP/1.1" 200 2 "https://sentry.blb.ventures/organizations/blbventures/issues/?project=4&project=2" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.75 Safari/537.36" "201.95.167.119"
nginx_1 | 192.168.64.8 - - [21/Oct/2020:17:32:59 +0000] "GET /api/0/organizations/blbventures/repos/?status= HTTP/1.1" 200 817 "https://sentry.blb.ventures/settings/blbventures/integrations/github/1/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.75 Safari/537.36" "201.95.167.119"
web_1 | 17:32:59 [INFO] sentry.integrations.client: integration.http_response (integration='github' status_string='401' error='401 Client Error: Unauthorized for url: https://api.github.com/app/installations/11300898/access_tokens')
snuba-sessions-consumer_1 | 2020-10-21 17:32:59,419 Completed processing <Batch: 1 message, open for 1.03 seconds>.
web_1 | Traceback (most recent call last):
web_1 | File "/usr/local/lib/python3.6/site-packages/sentry/shared_integrations/client.py", line 241, in _request
web_1 | resp.raise_for_status()
web_1 | File "/usr/local/lib/python3.6/site-packages/requests/models.py", line 940, in raise_for_status
web_1 | raise HTTPError(http_error_msg, response=self)
web_1 | requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: https://api.github.com/app/installations/11300898/access_tokens
web_1 |
web_1 | During handling of the above exception, another exception occurred:
web_1 |
web_1 | Traceback (most recent call last):
web_1 | File "/usr/local/lib/python3.6/site-packages/sentry/api/base.py", line 124, in handle_exception
web_1 | response = super(Endpoint, self).handle_exception(exc)
web_1 | File "/usr/local/lib/python3.6/site-packages/rest_framework/views.py", line 449, in handle_exception
web_1 | self.raise_uncaught_exception(exc)
web_1 | File "/usr/local/lib/python3.6/site-packages/sentry/api/base.py", line 237, in dispatch
web_1 | response = handler(request, *args, **kwargs)
web_1 | File "/usr/local/lib/python3.6/site-packages/sentry/api/endpoints/organization_integration_repos.py", line 36, in get
web_1 | repositories = install.get_repositories(request.GET.get("search"))
web_1 | File "/usr/local/lib/python3.6/site-packages/sentry/integrations/github/integration.py", line 89, in get_repositories
web_1 | for i in self.get_client().get_repositories()
web_1 | File "/usr/local/lib/python3.6/site-packages/sentry/integrations/github/client.py", line 42, in get_repositories
web_1 | repositories = self.get("/installation/repositories", params={"per_page": 100})
web_1 | File "/usr/local/lib/python3.6/site-packages/sentry/shared_integrations/client.py", line 286, in get
web_1 | return self.request("GET", *args, **kwargs)
web_1 | File "/usr/local/lib/python3.6/site-packages/sentry/integrations/github/client.py", line 75, in request
web_1 | "Authorization": "token %s" % self.get_token(),
web_1 | File "/usr/local/lib/python3.6/site-packages/sentry/integrations/github/client.py", line 94, in get_token
web_1 | res = self.create_token()
web_1 | File "/usr/local/lib/python3.6/site-packages/sentry/integrations/github/client.py", line 111, in create_token
web_1 | "Accept": "application/vnd.github.machine-man-preview+json",
web_1 | File "/usr/local/lib/python3.6/site-packages/sentry/shared_integrations/client.py", line 292, in post
web_1 | return self.request("POST", *args, **kwargs)
web_1 | File "/usr/local/lib/python3.6/site-packages/sentry/integrations/github/client.py", line 79, in request
web_1 | return self._request(method, path, headers=headers, data=data, params=params)
web_1 | File "/usr/local/lib/python3.6/site-packages/sentry/shared_integrations/client.py", line 257, in _request
web_1 | raise ApiError.from_response(resp, url=full_url)
web_1 | sentry.shared_integrations.exceptions.ApiUnauthorized: {"message":"A JSON web token could not be decoded","documentation_url":"https://docs.github.com/rest"}
nginx_1 | 192.168.64.8 - - [21/Oct/2020:17:32:59 +0000] "GET /api/0/organizations/blbventures/integrations/1/repos/ HTTP/1.1" 500 72 "https://sentry.blb.ventures/settings/blbventures/integrations/github/1/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.75 Safari/537.36" "201.95.167.119"
nginx_1 | 192.168.64.8 - - [21/Oct/2020:17:32:59 +0000] "POST /api/1/store/ HTTP/1.1" 200 41 "-" "sentry.python/0.17.8" "192.168.64.1"
nginx_1 | 2020/10/21 17:32:59 [warn] 6#6: *4773 a client request body is buffered to a temporary file /var/cache/nginx/client_temp/0000000023, client: 192.168.64.8, server: , request: "POST /api/1/events/1c93519fe7ec449b8da8eda4a0113e48/attachments/?sentry_key=752ca61777084ee580c575f1ff80910c&sentry_version=7&sentry_client=rrweb HTTP/1.1", host: "sentry.blb.ventures", referrer: "https://sentry.blb.ventures/settings/blbventures/integrations/github/1/"
nginx_1 | 192.168.64.8 - - [21/Oct/2020:17:33:00 +0000] "POST /api/1/store/?sentry_key=752ca61777084ee580c575f1ff80910c&sentry_version=7 HTTP/1.1" 200 41 "https://sentry.blb.ventures/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.75 Safari/537.36" "201.95.167.119"
nginx_1 | 192.168.64.8 - - [21/Oct/2020:17:33:00 +0000] "POST /api/1/events/1c93519fe7ec449b8da8eda4a0113e48/attachments/?sentry_key=752ca61777084ee580c575f1ff80910c&sentry_version=7&sentry_client=rrweb HTTP/1.1" 201 0 "https://sentry.blb.ventures/settings/blbventures/integrations/github/1/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.75 Safari/537.36" "201.95.167.119"
nginx_1 | 192.168.64.8 - - [21/Oct/2020:17:33:00 +0000] "GET /api/embed/error-page/?dsn=https://[email protected]/1&eventId=1c93519fe7ec449b8da8eda4a0113e48&dsn=https%3A%2F%2F752ca61777084ee580c575f1ff80910c%40sentry.blb.ventures%2F1 HTTP/1.1" 200 15256 "https://sentry.blb.ventures/settings/blbventures/integrations/github/1/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.75 Safari/537.36" "201.95.167.119"
ingest-consumer_1 | 17:33:01 [INFO] batching-kafka-consumer: Flushing 4 items (from {('ingest-events', 0): [581805, 581806], ('ingest-attachments', 0): [6, 7]}): forced:False size:False time:True
ingest-consumer_1 | 17:33:01 [INFO] batching-kafka-consumer: Worker flush took 340ms
nginx_1 | 192.168.64.8 - - [21/Oct/2020:17:33:01 +0000] "GET /_static/1603224574/sentry/dist/vendor.js.map HTTP/1.1" 200 1667328 "-" "sentry/20.10.1 (https://sentry.io)" "192.168.64.1"
snuba-transactions-consumer_1 | 2020-10-21 17:33:02,324 Completed processing <Batch: 1 message, open for 0.82 seconds>.
snuba-outcomes-consumer_1 | 2020-10-21 17:33:02,337 Completed processing <Batch: 2 messages, open for 0.83 seconds>.
snuba-consumer_1 | 2020-10-21 17:33:02,357 Completed processing <Batch: 1 message, open for 0.85 seconds>.
snuba-transactions-consumer_1 | 2020-10-21 17:33:03,328 Completed processing <Batch: 1 message, open for 1.00 seconds>.
snuba-outcomes-consumer_1 | 2020-10-21 17:33:03,350 Completed processing <Batch: 1 message, open for 1.01 seconds>.
snuba-consumer_1 | 2020-10-21 17:33:03,380 Completed processing <Batch: 1 message, open for 1.02 seconds>.
redis_1 | 1:M 21 Oct 2020 17:33:06.074 * 100 changes in 300 seconds. Saving...
redis_1 | 1:M 21 Oct 2020 17:33:06.076 * Background saving started by pid 268
redis_1 | 268:C 21 Oct 2020 17:33:06.111 * DB saved on disk
redis_1 | 268:C 21 Oct 2020 17:33:06.112 * RDB: 0 MB of memory used by copy-on-write
redis_1 | 1:M 21 Oct 2020 17:33:06.176 * Background saving terminated with success
nginx_1 | 192.168.64.8 - - [21/Oct/2020:17:33:10 +0000] "POST /api/2/envelope/ HTTP/1.1" 200 41 "-" "sentry.python/0.19.0" "161.35.186.66"
ingest-consumer_1 | 17:33:11 [INFO] batching-kafka-consumer: Flushing 1 items (from {('ingest-transactions', 0): [49781, 49781]}): forced:False size:False time:True
ingest-consumer_1 | 17:33:11 [INFO] batching-kafka-consumer: Worker flush took 4ms
snuba-outcomes-consumer_1 | 2020-10-21 17:33:12,315 Completed processing <Batch: 1 message, open for 1.00 seconds>.
snuba-consumer_1 | 2020-10-21 17:33:12,337 Completed processing <Batch: 1 message, open for 1.03 seconds>.
snuba-transactions-consumer_1 | 2020-10-21 17:33:12,339 Completed processing <Batch: 1 message, open for 1.03 seconds>.
Also, I submitted a crash report presented to me there just now so probably there'll be more info to you in your sentry installation also.
Do you need anything more?
bellini666
on 21 Oct 2020
@BYK also, I just noticed why you asked for more info. For some reason the I pasted the wrong url in my original comment about the internal issue I shared from my installation. Here it is again: https://sentry.blb.ventures/share/issue/db24c378bb2a498c95e7d19c16b5bc57/
bellini666
on 21 Oct 2020
@bellini666 - bingo! I think you just caught a bug in our Py3 version. Thanks for reporting again. A fix will be out soon 馃檪
BYK
on 21 Oct 2020
@BYK awesome! Tell me when it is out so I can update my installation and report back if it is solved.
bellini666
on 21 Oct 2020
@bellini666 once we merge a fix I'll reply here with the SHA it was merged so you can try it by running
SENTRY_PYTHON3=1 SENTRY_IMAGE=getsentry/sentry:<SHA> ./install.sh
Hi @bellini666 - can you try with the new image below:
PLEASE NOTE: This is a nightly build and has a backward-incompatible change for in-flight events so if your instance is having a lot of events, try this on a back up or somewhere else.
SENTRY_IMAGE=us.gcr.io/sentryio/sentry:84f47fa1d885e1821d5ff183e60ff0109ff45fce SENTRY_PYTHON3=1 ./install.sh
Hi @BYK ,
I confirm that that build fixes the issue!
I'm going to keep the installation running on that image since it has backward-incompatible changes (yeah, I deployed in production. I like to live dangerously =P). When can I ./install.sh again without the SENTRY_IMAGE making sure that it will have the changes there?
bellini666
on 22 Oct 2020
I confirm that that build fixes the issue!
Great, thanks for testing so quickly!
When can I ./install.sh again without the SENTRY_IMAGE making sure that it will have the changes there?
You'd ideally move on to a "stable" version where the next one comes out on November 16th. But once this patch is merged and built on master, you can run ./install.sh which will pull the latest nightly unless you are on a release branch/tag. Makes sense?
BYK
on 22 Oct 2020
You'd ideally move on to a "stable" version where the next one comes out on November 16th. But once this patch is merged and built on master, you can run ./install.sh which will pull the latest nightly unless you are on a release branch/tag. Makes sense?
Yeah, I'm using the default config for the release so I'm indeed using the nightlies.
I just saw that the commit was merged that. I think the proper question would be: How can I check the version of the latest commit built on master that would be used for the nightly build? Or even more simpler, how long does it take usually to build a new version after a commit is pushed to master?
bellini666
on 22 Oct 2020
Or even more simpler, how long does it take usually to build a new version after a commit is pushed to master?
Typically around 15 minutes. I've also checked the build and it is indeed pushed.
A surefire way is to search for the commit SHA you are interested in here: https://hub.docker.com/r/getsentry/sentry/tags
If it is there, it is on nightly 馃槈
BYK
on 22 Oct 2020
Related issues
adrielliu
路
3Comments
giggsey
路
3Comments
nature1995
路
4Comments
marbon87
路
5Comments
dotconnor
路
6Comments