Sentry: SnubaError: HTTPConnectionPool(host='localhost', port=1218): Max retries exceeded with url: /query

[bushiquanquan]

Important Details

How are you running Sentry?

• [ ] On-Premise docker [Version xyz]
• [ ] Saas (sentry.io)
• [√ ] Other [master source code]

Description

[What happened]

Steps to Reproduce

when response for url http://xxxx:9000/organizations/sentry/issues/?project=2， error occurs

Good items to include here include:

• Include a stacktrace or other logs when relevant
• Include a redacted version of your configuration when relevant
  Traceback (most recent call last):
  File "/root/sentry/src/sentry/api/base.py", line 90, in handle_exception
  response = super(Endpoint, self).handle_exception(exc)
  File "/root/sentry/src/sentry/api/base.py", line 190, in dispatch
  response = handler(request, *args, kwargs)
  File "/root/sentry/src/sentry/api/endpoints/organization_group_index.py", line 159, in get
  'date_from': start,
  File "/root/sentry/src/sentry/api/endpoints/organization_group_index.py", line 40, in _search
  result = search.query(query_kwargs)
  File "/root/sentry/src/sentry/search/snuba/backend.py", line 293, in query
  search_filters, date_from, date_to,
  File "/root/sentry/src/sentry/search/snuba/backend.py", line 524, in _query
  search_filters=search_filters,
  File "/root/sentry/src/sentry/search/snuba/backend.py", line 688, in snuba_search
  sample=1, # Don't use clickhouse sampling, even when in turbo mode.
  File "/root/sentry/src/sentry/utils/snuba.py", line 592, in raw_query
  raise SnubaError(err)

What you expected to happen

[What you think should be happening]

Possible Solution

[If you have an idea on how this could be solved include that detail here.]

[markstory]

What version of sentry are you using? The current state of master is not ready for public consumption in an on-premise deployment. You would need to stay on the 9.1.1 tag.

[dreamyeah]

9.1.1 tag also occurs this error for me.

[dreamyeah]

[9.1.1 tag also occurs this error for me.]
I found the reason.
I config the following in sentry.conf.py, remove this config then can run success.
SENTRY_SEARCH = 'sentry.search.snuba.SnubaSearchBackend'
SENTRY_TAGSTORE = 'sentry.tagstore.snuba.SnubaCompatibilityTagStorage'
SENTRY_TSDB = 'sentry.tsdb.redissnuba.RedisSnubaTSDB'
SENTRY_EVENTSTREAM = 'sentry.eventstream.snuba.SnubaEventStream'

[bushiquanquan]

[9.1.1 tag also occurs this error for me.]
I found the reason.
I config the following in sentry.conf.py, remove this config then can run success.
SENTRY_SEARCH = 'sentry.search.snuba.SnubaSearchBackend'
SENTRY_TAGSTORE = 'sentry.tagstore.snuba.SnubaCompatibilityTagStorage'
SENTRY_TSDB = 'sentry.tsdb.redissnuba.RedisSnubaTSDB'
SENTRY_EVENTSTREAM = 'sentry.eventstream.snuba.SnubaEventStream'

and then... can events be saved normally? no any function performs anomalously？

[lucas-zimerman]

those vars shouldnt be used with 9.1 but more with 10, if im sure

[bushiquanquan]

those vars shouldnt be used with 9.1 but more with 10, if im sure

yes, how can i fix it ?

[bushiquanquan]

What version of sentry are you using? The current state of master is not ready for public consumption in an on-premise deployment. You would need to stay on the 9.1.1 tag.

SnubaError: None: Max retries exceeded with url: /tests/eventstream (Caused by None)
06:24:17 [ERROR] celery.worker.job: Task sentry.tasks.store.save_event[918146c9-e9ea-48c3-bbfa-9b00b709efb1] raised unexpected: SnubaError(MaxRetryError('None: Max retries exceeded with url: /tests/eventstream (Caused by None)',),) (data={u'hostname': 'celery@instance-l327lsty', u'name': 'sentry.tasks.store.save_event', u'args': '[]', u'internal': False, u'kwargs': "{'event_id': u'bed5aeaa1f934dc2b743274ba8e308e0', 'cache_key': u'e:bed5aeaa1f934dc2b743274ba8e308e0:1', 'start_time': 1560925454.509667, 'project_id': 1L, 'data': None}", u'id': '918146c9-e9ea-48c3-bbfa-9b00b709efb1'})
Traceback (most recent call last):
File "/www/sentry/local/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
R = retval = fun(args, *kwargs)
File "/www/sentry/local/lib/python2.7/site-packages/celery/app/trace.py", line 438, in __protected_call__
return self.run(args, *kwargs)
File "/www/sentry/local/lib/python2.7/site-packages/sentry_sdk/integrations/celery.py", line 118, in _inner
reraise(exc_info)
File "/www/sentry/local/lib/python2.7/site-packages/sentry_sdk/integrations/celery.py", line 113, in _inner
return f(args, *kwargs)
File "/root/sentryperf/src/sentry/tasks/base.py", line 54, in _wrapped
result = func(args, *kwargs)
File "/root/sentryperf/src/sentry/tasks/store.py", line 546, in save_event
_do_save_event(cache_key, data, start_time, event_id, project_id, *kwargs)
File "/root/sentryperf/src/sentry/tasks/store.py", line 482, in _do_save_event
event = manager.save(project_id, assume_normalized=True)
File "/root/sentryperf/src/sentry/event_manager.py", line 968, in save
skip_consume=raw,
File "/root/sentryperf/src/sentry/utils/services.py", line 91, in
context[key] = (lambda f: lambda a, *k: getattr(self, f)(a, *k))(key)
File "/root/sentryperf/src/sentry/eventstream/snuba.py", line 268, in insert
primary_hash, skip_consume)
File "/root/sentryperf/src/sentry/eventstream/snuba.py", line 116, in insert
'skip_consume': skip_consume,
File "/root/sentryperf/src/sentry/eventstream/snuba.py", line 259, in _send
raise snuba.SnubaError(err)

[bushiquanquan]

What version of sentry are you using? The current state of master is not ready for public consumption in an on-premise deployment. You would need to stay on the 9.1.1 tag.
can u help me😭

[IvanBabushkin]

I also have this error on version 9.1.2 when i try merge more than one page errors.
Please help fix it.

[githubmann]

hey, man, how to fix it, please help me

[githubmann]

hey, man, how to fix it, please help me

please, man, beg you.

[trevorphillipscoding]

I'm in the same boat as @IvanBabushkin . I'm on 9.1.2 and it doesn't allow me to resolve/merge more than one page of issues.

[bkilchynskyi]

I also have this problem. Did upgrade database from sentry 9.1.2 to 10 version (installed sentry from PIP).
File "/srv/sentry/local/lib/python2.7/site-packages/sentry/migrations/0024_auto_20191230_2052.py", line 71, in backfill_eventstream skip_consume=True, File "/srv/sentry/local/lib/python2.7/site-packages/sentry/utils/services.py", line 104, in <lambda> context[key] = (lambda f: lambda *a, **k: getattr(self, f)(*a, **k))(key) File "/srv/sentry/local/lib/python2.7/site-packages/sentry/eventstream/snuba.py", line 261, in insert skip_consume, File "/srv/sentry/local/lib/python2.7/site-packages/sentry/eventstream/snuba.py", line 125, in insert "skip_consume": skip_consume, File "/srv/sentry/local/lib/python2.7/site-packages/sentry/eventstream/snuba.py", line 239, in _send raise snuba.SnubaError(err) sentry.utils.snuba.SnubaError: HTTPConnectionPool(host='localhost', port=1218): Max retries exceeded with url: /tests/events/eventstream (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f4618cd00d0>: Failed to establish a new connection: [Errno 111] Connection refused',))

Plese help me whit this :)

[heya0817]

I have the same problem when I jumped from 9.1.2 to 10. Can someone help with this? Thanks.

[klingac]

Can you check logs (primary snuba-api service)? I had similar error once.
I can't remember, but I think that problem was in inconsistency of clickhouse tables or sth. After runnig docker-compose run snuba-api bootstrap --force was DB repaired and now it works.

[m0n5t3r]

sorry to resurrect this ticket, but this is happening on 9.1.2 (the docker version) too, because grouping events goes to https://github.com/getsentry/sentry/blob/9.1.2/src/sentry/api/endpoints/organization_group_index.py, which, as you can see, uses snuba; https://github.com/getsentry/sentry/blob/9.1.2/src/sentry/api/endpoints/group_events.py looks at options and should work, but there isn't an obvious way to make sentry use it :-/