Sentry: [ERROR] celery.worker.job: Task sentry.tasks.store.save_event[c092365d-609b-454a-a66f-2b7039fb44d8] raised unexpected

[Fabriceli]

I build sentry and run it with docker, and i can create my project in my sentry website, but i can not get the events.
SENTRY: 9.0.0
OS: centos7
DOCKER: 1.13.1
DOCKER-COMPOSE: 1.22.0
PYTHON: 2.7.5
PIP: 18.0

I get the logs of the workers:

11:31:20 [ERROR] celery.worker.job: Task sentry.tasks.store.save_event[c092365d-609b-454a-a66f-2b7039fb44d8] raised unexpected: ProgrammingError("ProgrammingError('function sentry_increment_project_counter(integer, integer) does not exist\\nLINE 2: select sentry_increment_project_counter(1, 1...\\n ^\\nHINT: No function matches the given name and argument types. You might need to add explicit type casts.\\n',)\nSQL: \n select sentry_increment_project_counter(%s, %s)\n ",) (data={u'hostname': 'celery@0cbe655e887a', u'name': 'sentry.tasks.store.save_event', u'args': '[]', u'internal': False, u'kwargs': "{'event_id': u'ed0b17bd7f4341f6bcc7063c1c380257', 'cache_key': 'e:ed0b17bd7f4341f6bcc7063c1c380257:1', 'start_time': 1533123080.075781, 'project_id': 1, 'data': None}", u'id': 'c092365d-609b-454a-a66f-2b7039fb44d8'}) Traceback (most recent call last): File "/usr/local/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task R = retval = fun(*args, **kwargs) File "/usr/local/lib/python2.7/site-packages/celery/app/trace.py", line 438, in __protected_call__ return self.run(*args, **kwargs) File "/usr/local/lib/python2.7/site-packages/sentry/tasks/base.py", line 54, in _wrapped result = func(*args, **kwargs) File "/usr/local/lib/python2.7/site-packages/sentry/tasks/store.py", line 332, in save_event manager.save(project_id) File "/usr/local/lib/python2.7/site-packages/sentry/event_manager.py", line 760, in save event=event, hashes=hashes, release=release, **group_kwargs File "/usr/local/lib/python2.7/site-packages/sentry/event_manager.py", line 1080, in _save_aggregate short_id = project.next_short_id() File "/usr/local/lib/python2.7/site-packages/sentry/models/project.py", line 133, in next_short_id return Counter.increment(self) File "/usr/local/lib/python2.7/site-packages/sentry/models/counter.py", line 33, in increment return increment_project_counter(project, delta) File "/usr/local/lib/python2.7/site-packages/sentry/models/counter.py", line 47, in increment_project_counter ''', [project.id, delta] File "/usr/local/lib/python2.7/site-packages/raven/contrib/django/client.py", line 123, in execute return real_execute(self, sql, params) File "/usr/local/lib/python2.7/site-packages/django/db/backends/util.py", line 53, in execute return self.cursor.execute(sql, params) File "/usr/local/lib/python2.7/site-packages/django/db/utils.py", line 99, in __exit__ six.reraise(dj_exc_type, dj_exc_value, traceback) File "/usr/local/lib/python2.7/site-packages/django/db/backends/util.py", line 53, in execute return self.cursor.execute(sql, params) File "/usr/local/lib/python2.7/site-packages/sentry/db/postgres/decorators.py", line 80, in inner raise_the_exception(self.db, e) File "/usr/local/lib/python2.7/site-packages/sentry/db/postgres/decorators.py", line 78, in inner return func(self, *args, **kwargs) File "/usr/local/lib/python2.7/site-packages/sentry/db/postgres/decorators.py", line 22, in inner return func(self, *args, **kwargs) File "/usr/local/lib/python2.7/site-packages/sentry/db/postgres/decorators.py", line 101, in inner six.reraise(exc_info[0], exc_info[0](msg), exc_info[2]) File "/usr/local/lib/python2.7/site-packages/sentry/db/postgres/decorators.py", line 94, in inner return func(self, sql, *args, **kwargs) File "/usr/local/lib/python2.7/site-packages/sentry/db/postgres/base.py", line 47, in execute return self.cursor.execute(sql, params) ProgrammingError: ProgrammingError('function sentry_increment_project_counter(integer, integer) does not exist\nLINE 2: select sentry_increment_project_counter(1, 1...\n ^\nHINT: No function matches the given name and argument types. You might need to add explicit type casts.\n',) SQL: select sentry_increment_project_counter(%s, %s)

Anyone can help me, thanks a lot

[Fabriceli]

Well, I fixed this problem, it is not easy.

1. run this command and get in the postgres docker
   docker exec -it onpremise_postgres_1 bash

2. get in the postgre database
   psql -h 127.0.0.1 -d postgres -U postgres

3. run this part program:
   create or replace function sentry_increment_project_counter( project bigint, delta int) returns int as $$ declare new_val int; begin loop update sentry_projectcounter set value = value + delta where project_id = project returning value into new_val; if found then return new_val; end if; begin insert into sentry_projectcounter(project_id, value) values (project, delta) returning value into new_val; return new_val; exception when unique_violation then end; end loop; end $$ language plpgsql;

Then it works,

Thanks

[bushiquanquan]

i have one same problem,can u help me?
[ERROR] celery.worker.job: Task sentry.tasks.store.save_event[7555b03c-437a-4568-8c6e-8ce55c644f6f] raised unexpected: SnubaError(MaxRetryError('None: Max retries exceeded with url: /tests/eventstream (Caused by None)',),) (data={u'hostname': '[email protected]', u'name': 'sentry.tasks.store.save_event', u'args': '[]', u'internal': False, u'kwargs': "{'event_id': u'ffa575a4bf4146e3a47e7ac4132c80af', 'cache_key': u'e:ffa575a4bf4146e3a47e7ac4132c80af:1', 'start_time': 1557136137.281915, 'project_id': 1L, 'data': None}", u'id': '7555b03c-437a-4568-8c6e-8ce55c644f6f'}) Restoring 8 unacknowledged message(s). 09:48:59 [INFO] sentry.bgtasks: bgtask.stop (task_name=u'sentry.bgtasks.clean_dsymcache:clean_dsymcache')

[bushiquanquan]

Well, I fixed this problem, it is not easy.

1. run this command and get in the postgres docker
   docker exec -it onpremise_postgres_1 bash
2. get in the postgre database
   psql -h 127.0.0.1 -d postgres -U postgres
3. run this part program:
   create or replace function sentry_increment_project_counter( project bigint, delta int) returns int as $$ declare new_val int; begin loop update sentry_projectcounter set value = value + delta where project_id = project returning value into new_val; if found then return new_val; end if; begin insert into sentry_projectcounter(project_id, value) values (project, delta) returning value into new_val; return new_val; exception when unique_violation then end; end loop; end $$ language plpgsql;

Then it works,

Thanks
i have one same problem,can u help me?
[ERROR] celery.worker.job: Task sentry.tasks.store.save_event[7555b03c-437a-4568-8c6e-8ce55c644f6f] raised unexpected: SnubaError(MaxRetryError('None: Max retries exceeded with url: /tests/eventstream (Caused by None)',),) (data={u'hostname': '[email protected]', u'name': 'sentry.tasks.store.save_event', u'args': '[]', u'internal': False, u'kwargs': "{'event_id': u'ffa575a4bf4146e3a47e7ac4132c80af', 'cache_key': u'e:ffa575a4bf4146e3a47e7ac4132c80af:1', 'start_time': 1557136137.281915, 'project_id': 1L, 'data': None}", u'id': '7555b03c-437a-4568-8c6e-8ce55c644f6f'}) Restoring 8 unacknowledged message(s). 09:48:59 [INFO] sentry.bgtasks: bgtask.stop (task_name=u'sentry.bgtasks.clean_dsymcache:clean_dsymcache')

[Fabriceli]

Well, I fixed this problem, it is not easy.

1. run this command and get in the postgres docker
   docker exec -it onpremise_postgres_1 bash
2. get in the postgre database
   psql -h 127.0.0.1 -d postgres -U postgres
3. run this part program:
   create or replace function sentry_increment_project_counter( project bigint, delta int) returns int as $$ declare new_val int; begin loop update sentry_projectcounter set value = value + delta where project_id = project returning value into new_val; if found then return new_val; end if; begin insert into sentry_projectcounter(project_id, value) values (project, delta) returning value into new_val; return new_val; exception when unique_violation then end; end loop; end $$ language plpgsql;

Then it works,
Thanks
i have one same problem,can u help me?
[ERROR] celery.worker.job: Task sentry.tasks.store.save_event[7555b03c-437a-4568-8c6e-8ce55c644f6f] raised unexpected: SnubaError(MaxRetryError('None: Max retries exceeded with url: /tests/eventstream (Caused by None)',),) (data={u'hostname': '[email protected]', u'name': 'sentry.tasks.store.save_event', u'args': '[]', u'internal': False, u'kwargs': "{'event_id': u'ffa575a4bf4146e3a47e7ac4132c80af', 'cache_key': u'e:ffa575a4bf4146e3a47e7ac4132c80af:1', 'start_time': 1557136137.281915, 'project_id': 1L, 'data': None}", u'id': '7555b03c-437a-4568-8c6e-8ce55c644f6f'}) Restoring 8 unacknowledged message(s). 09:48:59 [INFO] sentry.bgtasks: bgtask.stop (task_name=u'sentry.bgtasks.clean_dsymcache:clean_dsymcache')

you can follow the steps, and which step you dont understand?

[bushiquanquan]

Well, I fixed this problem, it is not easy.

1. run this command and get in the postgres docker
   docker exec -it onpremise_postgres_1 bash
2. get in the postgre database
   psql -h 127.0.0.1 -d postgres -U postgres
3. run this part program:
   create or replace function sentry_increment_project_counter( project bigint, delta int) returns int as $$ declare new_val int; begin loop update sentry_projectcounter set value = value + delta where project_id = project returning value into new_val; if found then return new_val; end if; begin insert into sentry_projectcounter(project_id, value) values (project, delta) returning value into new_val; return new_val; exception when unique_violation then end; end loop; end $$ language plpgsql;

Then it works,
Thanks
i have one same problem,can u help me?
[ERROR] celery.worker.job: Task sentry.tasks.store.save_event[7555b03c-437a-4568-8c6e-8ce55c644f6f] raised unexpected: SnubaError(MaxRetryError('None: Max retries exceeded with url: /tests/eventstream (Caused by None)',),) (data={u'hostname': '[email protected]', u'name': 'sentry.tasks.store.save_event', u'args': '[]', u'internal': False, u'kwargs': "{'event_id': u'ffa575a4bf4146e3a47e7ac4132c80af', 'cache_key': u'e:ffa575a4bf4146e3a47e7ac4132c80af:1', 'start_time': 1557136137.281915, 'project_id': 1L, 'data': None}", u'id': '7555b03c-437a-4568-8c6e-8ce55c644f6f'}) Restoring 8 unacknowledged message(s). 09:48:59 [INFO] sentry.bgtasks: bgtask.stop (task_name=u'sentry.bgtasks.clean_dsymcache:clean_dsymcache')

you can follow the steps, and which step you dont understand?

i followed the steps, but unused . i think it is not the same question.

[bushiquanquan]

or mine is nothing to do with db？and i installed with source code, not with docker

[bushiquanquan]

when run command " SENTRY_CONF=/etc/sentry sentry run worker", error occours like above, "celery.worker.job: Task sentry.tasks.store.save_event" repeat again and agian

[jmolbach]

Any general solution on this? I experience the same behaviour.

[rafalandryanczyk]

I have the same issue after fresh installation with source code.

[Perlmint]

I experienced the same issue - SnubaError(MaxRetryError('None: Max retries exceeded with url: /tests/eventstream (Caused by None)',),).
The reason was that snuba is not running, but SENTRY_EVENTSTREAM is snuba in default.

I resolved this error with running snuba and clickhouse. docker-compose.yml for snuba is helpful to me.

[BYK]

Alright, I'm going to close this as it seems to be related with some DB configuration and this issue became a dumping ground for random errors. The best way to avoid these issues is by using our official on-premise repository for your installations and not jumping Sentry versions while upgrading. If you have similar issues, please use our forums as the issue board is only for the core Sentry code-related issues.

[STRATZ-Ken]

@Perlmint Can you explain what you did in detail? I am having the same issue, and don't wish to use Docker.

Thanks,

[Perlmint]

@STRATZ-Ken I simply run snuba along with my sentry server - I used docker and docker-compose.

I have never tried to run snuba without docker. So I can't help how to run snuba without docker. Anyway, run snuba on http://127.0.0.1:1218 (This is default config value of sentry).

[githubmann]

how to run snuba, i crashed

[BYK]

@githubmann you can find instructions around how to run a local version of Sentry (including Snuba) at https://docs.sentry.io/server/.