Onpremise: One of docker container restarting all the time from the fresh sentry install

[nuget1991]

Version Information

Sentry 20.11.0eb42277

Description

Docker container sentry_onpremise_snuba-subscription-consumer-events_1 restarting all the time. Into logs I am looking into repeating messages:

New partitions assigned: {Partition(topic=Topic(name='events'), index=0): 4}
Caught OffsetOutOfRange('KafkaError{code=OFFSET_OUT_OF_RANGE,val=1,str="Broker: Offset out of range"}'), shutting down...
Traceback (most recent call last):
File "/usr/local/bin/snuba", line 33, in
sys.exit(load_entry_point('snuba', 'console_scripts', 'snuba')())
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 722, in __call__
return self.main(args, *kwargs)
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 697, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 895, in invoke
return ctx.invoke(self.callback, *ctx.params)
File "/usr/local/lib/python3.8/site-packages/click/core.py", line 535, in invoke
return callback(args, **kwargs)
File "/usr/src/snuba/snuba/cli/subscriptions.py", line 224, in subscriptions
batching_consumer.run()
File "/usr/src/snuba/snuba/utils/streams/processing/processor.py", line 109, in run
self._run_once()
File "/usr/src/snuba/snuba/utils/streams/processing/processor.py", line 139, in _run_once
self.__message = self.__consumer.poll(timeout=1.0)
File "/usr/src/snuba/snuba/subscriptions/consumer.py", line 120, in poll
message = self.__consumer.poll(timeout)
File "/usr/src/snuba/snuba/utils/streams/synchronized.py", line 217, in poll
message = self.__consumer.poll(timeout)
File "/usr/src/snuba/snuba/utils/streams/backends/kafka.py", line 400, in poll
raise OffsetOutOfRange(str(error))
snuba.utils.streams.backends.abstract.OffsetOutOfRange: KafkaError{code=OFFSET_OUT_OF_RANGE,val=1,str="Broker: Offset out of range"}

• '[' s = - ']'
• snuba subscriptions --help
• set -- snuba subscriptions --auto-offset-reset=latest --consumer-group=snuba-events-subscriptions-consumers --topic=events --result-topic=events-subscription-results --dataset=events --commit-log-topic=snuba-commit-log --commit-log-group=snuba-consumers --delay-seconds=60 --schedule-ttl=60
• set gosu snuba snuba subscriptions --auto-offset-reset=latest --consumer-group=snuba-events-subscriptions-consumers --topic=events --result-topic=events-subscription-results --dataset=events --commit-log-topic=snuba-commit-log --commit-log-group=snuba-consumers --delay-seconds=60 --schedule-ttl=60
• exec gosu snuba snuba subscriptions --auto-offset-reset=latest --consumer-group=snuba-events-subscriptions-consumers --topic=events --result-topic=events-subscription-results --dataset=events --commit-log-topic=snuba-commit-log --commit-log-group=snuba-consumers --delay-seconds=60 --schedule-ttl=60
  2020-12-08 14:33:49,175 New partitions assigned: {Partition(topic=Topic(name='events'), index=0): 4}
  2020-12-08 14:33:49,178 Caught OffsetOutOfRange('KafkaError{code=OFFSET_OUT_OF_RANGE,val=1,str="Broker: Offset out of range"}'), shutting down...

I've already seen into https://github.com/getsentry/onpremise/issues/478 and that sollution didn't help me. Can someone have a look into it? During install I had the following logs:

Creating sentry_onpremise_snuba-api_run ... done

• '[' b = - ']'
• snuba bootstrap --help
• set -- snuba bootstrap --no-migrate --force
• set gosu snuba snuba bootstrap --no-migrate --force
• exec gosu snuba snuba bootstrap --no-migrate --force
  %3|1605784974.331|FAIL|rdkafka#producer-1| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.18.0.5:9092 failed: Connection refused (after 2ms in state CONNECT)
  %3|1605784975.329|FAIL|rdkafka#producer-1| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.18.0.5:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
  2020-11-19 11:22:55,329 Connection to Kafka failed (attempt 0)
  Traceback (most recent call last):
  File "/usr/src/snuba/snuba/cli/bootstrap.py", line 55, in bootstrap
  client.list_topics(timeout=1)
  cimpl.KafkaException: KafkaError{code=_TRANSPORT,val=-195,str="Failed to get metadata: Local: Broker transport failure"}
  %3|1605784976.331|FAIL|rdkafka#producer-2| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.18.0.5:9092 failed: Connection refused (after 0ms in state CONNECT)
  %3|1605784977.331|FAIL|rdkafka#producer-2| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.18.0.5:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
  2020-11-19 11:22:57,332 Connection to Kafka failed (attempt 1)
  Traceback (most recent call last):
  File "/usr/src/snuba/snuba/cli/bootstrap.py", line 55, in bootstrap
  client.list_topics(timeout=1)
  cimpl.KafkaException: KafkaError{code=_TRANSPORT,val=-195,str="Failed to get metadata: Local: Broker transport failure"}
  %3|1605784978.335|FAIL|rdkafka#producer-3| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.18.0.5:9092 failed: Connection refused (after 0ms in state CONNECT)
  %3|1605784979.335|FAIL|rdkafka#producer-3| [thrd:kafka:9092/bootstrap]: kafka:9092/bootstrap: Connect to ipv4#172.18.0.5:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
  2020-11-19 11:22:59,336 Connection to Kafka failed (attempt 2)
  Traceback (most recent call last):
  File "/usr/src/snuba/snuba/cli/bootstrap.py", line 55, in bootstrap
  client.list_topics(timeout=1)
  cimpl.KafkaException: KafkaError{code=_TRANSPORT,val=-195,str="Failed to get metadata: Local: Broker transport failure"}
  2020-11-19 11:23:01,312 Topic ingest-sessions created
  2020-11-19 11:23:01,312 Topic events created
  2020-11-19 11:23:01,312 Topic cdc created
  2020-11-19 11:23:01,312 Topic errors-replacements created
  2020-11-19 11:23:01,312 Topic outcomes created
  2020-11-19 11:23:01,312 Topic event-replacements created
  2020-11-19 11:23:01,312 Topic snuba-commit-log created
  Creating sentry_onpremise_snuba-api_run ...
  Creating sentry_onpremise_snuba-api_run ... done

[BYK]

The logs you shared for the install step are partial and they don't contain any useful information to see what might be going wrong. If this is truly a fresh install, and you don't have any concerns about data loss, simply run the following commands and then run ./install.sh again:

docker-compose down -v
docker volume rm sentry-kafka
docker volume rm sentry-zookeeper

[nuget1991]

after deleting data into kafka directory I've solved my problem, thanks. Now sentry looks fine all containers under monitoring and didn't report problems during the whole day.