Onpremise: Missing Error Events after Migration

[4lxndr]

Hello, i migrated Sentry from a server to another one. From the source server i grabbed all data using postgres dump and imported them successfully on the new server. Users, projects and all the stuff are working but the error events are missing. Is there a way to trigger the indexer for the error events manually? If a new error event pops up it also relates to older events of this kind and knows exactly when this event first occurred (so the data seems available).

Thanks in advance!

[BYK]

Which version of Sentry was this?

[4lxndr]

ah sorry, the setup is build with SENTRY_IMAGE=getsentry/sentry:10 ./install.sh (checked https://github.com/getsentry/onpremise/pull/407 out before).

Edit: Old one was the same version

[BYK]

There may be some issues regarding Clickhouse and Snuba. Is there any reason you are not using the latest master as we recommend?

Also are you seeing any errors in snuba-api, snuba-consumer or post-process-forwarder logs? If all are clean I'd check kafka and zookeeper logs.

[4lxndr]

Is there any reason you are not using the latest master as we recommend?

Because i experienced the same behavior described in https://github.com/getsentry/onpremise/issues/412

# docker-compose logs snuba-api
Attaching to sentry_snuba-api_1
snuba-api_1               | + '[' a = - ']'
snuba-api_1               | + snuba api --help
snuba-api_1               | + set -- snuba api
snuba-api_1               | + set gosu snuba snuba api
snuba-api_1               | + exec gosu snuba snuba api
snuba-api_1               | *** Starting uWSGI 2.0.18 (64bit) on [Fri May 22 10:32:00 2020] ***
snuba-api_1               | compiled with version: 8.3.0 on 18 May 2020 18:44:36
snuba-api_1               | os: Linux-4.9.0-9-amd64 #1 SMP Debian 4.9.168-1+deb9u3 (2019-06-16)
snuba-api_1               | nodename: b478cadc6dcf
snuba-api_1               | machine: x86_64
snuba-api_1               | clock source: unix
snuba-api_1               | pcre jit disabled
snuba-api_1               | detected number of CPU cores: 8
snuba-api_1               | current working directory: /usr/src/snuba
snuba-api_1               | detected binary path: /usr/local/bin/uwsgi
snuba-api_1               | your memory page size is 4096 bytes
snuba-api_1               | detected max file descriptor number: 1048576
snuba-api_1               | lock engine: pthread robust mutexes
snuba-api_1               | thunder lock: enabled
snuba-api_1               | uwsgi socket 0 bound to TCP address 0.0.0.0:1218 fd 3
snuba-api_1               | Python version: 3.7.7 (default, May 15 2020, 11:37:57)  [GCC 8.3.0]
snuba-api_1               | Set PythonHome to /usr/local
snuba-api_1               | Python main interpreter initialized at 0x559c49867830
snuba-api_1               | python threads support enabled
snuba-api_1               | your server socket listen backlog is limited to 100 connections
snuba-api_1               | your mercy for graceful operations on workers is 60 seconds
snuba-api_1               | mapped 145808 bytes (142 KB) for 1 cores
snuba-api_1               | *** Operational MODE: single process ***
snuba-api_1               | initialized 38 metrics
snuba-api_1               | spawned uWSGI master process (pid: 1)
snuba-api_1               | spawned uWSGI worker 1 (pid: 18, cores: 1)
snuba-api_1               | metrics collector thread started
snuba-api_1               | WSGI app 0 (mountpoint='') ready in 1 seconds on interpreter 0x559c49867830 pid: 18 (default app)

# docker-compose logs snuba-consumer
Attaching to sentry_snuba-consumer_1
snuba-consumer_1          | + '[' c = - ']'
snuba-consumer_1          | + snuba consumer --help
snuba-consumer_1          | + set -- snuba consumer --auto-offset-reset=latest --max-batch-time-ms 750
snuba-consumer_1          | + set gosu snuba snuba consumer --auto-offset-reset=latest --max-batch-time-ms 750
snuba-consumer_1          | + exec gosu snuba snuba consumer --auto-offset-reset=latest --max-batch-time-ms 750
snuba-consumer_1          | 2020-05-22 10:32:09,661 New partitions assigned: {Partition(topic=Topic(name='events'), index=0): 1713}
snuba-consumer_1          | 2020-05-22 10:32:19,834 Flushing 1 items (from {Partition(topic=Topic(name='events'), index=0): Offsets(lo=1713, hi=1713)}): forced:False size:False time:True
snuba-consumer_1          | 2020-05-22 10:32:19,952 Worker flush took 117ms



md5-518f92d7b2676663e8c52bfa32b09bcc



# docker-compose logs post-process-forwarder
Attaching to sentry_post-process-forwarder_1
post-process-forwarder_1  | 10:32:00 [WARNING] sentry.utils.geo: settings.GEOIP_PATH_MMDB not configured.
post-process-forwarder_1  | 10:32:05 [INFO] sentry.plugins.github: apps-not-configured



md5-39cf098f4fd55f40f4d7af52a838e867



# docker-compose logs kafka
Attaching to sentry_kafka_1
kafka_1                   | [2020-05-22 10:32:05,605] ERROR [GroupMetadataManager brokerId=1001] Error loading offsets from __consumer_offsets-25 (kafka.coordinator.group.GroupMetadataManager)
kafka_1                   | org.apache.kafka.common.KafkaException: Unknown group metadata version 3
kafka_1                   |     at kafka.coordinator.group.GroupMetadataManager$.schemaForGroupValue(GroupMetadataManager.scala:1077)
kafka_1                   |     at kafka.coordinator.group.GroupMetadataManager$.readGroupMessageValue(GroupMetadataManager.scala:1309)
kafka_1                   |     at kafka.coordinator.group.GroupMetadataManager$$anonfun$doLoadGroupsAndOffsets$2$$anonfun$apply$13.apply(GroupMetadataManager.scala:602)
kafka_1                   |     at kafka.coordinator.group.GroupMetadataManager$$anonfun$doLoadGroupsAndOffsets$2$$anonfun$apply$13.apply(GroupMetadataManager.scala:574)
kafka_1                   |     at scala.collection.Iterator$class.foreach(Iterator.scala:891)
kafka_1                   |     at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
kafka_1                   |     at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
kafka_1                   |     at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
kafka_1                   |     at kafka.coordinator.group.GroupMetadataManager$$anonfun$doLoadGroupsAndOffsets$2.apply(GroupMetadataManager.scala:574)
kafka_1                   |     at kafka.coordinator.group.GroupMetadataManager$$anonfun$doLoadGroupsAndOffsets$2.apply(GroupMetadataManager.scala:555)
kafka_1                   |     at scala.collection.Iterator$class.foreach(Iterator.scala:891)
kafka_1                   |     at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
kafka_1                   |     at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
kafka_1                   |     at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
kafka_1                   |     at kafka.coordinator.group.GroupMetadataManager.doLoadGroupsAndOffsets(GroupMetadataManager.scala:555)
kafka_1                   |     at kafka.coordinator.group.GroupMetadataManager.loadGroupsAndOffsets(GroupMetadataManager.scala:500)
kafka_1                   |     at kafka.coordinator.group.GroupMetadataManager$$anonfun$scheduleLoadGroupAndOffsets$1.apply$mcV$sp(GroupMetadataManager.scala:491)
kafka_1                   |     at kafka.utils.KafkaScheduler$$anonfun$1.apply$mcV$sp(KafkaScheduler.scala:114)
kafka_1                   |     at kafka.utils.CoreUtils$$anon$1.run(CoreUtils.scala:63)
kafka_1                   |     at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
kafka_1                   |     at java.util.concurrent.FutureTask.run(FutureTask.java:266)
kafka_1                   |     at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
kafka_1                   |     at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
kafka_1                   |     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
kafka_1                   |     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
kafka_1                   |     at java.lang.Thread.run(Thread.java:748)



md5-518f92d7b2676663e8c52bfa32b09bcc



# docker-compose logs zookeeper
Attaching to sentry_zookeeper_1
zookeeper_1               | ===> ENV Variables ...
zookeeper_1               | ALLOW_UNSIGNED=false
zookeeper_1               | COMPONENT=zookeeper
zookeeper_1               | CONFLUENT_DEB_VERSION=1
zookeeper_1               | CONFLUENT_MAJOR_VERSION=5
zookeeper_1               | CONFLUENT_MINOR_VERSION=1
zookeeper_1               | CONFLUENT_MVN_LABEL=
zookeeper_1               | CONFLUENT_PATCH_VERSION=2
zookeeper_1               | CONFLUENT_PLATFORM_LABEL=
zookeeper_1               | CONFLUENT_SUPPORT_METRICS_ENABLE=false
zookeeper_1               | CONFLUENT_VERSION=5.1.2
zookeeper_1               | CUB_CLASSPATH=/etc/confluent/docker/docker-utils.jar
zookeeper_1               | HOME=/root
zookeeper_1               | HOSTNAME=6460f6871716
zookeeper_1               | KAFKA_VERSION=2.1.1cp1
zookeeper_1               | LANG=C.UTF-8
zookeeper_1               | PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
zookeeper_1               | PWD=/
zookeeper_1               | PYTHON_PIP_VERSION=8.1.2
zookeeper_1               | PYTHON_VERSION=2.7.9-1
zookeeper_1               | SCALA_VERSION=2.11
zookeeper_1               | SHLVL=1
zookeeper_1               | ZOOKEEPER_CLIENT_PORT=2181
zookeeper_1               | ZOOKEEPER_LOG4J_ROOT_LOGLEVEL=WARN
zookeeper_1               | ZOOKEEPER_TOOLS_LOG4J_LOGLEVEL=WARN
zookeeper_1               | ZULU_OPENJDK_VERSION=8=8.30.0.1
zookeeper_1               | _=/usr/bin/env
zookeeper_1               | ===> User
zookeeper_1               | uid=0(root) gid=0(root) groups=0(root)
zookeeper_1               | ===> Configuring ...
zookeeper_1               | ===> Running preflight checks ... 
zookeeper_1               | ===> Check if /var/lib/zookeeper/data is writable ...
zookeeper_1               | ===> Check if /var/lib/zookeeper/log is writable ...
zookeeper_1               | ===> Launching ... 
zookeeper_1               | ===> Launching zookeeper ... 
zookeeper_1               | [2020-05-22 10:32:01,025] WARN Either no config or no quorum defined in config, running  in standalone mode (org.apache.zookeeper.server.quorum.QuorumPeerMain)

Btw is there a list containing a description what service is responsible for? I remember older versions with just a hand full of services and now we have 18 services.

thanks for your support

[BYK]

Is there any reason you are not using the latest master as we recommend?

Because i experienced the same behavior described in #412

What I meant was not using the SENTRY_IMAGE override _and_ using the latest master from this repo.

Btw is there a list containing a description what service is responsible for? I remember older versions with just a hand full of services and now we have 18 services.

This request has been made in a few more places and I've put it into our todo list. That said I cannot promise a rigid timeline to get this out.

In general:

• Nginx: Router, Reverse Proxy, SSL Terminator if you wish to
• Relay: Event ingestion (including rate limiting, filtering, scrubbing etc)
• Symbolicator: Symbolication of native crash dumps
• Clickhouse: Storing all event and event-related data
• Snuba: The API we use to interface with Clickhouse (this has multiple services that to some processing on events and Clickhouse)
• Kafka/Zookeeper: The core message bus across Relay, Sentry, and Snuba
• post-process-forwarder: The Kafka topic consumer that notifies Sentry about processed events by Snuba (if this has issues, you would ingest events but won't see them on the UI)
• worker: The general worker for Celery tasks, including sending e-mails etc.
• cron: The general cron (celerybeat) worker process for timed tasks
• web: The main sentry backend instance, including all web-facing tasks
• X-cleanup: The cleanup service for the service X to remove unused or temporary data periodically to save space.

---

Seems like your Kafka instance got corrupted or having issues. The easiest way would be to shut everything down, remove kafka and zookeeper related volumes and then bring things back up. After this you'll likely need to run docker-compose run --rm snuba-api bootstrap --force to create the Kafka topics again. At that point, it might be wiser to upgrading to the latest version of this repo and running ./install.sh and upgrade your instance along the way.

Hope this helps and looking forward to hearing back about the results.

[4lxndr]

Thank you for the explanation! Now its a bit clearer what service is responsible for.

Regarding your recommendation, i updated everything to the latest master version. The docker-compose logs look fine now and the kafka error is gone.

But (as always :D) there is something new. We use google auth for the organization and if i press the button it leads to an error:

 400: invalid_request
Invalid parameter value for redirect_uri: Missing scheme: /auth/sso/

so i compared the requests for each instance:

old one - working

redirect_uri: https://sentry.xyz.xy/auth/sso/

new one - not working anymor

redirect_uri:/auth/sso/

the difference is the missing scheme and domain. Is there an option where i can set this?

[BYK]

Is there an option where i can set this?

Could you be missing your system.url-prefix setting in your config.yml file?

[4lxndr]

its working, thank you very much for your input and time!