Onpremise: Cannot restore from json backup

[afaianswq]

Version Information

Version: 21.2.0

Steps to Reproduce

1. Install Sentry from scratch in a clean environment, i.e. in a newly created EC2 machine
2. Try importing a json backup

Expected Result

Backup successfully imported, configuration restored.

Actual Result

An exception is printed, configuration not restored.

Logs

[start of installation is cut up to this point]

Would you like to create a user account now? [Y/n]: n

Run `sentry createuser` to do this later.

Creating missing DSNs
Correcting Group.num_comments counter

▶ Migrating file storage ...
Unable to find image 'alpine:latest' locally
latest: Pulling from library/alpine
4c0d98bf9879: Already exists
Digest: sha256:08d6ca16c60fe7490c03d10dc339d9fd8ea67c6466dea8d558526b1330a85930
Status: Downloaded newer image for alpine:latest

▶ Generating Relay credentials ...
▶ Setting up GeoIP integration ...
Setting up IP address geolocation ...
IP address geolocation database already exists.
IP address geolocation is not configured for updates.
See https://develop.sentry.dev/self-hosted/geolocation/ for instructions.
Error setting up IP address geolocation.


-----------------------------------------------------------------

You're all done! Run the following command to get Sentry running:

  docker-compose up -d

-----------------------------------------------------------------

$ docker-compose run --rm -T web import /etc/sentry/backup.json
Starting sentry_onpremise_smtp_1         ... done
Starting sentry_onpremise_memcached_1    ... done
Starting sentry_onpremise_clickhouse_1   ... done
Starting sentry_onpremise_symbolicator_1 ... done
Starting sentry_onpremise_zookeeper_1    ... done
Starting sentry_onpremise_redis_1        ... done
Starting sentry_onpremise_postgres_1     ... done
Starting sentry_onpremise_kafka_1        ... done
Starting sentry_onpremise_snuba-sessions-consumer_1     ... done
Starting sentry_onpremise_snuba-outcomes-consumer_1     ... done
Starting sentry_onpremise_snuba-transactions-consumer_1 ... done
Starting sentry_onpremise_snuba-replacer_1              ... done
Starting sentry_onpremise_snuba-api_1                   ... done
Starting sentry_onpremise_snuba-consumer_1              ... done
Creating sentry_onpremise_web_run                       ... done
15:56:55 [INFO] sentry.plugins.github: apps-not-configured
* Unknown config option found: 'slack.legacy-app'
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/sentry/db/postgres/decorators.py", line 92, in inner
    return func(self, sql, *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/sentry/db/postgres/base.py", line 75, in execute
    return self.cursor.execute(sql, clean_bad_params(params))
psycopg2.errors.UniqueViolation: duplicate key value violates unique constraint "django_content_type_app_label_model_76bd3d3b_uniq"
DETAIL:  Key (app_label, model)=(nodestore, node) already exists.


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
  File "/usr/local/lib/python3.6/site-packages/sentry/db/postgres/decorators.py", line 78, in inner
    raise_the_exception(self.db, e)
  File "/usr/local/lib/python3.6/site-packages/sentry/db/postgres/decorators.py", line 76, in inner
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/sentry/db/postgres/decorators.py", line 19, in inner
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/sentry/db/postgres/decorators.py", line 96, in inner
    raise exc_info[0](msg).with_traceback(exc_info[2])
  File "/usr/local/lib/python3.6/site-packages/sentry/db/postgres/decorators.py", line 92, in inner
    return func(self, sql, *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/sentry/db/postgres/base.py", line 75, in execute
    return self.cursor.execute(sql, clean_bad_params(params))
psycopg2.errors.UniqueViolation: UniqueViolation('duplicate key value violates unique constraint "django_content_type_app_label_model_76bd3d3b_uniq"\nDETAIL:  Key (app_label, model)=(nodestore, node) already exists.\n',)
SQL: UPDATE "django_content_type" SET "app_label" = %s, "model" = %s WHERE "django_content_type"."id" = %s

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/bin/sentry", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.6/site-packages/sentry/runner/__init__.py", line 164, in main
    cli(prog_name=get_prog(), obj={}, max_content_width=100)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/click/decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/sentry/runner/decorators.py", line 28, in inner
    return ctx.invoke(f, *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/sentry/runner/commands/backup.py", line 14, in import_
    obj.save()
  File "/usr/local/lib/python3.6/site-packages/django/core/serializers/base.py", line 205, in save
    models.Model.save_base(self.object, using=using, raw=True, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/django/db/models/base.py", line 838, in save_base
    updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)
  File "/usr/local/lib/python3.6/site-packages/django/db/models/base.py", line 905, in _save_table
    forced_update)
  File "/usr/local/lib/python3.6/site-packages/django/db/models/base.py", line 955, in _do_update
    return filtered._update(values) > 0
  File "/usr/local/lib/python3.6/site-packages/django/db/models/query.py", line 667, in _update
    return query.get_compiler(self.db).execute_sql(CURSOR)
  File "/usr/local/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 1204, in execute_sql
    cursor = super(SQLUpdateCompiler, self).execute_sql(result_type)
  File "/usr/local/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 899, in execute_sql
    raise original_exception
  File "/usr/local/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 889, in execute_sql
    cursor.execute(sql, params)
  File "/usr/local/lib/python3.6/site-packages/sentry_sdk/integrations/django/__init__.py", line 493, in execute
    return real_execute(self, sql, params)
  File "/usr/local/lib/python3.6/site-packages/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
  File "/usr/local/lib/python3.6/site-packages/django/db/utils.py", line 94, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/usr/local/lib/python3.6/site-packages/django/utils/six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "/usr/local/lib/python3.6/site-packages/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
  File "/usr/local/lib/python3.6/site-packages/sentry/db/postgres/decorators.py", line 78, in inner
    raise_the_exception(self.db, e)
  File "/usr/local/lib/python3.6/site-packages/sentry/db/postgres/decorators.py", line 76, in inner
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/sentry/db/postgres/decorators.py", line 19, in inner
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/sentry/db/postgres/decorators.py", line 96, in inner
    raise exc_info[0](msg).with_traceback(exc_info[2])
  File "/usr/local/lib/python3.6/site-packages/sentry/db/postgres/decorators.py", line 92, in inner
    return func(self, sql, *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/sentry/db/postgres/base.py", line 75, in execute
    return self.cursor.execute(sql, clean_bad_params(params))
django.db.utils.IntegrityError: UniqueViolation('duplicate key value violates unique constraint "django_content_type_app_label_model_76bd3d3b_uniq"\nDETAIL:  Key (app_label, model)=(nodestore, node) already exists.\n',)
SQL: UPDATE "django_content_type" SET "app_label" = %s, "model" = %s WHERE "django_content_type"."id" = %s
ERROR: 1

[BYK]

Looks like you are not restoring this backup onto an empty database?

[afaianswq]

Not sure how is that possible on a fresh install on a just created EC2 instance.

[BYK]

@wedamija any ideas around why this might happen? (/cc @mattrobenolt)

[Agalin]

Just been affected by the same issue due to instance migration.

Database after running install.sh is not empty. There are already built-in projects inside. It fails not only on ContentTypes instances but also later on internal project... and then again on secret token whose scopes aren't serialized to json properly (at least on 20.12).

To restore (run those on NEW instance only):

• Get into Sentry Django shell (docker-compose run --rm web django shell).
• Remove Content Types.

>>> from django.contrib.contenttypes.models import ContentType
>>> ContentType.objects.all().delete()

• Remove all project options.

>>> from sentry.models.projectoption import ProjectOption
>>> ProjectOption.objects.all().delete()

• Fix token scopes in the backup file:
  Find all "scope": "['scope1', 'scope2']" instances (notice how 'scope1' uses single apostrophe?) and change to "scope": "[\"scope1\", \"scope2\"]" (scopes will differ for your keys).

[afaianswq]

Sigh.

[Agalin]

Small update: note that it might break id sequences in postgres so those might need to be manually updated to big enough value.

[BYK]

Heya, sorry for the trouble. I'm trying to understand how this is happening as I tried the backup/restore cycle quite recently (with 21.1.0) and it worked without any issues.

I'm guessing I used to minimal of a setup?

[Agalin]

It's most probably related to migrations removing Content Types (models).

Didn't check it in full detail but it seems that import process does not update Postgres sequences correctly (as I've got issues with one of Django's auth permissions sequences also) so problem would only appear for backups created for Sentry that has been updated a few times - and some models got removed (so there are free spaces between ids, i.e. SELECT count(*) FROM django_content_type is less than SELECT max(id) FROM django_content_type).
Import seems to reuse ids (not surprising) but then sequence is only incremented count(*) times and end with a value lower than max(id).

If my reasoning is correct then simple way of reproducing it might be:

• Create two projects.
• Delete the first project (one with lower id).
• Try backup-restore.

[Agalin]

Found an existing issue for that... https://github.com/getsentry/sentry/issues/3840 from 2016, still open.

[BYK]

Closing this one in favor of getsentry/sentry#3840. Will try to get that sorted out ASAP. Sorry and thanks @Agalin!

[Agalin]

It seems that getsentry/sentry#3840 is only part of the problem though. Sequence reset will fix project creation and Sentry upgrade errors related to restored backup... but not the other two problems described here:

• Backup containing entries created by just running install.sh (some of those are due to migrations, e.g. Content Types) which leads to uniqueness errors (I don't think there is any issue open in main repo, only fix would be to flush tables before importing probably).
• Wrong encoding of token roles (getsentry/sentry#23843).

[afaianswq]

I guess the only reliable way to backup sentry is to backup docker volumes.

[Agalin]

That would be very unfortunate as currently our Postgres volume alone takes over 30GB while backup json is only ~200KB. I wonder how would it work to do django dumpdata and then django loaddata i.e. built-in Django fixtures creation and load mechanism.

[BYK]

It seems that getsentry/sentry#3840 is only part of the problem though. Sequence reset will fix project creation and Sentry upgrade errors related to restored backup... but not the other two problems described here

Very true, my bad for closing. Reopened to track the "Content Types" issue.

[BYK]

@wedamija @dcramer I figured out the issue: our export command exports _any_ model in _any_ installed application, unless it has explicitly disabled the export via the __core__ = False mechanism. I feel like this is overly broad and I'm not sure if we should even include models from django.contrib apps. Any guidance around this would be very helpful resolving this issue.

[Agalin]

I have no idea if Sentry uses generic relations anywhere but if it does then content type id must match - which means that this table must also be exported.
Also it wouldn't fix the issue of install.sh providing a db with internal Sentry projects preconfigured which leads to data in some Sentry models also (mentioned ProjectOption or SentryOption, I think I had issues with both of those). Simplest solution would be to clear tables that are imported and fix sequences. But it would make import a destructive operation in itself.

[BYK]

Also it wouldn't fix the issue of install.sh providing a db with internal Sentry projects preconfigured which leads to data in some Sentry models

These are coming from the initial migrations so we cannot change this. Instead, we should either overwrite these by clearing the tables as you suggested, or not backup these at all which doesn't sound quite right.

[chadwhitacre]

@BYK One way or another we should probably transfer this to the sentry repo and assign to Ops since they own backup/restore, agree?

[mattrobenolt]

The ops team is not responsible for this. We have never and will never use this. I'd vote we delete it and promote better methods of backup imo. I don't remember this ever really working as intended in the past 5+ years.

[BYK]

Yup, ops doesn't maintain this, that said this is the main method for "light" migrations and moves from self-hosted to SaaS so we need to maintain it for now.

@chadwhitacre we'll own this as the open-source team for now but we may find a more suitable team in the future once the requirements and responsibilities are better defined.