Sentry: 0302 Migration error (8.12.0 -> 9.0)

[ashymanouski]

Hello,
we tried to update our local sentry 8.12.0 and faced with migrations issue:

```
sentry:0300_auto__add_processingissue__add_unique_processingissue_project_checksum
sentry:0301_auto__chg_field_environment_project_id__chg_field_releaseenvironment_p
sentry:0302_merge_environments
Error in migration: sentry:0302_merge_environments
The following content types are stale and need to be deleted:

sentry | dsymobject
sentry | dsymsymbol
sentry | dsymsdk
sentry | globaldsymfile
sentry | dsymbundle

Any objects related to these content types by a foreign key will also
be deleted. Are you sure you want to delete these content types?
If you're unsure, answer 'no'.

Type 'yes' to continue, or 'no' to cancel: no

Error in sys.excepthook:
Traceback (most recent call last):
File "/usr/local/lib/python2.7/site-packages/raven/base.py", line 274, in handle_exception
self.captureException(exc_info=exc_info, level='fatal')
File "/usr/local/lib/python2.7/site-packages/raven/base.py", line 814, in captureException
'raven.events.Exception', exc_info=exc_info, kwargs)
File "/usr/local/lib/python2.7/site-packages/sentry/utils/raven.py", line 46, in capture
return super(SentryInternalClient, self).capture(args, *kwargs)
File "/usr/local/lib/python2.7/site-packages/raven/contrib/django/client.py", line 297, in capture
result = super(DjangoClient, self).capture(event_type, *kwargs)
File "/usr/local/lib/python2.7/site-packages/raven/base.py", line 641, in capture
self.send(data)
File "/usr/local/lib/python2.7/site-packages/sentry/utils/raven.py", line 75, in send
key = ProjectKey.get_default(settings.SENTRY_PROJECT)
File "/usr/local/lib/python2.7/site-packages/sentry/models/projectkey.py", line 121, in get_default
)[0]
File "/usr/local/lib/python2.7/site-packages/django/db/models/query.py", line 132, in __getitem__
return list(qs)[0]
File "/usr/local/lib/python2.7/site-packages/django/db/models/query.py", line 96, in __iter__
self._fetch_all()
File "/usr/local/lib/python2.7/site-packages/django/db/models/query.py", line 857, in _fetch_all
self._result_cache = list(self.iterator())
File "/usr/local/lib/python2.7/site-packages/django/db/models/query.py", line 220, in iterator
for row in compiler.results_iter():
File "/usr/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 713, in results_iter
for rows in self.execute_sql(MULTI):
File "/usr/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 786, in execute_sql
cursor.execute(sql, params)
File "/usr/local/lib/python2.7/site-packages/raven/contrib/django/client.py", line 123, in execute
return real_execute(self, sql, params)
File "/usr/local/lib/python2.7/site-packages/django/db/backends/util.py", line 53, in execute
return self.cursor.execute(sql, params)
File "/usr/local/lib/python2.7/site-packages/django/db/utils.py", line 99, in __exit__
six.reraise(dj_exc_type, dj_exc_value, traceback)
File "/usr/local/lib/python2.7/site-packages/django/db/backends/util.py", line 53, in execute
return self.cursor.execute(sql, params)
File "/usr/local/lib/python2.7/site-packages/sentry/db/postgres/decorators.py", line 80, in inner
raise_the_exception(self.db, e)
File "/usr/local/lib/python2.7/site-packages/sentry/db/postgres/decorators.py", line 78, in inner
return func(self, *args, *kwargs)
File "/usr/local/lib/python2.7/site-packages/sentry/db/postgres/decorators.py", line 22, in inner
return func(self, args, *kwargs)
File "/usr/local/lib/python2.7/site-packages/sentry/db/postgres/decorators.py", line 101, in inner
six.reraise(exc_info[0], exc_info0, exc_info[2])
File "/usr/local/lib/python2.7/site-packages/sentry/db/postgres/decorators.py", line 94, in inner
return func(self, sql, args, *kwargs)
File "/usr/local/lib/python2.7/site-packages/sentry/db/postgres/base.py", line 47, in execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: ProgrammingError('column sentry_projectkey.rate_limit_count does not existnLINE 1: ...tkey"."status", "sentry_projectkey"."date_added", "sentry_pr...n ^n',)
SQL: SELECT "sentry_projectkey"."id", "sentry_projectkey"."project_id", "sentry_projectkey"."label", "sentry_projectkey"."public_key", "sentry_projectkey"."secret_key", "sentry_projectkey"."roles", "sentry_projectkey"."status", "sentry_projectkey"."date_added", "sentry_projectkey"."rate_limit_count", "sentry_projectkey"."rate_limit_window", "sentry_projectkey"."data" FROM "sentry_projectkey" WHERE ("sentry_projectkey"."project_id" = %s AND "sentry_projectkey"."status" = %s AND "sentry_projectkey"."roles" = ("sentry_projectkey"."roles" | 1)) LIMIT 1

Original exception was:
Traceback (most recent call last):
File "/usr/local/bin/sentry", line 11, in
sys.exit(main())
File "/usr/local/lib/python2.7/site-packages/sentry/runner/__init__.py", line 162, in main
cli(prog_name=get_prog(), obj={}, max_content_width=100)
File "/usr/local/lib/python2.7/site-packages/click/core.py", line 722, in __call__
return self.main(args, *kwargs)
File "/usr/local/lib/python2.7/site-packages/click/core.py", line 697, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python2.7/site-packages/click/core.py", line 1066, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python2.7/site-packages/click/core.py", line 895, in invoke
return ctx.invoke(self.callback, *ctx.params)
File "/usr/local/lib/python2.7/site-packages/click/core.py", line 535, in invoke
return callback(args, *kwargs)
File "/usr/local/lib/python2.7/site-packages/click/decorators.py", line 17, in new_func
return f(get_current_context(), *args, *kwargs)
File "/usr/local/lib/python2.7/site-packages/sentry/runner/decorators.py", line 36, in inner
return ctx.invoke(f, args, *kwargs)
File "/usr/local/lib/python2.7/site-packages/click/core.py", line 535, in invoke
return callback(args, *kwargs)
File "/usr/local/lib/python2.7/site-packages/click/decorators.py", line 17, in new_func
return f(get_current_context(), args, *kwargs)
File "/usr/local/lib/python2.7/site-packages/sentry/runner/commands/upgrade.py", line 67, in upgrade
_upgrade(not noinput, traceback, verbosity, not no_repair)
File "/usr/local/lib/python2.7/site-packages/sentry/runner/commands/upgrade.py", line 29, in _upgrade
verbosity=verbosity,
File "/usr/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 159, in call_command
return klass.execute(args, *defaults)
File "/usr/local/lib/python2.7/site-packages/django/core/management/base.py", line 285, in execute
output = self.handle(args, *options)
File "/usr/local/lib/python2.7/site-packages/south/management/commands/migrate.py", line 115, in handle
ignore_ghosts=ignore_ghosts,
File "/usr/local/lib/python2.7/site-packages/south/migration/__init__.py", line 234, in migrate_app
success = migrator.migrate_many(target, workplan, database)
File "/usr/local/lib/python2.7/site-packages/south/migration/migrators.py", line 256, in migrate_many
result = migrator.__class__.migrate_many(migrator, target, migrations, database)
File "/usr/local/lib/python2.7/site-packages/south/migration/migrators.py", line 341, in migrate_many
result = self.migrate(migration, database)
File "/usr/local/lib/python2.7/site-packages/south/migration/migrators.py", line 132, in migrate
result = self.run(migration, database)
File "/usr/local/lib/python2.7/site-packages/south/migration/migrators.py", line 291, in run
return super(Forwards, self).run(migration, database)
File "/usr/local/lib/python2.7/site-packages/south/migration/migrators.py", line 114, in run
return self.run_migration(migration, database)
File "/usr/local/lib/python2.7/site-packages/south/migration/migrators.py", line 85, in run_migration
migration_function()
File "/usr/local/lib/python2.7/site-packages/south/migration/migrators.py", line 61, in
return (lambda: direction(orm))
File "/usr/local/lib/python2.7/site-packages/sentry/south_migrations/0302_merge_environments.py", line 12, in forwards
self._forwards(orm)
File "/usr/local/lib/python2.7/site-packages/sentry/south_migrations/0302_merge_environments.py", line 30, in _forwards
for name, organization_id in dupe_envs:
ValueError: too many values to unpack
```
No matter what I choose, delete or not, the result is the same

I would be grateful for any advice

[melghors]

Same problem

[melghors]

I tried to run migrations from 9.0 sentry to db schema v. 8.12. I resolved it running migrations from 8.22 sentry and then 9.0.

[LJxFFFF]

same problem, I ran migrations from 8.9.0(Python) to 9.0(docker-compose), and I could not resloved it...
I tried to print dupe_envs on web shell:

here's 0302_merge_environments.py (about 30 line):

 -*- coding: utf-8 -*-
from south.utils import datetime_utils as datetime
from south.db import db
from south.v2 import DataMigration
from django.db import IntegrityError, models, transaction


class Migration(DataMigration):
    def forwards(self, orm):
        db.commit_transaction()
        try:
            self._forwards(orm)
        except Exception:
            # Explicitly resume the transaction because
            # South is going to try and roll it back, but when
            # it can't find one, it'll error itself, masking
            # the actual exception being raised
            #
            # See https://github.com/getsentry/sentry/issues/5035
            db.start_transaction()
            raise
        db.start_transaction()

    def _forwards(self, orm):
        "Write your forwards methods here."
        dupe_envs = orm.Environment.objects.values_list('name', 'organization_id')\
                                           .annotate(ecount=models.Count('id'))\
                                           .filter(ecount__gt=1)

        for name, organization_id in dupe_envs:
            envs = list(
                orm.Environment.objects.filter(
                    name=name,
                    organization_id=organization_id,
                ).order_by('date_added')
            )
            to_env = envs[0]
            from_envs = envs[1:]

It sure throws VALUE ERROR because data struct is incorrect....

[mgedmin]

Reverting commit 86df905fa67c0751944ae8dadc2ced2dd7fe7607 got me past this migration.

(And by "reverting" I mean I manually edited lib/python2.7/site-packages/sentry/south_migrations/0302_merge_environments.py inside my virtualenv with vi because YOLO.)

[bwesen]

Got the same problem in a 8-9.1 migration, 0302 fails with Too many values to unpack... :/

[bwesen]

This just seems to be broken, I verified the same problem as the above poster @LiuJinxuan , the .values_list adds ecount in an annotation so you get 3-value tuples for each dupe_envs list entry. The original code was correct (unpacking by key), then someone patched it as @mgedmin found, which simply broke it. Might it be that the testcases don't have any duped envs and thus haven't found this problem?

The fix is to revert this commit

or as a quick fix if you're in a Docker container (as I was):

# cd /usr/local/lib/python2.7/site-packages/sentry/south_migrations
# sed -i 's/in dupe_envs/, ecount in dupe_envs/g' 0302_merge_environments.py

[BYK]

Closing this issue as we cannot go back in time and fix this unfortunately :(

@bwesen - thanks for sharing your solution which I hope would help anyone else encountering this.

[bwesen]

@BYK Why not just fix the 0302_ migration file now that we know why this happens? The bug will hit anybody out there migrating from Sentry 8 to 9 or so. Granted, most might have gone through the migrations already but me for example hadn't and I'm sure more can follow :)

[BYK]

@BYK Why not just fix the 0302_ migration file now that we know why this happens? The bug will hit anybody out there migrating from Sentry 8 to 9 or so.

Because we have just released Sentry 10, which requires Sentry 9.1.2 to upgrade and we are not going to release a new patch version for 9.0 or 9.1 :D

Granted, most might have gone through the migrations already but me for example hadn't and I'm sure more can follow :)

This is also another strong reason :)

[kpshameem]

will do the job
diff --git a/Dockerfile b/Dockerfile
index 08d8252..cbf9599 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -1,2 +1,4 @@
ARG SENTRY_IMAGE
FROM ${SENTRY_IMAGE:-sentry:9.1.2}-onbuild
+
+RUN sed -i 's/in dupe_envs/, ecount in dupe_envs/g' /usr/local/lib/python2.7/site-packages/sentry/south_migrations/0302_merge_environments.py