Onpremise: Upgrade 9.1.2 to 10. Migration error

Created on 27 Jan 2020  路  13Comments  路  Source: getsentry/onpremise

Hello!
I've started to use sentry from version 9.0.0. On prod running sentry 9.1.2 now.
I decided install stage with onpremise. Also migrated database from prod. Version 9.1.2 works fine.
When upgrading to 10 I've got an error during migration 0024_auto_20191230_2052:

Operations to perform:
  Apply all migrations: admin, auth, contenttypes, jira_ac, nodestore, sentry, sessions, sites, social_auth
Running migrations:
  Applying sentry.0024_auto_20191230_2052.../usr/local/lib/python2.7/site-packages/django/db/models/fields/__init__.py:1451: RuntimeWarning: DateTimeField Event.datetime received a naive datetime (2019-10-29 11:37:43.582226) while time zone support is active.
  RuntimeWarning)
/usr/local/lib/python2.7/site-packages/django/db/models/fields/__init__.py:1451: RuntimeWarning: DateTimeField Event.datetime received a naive datetime (2020-01-27 11:37:43.582226) while time zone support is active.
  RuntimeWarning)
Events to process: 131482

Traceback (most recent call last):
  File "/usr/local/bin/sentry", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python2.7/site-packages/sentry/runner/__init__.py", line 164, in main
    cli(prog_name=get_prog(), obj={}, max_content_width=100)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/sentry/runner/decorators.py", line 30, in inner
    return ctx.invoke(f, *args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/sentry/runner/commands/upgrade.py", line 168, in upgrade
    _upgrade(not noinput, traceback, verbosity, not no_repair)
  File "/usr/local/lib/python2.7/site-packages/sentry/runner/commands/upgrade.py", line 130, in _upgrade
    ignore_ghost_migrations=True,
  File "/usr/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 131, in call_command
    return command.execute(*args, **defaults)
  File "/usr/local/lib/python2.7/site-packages/django/core/management/base.py", line 330, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lib/python2.7/site-packages/django/core/management/commands/migrate.py", line 204, in handle
    fake_initial=fake_initial,
  File "/usr/local/lib/python2.7/site-packages/django/db/migrations/executor.py", line 115, in migrate
    state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
  File "/usr/local/lib/python2.7/site-packages/django/db/migrations/executor.py", line 145, in _migrate_all_forwards
    state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
  File "/usr/local/lib/python2.7/site-packages/sentry/new_migrations/monkey/executor.py", line 26, in apply_migration
    state, migration, fake=fake, fake_initial=fake_initial
  File "/usr/local/lib/python2.7/site-packages/django/db/migrations/executor.py", line 244, in apply_migration
    state = migration.apply(state, schema_editor)
  File "/usr/local/lib/python2.7/site-packages/django/db/migrations/migration.py", line 129, in apply
    operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
  File "/usr/local/lib/python2.7/site-packages/django/db/migrations/operations/special.py", line 193, in database_forwards
    self.code(from_state.apps, schema_editor)
  File "/usr/local/lib/python2.7/site-packages/sentry/migrations/0024_auto_20191230_2052.py", line 70, in backfill_eventstream
    primary_hash = event.get_primary_hash()
AttributeError: 'Event' object has no attribute 'get_primary_hash'

How can I fix it?

Bug
Source
picture artful88533
👍2

Most helpful comment

Hey all, @lynnagara and I are looking into this. The issue only affects people migrating from 9.1.2, fresh installs are unaffected. Although we hope to resolve the issue by EOD today, you can use SENTRY_IMAGE=getsentry/sentry:b0ea6ea032d886edf24a70c1d1ac0d79e1462b39 ./install.sh (this is using a slightly older version of Sentry) to upgrade and then switch to latest afterwards in the meantime.

picture BYK on 30 Jan 2020
👍2

All 13 comments

Heya, thanks for reporting - we are looking into this.

BYK picture BYK on 27 Jan 2020
👍1

I've got one more question.
I used Ansible for deploying all of the sentry's services.
What is more correct: to build an image from this Docker file https://github.com/getsentry/sentry/blob/10.0.0/docker/Dockerfile or use a prepared image from dockerhub getsentry/sentry:10.0.0 ?

artful88533 picture artful88533 on 28 Jan 2020

@artful88533 forums are a better place to discuss these kind of questions ;)

BYK picture BYK on 28 Jan 2020
👀1

I'm having this issue too,
@BYK thank you for the fix, can you please tell when this fix will be available in DockerHub?

Operations to perform:
  Apply all migrations: admin, auth, contenttypes, jira_ac, nodestore, sentry, sessions, sites, social_auth
Running migrations:
  Applying sentry.0024_auto_20191230_2052.../usr/local/lib/python2.7/site-packages/django/db/models/fields/__init__.py:1451: RuntimeWarning: DateTimeField Event.datetime received a naive datetime (2019-10-31 14:34:41.644628) while time zone support is active.
  RuntimeWarning)
/usr/local/lib/python2.7/site-packages/django/db/models/fields/__init__.py:1451: RuntimeWarning: DateTimeField Event.datetime received a naive datetime (2020-01-29 14:34:41.644628) while time zone support is active.
  RuntimeWarning)
Events to process: 6459

Traceback (most recent call last):
  File "/usr/local/bin/sentry", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python2.7/site-packages/sentry/runner/__init__.py", line 164, in main
    cli(prog_name=get_prog(), obj={}, max_content_width=100)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/sentry/runner/decorators.py", line 30, in inner
    return ctx.invoke(f, *args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/sentry/runner/commands/upgrade.py", line 168, in upgrade
    _upgrade(not noinput, traceback, verbosity, not no_repair)
  File "/usr/local/lib/python2.7/site-packages/sentry/runner/commands/upgrade.py", line 130, in _upgrade
    ignore_ghost_migrations=True,
  File "/usr/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 131, in call_command
    return command.execute(*args, **defaults)
  File "/usr/local/lib/python2.7/site-packages/django/core/management/base.py", line 330, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lib/python2.7/site-packages/django/core/management/commands/migrate.py", line 204, in handle
    fake_initial=fake_initial,
  File "/usr/local/lib/python2.7/site-packages/django/db/migrations/executor.py", line 115, in migrate
    state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
  File "/usr/local/lib/python2.7/site-packages/django/db/migrations/executor.py", line 145, in _migrate_all_forwards
    state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
  File "/usr/local/lib/python2.7/site-packages/sentry/new_migrations/monkey/executor.py", line 26, in apply_migration
    state, migration, fake=fake, fake_initial=fake_initial
  File "/usr/local/lib/python2.7/site-packages/django/db/migrations/executor.py", line 244, in apply_migration
    state = migration.apply(state, schema_editor)
  File "/usr/local/lib/python2.7/site-packages/django/db/migrations/migration.py", line 129, in apply
    operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
  File "/usr/local/lib/python2.7/site-packages/django/db/migrations/operations/special.py", line 193, in database_forwards
    self.code(from_state.apps, schema_editor)
  File "/usr/local/lib/python2.7/site-packages/sentry/migrations/0024_auto_20191230_2052.py", line 109, in backfill_eventstream
    skip_consume=True,
  File "/usr/local/lib/python2.7/site-packages/sentry/utils/services.py", line 104, in <lambda>
    context[key] = (lambda f: lambda *a, **k: getattr(self, f)(*a, **k))(key)
  File "/usr/local/lib/python2.7/site-packages/sentry/eventstream/snuba.py", line 114, in insert
    "message": event.search_message,
AttributeError: 'Event' object has no attribute 'search_message'
BuSHari picture BuSHari on 29 Jan 2020
👀1 👍1

@bushari - the fix for the original issue is already on Docker Hub. Yours seem to be a different error. I'll look into it.

BYK picture BYK on 29 Jan 2020

@BYK Ok, thank you...

BuSHari picture BuSHari on 29 Jan 2020

Not sure if it's the same bug, but I've got yet another exception message from the same migration. When I've tested the upgrade procedure a few days ago it worked.

[...]
  Applying sentry.0023_hide_environment_none_20191126... OK
  Applying sentry.0024_auto_20191230_2052.../usr/local/lib/python2.7/site-packages/django/db/models/fields/__init__.py:1451: RuntimeWarning: DateTimeField Event.datetime received a naive datetime (2019-10-31 21:05:15.196611) while time zone support is active.
  RuntimeWarning)
/usr/local/lib/python2.7/site-packages/django/db/models/fields/__init__.py:1451: RuntimeWarning: DateTimeField Event.datetime received a naive datetime (2020-01-29 21:05:15.196611) while time zone support is active.
  RuntimeWarning)
Events to process: 104594

Traceback (most recent call last):
  File "/usr/local/bin/sentry", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python2.7/site-packages/sentry/runner/__init__.py", line 164, in main
    cli(prog_name=get_prog(), obj={}, max_content_width=100)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/sentry/runner/decorators.py", line 30, in inner
    return ctx.invoke(f, *args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/sentry/runner/commands/upgrade.py", line 168, in upgrade
    _upgrade(not noinput, traceback, verbosity, not no_repair)
  File "/usr/local/lib/python2.7/site-packages/sentry/runner/commands/upgrade.py", line 130, in _upgrade
    ignore_ghost_migrations=True,
  File "/usr/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 131, in call_command
    return command.execute(*args, **defaults)
  File "/usr/local/lib/python2.7/site-packages/django/core/management/base.py", line 330, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lib/python2.7/site-packages/django/core/management/commands/migrate.py", line 204, in handle
    fake_initial=fake_initial,
  File "/usr/local/lib/python2.7/site-packages/django/db/migrations/executor.py", line 115, in migrate
    state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
  File "/usr/local/lib/python2.7/site-packages/django/db/migrations/executor.py", line 145, in _migrate_all_forwards
    state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
  File "/usr/local/lib/python2.7/site-packages/sentry/new_migrations/monkey/executor.py", line 26, in apply_migration
    state, migration, fake=fake, fake_initial=fake_initial
  File "/usr/local/lib/python2.7/site-packages/django/db/migrations/executor.py", line 244, in apply_migration
    state = migration.apply(state, schema_editor)
  File "/usr/local/lib/python2.7/site-packages/django/db/migrations/migration.py", line 129, in apply
    operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
  File "/usr/local/lib/python2.7/site-packages/django/db/migrations/operations/special.py", line 193, in database_forwards
    self.code(from_state.apps, schema_editor)
  File "/usr/local/lib/python2.7/site-packages/sentry/migrations/0024_auto_20191230_2052.py", line 96, in backfill_eventstream
    primary_hash = _get_primary_hash(event)
  File "/usr/local/lib/python2.7/site-packages/sentry/migrations/0024_auto_20191230_2052.py", line 69, in _get_primary_hash
    for hash in (x.get_hash() for x in _get_grouping_variants(event).values())
  File "/usr/local/lib/python2.7/site-packages/sentry/migrations/0024_auto_20191230_2052.py", line 62, in _get_grouping_variants
    return get_grouping_variants_for_event(event, config)
  File "/usr/local/lib/python2.7/site-packages/sentry/grouping/api.py", line 196, in get_grouping_variants_for_event
    components = _get_calculated_grouping_variants_for_event(event, config)
  File "/usr/local/lib/python2.7/site-packages/sentry/grouping/api.py", line 143, in _get_calculated_grouping_variants_for_event
    rv = strategy.get_grouping_component_variants(event, config=config)
  File "/usr/local/lib/python2.7/site-packages/sentry/grouping/strategies/base.py", line 122, in get_grouping_component_variants
    component = self.get_grouping_component(event, variant, config)
  File "/usr/local/lib/python2.7/site-packages/sentry/grouping/strategies/base.py", line 98, in get_grouping_component
    iface = event.interfaces.get(iface_path)
AttributeError: 'Event' object has no attribute 'interfaces'
Cleaning up...
fnkr picture fnkr on 29 Jan 2020

It seems that there are some problems with that migration file, i tried to do a clean install on a different machine and i got a small error but most of the migration prcesses been done

Here is the error of sentry.0024_auto_20191230_2052

Applying sentry.0024_auto_20191230_2052.../usr/local/lib/python2.7/site-packages/django/db/models/fields/__init__.py:1451: RuntimeWarning: DateTimeField Event.datetime received a naive datetime (2019-10-31 15:57:56.376485) while time zone support is active.
  RuntimeWarning)
/usr/local/lib/python2.7/site-packages/django/db/models/fields/__init__.py:1451: RuntimeWarning: DateTimeField Event.datetime received a naive datetime (2020-01-29 15:57:56.376485) while time zone support is active.
  RuntimeWarning)
Nothing to do, skipping migration.

Here is the full migration process

Operations to perform:
  Apply all migrations: admin, auth, contenttypes, jira_ac, nodestore, sentry, sessions, sites, social_auth
Running migrations:
  Applying sentry.0001_initial... OK
  Applying contenttypes.0001_initial... OK
  Applying admin.0001_initial... OK
  Applying admin.0002_logentry_remove_auto_add... OK
  Applying contenttypes.0002_remove_content_type_name... OK
  Applying auth.0001_initial... OK
  Applying auth.0002_alter_permission_name_max_length... OK
  Applying auth.0003_alter_user_email_max_length... OK
  Applying auth.0004_alter_user_username_opts... OK
  Applying auth.0005_alter_user_last_login_null... OK
  Applying auth.0006_require_contenttypes_0002... OK
  Applying auth.0007_alter_validators_add_error_messages... OK
  Applying auth.0008_alter_user_username_max_length... OK
  Applying jira_ac.0001_initial... OK
  Applying nodestore.0001_initial... OK
  Applying sentry.0002_912_to_recent... OK
  Applying sentry.0003_auto_20191022_0122... OK
  Applying sentry.0004_bitfieldtestmodel_blankjsonfieldtestmodel_callabledefaultmodel_jsonfieldtestmodel_jsonfieldwithdefau... OK
  Applying sentry.0005_fix_content_types... OK
  Applying sentry.0006_sentryapp_date_published... OK
  Applying sentry.0007_auto_20191029_0131... OK
  Applying sentry.0008_auto_20191030_0016... OK
  Applying sentry.0009_auto_20191101_1608... OK
  Applying sentry.0010_auto_20191104_1641... OK
  Applying sentry.0011_remove_pagerdutyservice_service_id_from_state... OK
  Applying sentry.0012_remove_pagerdutyservice_service_id... OK
  Applying sentry.0013_auto_20191111_1829... OK
  Applying sentry.0014_delete_sentryappwebhookerror... OK
  Applying sentry.0015_delete_sentryappwebhookerror_db... OK
  Applying sentry.0016_delete_alert_rule_deprecated_fields... OK
  Applying sentry.0017_incident_aggregation... OK
  Applying sentry.0018_discoversavedquery_version... OK
  Applying sentry.0019_auto_20191114_2040... OK
  Applying sentry.0020_auto_20191125_1420... OK
  Applying sentry.0021_auto_20191203_1803... OK
  Applying sentry.0021_auto_20191202_1716... OK
  Applying sentry.0022_merge... OK
  Applying sentry.0023_hide_environment_none_20191126... OK
  Applying sentry.0024_auto_20191230_2052.../usr/local/lib/python2.7/site-packages/django/db/models/fields/__init__.py:1451: RuntimeWarning: DateTimeField Event.datetime received a naive datetime (2019-10-31 15:57:56.376485) while time zone support is active.
  RuntimeWarning)
/usr/local/lib/python2.7/site-packages/django/db/models/fields/__init__.py:1451: RuntimeWarning: DateTimeField Event.datetime received a naive datetime (2020-01-29 15:57:56.376485) while time zone support is active.
  RuntimeWarning)
Nothing to do, skipping migration.

 OK
  Applying sentry.0025_organizationaccessrequest_requester... OK
  Applying sentry.0026_delete_event... OK
  Applying sentry.0027_exporteddata... OK
  Applying sessions.0001_initial... OK
  Applying sites.0001_initial... OK
  Applying sites.0002_alter_domain_unique... OK
  Applying social_auth.0001_initial... OK
BuSHari picture BuSHari on 30 Jan 2020

Hey all, @lynnagara and I are looking into this. The issue only affects people migrating from 9.1.2, fresh installs are unaffected. Although we hope to resolve the issue by EOD today, you can use SENTRY_IMAGE=getsentry/sentry:b0ea6ea032d886edf24a70c1d1ac0d79e1462b39 ./install.sh (this is using a slightly older version of Sentry) to upgrade and then switch to latest afterwards in the meantime.

BYK picture BYK on 30 Jan 2020
👍2

Hi,

I was able to successfully upgrade our Sentry instance using the older version (https://github.com/getsentry/sentry/commit/b0ea6ea032d886edf24a70c1d1ac0d79e1462b39). Before doing that I've tried the upgrade Sentry using the latest available version (https://github.com/getsentry/sentry/commit/1e17bb461d26999c5cd89f6f2e274923ce0010d9) but ran into another bug:

  Applying sentry.0022_merge... OK
  Applying sentry.0023_hide_environment_none_20191126... OK
  Applying sentry.0024_auto_20191230_2052.../usr/local/lib/python2.7/site-packages/django/db/models/fields/__init__.py:1451: RuntimeWarning: DateTimeField Event.datetime received a naive datetime (2019-11-03 19:06:53.816964) while time zone support is active.
  RuntimeWarning)
/usr/local/lib/python2.7/site-packages/django/db/models/fields/__init__.py:1451: RuntimeWarning: DateTimeField Event.datetime received a naive datetime (2020-02-01 19:06:53.816964) while time zone support is active.
  RuntimeWarning)
Events to process: 106367

Traceback (most recent call last):
  File "/usr/local/bin/sentry", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python2.7/site-packages/sentry/runner/__init__.py", line 164, in main
    cli(prog_name=get_prog(), obj={}, max_content_width=100)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/sentry/runner/decorators.py", line 30, in inner
    return ctx.invoke(f, *args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/sentry/runner/commands/upgrade.py", line 168, in upgrade
    _upgrade(not noinput, traceback, verbosity, not no_repair)
  File "/usr/local/lib/python2.7/site-packages/sentry/runner/commands/upgrade.py", line 130, in _upgrade
    ignore_ghost_migrations=True,
  File "/usr/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 131, in call_command
    return command.execute(*args, **defaults)
  File "/usr/local/lib/python2.7/site-packages/django/core/management/base.py", line 330, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lib/python2.7/site-packages/django/core/management/commands/migrate.py", line 204, in handle
    fake_initial=fake_initial,
  File "/usr/local/lib/python2.7/site-packages/django/db/migrations/executor.py", line 115, in migrate
    state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
  File "/usr/local/lib/python2.7/site-packages/django/db/migrations/executor.py", line 145, in _migrate_all_forwards
    state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
  File "/usr/local/lib/python2.7/site-packages/sentry/new_migrations/monkey/executor.py", line 26, in apply_migration
    state, migration, fake=fake, fake_initial=fake_initial
  File "/usr/local/lib/python2.7/site-packages/django/db/migrations/executor.py", line 244, in apply_migration
    state = migration.apply(state, schema_editor)
  File "/usr/local/lib/python2.7/site-packages/django/db/migrations/migration.py", line 129, in apply
    operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
  File "/usr/local/lib/python2.7/site-packages/django/db/migrations/operations/special.py", line 193, in database_forwards
    self.code(from_state.apps, schema_editor)
  File "/usr/local/lib/python2.7/site-packages/sentry/migrations/0024_auto_20191230_2052.py", line 85, in backfill_eventstream
    skip_consume=True,
  File "/usr/local/lib/python2.7/site-packages/sentry/utils/services.py", line 104, in <lambda>
    context[key] = (lambda f: lambda *a, **k: getattr(self, f)(*a, **k))(key)
  File "/usr/local/lib/python2.7/site-packages/sentry/eventstream/snuba.py", line 116, in insert
    "datetime": event.datetime,
  File "/usr/local/lib/python2.7/site-packages/sentry/eventstore/models.py", line 101, in datetime
    date = datetime.fromtimestamp(timestamp)
TypeError: a float is required
Cleaning up...
fnkr picture fnkr on 1 Feb 2020

Thanks for trying fixing it.
I don't know if you deployed that fix to the last version, but the error is still presist but with different DateTime

Operations to perform:
  Apply all migrations: admin, auth, contenttypes, jira_ac, nodestore, sentry, sessions, sites, social_auth
Running migrations:
  Applying sentry.0024_auto_20191230_2052.../usr/local/lib/python2.7/site-packages/django/db/models/fields/__init__.py:1451: RuntimeWarning: DateTimeField Event.datetime received a naive datetime (2019-11-04 11:39:56.970526) while time zone support is active.
  RuntimeWarning)
/usr/local/lib/python2.7/site-packages/django/db/models/fields/__init__.py:1451: RuntimeWarning: DateTimeField Event.datetime received a naive datetime (2020-02-02 11:39:56.970526) while time zone support is active.
  RuntimeWarning)
Events to process: 6244

Traceback (most recent call last):
  File "/usr/local/bin/sentry", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python2.7/site-packages/sentry/runner/__init__.py", line 164, in main
    cli(prog_name=get_prog(), obj={}, max_content_width=100)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/sentry/runner/decorators.py", line 30, in inner
    return ctx.invoke(f, *args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/sentry/runner/commands/upgrade.py", line 168, in upgrade
    _upgrade(not noinput, traceback, verbosity, not no_repair)
  File "/usr/local/lib/python2.7/site-packages/sentry/runner/commands/upgrade.py", line 130, in _upgrade
    ignore_ghost_migrations=True,
  File "/usr/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 131, in call_command
    return command.execute(*args, **defaults)
  File "/usr/local/lib/python2.7/site-packages/django/core/management/base.py", line 330, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lib/python2.7/site-packages/django/core/management/commands/migrate.py", line 204, in handle
    fake_initial=fake_initial,
  File "/usr/local/lib/python2.7/site-packages/django/db/migrations/executor.py", line 115, in migrate
    state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
  File "/usr/local/lib/python2.7/site-packages/django/db/migrations/executor.py", line 145, in _migrate_all_forwards
    state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
  File "/usr/local/lib/python2.7/site-packages/sentry/new_migrations/monkey/executor.py", line 26, in apply_migration
    state, migration, fake=fake, fake_initial=fake_initial
  File "/usr/local/lib/python2.7/site-packages/django/db/migrations/executor.py", line 244, in apply_migration
    state = migration.apply(state, schema_editor)
  File "/usr/local/lib/python2.7/site-packages/django/db/migrations/migration.py", line 129, in apply
    operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
  File "/usr/local/lib/python2.7/site-packages/django/db/migrations/operations/special.py", line 193, in database_forwards
    self.code(from_state.apps, schema_editor)
  File "/usr/local/lib/python2.7/site-packages/sentry/migrations/0024_auto_20191230_2052.py", line 109, in backfill_eventstream
    skip_consume=True,
  File "/usr/local/lib/python2.7/site-packages/sentry/utils/services.py", line 104, in <lambda>
    context[key] = (lambda f: lambda *a, **k: getattr(self, f)(*a, **k))(key)
  File "/usr/local/lib/python2.7/site-packages/sentry/eventstream/snuba.py", line 114, in insert
    "message": event.search_message,
AttributeError: 'Event' object has no attribute 'search_message'
BuSHari picture BuSHari on 2 Feb 2020

@fnkr thanks for the report and sorry for the issue. Submitted another fix. @BuSHari you seem to be using an older version without the latest fix, can you try again with the latest fix?

BYK picture BYK on 3 Feb 2020

@BYK I've found an instance where the upgrade to 9.1.2 still doesn't work (I'll be putting up a PR to sentry to fix this shortly). This is specifically the same issue that @fnkr encountered.

The problem:
If you have been on Sentry 9.1.2 for less than 90 days (because you're upgrading in rapid succession from 9.0 to 9.1.2 to 10x) you likely have events that don't have an existing timestamp in the eventstore. So when it tries to insert the new event it takes the event.datetime and errors out because None is not a float.

If you comment out the try catch in the migration you'll get a stack trace that looks something like this (line numbers are slightly off due to added debugging statements)

Traceback (most recent call last):
  File "/usr/local/bin/sentry", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python2.7/site-packages/sentry/runner/__init__.py", line 166, in main
    cli(prog_name=get_prog(), obj={}, max_content_width=100)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/sentry/runner/decorators.py", line 30, in inner
    return ctx.invoke(f, *args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/sentry/runner/commands/upgrade.py", line 174, in upgrade
    _upgrade(not noinput, traceback, verbosity, not no_repair, with_nodestore)
  File "/usr/local/lib/python2.7/site-packages/sentry/runner/commands/upgrade.py", line 130, in _upgrade
    ignore_ghost_migrations=True,
  File "/usr/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 131, in call_command
    return command.execute(*args, **defaults)
  File "/usr/local/lib/python2.7/site-packages/django/core/management/base.py", line 330, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lib/python2.7/site-packages/django/core/management/commands/migrate.py", line 204, in handle
    fake_initial=fake_initial,
  File "/usr/local/lib/python2.7/site-packages/django/db/migrations/executor.py", line 115, in migrate
    state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
  File "/usr/local/lib/python2.7/site-packages/django/db/migrations/executor.py", line 145, in _migrate_all_forwards
    state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
  File "/usr/local/lib/python2.7/site-packages/sentry/new_migrations/monkey/executor.py", line 26, in apply_migration
    state, migration, fake=fake, fake_initial=fake_initial
  File "/usr/local/lib/python2.7/site-packages/django/db/migrations/executor.py", line 244, in apply_migration
    state = migration.apply(state, schema_editor)
  File "/usr/local/lib/python2.7/site-packages/django/db/migrations/migration.py", line 129, in apply
    operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
  File "/usr/local/lib/python2.7/site-packages/django/db/migrations/operations/special.py", line 193, in database_forwards
    self.code(from_state.apps, schema_editor)
  File "/usr/local/lib/python2.7/site-packages/sentry/migrations/0024_auto_20191230_2052.py", line 102, in backfill_eventstream
    print(float(event.datetime.strftime("%s")))
  File "/usr/local/lib/python2.7/site-packages/sentry/eventstore/models.py", line 103, in datetime
    date = datetime.fromtimestamp(timestamp)
ThomasWunderlich picture ThomasWunderlich on 16 Sep 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

eandersons picture eandersons  路  5Comments
WoLpH picture WoLpH  路  3Comments
dotconnor picture dotconnor  路  6Comments
harshulj picture harshulj  路  3Comments
NullIsNot0 picture NullIsNot0  路  5Comments