Onpremise: migration error on 9.1.2 --> 10

[bmaeser]

when i upgrade from 9.1.2 to the newest version (10) and run install.sh i get a bunch of migration errors on the sentry.0024_auto_20191230_2052 migration.

here is a snippet from my logs:

######### snip

Operations to perform:
  Apply all migrations: admin, auth, contenttypes, jira_ac, nodestore, sentry, sessions, sites, social_auth
Running migrations:
  Applying admin.0002_logentry_remove_auto_add... OK
  Applying auth.0007_alter_validators_add_error_messages... OK
  Applying auth.0008_alter_user_username_max_length... OK
  Applying sentry.0002_912_to_recent... OK
  Applying sentry.0003_auto_20191022_0122... OK
  Applying sentry.0004_bitfieldtestmodel_blankjsonfieldtestmodel_callabledefaultmodel_jsonfieldtestmodel_jsonfieldwithdefau... OK
  Applying sentry.0005_fix_content_types... OK
  Applying sentry.0006_sentryapp_date_published... OK
  Applying sentry.0007_auto_20191029_0131... OK
  Applying sentry.0008_auto_20191030_0016... OK
  Applying sentry.0009_auto_20191101_1608... OK
  Applying sentry.0010_auto_20191104_1641... OK
  Applying sentry.0011_remove_pagerdutyservice_service_id_from_state... OK
  Applying sentry.0012_remove_pagerdutyservice_service_id... OK
  Applying sentry.0013_auto_20191111_1829... OK
  Applying sentry.0014_delete_sentryappwebhookerror... OK
  Applying sentry.0015_delete_sentryappwebhookerror_db... OK
  Applying sentry.0016_delete_alert_rule_deprecated_fields... OK
  Applying sentry.0017_incident_aggregation... OK
  Applying sentry.0018_discoversavedquery_version... OK
  Applying sentry.0019_auto_20191114_2040... OK
  Applying sentry.0020_auto_20191125_1420... OK
  Applying sentry.0021_auto_20191203_1803... OK
  Applying sentry.0021_auto_20191202_1716... OK
  Applying sentry.0022_merge... OK
  Applying sentry.0023_hide_environment_none_20191126... OK
  Applying sentry.0024_auto_20191230_2052...Events to process: 134509

An error occured while trying to instert the following event: <sentry.eventstore.models.Event object at 0x7f0537cb1ed0>
.----
a float is required
An error occured while trying to instert the following event: <sentry.eventstore.models.Event object at 0x7f0537bb2fd0>
.----
a float is required
An error occured while trying to instert the following event: <sentry.eventstore.models.Event object at 0x7f0537cb1ed0>
.----
a float is required
An error occured while trying to instert the following event: <sentry.eventstore.models.Event object at 0x7f0537bb2fd0>
.----
a float is required
An error occured while trying to instert the following event: <sentry.eventstore.models.Event object at 0x7f0537cb1ed0>
.----
######### snip ( .... a lot more of those erros )

any idea how to fix this?

at the moment i reverted back to 9.1.2 and restored the postgres database from i backup i pulled prior to upgrading to v10

thanks a lot,
bernhard

[BYK]

Heya @bmaeser sorry I missed this. Are you still having issues?

[bmaeser]

i did not try it since i opened this issue. has there been any changes to this?

[BYK]

has there been any changes to this?

Yup, I think this got fixed. Would you mind giving this another shot with latest master?

[bmaeser]

yeah, will do as soon as i can

[bmaeser]

nope, still failing. same error, started from a fresh checkout from github (master)

[Skyross]

Just to confirm, we have the same error as well. I am trying to upgrade from 9.1.2.

UPD: I've patched 0024_auto_20191230_2052.py migration with debug info and found that somehow there are "broken events" with empty data at the DB (like {'event_id': u'c17fea391e4445ddbcdbe7550fcce9e7', 'project_id': 2, 'data': {}, 'group_id': 86899)

The data was captured before creating NewEvent instance:

event = NewEvent(
    project_id=e.project_id, event_id=e.event_id, group_id=e.group_id, data=e.data.data
)

UPD2: Looks like the issue lays at the code:

eventstream.insert(
    group=event.group,
    event=event,
    is_new=False,
    is_regression=False,
    is_new_group_environment=False,
    primary_hash=primary_hash,
    received_timestamp=event.data.get("received")
                       or float(event.datetime.strftime("%s")),
    skip_consume=True,
)

at event.datetime specifically, because it used value, not from the original event e...

[BYK]

@Skyross can you provide more details about your finding? I don't fully understand what you mean but seems like you are onto something and I want to get that fixed :)

[BYK]

Closing due to lack of response.

[bmaeser]

this is still an issue, i still can not migrate form 9.1.2 to 10

if you tell me what information you need, i gladly provide it - but i dont see anything i can do than posting logs.

[BYK]

@bmaeser what exactly is the issue? Can you please file a new issue with more details?

[bmaeser]

nothing has changed from the first comment on this issue. unable to upgrade from 9.1.2 to the newest version (10) sentry.0024_auto_20191230_2052throws a ton of a float is required (see log-snippet on the very top of this thread).

i dont have any more detail. i would provide it if i could.

[BYK]

nothing has changed from the first comment on this issue.

Oh, sorry I didn't realize you to be the author of the issue :) Reopening, will try to investigate.

[jgelens]

I encounter the same issue when migrating to 10 :(

[beevelop]

Same issue on our site when trying to upgrade from 9.0.0 > 9.1.2 > 20.7.1. Is there any workaround / manual step in the meantime?

[jgelens]

@BYK any update on this or maybe some workaround? Would love to migrate to the latest version :)

[BYK]

@jgelens we want to get this ready by the end of this week and there's a PR for it getsentry/sentry#19881. If we don't hear back from @vladborovtsov by tomorrow, we'll try to finalize it ourselves.

You may also try this workaround for now suggested and tried by @shulcsm: https://github.com/getsentry/onpremise/issues/591#issuecomment-659871415

[jgelens]

@BYK Thanks a lot, waiting for the official PR :)

[BYK]

Seems like this is fixed with getsentry/sentry#20512 so closing. Crossed fingers for 20.9.0 release!

[fklingler]

This is still not working for me. I'm not sure the install script is loading the good sentry docker image.

I tried using SENTRY_IMAGE=getsentry/sentry:20.9.0 ./install.sh but I'm still having these logs mentioning latest:

Fetching and updating Docker images...

Some service image(s) must be built from source by running:
    docker-compose build snuba-cleanup post-process-forwarder sentry-cleanup symbolicator-cleanup web worker cron ingest-consumer
latest: Pulling from getsentry/sentry
Digest: sha256:9766553fd216618576a64cf27358c6886ecd51d44c0ba29d8b060ef80837c490
Status: Image is up to date for getsentry/sentry:latest
docker.io/getsentry/sentry:latest

And I'm getting the same errors that are apparently supposed to be fixed:

Running migrations:
  Applying sentry.0024_auto_20191230_2052...Events to process: 258

An error occured while trying to instert the following event: <sentry.eventstore.models.Event object at 0x7f32d2b9c310>
.----
a float is required
An error occured while trying to instert the following event: <sentry.eventstore.models.Event object at 0x7f32d231e690>
.----
a float is required
An error occured while trying to instert the following event: <sentry.eventstore.models.Event object at 0x7f32d2b9c310>
.----
a float is required
An error occured while trying to instert the following event: <sentry.eventstore.models.Event object at 0x7f32d22f0a10>
.----
a float is required
An error occured while trying to instert the following event: <sentry.eventstore.models.Event object at 0x7f32d2b9c310>

What am I doing wrong?

[rrouene]

Hi,

Same problem here.
Even with the last release.

Thank you

[BYK]

@fklingler @Sowappy - I'd just download the zip or tar archive from https://github.com/getsentry/onpremise/releases/tag/20.9.0, replace the contents of your onpremise folder and run ./install.sh without anything extra.

[mzmrk]

I have same issue.

I did some digging and concluded that events that contains breadcrumbs are not migrated.
It seems that those kinds of events are missing some fields like timestamp, platform, event_id, environment etc.

Attached some examples with redacted data.

I already migrated my data since I really needed Sentry R8 obfuscation support on Android.

this-event-is-not-migrating.txt
this-one-migrates.txt

Edit: Sorry, I named the files opposite way. And I can't upload again with correct names to fix in this post.

[BYK]

@mzmrk not sure when you tried this but this should be resolved by getsentry/sentry#20512 now.

[fklingler]

@BYK I made all my tests with a clone of this repository (onpremise), on master.
I just tried with the 20.9.0 tag but I have the same errors.

[BYK]

@BYK I made all my tests with a clone of this repository (onpremise), on master.

That's why you are seeing references to latest, master follows latest.

I just tried with the 20.9.0 tag but I have the same errors.

Looking closer, we just report the error but move forward with the migration. If the event is missing data, we cannot migrate it.

[fklingler]

I can confirm the migration did not move forward. I first aborted the command myself when I saw the errors, but I just restarted it and it stopped after all those errors:

.----
a float is required
An error occured while trying to instert the following event: <sentry.eventstore.models.Event object at 0x7f193bb82990>
.----
a float is required
Traceback (most recent call last):
  File "/usr/local/bin/sentry", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python2.7/site-packages/sentry/runner/__init__.py", line 166, in main
    cli(prog_name=get_prog(), obj={}, max_content_width=100)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/sentry/runner/decorators.py", line 30, in inner
    return ctx.invoke(f, *args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/sentry/runner/commands/upgrade.py", line 174, in upgrade
    _upgrade(not noinput, traceback, verbosity, not no_repair, with_nodestore)
  File "/usr/local/lib/python2.7/site-packages/sentry/runner/commands/upgrade.py", line 130, in _upgrade
    ignore_ghost_migrations=True,
  File "/usr/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 131, in call_command
    return command.execute(*args, **defaults)
  File "/usr/local/lib/python2.7/site-packages/django/core/management/base.py", line 330, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lib/python2.7/site-packages/django/core/management/commands/migrate.py", line 204, in handle
    fake_initial=fake_initial,
  File "/usr/local/lib/python2.7/site-packages/django/db/migrations/executor.py", line 115, in migrate
    state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
  File "/usr/local/lib/python2.7/site-packages/django/db/migrations/executor.py", line 145, in _migrate_all_forwards
    state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
  File "/usr/local/lib/python2.7/site-packages/sentry/new_migrations/monkey/executor.py", line 26, in apply_migration
    state, migration, fake=fake, fake_initial=fake_initial
  File "/usr/local/lib/python2.7/site-packages/django/db/migrations/executor.py", line 244, in apply_migration
    state = migration.apply(state, schema_editor)
  File "/usr/local/lib/python2.7/site-packages/django/db/migrations/migration.py", line 129, in apply
    operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
  File "/usr/local/lib/python2.7/site-packages/django/db/migrations/operations/special.py", line 193, in database_forwards
    self.code(from_state.apps, schema_editor)
  File "/usr/local/lib/python2.7/site-packages/sentry/migrations/0024_auto_20191230_2052.py", line 103, in backfill_eventstream
    "Cannot migrate any event. If this is okay, re-run migrations with SENTRY_SKIP_EVENTS_BACKFILL_FOR_10 environment variable set to skip this step."
Exception: Cannot migrate any event. If this is okay, re-run migrations with SENTRY_SKIP_EVENTS_BACKFILL_FOR_10 environment variable set to skip this step.
An error occurred, caught SIGERR on line 280
Cleaning up...

Should I just use the SENTRY_SKIP_EVENTS_BACKFILL_FOR_10 environment variable? For now I still want to migrate everything I can.

[BYK]

Should I just use the SENTRY_SKIP_EVENTS_BACKFILL_FOR_10 environment variable? For now I still want to migrate everything I can.

I'd recommend so as those events don't seem recoverable/migratable. You'll still have all your settings, projects etc. migrated. Just not your event data.

[mzmrk]

@fklingler You can migrate with those errors. It will just migrate the events it can. Events with errors are skipped.

@BYK It's weird because those same exact events are being displayed correctly in Sentry version 9.1.2. For now I'm just keeping old VM with older version of sentry before migration, in case I want to see some older events.

Edit: I kinda moved on with loss of some events. If someone else wants to investigate migration errors of events that contain breadcrumbs feel free to do so. (I checked 5 non-migrated and 5 migrated events. All non-migrated had breadcrumbs in it. I was simply printing out event data that was already pulled by migration script from database.)

[fklingler]

Ok, thanks a lot for your help.
Migration just passed for me, I finally have an up-to-date version of Sentry 🎉

[BYK]

@mzmrk

@BYK It's weird because those same exact events are being displayed correctly in Sentry version 9.1.2. For now I'm just keeping old VM with older version of sentry before migration, in case I want to see some older events.

Not weird at all. Sentry 10+ uses a completely different search and storage system for events. This is why we need the migration in the first place.