Onpremise: Migrating from python install

[tarjei]

Hi, I tried to upgrade from 8.13 to 9.0 and at the same time go from a local install to a docker install.

These are my notes.

Please note that I am no postgresql expert so I might have done things in an extra hard or stupid way.

TLDR;

• The postgres sentry user must be a superuser to run migrations
• You must upgrade and run all migrations up to 8.22 before upgrading to 9.0.

Importing the db

1. I created a separate import volume and copied my old db dump into that.

```docker-compose.yml:
...
postgres:
restart: unless-stopped
image: postgres:9.5
volumes:
- sentry-postgres:/var/lib/postgresql/data
- import:/import

...
volumes:
sentry-data:
sentry-postgres:
import:

And copied in the db:

docker volume inspect sentry_sentry-postgres
sudo cp sentry.sql /var/lib/docker/volumes/sentry_sentry-postgres-import/_data/

2. Then I created the sentry role within postgres bc that was the name of the role that has used this db earlier.

docker-compose start postgres postgres
docker-compose exec -u postgres postgres bash

And within the docker container:

createuser -d -sEP sentry
psql < import/sentry.sql

## Migration
To migrate the DB I hit a lot of snags until i understood that I had to upgrade top `8.22` first. 

So I edited the Dockerfile to:

FROM sentry:8.22-onbuild

Then rebuild everything:

make build
docker-compose build
`` And then ran the upgrade: docker-compose run --rm web upgrade`

Then upgrade to next version by editing the dockerfile, running a rebuild and then an upgrade.

Odds and ends

Sentry upgrade didn't relate to the changes i did in the config file so I used:
docker-compose run --rm web --config=/usr/src/sentry/sentry.conf.py upgrade

General ideas:

It would be very useful to have a command to print out the configuration in the container and also a command to test all connections to see that things are configured correctly.

[BYK]

@tarjei - I know this is quite late but thanks a lot for documenting your journey! It sure will help us create an official migration doc soon.

[stayallive]

Now with v10 out I was wondering if some kind of migration guide already exists? I would very much like to "migrate"/upgrade to the Docker version and Sentry 10. But I am unsure how to go about this and/or if the above is a good way to do it.

[BYK]

Hi @stayallive - the reason I kept this issue is to eventually document this process. The challenge is the diversity of Python-based setups. For now I'd say all you'd need would be the sentry-postgres and sentry-data volumes and then run ./install.sh as long as you are on 9.1.2.

The DB migration method @tarjei shared makes sense. Alternatively, you may try copying the files through a volume mount into the sentry-postgres volume. Same for sentry-data if you want to keep uploaded avatars and anything else you've uploaded to Sentry such as attachments, source maps etc.

[to-be-the-best]

@BYK Do you have any suggestions for if you do not have 9.1.2?
I want to update from from python 8.19.0 to docker-compose v10.

I have already spend 2 days trying to get it working, trying both the master branch and commit ea47a628 which is the last commit before tarjei's message.
Are there any particular commits you know of i have to look out for between those 2 commits with breaking changes, that make me want to hit my head against a wall?

Thanks in advance,
Tom

[tarjei]

Do it in stages. First upgrade the db to the latest v8, then to 9 and then
to 10. The hard part is getting the migrations done.

fre. 17. jan. 2020 kl. 20:37 skrev Tom Reinders [e-mail]:

@BYK https://github.com/BYK Do you have any suggestions for if you do
not have 9.1.2?
I want to update from from python 8.19.0 to docker-compose v10.

I have already spend 2 days trying to get it working, trying both the
master branch and commit ea47a62
https://github.com/getsentry/onpremise/commit/ea47a6283f089d0f5c8202fed7b573e11edd8122
which is the last commit before tarjei's message.
Are there any particular commits you know of i have to look out for
between those 2 commits with breaking changes, that make me want to hit my
head against a wall?

Thanks in advance,
Tom

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/getsentry/onpremise/issues/128?email_source=notifications&email_token=AABTSP64N5HT3KCLSIJOB2LQ6ICILA5CNFSM4FXJ6UG2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJIX5CA#issuecomment-575766152,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AABTSP2RWYLT47YELHFGQFDQ6ICILANCNFSM4FXJ6UGQ
.

>

---

Tarjei Huse
Mobil: 920 63 413

[Apollon77]

@BYK

For now I'd say all you'd need would be the sentry-postgres and sentry-data volumes and then run ./install.sh as long as you are on 9.1.2.
What is the docker mount point for sentry-data? I anticipated /data in the sentry-web container from getsentry/sentry docker image but there is not data written in so far for me ... (I only installed it and adjusted some docker env vars and created first project and such)..

[Apollon77]

The described way above did not worked for me. :-(

I copied the 9.1.2 postgres data into sentry 10 volume and tried to do migration ... in fact it initialized the database new :-( So it seems to have no migraton from 9.1.2 to 10 ... or do I miss something?

Log:

root@sentry:~/sentry/onpremise# docker-compose run --rm web upgrade
Starting sentry_onpremise_smtp_1_7c412370067f ... done
Starting sentry_onpremise_redis_1_b84e6a77d347 ... done
Starting sentry_onpremise_kafka_1_299ff441bd72        ... done
Starting sentry_onpremise_postgres_1_77b6b0de020d     ... done
Starting sentry_onpremise_memcached_1_1d65f926c1a9    ... done
Starting sentry_onpremise_kafka_1_299ff441bd72        ... done
Starting sentry_onpremise_snuba-replacer_1_13b9e8388e1d ... done
Starting sentry_onpremise_snuba-api_1_d406da8e7046      ... done
21:02:29 [WARNING] sentry.utils.geo: settings.GEOIP_PATH_MMDB not configured.
21:02:34 [INFO] sentry.plugins.github: apps-not-configured
Operations to perform:
  Apply all migrations: admin, auth, contenttypes, jira_ac, nodestore, sentry, sessions, sites, social_auth
Running migrations:
  Applying sentry.0001_initial... OK
  Applying contenttypes.0001_initial... OK
  Applying admin.0001_initial... OK
  Applying admin.0002_logentry_remove_auto_add... OK
  Applying contenttypes.0002_remove_content_type_name... OK
  Applying auth.0001_initial... OK
  Applying auth.0002_alter_permission_name_max_length... OK
  Applying auth.0003_alter_user_email_max_length... OK
  Applying auth.0004_alter_user_username_opts... OK
  Applying auth.0005_alter_user_last_login_null... OK
  Applying auth.0006_require_contenttypes_0002... OK
  Applying auth.0007_alter_validators_add_error_messages... OK
  Applying auth.0008_alter_user_username_max_length... OK
  Applying jira_ac.0001_initial... OK
  Applying nodestore.0001_initial... OK
  Applying sentry.0002_912_to_recent... OK
  Applying sentry.0003_auto_20191022_0122... OK
  Applying sentry.0004_bitfieldtestmodel_blankjsonfieldtestmodel_callabledefaultmodel_jsonfieldtestmodel_jsonfieldwithdefau... OK
  Applying sentry.0005_fix_content_types... OK
  Applying sentry.0006_sentryapp_date_published... OK
  Applying sentry.0007_auto_20191029_0131... OK
  Applying sentry.0008_auto_20191030_0016... OK
  Applying sentry.0009_auto_20191101_1608... OK
  Applying sentry.0010_auto_20191104_1641... OK
  Applying sentry.0011_remove_pagerdutyservice_service_id_from_state... OK
  Applying sentry.0012_remove_pagerdutyservice_service_id... OK
  Applying sentry.0013_auto_20191111_1829... OK
  Applying sentry.0014_delete_sentryappwebhookerror... OK
  Applying sentry.0015_delete_sentryappwebhookerror_db... OK
  Applying sentry.0016_delete_alert_rule_deprecated_fields... OK
  Applying sentry.0017_incident_aggregation... OK
  Applying sentry.0018_discoversavedquery_version... OK
  Applying sentry.0019_auto_20191114_2040... OK
  Applying sentry.0020_auto_20191125_1420... OK
  Applying sentry.0021_auto_20191203_1803... OK
  Applying sentry.0021_auto_20191202_1716... OK
  Applying sentry.0022_merge... OK
  Applying sentry.0023_hide_environment_none_20191126... OK
  Applying sentry.0024_auto_20191230_2052.../usr/local/lib/python2.7/site-packages/django/db/models/fields/__init__.py:1451: RuntimeWarning: DateTimeField Event.datetime received a naive datetime (2019-10-30 21:03:29.604312) while time zone support is active.
  RuntimeWarning)
/usr/local/lib/python2.7/site-packages/django/db/models/fields/__init__.py:1451: RuntimeWarning: DateTimeField Event.datetime received a naive datetime (2020-01-28 21:03:29.604312) while time zone support is active.
  RuntimeWarning)
Nothing to do, skipping migration.

 OK
  Applying sentry.0025_organizationaccessrequest_requester... OK
  Applying sentry.0026_delete_event... OK
  Applying sentry.0027_exporteddata... OK
  Applying sessions.0001_initial... OK
  Applying sites.0001_initial... OK
  Applying sites.0002_alter_domain_unique... OK
  Applying social_auth.0001_initial... OK
21:03:31 [WARNING] sentry: Cannot initiate onboarding for organization (1) due to missing owners
Created internal Sentry project (slug=internal, id=1)

Would you like to create a user account now? [Y/n]: y
Email: XXXXXX
Password: 
Repeat for confirmation: 
User created: XXXXX
Added to organization: sentry
Creating missing DSNs
Correcting Group.num_comments counter
root@sentry:~/sentry/onpremise# 

[BYK]

So it seems to have no migraton from 9.1.2 to 10 ... or do I miss something?

You definitely miss something as we support migrating from 9.1.2 to 10.

I copied the 9.1.2 postgres data into sentry 10 volume and tried to do migration

Seems like this didn't go as smoothly or there's a confusion around volume names. If you can provide more details about how you did this, we can identify the root cause.

[Apollon77]

Ok, I did the following:

• stopped 9.1.2 postgres
• took file from postgres data volume from 9.12 and copied it 1:1 into /var/lib/docker/volume/sentry-postgres/_data (structure was basically the same) of the sentry-10 system
• then I did a try with install.sh, but also directly tried the commands from above here - both lead to same result ... see logs

I'm sure that in general the DB files where used because I needed to change the config file to add the postgres db credentials like defined for the "old" 9.1.2 container because else I got erros when the processes tried to connect to the db. with the existing user "sentry" and his password it worked.

[to-be-the-best]

I had the same issue.
You will have to make a dump from the 9 version and import that in the database, sort of following "Importing the db" step of @tarjei message combined with stealing some commands from install.sh to get the database running in the first place without starting the migrations.
Then you may try to run install.sh hoping it is not bringing as many problems as it did to me.
I think i got mine working yesterday, after so many hours.

PS: if you have a lot of data in the (i'm not sure but i believe it was called nodestore?) you may want to get that data lost.
Apparently there is a migration that does not like running over 2.4 million rows for some reason... why whould that be..... (Started the migration on friday evening and it was still running on monday afternoon.......)

[Apollon77]

I solved it for me by yimply starting over new because I only had one project in it for the start and got that "switch to 10" point very early, so "solve" for me, but I think many other users need that

[samuelebistoletti]

Hi,
tried to upgrade a 9 install to 10, something brokes during migration:

Applying sentry.0023_hide_environment_none_20191126... OK
Applying sentry.0024_auto_20191230_2052.../usr/local/lib/python2.7/site-packages/django/db/models/fields/__init__.py:1451: RuntimeWarning: DateTimeField Event.datetime received a naive datetime (2019-11-01 14:59:52.039221) while time zone support is active.
RuntimeWarning)
/usr/local/lib/python2.7/site-packages/django/db/models/fields/__init__.py:1451: RuntimeWarning: DateTimeField Event.datetime received a naive datetime (2020-01-30 14:59:52.039221) while time zone support is active.
RuntimeWarning)
Events to process: 20791

Traceback (most recent call last):
File "/usr/local/bin/sentry", line 8, in
sys.exit(main())
File "/usr/local/lib/python2.7/site-packages/sentry/runner/__init__.py", line 164, in main
cli(prog_name=get_prog(), obj={}, max_content_width=100)
File "/usr/local/lib/python2.7/site-packages/click/core.py", line 722, in __call__
return self.main(args, *kwargs)
File "/usr/local/lib/python2.7/site-packages/click/core.py", line 697, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python2.7/site-packages/click/core.py", line 1066, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python2.7/site-packages/click/core.py", line 895, in invoke
return ctx.invoke(self.callback, *ctx.params)
File "/usr/local/lib/python2.7/site-packages/click/core.py", line 535, in invoke
return callback(args, *kwargs)
File "/usr/local/lib/python2.7/site-packages/click/decorators.py", line 17, in new_func
return f(get_current_context(), *args, *kwargs)
File "/usr/local/lib/python2.7/site-packages/sentry/runner/decorators.py", line 30, in inner
return ctx.invoke(f, args, *kwargs)
File "/usr/local/lib/python2.7/site-packages/click/core.py", line 535, in invoke
return callback(args, *kwargs)
File "/usr/local/lib/python2.7/site-packages/click/decorators.py", line 17, in new_func
return f(get_current_context(), args, *kwargs)
File "/usr/local/lib/python2.7/site-packages/sentry/runner/commands/upgrade.py", line 168, in upgrade
_upgrade(not noinput, traceback, verbosity, not no_repair)
File "/usr/local/lib/python2.7/site-packages/sentry/runner/commands/upgrade.py", line 130, in _upgrade
ignore_ghost_migrations=True,
File "/usr/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 131, in call_command
return command.execute(args, *defaults)
File "/usr/local/lib/python2.7/site-packages/django/core/management/base.py", line 330, in execute
output = self.handle(args, *options)
File "/usr/local/lib/python2.7/site-packages/django/core/management/commands/migrate.py", line 204, in handle
fake_initial=fake_initial,
File "/usr/local/lib/python2.7/site-packages/django/db/migrations/executor.py", line 115, in migrate
state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
File "/usr/local/lib/python2.7/site-packages/django/db/migrations/executor.py", line 145, in _migrate_all_forwards
state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
File "/usr/local/lib/python2.7/site-packages/sentry/new_migrations/monkey/executor.py", line 26, in apply_migration
state, migration, fake=fake, fake_initial=fake_initial
File "/usr/local/lib/python2.7/site-packages/django/db/migrations/executor.py", line 244, in apply_migration
state = migration.apply(state, schema_editor)
File "/usr/local/lib/python2.7/site-packages/django/db/migrations/migration.py", line 129, in apply
operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
File "/usr/local/lib/python2.7/site-packages/django/db/migrations/operations/special.py", line 193, in database_forwards
self.code(from_state.apps, schema_editor)
File "/usr/local/lib/python2.7/site-packages/sentry/migrations/0024_auto_20191230_2052.py", line 109, in backfill_eventstream
skip_consume=True,
File "/usr/local/lib/python2.7/site-packages/sentry/utils/services.py", line 104, in
context[key] = (lambda f: lambda a, *k: getattr(self, f)(a, *k))(key)
File "/usr/local/lib/python2.7/site-packages/sentry/eventstream/snuba.py", line 114, in insert
"message": event.search_message,
AttributeError: 'Event' object has no attribute 'search_message'

[BYK]

@samuelebistoletti - please see #366 for your issue.

@to-be-the-best

Apparently there is a migration that does not like running over 2.4 million rows for some reason... why whould that be..... (Started the migration on friday evening and it was still running on monday afternoon.......)

That is a lot of events! Maybe you can reduce event retention days first to shed some of that data?

[to-be-the-best]

Nah, TRUNCATE nodestore_node; works too.
Also don't really want to do it all again.....
I already spend like 75 hours on this.

[github-actions[bot]]

This issue has gone three weeks without activity. In another week, I will close it.

But! If you comment or otherwise update it, I will reset the clock, and if you label it Status: Accepted, I will leave it alone ... forever!

---

"A weed is but an unloved flower." ― _Ella Wheeler Wilcox_ 🥀