Onpremise: Postgres errors when upgrading from 9.1.1 to 10.0

[tchwpkgorg]

Code (master branch) downloaded today (2010-01-08).

Running:

# docker-compose down
# ./install.sh 

Fails with:

fixing permissions on existing directory /var/lib/postgresql/9.6/data ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default timezone ... Etc/UTC
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok

Success. You can now start the database server using:

    pg_ctl -D /var/lib/postgresql/9.6/data -l logfile start


WARNING: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.

*failure*
Consult the last few lines of "pg_upgrade_server.log" for
the probable cause of the failure.

There seems to be a postmaster servicing the old cluster.
Please shutdown that postmaster and try again.
Failure, exiting
Cleaning up...

Not sure how to proceed.

[Fogapod]

I had the same error, running script again fixed it

[tchwpkgorg]

I've tried to run it several times already, but it still errors.

Also stopped all docker containers, but it still errors.

The last idea was to do:

rm /var/lib/docker/volumes/sentry-postgres/_data/postmaster.pid

Then install.sh errors with:

fixing permissions on existing directory /var/lib/postgresql/9.6/data ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default timezone ... Etc/UTC
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok

Success. You can now start the database server using:

    pg_ctl -D /var/lib/postgresql/9.6/data -l logfile start


WARNING: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Performing Consistency Checks
-----------------------------
Checking cluster versions                                   ok

The source cluster was not shut down cleanly.
Failure, exiting
Cleaning up...

[tchwpkgorg]

Had to do "docker-compose down" in the sources dir of sentry 9.1. Only this would shut down postgres cleanly.

Doing "docker-compose down" in the sources dir of sentry 10.0 would not stop postgres cleanly and would result in above errors.

After this, ./install.sh executed in sources dir of sentry 10.0 failed with exit code 1 and:

Creating sentry_onpremise_smtp_1           ... done
Creating sentry_onpremise_postgres_1       ... done

ERROR: for sentry_onpremise_snuba-replacer_1  UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)

ERROR: for sentry_onpremise_snuba-consumer_1  UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)

ERROR: for sentry_onpremise_snuba-api_1  UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)

ERROR: for snuba-replacer  UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)

ERROR: for snuba-consumer  UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)

ERROR: for snuba-api  UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)
An HTTP request took too long to complete. Retry with --verbose to obtain debug information.
If you encounter this issue regularly because of slow network conditions, consider setting COMPOSE_HTTP_TIMEOUT to a higher value (current value: 60).
Cleaning up...

[tchwpkgorg]

And after running ./install.sh it failed with:

aking migration for admin.0001_initial
Faking migration for sessions.0001_initial
Operations to perform:
  Apply all migrations: nodestore, sessions, admin, sentry, jira_ac, sites, auth, social_auth, contenttypes
Running migrations:
  Rendering model states... DONE
  Applying admin.0002_logentry_remove_auto_add... OK
  Applying auth.0007_alter_validators_add_error_messages... OK
  Applying sentry.0002_912_to_recent...Traceback (most recent call last):
  File "/usr/local/bin/sentry", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python2.7/site-packages/sentry/runner/__init__.py", line 164, in main
    cli(prog_name=get_prog(), obj={}, max_content_width=100)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/sentry/runner/decorators.py", line 30, in inner
    return ctx.invoke(f, *args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/sentry/runner/commands/upgrade.py", line 168, in upgrade
    _upgrade(not noinput, traceback, verbosity, not no_repair)
  File "/usr/local/lib/python2.7/site-packages/sentry/runner/commands/upgrade.py", line 130, in _upgrade
    ignore_ghost_migrations=True,
  File "/usr/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 119, in call_command
    return command.execute(*args, **defaults)
  File "/usr/local/lib/python2.7/site-packages/django/core/management/base.py", line 399, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lib/python2.7/site-packages/django/core/management/commands/migrate.py", line 200, in handle
    executor.migrate(targets, plan, fake=fake, fake_initial=fake_initial)
  File "/usr/local/lib/python2.7/site-packages/django/db/migrations/executor.py", line 92, in migrate
    self._migrate_all_forwards(plan, full_plan, fake=fake, fake_initial=fake_initial)
  File "/usr/local/lib/python2.7/site-packages/django/db/migrations/executor.py", line 121, in _migrate_all_forwards
    state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
  File "/usr/local/lib/python2.7/site-packages/sentry/new_migrations/monkey/executor.py", line 26, in apply_migration
    state, migration, fake=fake, fake_initial=fake_initial
  File "/usr/local/lib/python2.7/site-packages/django/db/migrations/executor.py", line 198, in apply_migration
    state = migration.apply(state, schema_editor)
  File "/usr/local/lib/python2.7/site-packages/django/db/migrations/migration.py", line 123, in apply
    operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
  File "/usr/local/lib/python2.7/site-packages/django/db/migrations/operations/fields.py", line 201, in database_forwards
    schema_editor.alter_field(from_model, from_field, to_field)
  File "/usr/local/lib/python2.7/site-packages/django/db/backends/base/schema.py", line 482, in alter_field
    old_db_params, new_db_params, strict)
  File "/usr/local/lib/python2.7/site-packages/django/db/backends/postgresql/schema.py", line 116, in _alter_field
    new_db_params, strict,
  File "/usr/local/lib/python2.7/site-packages/django/db/backends/base/schema.py", line 634, in _alter_field
    params,
  File "/usr/local/lib/python2.7/site-packages/django/db/backends/base/schema.py", line 110, in execute
    cursor.execute(sql, params)
  File "/usr/local/lib/python2.7/site-packages/sentry_sdk/integrations/django/__init__.py", line 472, in execute
    return real_execute(self, sql, params)
  File "/usr/local/lib/python2.7/site-packages/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
  File "/usr/local/lib/python2.7/site-packages/django/db/utils.py", line 95, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/usr/local/lib/python2.7/site-packages/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
  File "/usr/local/lib/python2.7/site-packages/sentry/db/postgres/decorators.py", line 81, in inner
    raise_the_exception(self.db, e)
  File "/usr/local/lib/python2.7/site-packages/sentry/db/postgres/decorators.py", line 79, in inner
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/sentry/db/postgres/decorators.py", line 22, in inner
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/sentry/db/postgres/decorators.py", line 99, in inner
    six.reraise(exc_info[0], exc_info[0](msg), exc_info[2])
  File "/usr/local/lib/python2.7/site-packages/sentry/db/postgres/decorators.py", line 95, in inner
    return func(self, sql, *args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/sentry/db/postgres/base.py", line 76, in execute
    return self.cursor.execute(sql, clean_bad_params(params))
django.db.utils.IntegrityError: NotNullViolation('column "project_id" contains null values\n',)
SQL: ALTER TABLE "sentry_groupedmessage" ALTER COLUMN "project_id" SET NOT NULL
Cleaning up...

So looks the upgrade broke my installation, not that good :(

[tchwpkgorg]

Fixed the last one with:

postgres=# select * from sentry_groupedmessage where project_id is NULL;

postgres=# update sentry_groupedmessage set project_id = 3 where id = <ID>;
UPDATE 1

After which ./install.sh finished successfully.

[tchwpkgorg]

However, "docker-compose up -d" fails:

# docker-compose up -d
Starting sentry_onpremise_postgres_1             ... done
Starting sentry_onpremise_memcached_1            ... done
Starting sentry_onpremise_zookeeper_1            ... done
Starting sentry_onpremise_symbolicator_1         ... done
Starting sentry_onpremise_redis_1                ... done
Creating sentry_onpremise_symbolicator-cleanup_1 ... done
Starting sentry_onpremise_smtp_1                 ... done
Starting sentry_onpremise_clickhouse_1           ... done
Starting sentry_onpremise_kafka_1                ... done
Starting sentry_onpremise_snuba-api_1            ... done
Starting sentry_onpremise_snuba-consumer_1       ... done
Starting sentry_onpremise_snuba-replacer_1       ... done
Creating sentry_onpremise_snuba-cleanup_1          ... done
Creating sentry_onpremise_web_1                  ... 
Creating sentry_onpremise_cron_1                 ... 
Creating sentry_onpremise_sentry-cleanup_1         ... done
Creating sentry_onpremise_post-process-forwarder_1 ... 
Creating sentry_onpremise_worker_1                 ... 

ERROR: for sentry_onpremise_post-process-forwarder_1  UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)

ERROR: for sentry_onpremise_web_1  UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)

ERROR: for sentry_onpremise_cron_1  UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)

ERROR: for sentry_onpremise_worker_1  UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)

ERROR: for post-process-forwarder  UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)

ERROR: for web  UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)

ERROR: for cron  UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)

ERROR: for worker  UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)
ERROR: An HTTP request took too long to complete. Retry with --verbose to obtain debug information.
If you encounter this issue regularly because of slow network conditions, consider setting COMPOSE_HTTP_TIMEOUT to a higher value (current value: 60).

[BYK]

@tchwpkgorg sorry for the troubles you had and thanks a lot for sharing the journey here!

Had to do "docker-compose down" in the sources dir of sentry 9.1. Only this would shut down postgres cleanly.

Doing "docker-compose down" in the sources dir of sentry 10.0 would not stop postgres cleanly and would result in above errors.

This is correct as the older versions did not have an explicit project name. This made docker-compose use the directory name as the project name. Since the new version has an explicit name, which is very likely do be different from the directory name. Just running docker-compose down after pulling does not work. That said I tried to protect against this in the install script: https://github.com/getsentry/onpremise/blob/b2080c14467fd44d10c3faffc7efc80a9d98c96d/install.sh#L65-L69

It was working for my local upgrade tests, not sure why it failed for you.

Regarding the failed migration, I'd guess it was because you were on 9.1.1 and not 9.1.2. I should add that as a warning to the readme as the v10 migrations need you to be on 9.1.2 first for a clean upgrade.

Finally, I'd guess sentry just takes some time to get up. I recommend simply running docker-compose up -d again and see if it works. Alternatively you can do that by increasing the timeout so it just waits longer: COMPOSE_HTTP_TIMEOUT=300 docker-compose up -d.

Hope this helps!

[Christophe31]

I had same db issue, I changed pg version back to the old to start & stop cleanly after i deleted the pid. How to get to 9.1.2 version on premise?

found:
in docker compose use sentry in the image field of the web service then run sentry upgrade

[BYK]

How to get to 9.1.2 version on premise?

Ideal way would be to checkout 9.1.2 tag from this repo and then run SENTRY_IMAGE=sentry:9.1.2 ./install.sh.

[tchwpkgorg]

I recommend simply running docker-compose up -d again and see if it works.
COMPOSE_HTTP_TIMEOUT=300 docker-compose up -d.

Indeed! And thanks for the tip!

[zanechua]

I was on 9.1.2 and still faced the same errors mentioned in the first post of this thread.

Had to follow the same steps as @tchwpkgorg but got a different error about removing sentry-postgres volume

Error response from daemon: unable to remove volume: remove sentry-postgres: volume is in use

EDIT: So I guess something really wonky happened and required me to manually remove the volume. Even using the --force flag wouldn't allow for the removal of the volume. Had to resort to the first answer in the following thread.

https://serverfault.com/questions/892656/docker-volume-rm-force-has-no-impact

Had to manually run the commands in the install.sh script and recreate the volume folders. Subsequently everything else in the script ran fine. Sentry seems to have booted up fine..

[josh-degraw]

Fixed the last one with:

postgres=# select * from sentry_groupedmessage where project_id is NULL;

postgres=# update sentry_groupedmessage set project_id = 3 where id = <ID>;
UPDATE 1

After which ./install.sh finished successfully.

I'm stuck on this step. I can't figure out how to get to that command prompt (never needed to before, and never used postgres until now). How do I go about getting to that spot?

Figured it out. Documenting in case anyone else had trouble:

docker exec -it sentry_onpremise_postgres_1 bash
psql -h 127.0.0.1 -d postgres -U postgres