Onpremise: Upgrade from 9.1.2 to 10 fail because of postgres 9.6 with data created with 9.5

[Rebolon]

Hi,

i've tried the upgrade today, but this is a fail.
I've read some issues (#301 and others) that explain that we have to docker-compose down but even with this when i run insall.sh it always stop at:

Error response from daemon: get sentry-postgres-new: no such volume
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.utf8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /var/lib/postgresql/9.6/data ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default timezone ... Etc/UTC
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ...
WARNING: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
ok

Success. You can now start the database server using:

    pg_ctl -D /var/lib/postgresql/9.6/data -l logfile start


*failure*
Consult the last few lines of "pg_upgrade_server.log" for
the probable cause of the failure.

There seems to be a postmaster servicing the old cluster.
Please shutdown that postmaster and try again.
Failure, exiting
Cleaning up...

and if i do a docker-compose up then i can see in the log:

postgres_1                | PostgreSQL Database directory appears to contain a database; Skipping initialization
postgres_1                |
postgres_1                | FATAL:  database files are incompatible with server
postgres_1                | DETAIL: The data directory was initialized by PostgreSQL version 9.5, which is not compatible with this version 9.6.16.

I may crash the server and install from scratch :-(

[BYK]

Seems like there were multiple issues during your upgrade:

Missing sentry-postgres-new

Error response from daemon: get sentry-postgres-new: no such volume

This means sentry-postgres-new volume was not created for some reason. Is there anything before this that would indicate the reason for this failure?

Unclean shutdown of old Postgres server

*failure*
Consult the last few lines of "pg_upgrade_server.log" for
the probable cause of the failure.

There seems to be a postmaster servicing the old cluster.
Please shutdown that postmaster and try again.
Failure, exiting

This means your previous installation did not shutdown properly. The install script tries to do this automatically but seems like it failed. This is the reason for the failures.

There was someone in the issues faced with a similar problem and recovered: https://github.com/getsentry/onpremise/issues/325#issuecomment-572008510. Following these steps may help.

[BYK]

Closing due to lack of response.

[Rebolon]

I tryed different thing, but it always failed. So i decided to crash all docker volumes and container and install a fresh sentry.
Hard decision, but i have only 10 channels and 5 users, so it was done in 15 minutes, whereas i tryed during 5hours the upgrade.

[sandrotanner]

I had the same issue (even though postgress was stopped gracefully) and managed to work around it by dumping the db in 9.5 and restoring it in 9.6:

_set postgres version to 9.5 in docker-compose.yml_

docker exec -t sentry_onpremise_postgres_1 docpg_dumpall -c -U postgres > dump_`date +%d-%m-%Y"_"%H_%M_%S`.sql
mv /var/lib/docker/volumes/sentry-postgres/_data/* SOME_BACKUP_LOCATION

_set postgres back to version 9.6 in docker-compose.yml_

./install.sh
docker-compose up -d postgres
cat dump_11-02-2020_13_59_41.sql | docker exec -i sentry_onpremise_postgres_1 psql -U postgres
docker-compose up -d

[Freundschaft]

I had the same issue (even though postgress was stopped gracefully) and managed to work around it by dumping the db in 9.5 and restoring it in 9.6:

_set postgres version to 9.5 in docker-compose.yml_

docker exec -t sentry_onpremise_postgres_1 docpg_dumpall -c -U postgres > dump_`date +%d-%m-%Y"_"%H_%M_%S`.sql
mv /var/lib/docker/volumes/sentry-postgres/_data/* SOME_BACKUP_LOCATION

_set postgres back to version 9.6 in docker-compose.yml_

./install.sh
docker-compose up -d postgres
cat dump_11-02-2020_13_59_41.sql | docker exec -i sentry_onpremise_postgres_1 psql -U postgres
docker-compose up -d

thats what I did, and it only kind of worked, now all my projects are gone :D

[omgitsbillryan]

Probably a dumb question - is install.sh meant for upgrading only? (as opposed to a new sentry installation)

I'm just trying to get a local installation running to poke around but I'm running into the above error.

[BYK]

Probably a dumb question - is install.sh meant for upgrading only? (as opposed to a new sentry installation)

No dumb questions :) install.sh is mean for both installing from scratch and upgrading.

I'm just trying to get a local installation running to poke around but I'm running into the above error.

Can you file a new issue with your install logs?

[omgitsbillryan]

docker system prune -a --volumes did the trick.

I must have somehow got a postgres volume in a stuck state, doh.

[hhoechtl]

@Freundschaft I have the same issue with missing projects even tough the postgres import seemed to have finished properly.
What did you do to solve the issue?