Awx: Restarting AWX docker from installer playbook fails on postgres data.

[one1zero1one]

ISSUE TYPE

• Bug Report

COMPONENT NAME

• Installer

SUMMARY

Restarting the AWX environment (by removing containers and re-running installer playbook) fails to start postgres container.

ENVIRONMENT

• AWX version: 1.0.0.591
• AWX install method: docker for mac
• Ansible version: ansible 2.4.0.0
• Operating System: osx 10.11.6 (15G1611)
• Web Browser: Chrome

STEPS TO REPRODUCE

Run installer playbook.
Wait for migration, operate any change within AWX (e.g. add inventory).
Remove all awx containers.
Re-run installer playbook.

EXPECTED RESULTS

AWX starts and previous changed data within is available (e.g. added inventory).

ACTUAL RESULTS

AWX fails to start (internal server error) with postgres container failing
"initdb: directory "/var/lib/postgresql/data" exists but is not empty"

ADDITIONAL INFORMATION

Seems that postgres doesn't like the root of a mountpoint for its data (_Best practice is to create a directory within the mount-point directory that is owned by the PostgreSQL user, and then create the data directory within that._)

Since the mountpoint of the alpine postgres is /var/lib/postgresql/data/, the workaround I found in the installer playbook is pointing the postgres container to /var/lib/postgresql/data/pgdata (toghether with the env variable PGDATA: /var/lib/postgresql/data/pgdata env) => postgres restarted OK, data persisted.

[matburt]

I think this is a good solution, it's a pretty easy change to make to the install playbook I'm just a little worried about migrating existing deployments, but maybe that's not a big deal?

[one1zero1one]

I'm afraid mitigating _existing deployments_ would require a backup with pg_dump inside the postgres container _prior to restarting_, followed by restore after the new installer brings the container back - with the new definition. (did not test it, my postgres-fu is limited, may be another way to deal with it - I only tested persistence _after_ the change, which was ok)

[matburt]

possibly... I think if we aren't actually recreating the container then maybe we'll be fine?

[matburt]

Fixed in #535

There may be some upgrade implications see #535 for details.

[amostech]

I just ran into this issue. Basically I had an AWX installation running and then tried to login to it this morning and the authentication to the dashboard was failing.

Tried to restart the VM that contains the docker containers and now the postgres docker container is in a loop showing the message stated here:

2018-06-25T18:54:28.465651000Z initdb: directory "/var/lib/postgresql/data/pgdata" exists but is not empty
2018-06-25T18:54:28.465920000Z If you want to create a new database system, either remove or empty

Is there any workaround that wouldn't require reinstalling the containers or AWX? All I need is to restart the postgres container.

Also, as a side note would AWX be suitable for production use? Will I lose the data that I had in my postgres database?

[cpare]

I just ran into this issue. Basically I had an AWX installation running and then tried to login to it this morning and the authentication to the dashboard was failing.

Tried to restart the VM that contains the docker containers and now the postgres docker container is in a loop showing the message stated here:

2018-06-25T18:54:28.465651000Z initdb: directory "/var/lib/postgresql/data/pgdata" exists but is not empty
2018-06-25T18:54:28.465920000Z If you want to create a new database system, either remove or empty

Is there any workaround that wouldn't require reinstalling the containers or AWX? All I need is to restart the postgres container.

Also, as a side note would AWX be suitable for production use? Will I lose the data that I had in my postgres database?

Did you ever find a solution for this - I am facing the same issue and while it's not a lot of work to recreate, I would prefer to recover.