Awx: Can't install project using docker-compose on Mac from devel or 13.0.0

[geerlingguy]

ISSUE TYPE

• Bug Report

SUMMARY

I've tried cloning the project and following the Docker Compose installation instructions for both devel and 13.0.0, and neither version results in a working installation on my Mac.

ENVIRONMENT

• AWX version: 13.0.0 or devel
• AWX install method: docker for mac
• Ansible version: 2.9.10
• Operating System: macOS (10.15.5)
• Web Browser: Safari

STEPS TO REPRODUCE

1. git clone https://github.com/ansible/awx.git
2. cd awx/installer
3. ansible-playbook -i inventory install.yml

EXPECTED RESULTS

After some time for database migration to complete, the AWX installation should be accessible at http://localhost/

ACTUAL RESULTS

1. The page http://localhost/ remains stuck at 'Server Error' indefinitely (waited 20 minutes).

2. The awx_task and awx_web container logs show:

   django.db.utils.OperationalError: FATAL: no pg_hba.conf entry for host "172.24.0.5", user "awx", database "awx", SSL off

3. The awx_posgres container logs show:

   2020-07-21 20:43:10.990 UTC [1466] FATAL: no pg_hba.conf entry for host "172.24.0.5", user "awx", database "awx", SSL off

ADDITIONAL INFORMATION

Related issue: https://github.com/ansible/awx/issues/4736

Full initial startup log of the postgres container (using docker logs awx_postgres):

The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.utf8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /var/lib/postgresql/data/pgdata ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default timezone ... Etc/UTC
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... 
PostgreSQL Database directory appears to contain a database; Skipping initialization

2020-07-21 21:54:15.512 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
2020-07-21 21:54:15.512 UTC [1] LOG:  listening on IPv6 address "::", port 5432
2020-07-21 21:54:15.516 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2020-07-21 21:54:15.571 UTC [29] LOG:  database system was shut down at 2020-07-21 21:54:11 UTC
2020-07-21 21:54:15.593 UTC [1] LOG:  database system is ready to accept connections
2020-07-21 21:54:15.735 UTC [36] LOG:  incomplete startup packet
2020-07-21 21:54:15.738 UTC [37] LOG:  incomplete startup packet
2020-07-21 21:54:16.726 UTC [38] FATAL:  no pg_hba.conf entry for host "172.28.0.4", user "awx", database "postgres", SSL off
2020-07-21 21:54:16.726 UTC [39] FATAL:  no pg_hba.conf entry for host "172.28.0.2", user "awx", database "postgres", SSL off
2020-07-21 21:54:19.370 UTC [40] FATAL:  no pg_hba.conf entry for host "172.28.0.4", user "awx", database "awx", SSL off
...

WORKAROUND

I reset the entire environment using:

cd ~/.awx/awxcompose
docker-compose down -v
cd ~
rm -rf ~/.awx

Then I initialized the environment again. Same issue. But then I deleted the local pgdocker directory and restarted the postgres container, and it seems on second try, the database was initialized correctly:

cd ~/.awx/awxcompose
rm -rf ~/.awx/pgdocker
docker-compose restart postgres

At this point, the 'AWX is upgrading' message starts appearing when I access http://localhost/, and the initialization process works correctly, resulting in a running AWX installation.

[geerlingguy]

Two other issues that could be tangentially related: https://github.com/ansible/awx/pull/7698 and https://github.com/docker-library/postgres/issues/482

Though I'm trying to see what else might be the problem. Maybe the use of a bind mount for this volume (:Z) with Docker for Mac is causing the volume to initialize in a way that Postgres doesn't use it for database creation the same way the first time the volume is created?

[geerlingguy]

It seems like another alternative is to ditch the bind mount and just use a Docker volume instead. Performance is actually improved quite a bit on the Mac with this setup too. I'm not sure what the reason is for the bind mount regardless—a docker volume would still be persistent (so the database wouldn't be blown away)... maybe it's for backup purposes (though it's easy enough to back up volumes?

I'm going to file a PR that modifies the mount to use a Docker volume instead of bind mounting it, but maybe another alternative would be to allow the mount to be configurable? An inventory setting that would allow you to specify the mount configuration on your own?

[geerlingguy]

The above PR (#7701) fixes the issue of the postgres initialization, but the default awx images used with all the default settings still break, resulting in the task container hitting an endless loop of:

  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "main_instance" does not exist
LINE 1: SELECT (1) AS "a" FROM "main_instance" WHERE "main_instance"...
                               ^

2020-07-21 22:14:36,796 INFO exited: dispatcher (exit status 1; not expected)
2020-07-21 22:14:36,796 INFO exited: dispatcher (exit status 1; not expected)

So I also had to restart the awx_task container:

cd ~/.awx/awxcompose
docker-compose restart task

And then the database migration kicked off and everything came up after about one minute.

[geerlingguy]

In the PR, I also added instructions for migrating your AWX Postgres database from a bind mount to a docker volume, in case anyone else is interested in testing that migration path.

[spam-n-eggs]

@geerlingguy I know that this probably isn't the right place to ask this, but I didn't even get this far. How did you get past it trying to run PowerShell instead of python on the install?

[isuftin]

I'm seeing the same issues from within a CentOS 7 VM in VirtualBox using 13.0.0 (on a MacOS host, if that matters).

[isuftin]

Interestingly, as root in the VM, cd ~/.awx/awxcompose/ && docker-compose down && docker-compose up -d seems to allow the process to continue and finish :/

[loquitoslack]

I have a similar problem:

AWX version: 14.0.0 or devel
AWX install method: docker for mac
Ansible version: 2.9.2
Python version: 3.8.0
Operating System: macOS (10.15.6)
Web Browser: google-chrome

Previews steps
1) pip3 install docker
2) pip3 install docker-compose
3) git clone https://github.com/ansible/awx.git
4) cd awx/installer
5) ansible-playbook -i inventory install.yml
6) cd ~/.awx/awxcompose
8) rm -rf ~/.awx/pgdocker
9) docker-compose restart postgres
10) docker-compose restart task ( upgrading )

Autor: @geerlingguy

Awx running in my appel mac, thanks @geerlingguy by instructions

[digitole]

I had the same problem on Docker for Mac, also got it working with the steps loquitoslack described.

Would be great to @geerlingguy 's PR merged 👍

[silid]

@geerlingguy I just spent all morning trying to get this installed on a colleagues mac in version 15.0.0 (where it had worked fine on mine) so thanks for solving it. What can we do to get the PR accepted? Do you need any testers?

[geerlingguy]

I think the main thing is trying to add some functionality to the installer to try to do a migration for anyone who has a docker volume using the old method (see this comment: https://github.com/ansible/awx/pull/7701#issuecomment-664577116).

Right now my focus is elsewhere, so unfortunately I can't help push this past the finish line, but it's still the only way to get AWX to work on macOS.

[Apocrathia]

I just ran into this same issue on Ubuntu 20.04.1 using release 15.0.0. So, it's still a persistent issue for Docker Compose installs.

I just added the line host all all 172.16.0.0/12 trust to the pg_hba.conf file, restarted the stack, and everything appears to be working. Using 172.16.0.0/12 (126.16.0.1 - 126.31.255.254) rather than 0.0.0.0/0 to make some attempt at restricting traffic to the Docker network

[infraengineer-sre]

The same for macOS Catalina 10.15.7 (19H2)

2020-10-21 13:56:18.150 UTC [835] FATAL: no pg_hba.conf entry for host "172.18.0.4", user "awx", database "awx", SSL off

ENVIRONMENT

AWX version: 14.1.0
AWX install method: docker Desktop for macOS version 19.03.13, build 4484c46d9d
Ansible version: ansible 2.10.2

config file = None
configured module search path = ['/Users/admin/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
ansible python module location = /usr/local/lib/python3.8/site-packages/ansible
executable location = /usr/local/bin/ansible
python version = 3.8.6 (default, Oct 8 2020, 14:06:32) [Clang 12.0.0 (clang-1200.0.32.2)]
Operating System: macOS 10.15.7 (19H2)
Web Browser: Firefox 81.0.2 build ID 20201012085804

STEPS TO REPRODUCE

git clone -b 14.1.0 https://github.com/ansible/awx.git
cd awx/installer
ansible-playbook -i inventory install.yml

django.db.utils.OperationalError: FATAL: no pg_hba.conf entry for host "172.18.0.4", user "awx", database "awx", SSL off