Awx: Internal Server Error with no explanation of what is wrong

[benapetr]

ISSUE TYPE

• Bug Report

COMPONENT NAME

• API
• UI

SUMMARY

When opening the website I get only text: Internal Server Error

ENVIRONMENT

• AWX version: 1.0.1
• AWX install method: docker on linux
• Ansible version: 2.3.2.0
• Operating System: RedHat 7.4
• Web Browser: Firefox

STEPS TO REPRODUCE

Open a website

EXPECTED RESULTS

I expect at least some verbose explanation of what went wrong not just "error", actually I would think programmers in 2017 can do better error message than this...

ACTUAL RESULTS

Just a nothing-saying error message

ADDITIONAL INFORMATION

Is there any way to get AWX to produce more verbose error messages?

From

TEST [root@ansible-test-rhel7 ~]# docker ps
CONTAINER ID        IMAGE                     COMMAND                  CREATED             STATUS                         PORTS                                NAMES
c6fc4305906d        ansible/awx_task:latest   "/tini -- /bin/sh -c "   13 days ago         Up 8 seconds                   8052/tcp                             awx_task
2f10d502e530        ansible/awx_web:latest    "/tini -- /bin/sh -c "   13 days ago         Up 9 seconds                   0.0.0.0:80->8052/tcp                 awx_web
c835cbc75dae        memcached:alpine          "docker-entrypoint.sh"   13 days ago         Up 10 seconds                  11211/tcp                            memcached
7ef2305537c9        rabbitmq:3                "docker-entrypoint.sh"   13 days ago         Up 10 seconds                  4369/tcp, 5671-5672/tcp, 25672/tcp   rabbitmq
c954e46025fa        postgres:9.6              "docker-entrypoint.sh"   13 days ago         Restarting (1) 3 seconds ago   5432/tcp                             postgres

I am judging that problem is with postgres container? I didn't do anything to it though. It probably started crashing on its own.

[matburt]

It'd be super helpful if you could grab the error messages from the console logs of the awx_web container.

[benapetr]

@matburt how do I do it?

[vkovalchuk]

The same symptoms. Command (at the host)

sudo docker logs awx_task

prints

[2017-12-13 12:34:49,469: CRITICAL/MainProcess] Unrecoverable error: OperationalError('could not connect to server: Connection refused\n\tIs the server running on host "postgres" (172.17.0.3) and accepting\n\tTCP/IP connections on port 5432?\n',)

• It's hard to debug postgres; logs say

initdb: directory "/var/lib/postgresql/data/pgdata" exists but is not empty
If you want to create a new database system, either remove or empty
the directory "/var/lib/postgresql/data/pgdata" or run initdb
with an argument other than "/var/lib/postgresql/data/pgdata".
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

• and it says "Restarting" constantly (I can't exec bash in it, Container so-and-so is restarting, wait until the container is running);
• I guess mysql might be more robust on docker;
• I still believe there is architecture error that awx_web does not request awx_task what's wrong with it.

[matburt]

This looks like something I fixed a little while ago:

https://github.com/ansible/awx/pull/535

which fixed:

https://github.com/ansible/awx/issues/438

[alizsyed]

This is exactly what was happening to me @matburt, I had emailed you about it. You had mentioned that my porting was wrong... the postgres container is just constantly looping and restarting.

[matburt]

@alizsyed gotcha... you didn't say that in your emails.

[krispayne]

This creeped up on me today on an install from January, using the release from December.

[apple4ever]

@krispayne Did you fix it or is it still happening?

I'm running 1.0.2.327 and after a couple of days of working, I start getting this issue. I usually just restart the entire server and its fine.

[krispayne]

@apple4ever no. We're still in testing for awx, so I didn't have anything but time put into it, so I've ended up starting over to see if maybe I missed some step. Maybe I ran the installer twice or something, I don't know. This is my first dip into docker, too.

Interestingly enough, after deleting everything and starting fresh, docker-entrypoint.sh still has trouble and postgres is constantly restarting. It may not even be related to this issue's problems, but I don't know enough about the system to know that.

Does Tower have these problems?

[krispayne]

Looks like my issue is possibly related to https://github.com/ansible/awx/issues/677

[maksimluzik]

This is closed, but I am experiencing completely same issue. This is already second time this is happening with AWX + Postgres. First time I nuked the data as it happened when I was playing with AWX, but now it happened with real data. I am just worrying that even if I restore the Postgres data this will happen again. What I learned from this it seems that it is somehow related to the /tmp/pgdocker shared volume/directory.

First time when this happened I received: "initdb: directory "/var/lib/postgresql/data" exists but is not empty"

Second time when this happened I received: FATAL: could not open file "base/16384/2601": No such file or directory

Wondering as well, that why postgres data is put into hosts /tmp/ folder, isn't that just causing the issue as host may clean "unnecessary" files from tmp folder thus causing data corruption? Maybe /home/username/.pgdocker or /root/pgdocker could be better place?

[AdrianDutu]

This is still happening in 2.0.1. My POC blew up after 1 month. I'll go with a native db in production but come on guys, one year later this is still an issue and the bug is closed.

[ssaammoo]

Hi everyone and happy new year ...
it is happening to me too and for 2nd time ... I left AWX working before holidays and now it is not working and postgres keep restarting

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
3536b5be34a9 ansible/awx_web:2.1.0 "/tini -- /bin/sh -c…" 2 weeks ago Up 33 minutes 0.0.0.0:80->8052/tcp awx_web
fe649687fb07 ansible/awx_task:2.1.0 "/tini -- /bin/sh -c…" 3 weeks ago Up 33 minutes 8052/tcp awx_task
e08a3bc5bb06 memcached:alpine "docker-entrypoint.s…" 3 weeks ago Up 33 minutes 11211/tcp memcached
ac69051adb5b ansible/awx_rabbitmq:3.7.4 "docker-entrypoint.s…" 3 weeks ago Up 33 minutes 4369/tcp, 5671-5672/tcp, 15671-15672/tcp, 25672/tcp rabbitmq
67be5532ba31 postgres:9.6 "docker-entrypoint.s…" 3 weeks ago Restarting (1) 25 seconds ago postgres

[ryanpetrello]

@ssaammoo could you share any errors in:

~ docker logs awx_web

[jdhayes]

This is also my issue. The postgres container is in boot loop, and the errors clearly state that an init script is attempting to initialize the data directory, however it cannot proceed due to that directory is not empty.

initdb: directory "/var/lib/postgresql/data/pgdata" exists but is not empty

I can move the directory out of the way and create an empty one here:
/tmp/pgdocker/pgdata
However, now when the postgres container is started it seems to start a migration process that runs for hours (not sure if it ever finishes).

[AdrianDutu]

You should not use the /tmp/pgdocker/pgdata location. The tmp folder is
regularly cleaned by the OS. If you had your data there, it's gone now.
When the container now sees the empty directory is thinks it's a fresh
installation and starts the migration process (which it does every time for
a fresh installation). It should finish, it definitely doesn't take hours.
If it's stuck, check the awx_task logs to see what's wrong with the
migration.
But before everything else. you should change the postgres_data_dir in your
inventory file to some other dir not in tmp and run install.yml again. (Or
just change the docker container mount point if you know how).

On Mon, Jan 14, 2019 at 11:53 PM Jordan Hayes [e-mail]
wrote:

This is also my issue. The postgres container is in boot loop, and the
errors clearly state that an init script is attempting to initialize the
data directory, however it cannot proceed due to that directory is not
empty.

initdb: directory "/var/lib/postgresql/data/pgdata" exists but is not empty

I can move the directory out of the way and create an empty one here:
/tmp/pgdocker/pgdata
However, now when the postgres container is started it seems to start a
migration process that runs for hours (not sure if it ever finishes).

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/ansible/awx/issues/796#issuecomment-454192237, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AEmcicLLfmGCsmc_d4vlAZ7d5UA8PTRrks5vDQpNgaJpZM4Q-rQs
.

[jdhayes]

There were some errors in the awx_task logs, so I just redeployed everything.
Also I changed the pgdata location under opt, thanks for the help.