Awx: Failed to save settings. Returned status: 504

[TonyTwoStep]

ISSUE TYPE

• Bug Report

COMPONENT NAME

• Installer

SUMMARY

Can't save any system settings after using the lastest installer. Results in a long wait and then 504 error. Reproduced this issue on CentOS 8 and Ubuntu 18.04.

ENVIRONMENT

• AWX version: AWX 9.3.0
• AWX install method: docker on linux
• Ansible version: 2.9.5
• Operating System: Ubuntu and CentOS 8
• Web Browser: FireFox and Chrome

STEPS TO REPRODUCE

Install the latest version of AWX from source.
Attempt to save a system setting such as Authentication or even UI.

EXPECTED RESULTS

Setting to save.

ACTUAL RESULTS

Long wait time and error message:
"Failed to save settings. Returned status: 504"

ADDITIONAL INFORMATION

[hafager]

I experience the same issue on CentOS 8 as well.

Changed version to 9.2.0 and it works like a charm.

[codekoala]

@hafager thank you for that pointer. I've been trying to use AWX for the first time, and nothing seemed to work at all. Switching to 9.2.0 has made the experience far less frustrating!

[hsld]

Same here. I've done some digging and believe this has something to do with the wsgi used in the nginx docker image. The problem seem come up whenever a request to wsgi happens. See the below log entries from the nginx container.

2020/03/26 11:43:50 [error] 123#0: *391 upstream timed out (110: Connection timed out) while reading response header from upstream, client: xxx.xxx.xxx.xxx, server: awxweb, request: "POST /api/v2/groups/ HTTP/1.1", upstream: "uwsgi://127.0.0.1:8050", host: "xxx.xxx.xxx.xxx"
xxx.xxx.xxx.xxx - admin [26/Mar/2020:11:43:50 +0000] "POST /api/v2/groups/ HTTP/1.1" 504 176 "-" "python-requests/2.22.0" "-"
Thu Mar 26 11:43:51 2020 - *** HARAKIRI ON WORKER 5 (pid: 440, try: 1) ***
Thu Mar 26 11:43:51 2020 - HARAKIRI !!! worker 5 status !!!
Thu Mar 26 11:43:51 2020 - HARAKIRI [core 0] xxx.xxx.xxx.xxx - POST /api/v2/groups/ since 1585222910
Thu Mar 26 11:43:51 2020 - HARAKIRI !!! end of worker 5 status !!!

To me this looks like there's no or delayed respone from wsgi and then the worker gets killed.

I've found this bug report from March 2018 which seems to deal with a similar / same issue.
Bug Report

I've followed the suggestions in this bug report and changed the server name in the nginx configuration but it didn't fix or change the problem. I've also done a comple fresh AWX installation and am running into the same problem.

[mjcotner]

Thank you @hafager. I had the same issue with 9.3.0.

[virtualhobbit]

Experiencing the same issue with 9.2.0 :-(

[hsld]

@virtualhobbit Is this a new install ? If so, did you by any chance change the host names of any of the docker containers ? If you did please switch back to the original host names, docker compose and try again. I've had the issue with 9.2.0 as well and pulled a fresh 9.2.0 from git, I used the original inventory file, installed it and imported a backup. Works fine again after that.

[virtualhobbit]

New install, haven't changed any names of any containers

[musil]

New install, haven't changed any names of any containers

same happen to me. fresh install centos8 (minimal) . AWX 9.3.0 no changes to names of any containters. for installation I used this howto https://www.techrepublic.com/article/how-to-install-the-awx-ansible-web-gui-on-centos-8/

[musil]

New install, haven't changed any names of any containers

same happen to me. fresh install centos8 (minimal) . AWX 9.3.0 no changes to names of any containters. for installation I used this howto https://www.techrepublic.com/article/how-to-install-the-awx-ansible-web-gui-on-centos-8/

Installed 9.2.0 and there was no issue.

[virtualhobbit]

I just re-checked this on 9.2.0, and it appears there are multiple issues.

I'm trying to save the LDAP auth settings. These appear to save but don't. Apparently this is know issue that has existed since # 1914 :-(

What was confusing me was that if I made and change to the "LDAP Group Type Parameters" it threw the 504 error.

[hsld]

I just re-checked this on 9.2.0, and it appears there are multiple issues.

I'm trying to save the LDAP auth settings. These appear to save but don't. Apparently this is know issue that has existed since # 1914 :-(

What was confusing me was that if I made and change to the "LDAP Group Type Parameters" it threw the 504 error.

This is odd. I've not had this issue with 9.2.0. After I did a clean install it worked fine again. My guess about the 504 is that's there to tell you that your request was not handled in time (or at all) by something upstream. This usually refers to a service which did not handle your request (save data) properly.

As mentioned in my earlier post it may have to do with uWSGI but someone more involved will need to look into that.

[ryanpetrello]

Those of you encountering this issue in 9.3.0, would you mind giving the latest release (10.0.0) a whirl? I expect some of you were running into deployment issues unintentionally mixing 9.3.0 and devel, and now that we've released 10.0.0 (which doesn't include RabbitMQ, but the new redis changes - https://github.com/ansible/awx/issues/5443), I expect this issue will be resolved for you.

[aak1989]

Hey, i had a similar issue, i am running AWX on kubernetes, i used the 10.0.0 version and it's working for me, however AWX UI no longer appears to refresh automatically,The job appears to stay in pending until the browser is manually refreshed.

[ryanpetrello]

@aak1989 would you mind filing a new issue that describes your setup, and what you're seeing?

Thanks!