Awx: AWX UI no longer appears to refresh automatically

Created on 14 Mar 2018 · 29Comments · Source: ansible/awx

ISSUE TYPE

Bug Report

COMPONENT NAME

SUMMARY

The AWX UI doesn't appear to refresh for on-going tasks. For example, starting a job, the job always remains at pending unless you manually refresh the page.

ENVIRONMENT

AWX version: 1.0.4.70
AWX install method: docker on linux
Ansible version: 2.4.3.0
Operating System: RHEL7 Docker
Web Browser: Chrome and Firefox

STEPS TO REPRODUCE

Start a job of any kind.

EXPECTED RESULTS

The UI should automatically refresh as the job goes from Pending to In Progress to Complete and the activity log to automatically update.

ACTUAL RESULTS

The job appears to stay in pending until the browser is manually refreshed.

This did work in earlier versions of AWX. Is this a new regression?

api ui high needs_devel bug

Source

seafoodbuffet

Most helpful comment

If you are using reverse proxy in front of awx, make shure that your proxy can process websocket connections correctly. In case of nginx, put this into your config:
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;

this helps for my case.

jjjbushjjj on 20 Mar 2018

👍4 🎉1

All 29 comments

seeing this issue as well

badstreff on 15 Mar 2018

@seafoodbuffet @BadStreff Can you try clearing your cookies and trying again?

jaredevantabor on 16 Mar 2018

Still no luck after clearing cookies, tried clearing everything from the browser and that didn't help either.

badstreff on 19 Mar 2018

Same here, I've tried without cookies and in private browsing mode, both with the same result.

seafoodbuffet on 20 Mar 2018

this helps for my case.

jjjbushjjj on 20 Mar 2018

👍4 🎉1

I'm using the standard docker-compose setup, which runs awx_web directly without a proxy. Also, this exact same setup worked on previous versions of the awx_web image

seafoodbuffet on 20 Mar 2018

I have the exact same setup as listed above. Also using RHEL7 as the host.

badstreff on 21 Mar 2018

+1. Same issue, was working prior to upgrade. No changes to nginx proxy. Version 1.0.4.109

jleach8 on 21 Mar 2018

So I ended up deleting my docker containers, updating the repo (git pull), and running the install.yml file again. The pages appear to be refreshing as expected now.

badstreff on 21 Mar 2018

Like @BadStreff I basically restarted all the docker containers (except in my case it was because I was an idiot and forgot to exclude docker from an automatic yum update). After all the awx containers auto-restarted (after docker engine restarted) everything is auto-refreshing for me once more. No changes, didn't clear cookies or anything.

seafoodbuffet on 22 Mar 2018

I've noticed this keeps popping up occasionally. On 1.0.5.19 it's now back to not updating. On 1.0.4.105 it's fine, 1.0.4.106 it's not updating.

JSkier21 on 2 Apr 2018

Yep, mine are back to not updating again. I haven't cared enough to refresh the containers again as it is only a minor annoyance at first. After a few days though...

badstreff on 2 Apr 2018

There were some bugs that were fixed related to the websockets that recently landed in devel. Would you mind retesting with the latest devel? @seafoodbuffet

jaredevantabor on 17 May 2018

What are right wstunnel proxypass directive for the Apache2 configuration?

RogerSik on 6 Sep 2018

Hello,

I have this problem. The job progress don't refresh and I need to refresh manually my browser.

AWX: 2.1.0
Ansible: 2.7.2

I have installed with your guidelines here: https://github.com/ansible/awx/blob/devel/INSTALL.md

Already tried to stop docker and restart it.

Lutaksur on 6 Dec 2018

What browser and version are you using?

jakemcdermott on 6 Dec 2018

Safari Version 12.0 (14606.1.36.1.9)
Chrome Version 70.0.3538.110 (Official Build) (64-bit)

Lutaksur on 6 Dec 2018

Do you see any tracebacks in the chrome console? Do you see any tracebacks in the container logs for the task or web containers?

jakemcdermott on 6 Dec 2018

I saw nothing on the browser console.

On awx task from the beginning and the end of the job execution:

2018-12-06 18:34:00,412 DEBUG    awx.main.dispatch delivered 50c5e3c5-1167-4615-afb2-9a5f1821e7ae to worker[224] qsize 0
RESULT 2
OKREADY
2018-12-06 18:34:00,414 DEBUG    awx.main.dispatch task 50c5e3c5-1167-4615-afb2-9a5f1821e7ae starting awx.main.tasks.update_inventory_computed_fields(*[2, True])
2018-12-06 18:34:00,432 DEBUG    awx.main.models.inventory Going to update inventory computed fields
2018-12-06 18:34:00,479 DEBUG    awx.main.models.inventory Finished updating inventory computed fields
2018-12-06 18:34:00,483 DEBUG    awx.main.dispatch task 50c5e3c5-1167-4615-afb2-9a5f1821e7ae is finished
2018-12-06 18:34:00,698 DEBUG    awx.main.dispatch delivered d7e31d33-2ee4-4f80-9b33-033546649be2 to worker[222] qsize 0
2018-12-06 18:34:00,700 DEBUG    awx.main.dispatch task d7e31d33-2ee4-4f80-9b33-033546649be2 starting awx.main.scheduler.tasks.run_job_launch(*[96])
2018-12-06 18:34:00,712 DEBUG    awx.main.scheduler Starting Scheduler
2018-12-06 18:34:00,859 DEBUG    awx.main.scheduler Starting dependent project_update 97 (pending) in group tower instance awx
2018-12-06 18:34:00,862 INFO     awx.main.scheduler Submitting project_update 97 (waiting) to <instance group, instance> <1,awx>.
2018-12-06 18:34:00,893 DEBUG    awx.main.scheduler project_update 97 (waiting) consumed 1 capacity units from tower with prior total of 0
2018-12-06 18:34:00,894 DEBUG    awx.main.scheduler job 96 (pending) is blocked from running
2018-12-06 18:34:00,915 DEBUG    awx.main.dispatch publish awx.main.tasks.RunProjectUpdate(a87c82c9-b109-474b-a002-466d7a870a59, queue=awx)
2018-12-06 18:34:00,930 DEBUG    awx.main.dispatch delivered a87c82c9-b109-474b-a002-466d7a870a59 to worker[226] qsize 0
2018-12-06 18:34:00,931 DEBUG    awx.main.dispatch task d7e31d33-2ee4-4f80-9b33-033546649be2 is finished
2018-12-06 18:34:00,936 DEBUG    awx.main.dispatch task a87c82c9-b109-474b-a002-466d7a870a59 starting awx.main.tasks.RunProjectUpdate(*[97])
2018-12-06 18:34:02,904 DEBUG    awx.main.dispatch publish awx.main.scheduler.tasks.run_task_manager(a3cf5917-ceb8-49e9-8dd7-f4448809905c, queue=awx_private_queue)
[2018-12-06 18:34:02,904: DEBUG/Process-1] publish awx.main.scheduler.tasks.run_task_manager(a3cf5917-ceb8-49e9-8dd7-f4448809905c, queue=awx_private_queue)
2018-12-06 18:34:02,931 DEBUG    awx.main.dispatch delivered a3cf5917-ceb8-49e9-8dd7-f4448809905c to worker[222] qsize 0
2018-12-06 18:34:02,933 DEBUG    awx.main.dispatch task a3cf5917-ceb8-49e9-8dd7-f4448809905c starting awx.main.scheduler.tasks.run_task_manager(*[])
2018-12-06 18:34:02,934 DEBUG    awx.main.scheduler Running Tower task manager.
2018-12-06 18:34:02,945 DEBUG    awx.main.scheduler Starting Scheduler
2018-12-06 18:34:02,996 DEBUG    awx.main.scheduler Dependent project_update 97 (running) is blocked from running
2018-12-06 18:34:02,997 DEBUG    awx.main.scheduler job 96 (pending) is blocked from running
2018-12-06 18:34:03,000 DEBUG    awx.main.dispatch task a3cf5917-ceb8-49e9-8dd7-f4448809905c is finished
2018-12-06 18:34:03,843 DEBUG    awx.main.dispatch delivered dc97ce6f-3669-4f74-959f-b521a32cde77 to worker[221] qsize 0
2018-12-06 18:34:03,848 DEBUG    awx.main.dispatch delivered f9dfea08-056b-41d7-95f2-6bf6f8809b62 to worker[225] qsize 0
2018-12-06 18:34:03,862 DEBUG    awx.main.dispatch task dc97ce6f-3669-4f74-959f-b521a32cde77 is finished
2018-12-06 18:34:03,870 DEBUG    awx.main.dispatch task f9dfea08-056b-41d7-95f2-6bf6f8809b62 is finished
2018-12-06 18:34:04,358 DEBUG    awx.main.dispatch delivered 064abc7a-d200-4e55-88e4-449ddd181482 to worker[225] qsize 0
2018-12-06 18:34:04,376 DEBUG    awx.main.dispatch task 064abc7a-d200-4e55-88e4-449ddd181482 is finished
2018-12-06 18:34:04,420 DEBUG    awx.main.dispatch delivered 0242ac12-0006-de93-f6aa-000000000006 to worker[225] qsize 0
2018-12-06 18:34:04,437 DEBUG    awx.main.dispatch delivered 0242ac12-0006-de93-f6aa-000000000008 to worker[221] qsize 0
2018-12-06 18:34:04,439 DEBUG    awx.main.dispatch task 0242ac12-0006-de93-f6aa-000000000006 is finished
2018-12-06 18:34:04,454 DEBUG    awx.main.dispatch task 0242ac12-0006-de93-f6aa-000000000008 is finished
2018-12-06 18:34:04,485 DEBUG    awx.main.dispatch delivered 1447d9d4-7545-4ac7-b75b-05a61b39e61c to worker[221] qsize 0
2018-12-06 18:34:04,493 DEBUG    awx.main.dispatch delivered 0242ac12-0006-de93-f6aa-00000000000a to worker[225] qsize 0
2018-12-06 18:34:04,505 DEBUG    awx.main.dispatch task 1447d9d4-7545-4ac7-b75b-05a61b39e61c is finished
2018-12-06 18:34:04,512 DEBUG    awx.main.dispatch task 0242ac12-0006-de93-f6aa-00000000000a is finished
2018-12-06 18:34:04,524 DEBUG    awx.main.dispatch delivered 6d7eece3-1628-4982-aa14-4ad186178232 to worker[225] qsize 0
2018-12-06 18:34:04,541 DEBUG    awx.main.dispatch task 6d7eece3-1628-4982-aa14-4ad186178232 is finished
2018-12-06 18:34:04,547 DEBUG    awx.main.dispatch delivered 0242ac12-0006-de93-f6aa-00000000000c to worker[221] qsize 0
2018-12-06 18:34:04,564 DEBUG    awx.main.dispatch task 0242ac12-0006-de93-f6aa-00000000000c is finished
2018-12-06 18:34:13,124 DEBUG    awx.main.dispatch publish awx.main.tasks.awx_periodic_scheduler(d5fd2282-243b-4d07-96aa-1347a2e45b36, queue=awx_private_queue)
[2018-12-06 18:34:13,124: DEBUG/Process-1] publish awx.main.tasks.awx_periodic_scheduler(d5fd2282-243b-4d07-96aa-1347a2e45b36, queue=awx_private_queue)
2018-12-06 18:34:13,140 DEBUG    awx.main.dispatch delivered d5fd2282-243b-4d07-96aa-1347a2e45b36 to worker[222] qsize 0
2018-12-06 18:34:13,142 DEBUG    awx.main.dispatch task d5fd2282-243b-4d07-96aa-1347a2e45b36 starting awx.main.tasks.awx_periodic_scheduler(*[])
2018-12-06 18:34:13,149 DEBUG    awx.main.tasks Last scheduler run was: 2018-12-06 18:33:43.150484+00:00
2018-12-06 18:34:13,160 DEBUG    awx.main.dispatch task d5fd2282-243b-4d07-96aa-1347a2e45b36 is finished

Same from aux_web:

> 127.0.0.1:45886 - - [06/Dec/2018:18:37:17] "WSCONNECT /websocket/" - -
127.0.0.1:45886 - - [06/Dec/2018:18:37:18] "WSDISCONNECT /websocket/" - -
127.0.0.1:45896 - - [06/Dec/2018:18:37:18] "WSCONNECTING /websocket/" - -
127.0.0.1:45896 - - [06/Dec/2018:18:37:18] "WSCONNECT /websocket/" - -
127.0.0.1:45896 - - [06/Dec/2018:18:37:19] "WSDISCONNECT /websocket/" - -
127.0.0.1:45912 - - [06/Dec/2018:18:37:19] "WSCONNECTING /websoc10.100.21.213 - - [06/Dec/2018:18:37:19 +0000] "GET /websocket/ HTTP/1.1" 101 29 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.110 Safari/537.36" "-"
10.100.21.213 - - [06/Dec/2018:18:37:20 +0000] "GET /websocket/ HTTP/1.1" 101 29 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.110 Safari/537.36" "-"
10.100.21.213 - - [06/Dec/2018:18:37:20 +0000] "GET / HTTP/1.1" 200 11337 "http://10.42.20.167:8080/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.110 Safari/537.36" "-"
[pid: 135|app: 0|req: 397/1590] 10.100.21.213 () {46 vars in 2287 bytes} [Thu Dec  6 18:37:20 2018] GET / => generated 11337 bytes in 45 msecs (HTTP/1.1 200) 7 headers in 312 bytes (1 switches on core 0)
[pid: 134|app: 0|req: 325/1591] 10.100.21.213 () {44 vars in 2216 bytes} [Thu Dec  6 18:37:20 2018] GET /api/ => generated 173 bytes in 45 msecs (HTTP/1.1 200) 11 headers in 471 bytes (1 switches on core 0)
10.100.21.213 - - [06/Dec/2018:18:37:20 +0000] "GET /api/ HTTP/1.1" 200 173 "http://10.42.20.167:8080/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.110 Safari/537.36" "-"
10.100.21.213 - - [06/Dec/2018:18:37:20 +0000] "GET /static/assets/favicon.ico?v=2.1.0 HTTP/1.1" 200 15086 "http://10.42.20.167:8080/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.110 Safari/537.36" "-"
10.100.21.213 - - [06/Dec/2018:18:37:20 +0000] "OPTIONS /api/v2/unified_jobs/ HTTP/1.1" 200 10271 "http://10.42.20.167:8080/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.110 Safari/537.36" "-"
[pid: 136|app: 0|req: 270/1592] 10.100.21.213 () {46 vars in 2291 bytes} [Thu Dec  6 18:37:20 2018] OPTIONS /api/v2/unified_jobs/ => generated 10271 bytes in 111 msecs (HTTP/1.1 200) 10 headers in 376 bytes (1 switches on core 0)
10.100.21.213 - - [06/Dec/2018:18:37:20 +0000] "GET /api/ HTTP/1.1" 200 173 "http://10.42.20.167:8080/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.110 Safari/537.36" "-"
[pid: 134|app: 0|req: 326/1593] 10.100.21.213 () {46 vars in 2238 bytes} [Thu Dec  6 18:37:20 2018] GET /api/ => generated 173 bytes in 67 msecs (HTTP/1.1 200) 11 headers in 471 bytes (1 switches on core 0)
[pid: 137|app: 0|req: 311/1594] 10.100.21.213 () {44 vars in 2236 bytes} [Thu Dec  6 18:37:20 2018] GET /api/v2/config/ => generated 505 bytes in 79 msecs (HTTP/1.1 200) 10 headers in 388 bytes (1 switches on core 0)
10.100.21.213 - - [06/Dec/2018:18:37:21 +0000] "GET /api/v2/config/ HTTP/1.1" 200 505 "http://10.42.20.167:8080/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.110 Safari/537.36" "-"
[pid: 135|app: 0|req: 398/1595] 10.100.21.213 () {46 vars in 2244 bytes} [Thu Dec  6 18:37:21 2018] GET /api/v2/ => generated 1545 bytes in 63 msecs (HTTP/1.1 200) 10 headers in 375 bytes (1 switches on core 0)
10.100.21.213 - - [06/Dec/2018:18:37:21 +0000] "GET /api/v2/ HTTP/1.1" 200 1545 "http://10.42.20.167:8080/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.110 Safari/537.36" "-"
10.100.21.213 - - [06/Dec/2018:18:37:21 +0000] "GET /websocket/ HTTP/1.1" 101 29 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.110 Safari/537.36" "-"
[pid: 133|app: 0|req: 291/1596] 10.100.21.213 () {44 vars in 2355 bytes} [Thu Dec  6 18:37:20 2018] GET /api/v2/unified_jobs/?page_size=20&order_by=-finished&not__launch_type=sync => generated 46711 bytes in 717 msecs (HTTP/1.1 200) 10 headers in 376 bytes (1 switches on core 0)
10.100.21.213 - - [06/Dec/2018:18:37:21 +0000] "GET /api/v2/unified_jobs/?page_size=20&order_by=-finished&not__launch_type=sync HTTP/1.1" 200 46711 "http://10.42.20.167:8080/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.110 Safari/537.36" "-"
10.100.21.213 - - [06/Dec/2018:18:37:21 +0000] "GET /api/v2/jobs/99/job_events/?order_by=-counter&page=1&page_size=50 HTTP/1.1" 200 52 "http://10.42.20.167:8080/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.110 Safari/537.36" "-"
[pid: 134|app: 0|req: 327/1597] 10.100.21.213 () {44 vars in 2335 bytes} [Thu Dec  6 18:37:21 2018] GET /api/v2/jobs/99/job_events/?order_by=-counter&page=1&page_size=50 => generated 52 bytes in 53 msecs (HTTP/1.1 200) 11 headers in 396 bytes (1 switches on core 0)
10.100.21.213 - - [06/Dec/2018:18:37:21 +0000] "GET /api/v2/jobs/99/ HTTP/1.1" 200 3494 "http://10.42.20.167:8080/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.110 Safari/537.36" "-"
[pid: 135|app: 0|req: 399/1598] 10.100.21.213 () {44 vars in 2238 bytes} [Thu Dec  6 18:37:21 2018] GET /api/v2/jobs/99/ => generated 3494 bytes in 118 msecs (HTTP/1.1 200) 10 headers in 383 bytes (1 switches on core 0)
10.100.21.213 - - [06/Dec/2018:18:37:21 +0000] "OPTIONS /api/v2/jobs/99/ HTTP/1.1" 200 14018 "http://10.42.20.167:8080/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.110 Safari/537.36" "-"
[pid: 137|app: 0|req: 312/1599] 10.100.21.213 () {46 vars in 2281 bytes} [Thu Dec  6 18:37:21 2018] OPTIONS /api/v2/jobs/99/ => generated 14018 bytes in 192 msecs (HTTP/1.1 200) 10 headers in 384 bytes (1 switches on core 0)
10.100.21.213 - - [06/Dec/2018:18:37:22 +0000] "GET /api/v2/jobs/99/job_events/?order_by=-counter&page=1&page_size=50 HTTP/1.1" 200 52 "http://10.42.20.167:8080/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.110 Safari/537.36" "-"
[pid: 133|app: 0|req: 292/1600] 10.100.21.213 () {44 vars in 2335 bytes} [Thu Dec  6 18:37:22 2018] GET /api/v2/jobs/99/job_events/?order_by=-counter&page=1&page_size=50 => generated 52 bytes in 56 msecs (HTTP/1.1 200) 11 headers in 396 bytes (1 switches on core 0)
10.100.21.213 - - [06/Dec/2018:18:37:22 +0000] "GET /websocket/ HTTP/1.1" 101 29 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.110 Safari/537.36" "-"
10.100.21.213 - - [06/Dec/2018:18:37:23 +0000] "GET /websocket/ HTTP/1.1" 101 29 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.110 Safari/537.36" "-"
10.100.21.213 - - [06/Dec/2018:18:37:24 +0000] "GET /websocket/ HTTP/1.1" 101 29 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.110 Safari/537.36" "-"
10.100.21.213 - - [06/Dec/2018:18:37:25 +0000] "GET /websocket/ HTTP/1.1" 101 29 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.110 Safari/537.36" "-"
10.100.21.213 - - [06/Dec/2018:18:37:26 +0000] "GET /websocket/ HTTP/1.1" 101 29 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.110 Safari/537.36" "-"
10.100.21.213 - - [06/Dec/2018:18:37:27 +0000] "GET /websocket/ HTTP/1.1" 101 29 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.110 Safari/537.36" "-"
10.100.21.213 - - [06/Dec/2018:18:37:28 +0000] "GET /websocket/ HTTP/1.1" 101 29 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.110 Safari/537.36" "-"
10.100.21.213 - - [06/Dec/2018:18:37:29 +0000] "GET /websocket/ HTTP/1.1" 101 29 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.110 Safari/537.36" "-"
10.100.21.213 - - [06/Dec/2018:18:37:30 +0000] "GET /websocket/ HTTP/1.1" 101 29 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.110 Safari/537.36" "-"
10.100.21.213 - - [06/Dec/2018:18:37:31 +0000] "GET /websocket/ HTTP/1.1" 101 29 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.110 Safari/537.36" "-"
10.100.21.213 - - [06/Dec/2018:18:37:32 +0000] "GET /websocket/ HTTP/1.1" 101 29 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.110 Safari/537.36" "-"
10.100.21.213 - - [06/Dec/2018:18:37:33 +0000] "GET /websocket/ HTTP/1.1" 101 29 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.110 Safari/537.36" "-"
10.100.21.213 - - [06/Dec/2018:18:37:35 +0000] "GET /websocket/ HTTP/1.1" 101 29 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.110 Safari/537.36" "-"
10.100.21.213 - - [06/Dec/2018:18:37:36 +0000] "GET /websocket/ HTTP/1.1" 101 29 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.110 Safari/537.36" "-"
10.100.21.213 - - [06/Dec/2018:18:37:37 +0000] "GET /websocket/ HTTP/1.1" 101 29 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.110 Safari/537.36" "-"
10.100.21.213 - - [06/Dec/2018:18:37:38 +0000] "GET /websocket/ HTTP/1.1" 101 29 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.110 Safari/537.36" "-"
ket/" - -
127.0.0.1:45912 - - [06/Dec/2018:18:37:19] "WSCONNECT /websocket/" - -
127.0.0.1:45912 - - [06/Dec/2018:18:37:20] "WSDISCONNECT /websocket/" - -
127.0.0.1:45922 - - [06/Dec/2018:18:37:20] "WSCONNECTING /websocket/" - -
127.0.0.1:45922 - - [06/Dec/2018:18:37:20] "WSCONNECT /websocket/" - -
127.0.0.1:45922 - - [06/Dec/2018:18:37:21] "WSDISCONNECT /websocket/" - -
127.0.0.1:45972 - - [06/Dec/2018:18:37:21] "WSCONNECTING /websocket/" - -
127.0.0.1:45972 - - [06/Dec/2018:18:37:21] "WSCONNECT /websocket/" - -
127.0.0.1:45972 - - [06/Dec/2018:18:37:22] "WSDISCONNECT /websocket/" - -
127.0.0.1:46010 - - [06/Dec/2018:18:37:22] "WSCONNECTING /websocket/" - -
127.0.0.1:46010 - - [06/Dec/2018:18:37:22] "WSCONNECT /websocket/" - -
127.0.0.1:46010 - - [06/Dec/2018:18:37:23] "WSDISCONNECT /websocket/" - -
127.0.0.1:46026 - - [06/Dec/2018:18:37:23] "WSCONNECTING /websocket/" - -
127.0.0.1:46026 - - [06/Dec/2018:18:37:23] "WSCONNECT /websocket/" - -

10.100.21.213 is my computer private ip.
10.42.20.167 is one IP of the container host.
127.0.0.1 is obviously awx in the container.

Lutaksur on 6 Dec 2018

Not sure if the same issue, but I'm seeing this as well. I get this in the console (10 times usually before it seems to give up):

WebSocket connection to 'ws://some.host/websocket/' failed: WebSocket is closed before the connection is established.
WebSocket connection to 'ws://some.host/websocket/' failed: WebSocket is closed before the connection is established.
WebSocket connection to 'ws://some.host/websocket/' failed: WebSocket is closed before the connection is established.
WebSocket connection to 'ws://some.host/websocket/' failed: WebSocket is closed before the connection is established.
WebSocket connection to 'ws://some.host/websocket/' failed: WebSocket is closed before the connection is established.

And this in awx_web (matching the above):

172.22.0.6 - - [09/Dec/2018:22:23:26 +0000] "GET /websocket/ HTTP/1.1" 502 575 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.67 Safari/537.36" "1.1.1.1, 1.1.1.1"
2018/12/09 22:23:26 [error] 130#0: *14512 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 172.22.0.6, server: _, request: "GET /websocket/ HTTP/1.1", upstream: "http://127.0.0.1:8051/websocket/", host: "some.host"

lingfish on 10 Dec 2018

@lingfish +1

I'm getting the exact same websocket issues. Any updates on getting a fix?

<  AWX 2.1.1.0  >
 --------------- 
        \   ^__^
         \  (oo)\_______
            (__)      A )\/\
                ||----w |
                ||     ||
Ansible 2.7.2

jgibbarduk on 13 Dec 2018

This seems to start occurring x +y days after containers are started. At least 1 day after. Makes no difference if I go via my haproxy, or direct.

lingfish on 17 Dec 2018

Perhaps we need to reopen this issue? I see the same behavior here - seems like the UI worked properly for a few days, then stopped working. My most recent deploy of 2.1.2.0 is exhibiting the same behavior (ran fine for approximately 1 week, then stopped refreshing). I was previously running 1.6.0.49 - saw similar behavior.

 _______________ 
<  AWX 2.1.2.0  >
 --------------- 
        \   ^__^
         \  (oo)\_______
            (__)      A )\/\
                ||----w |
                ||     ||

Ansible 2.7.4 
Copyright © 2018 Red Hat, Inc. 
Visit Ansible.com for more information.

I'm deploying in Docker CE 18.x on AWS. There's an ALB in front for SSL termination, but otherwise a fairly straightforward deploy.

mikelaws on 28 Jan 2019

👍3

This is still happening on 2.1.0 and it's not the same as #1861, as in #1861 the issue was the log was not being written to postgres.

In my case, after exactly 1 day the stdout stops refreshing automatically, I believe this is related to daphne's default 86400 seconds websocket_timeout. What worked to me was to edit /supervisor.conf on awx_web container and modify daphne's startup command to look like:

command = /var/lib/awx/venv/awx/bin/daphne -b 127.0.0.1 -p 8051 --websocket_timeout -1 awx.asgi:channel_layer

this prevents daphne from closing connections with the built-in nginx after 1 day.

falencastro on 13 Mar 2019

proxy_set_header Upgrade $http_upgrade;

Centos 7, docker, AWX 9.1, nginx proxying to port 443 with a cert, had to add this here in my nginx config, adding both lines didn't work. This is now working for refreshing the output.

thediymaker on 6 Jan 2020

What are right wstunnel proxypass directive for the Apache2 configuration?

Anyone have a working Apache2 configuration?

routenull0 on 14 Jan 2020

I don't have it. @routenull0

I moved it from Apache to Traefik. Since then life with Reverse Proxy was getting much more easier. No need more to google specific Apache2 configuration for many services.

RogerSik on 14 Jan 2020

If you are using reverse proxy in front of awx, make shure that your proxy can process websocket connections correctly. In case of nginx, put this into your config:
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;

this helps for my case.

In case anyone found this to fix the issue, i had to change
THIS: proxy_set_header Connection $connection_upgrade;
INTO THIS: proxy_set_header Connection "upgrade";
source: https://www.serverlab.ca/tutorials/linux/web-servers-linux/how-to-configure-nginx-for-websockets/

neuromancer-es on 28 Jul 2020

If you are looking to use Kubernetes annotations for latest Nginx ingress, nginx.ingress.kubernetes.io/configuration-snippet didn't work for me. Had to use:

nginx.org/location-snippets:
  proxy_set_header Upgrade $http_upgrade;
  proxy_set_header Connection "upgrade";

pipoe2h on 22 Nov 2020

Was this page helpful?

0 / 5 - 0 ratings

Related issues

Add examples in documentation on how to use awxkit from Python

beenje · 3Comments

Use semver for AWX version numbers

Gui13 · 3Comments

Setting a custom logo doesn't replace AWX logo in top-left of UI

agaffney · 3Comments

RFE: Add 'doas' in the drop down list of privilege escalation methods in Credential.

IshwarKanse · 3Comments

"Invalid search term entered. GET returned: 400 無効な credential_type ID: NaN" when click CREDENTIAL at Templates (LANG Japanese only)

kakkotetsu · 3Comments