Awx: Invalid search term entered. GET returned: -1 when trying to access Jobs list.

Created on 4 Nov 2019 · 16Comments · Source: ansible/awx

ISSUE TYPE

Bug Report

SUMMARY

Clicking on web ui elements after remaining idle on the current page for around 2 minutes, produces Invalid search term entered. GET returned: -1.

Docker container awx_web shows the following error message:

x.x.x.x - - [04/Nov/2019:16:04:10 +0000] "OPTIONS /api/v2/unified_jobs/ HTTP/1.1" 200 10336 "https://awx.local/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:70.0) Gecko/20100101 Firefox/70.0" "x.x.x.x"
[pid: 131|app: 0|req: 282/1127] x.x.x.x () {48 vars in 2502 bytes} [Mon Nov  4 16:04:09 2019] OPTIONS /api/v2/unified_jobs/ => generated 10336 bytes in 382 msecs (HTTP/1.1 200) 10 headers in 398 bytes (1 switches on core 0)
2019/11/04 16:04:11 [crit] 126#0: *1416 open() "/var/lib/nginx/tmp/uwsgi/2/03/0000000032" failed (13: Permission denied) while reading upstream, client: x.x.x.x, server: _, request: "GET /api/v2/unified_jobs/?page_size=20&order_by=-finished&not__launch_type=sync HTTP/1.1", upstream: "uwsgi://127.0.0.1:8050", host: "awx.local", referrer: "https://awx.local/"
x.x.x.x - - [04/Nov/2019:16:04:11 +0000] "GET /api/v2/unified_jobs/?page_size=20&order_by=-finished&not__launch_type=sync HTTP/1.1" 200 47895 "https://awx.local/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:70.0) Gecko/20100101 Firefox/70.0" "x.x.x.x"
[pid: 130|app: 0|req: 246/1128] x.x.x.x () {48 vars in 2590 bytes} [Mon Nov  4 16:04:09 2019] GET /api/v2/unified_jobs/?page_size=20&order_by=-finished&not__launch_type=sync => generated 85316 bytes in 1535 msecs (HTTP/1.1 200) 8 headers in 233 bytes (1 switches on core 0)

It seems to only happen when you are on the job template page and you want to navigate to the job list afterwards.
A full refresh of the website: Dashboard and ctrl + f5 seems to fix the problem.

ENVIRONMENT

AWX version: 9.0.0
AWX install method: docker-compose
Ansible version: 2.8.5
Operating System: Ubuntu 16.04
Web Browser: Firefox/Chrome

STEPS TO REPRODUCE

clicking on Job Templates -> Waiting around 1-2 minutes -> clicking on Jobs

EXPECTED RESULTS

Navigating through the web ui is working correctly.

ACTUAL RESULTS

Navigating after remaining on the web ui without any action produces the error.

ADDITIONAL INFORMATION

grafik

bug

Source

jon-rei

👍1

Most helpful comment

Just upgraded from 6.1.0 to 9.1.0 and same error appeared. @AntonSergievitch: your approach fixed the issue. Not a permanent solution though.

docker exec -it awx_web bash -c "chown -R nginx:nginx /var/lib/nginx"

AntonSergievitch on 17 Jan 2020

👍8 ❤4

All 16 comments

Couple of questions:

Are you using social auth of any kind?
After a hard refresh are you throw back to the login page?

shanemcd on 4 Nov 2019

I'm currently using LDAP Auth for all users
No, AWX doesn't throw me back to the login page. I'm remaining logged in. But some of my colleagues are reporting that they are being logged out.

jon-rei on 4 Nov 2019

I think I have found a fix for this issue.

As you can see from the logs the access to a subfolder under /var/lib/nginx/... is forbidden.
[crit] 126#0: *1416 open() "/var/lib/nginx/tmp/uwsgi/2/03/0000000032" failed (13: Permission denied) while reading upstream

I just went into the docker container and changed the file and group ownership from the whole folder structure under /var/lib/nginx to nginx:nginx
After my permission change the error didn't occured again.

The previous ownership was something like systemd-baseos, as the user and root as the group.

jon-rei on 5 Nov 2019

👍4 🚀1

I was seeing the same issue but the ownership was systemd-coredump
The folders seemed to have changed permissions back overnight and I'm no longer seeing any issues in the UI

AWX 9.0.1

bash-4.4# ls -lah
total 0
drwxrwx--- 7 systemd-coredump root 78 Nov  5 03:09 .
drwxrwx--- 3 systemd-coredump root 17 Nov  5 03:05 ..
drwx------ 2 nginx            root  6 Nov  5 03:09 client_body
drwx------ 2 nginx            root  6 Nov  5 03:09 fastcgi
drwx------ 2 nginx            root  6 Nov  5 03:09 proxy
drwx------ 2 nginx            root  6 Nov  5 03:09 scgi
drwx------ 2 nginx            root  6 Nov  5 03:09 uwsgi
bash-4.4# pwd
/var/lib/nginx/tmp

marshyonline on 5 Nov 2019

The error cropped back up for me today.
Any page load would throw the error.

2019/11/06 01:57:28 [crit] 123#0: *77 open() "/var/lib/nginx/tmp/uwsgi/0/01/0000000010" failed (13: Permission denied) while reading upstream, client: 172.19.8.170, server: _, request: "GET /api/ HTTP/1.1", upstream: "uwsgi://127.0.0.1:8050", host: "awx-dev.*****.com.au", referrer: "https://awx-dev.*****.com.au/"

I have re-deployed the container's and shall test if the issue happens again

Extra info:
Using LDAP
No users are logged out when the error shows

Fresh perms should be

bash-4.4# cd /var/lib/nginx/
bash-4.4# ls -lah
total 0
drwxrwx--- 3 nginx root 17 Nov  6 02:11 .
drwxr-xr-x 1 root  root 17 Nov  5 00:49 ..
drwxrwx--- 2 nginx root  6 Oct  7 21:17 tmp
bash-4.4# cd tmp/
bash-4.4# ls -lah
total 0
drwxrwx--- 7 nginx root 78 Nov  6 02:12 .
drwxrwx--- 3 nginx root 17 Nov  6 02:11 ..
drwx------ 2 nginx root  6 Nov  6 02:12 client_body
drwx------ 2 nginx root  6 Nov  6 02:12 fastcgi
drwx------ 2 nginx root  6 Nov  6 02:12 proxy
drwx------ 2 nginx root  6 Nov  6 02:12 scgi
drwx------ 2 nginx root  6 Nov  6 02:12 uwsgi

marshyonline on 6 Nov 2019

A Rebuild of the containers has resolved this issue for me

marshyonline on 6 Nov 2019

@marshyonline Did you rebuild the containers based on the latest devel branch? Or did you used the newest version from docker hub?

jon-rei on 7 Nov 2019

Sorry, I should have supplyed more information.
Running local-docker/Docker Compose.

The below steps are what I did.

Removed All containers
Checked out 9.0.1 from git
Re-created installer/inventory with the same options as the orginal install
re-ran installer/install.yml

marshyonline on 7 Nov 2019

I think I have found a fix for this issue.

As you can see from the logs the access to a subfolder under /var/lib/nginx/... is forbidden.
[crit] 126#0: *1416 open() "/var/lib/nginx/tmp/uwsgi/2/03/0000000032" failed (13: Permission denied) while reading upstream

I just went into the docker container and changed the file and group ownership from the whole folder structure under /var/lib/nginx to nginx:nginx
After my permission change the error didn't occured again.

The previous ownership was something like systemd-baseos, as the user and root as the group.

Previous ounership is systemd-coredump root in docker for awx 9.1.0. Call chown -R nginx:nginx /var/lib/nginx solve problem on viewing finished jobs with 100+ hosts.

AntonSergievitch on 7 Jan 2020

Just upgraded from 6.1.0 to 9.1.0 and same error appeared. @AntonSergievitch: your approach fixed the issue. Not a permanent solution though.

nicolaibaralmueller on 13 Jan 2020

Just upgraded from 6.1.0 to 9.1.0 and same error appeared. @AntonSergievitch: your approach fixed the issue. Not a permanent solution though.

docker exec -it awx_web bash -c "chown -R nginx:nginx /var/lib/nginx"

AntonSergievitch on 17 Jan 2020

👍8 ❤4

I deployed AWX 9.3.0 (from the 9.3.0.tar.gz archive) on 2 servers (staging, prod) with a couple of days between both deployments.
Same inventory file for both servers, with dockerhub_base=ansible.

192.168.x.x - - [26/Mar/2020:10:36:48 +0000] "OPTIONS /api/v2/inventories/ HTTP/1.1" 200 13914 "https://awx.local/"
[pid: 136|app: 0|req: 209/1550] 192.168.x.x () {64 vars in 2777 bytes} [Thu Mar 26 10:36:48 2020] OPTIONS /api/v2/inventories/ => generated 13914 bytes in 90 msecs (HTTP/1.1 200) 8 headers in 239 bytes (1 switches on core 0)
2020/03/26 10:36:49 [crit] 133#0: *1988 open() "/var/lib/nginx/tmp/uwsgi/9/05/0000000059" failed (13: Permission denied) while reading upstream, client: 192.168.x.x, server: _, request: "GET /api/v2/inventories/?page_size=20&order_by=name HTTP/1.1", upstream: "uwsgi://127.0.0.1:8050", host: "awx.local", referrer: "https://awx.local/"

Server A: