Awx: [15.0.1] awx_redis: Fatal error, can't open config file '/usr/local/etc/redis/redis.conf': Permission denied when docker_compose_dir permissions are changed after AWX installation

[jean-christophe-manciot]

ISSUE TYPE

• Bug Report

SUMMARY

AWX redis container keeps restarting due to the aforementioned error.

ENVIRONMENT

• AWX version: 15.0.1
• AWX install method: docker compose on linux
• Ansible version: 2.10.2
• Operating System: Debian bullseye
• docker-ce: 5:19.03.13~3-0~debian-buster
• docker-compose: 1.25.4 or 1.27.3

STEPS TO REPRODUCE

0.1. Run inside Python3 virtual env
0.2. Install latest ansible, wheel, docker-compose and redis with pip3

1. Checkout 15.0.1 & create new build branch
2. Modify installer/inventory with:
   
   
   
   • awx_official=true
   
   
   • docker_compose_dir=/some/folder
   
   
   • host_port=
   
   
   • logging_driver=journald
   
   
   • pg_database=awx
   
   
   • pg_hostname=
   
   
   • pg_password=some_password
   
   
   • pg_port=<
   
   
   • pg_sslmode=require
   
   
   • pg_username=awx
   
   
   • postgres_data_dir=''
   
   
   • project_data_dir=/some/other/folder
   
   
   • secret_key=some_password
   
   
   • use_docker_compose=true
   
   
3. Restart PostgreSQL 10 AWX Cluster
4. Build as non-root with:

export awx_version="15.0.1"
cd installer
ansible-playbook -i inventory install.yml

EXPECTED RESULTS

All AWX docker containers should be created & running smoothly

ACTUAL RESULTS

# docker ps --all
CONTAINER ID        IMAGE                COMMAND                  CREATED             STATUS                          PORTS                    NAMES
7af443a35715        ansible/awx:15.0.1   "/usr/bin/tini -- /u…"   7 minutes ago       Up 7 minutes                    8052/tcp                 awx_task
8cc45e4c78dd        ansible/awx:15.0.1   "/usr/bin/tini -- /b…"   7 minutes ago       Up 7 minutes                    0.0.0.0:8000->8052/tcp   awx_web
2a36f8729acf        redis                "docker-entrypoint.s…"   8 minutes ago       Restarting (1) 36 seconds ago                            awx_redis

# docker logs --details -f awx_redis
...
 1:M 26 Oct 2020 11:01:22.670 # Redis is now ready to exit, bye bye...
 1:C 26 Oct 2020 11:01:39.596 # Fatal error, can't open config file '/usr/local/etc/redis/redis.conf': Permission denied
...[Fatal error repeated]

[phospi]

ran into the same issue with the following specs:

ENVIRONMENT

• AWX version: 15.0.1
• AWX install method: docker compose on linux
• Ansible version: 2.9
• Operating System: CentOS 7
• docker-ce: 1.13.1, build 64e9980/1.13.1
• docker-compose: 1.26.2, build unknown
• redis docker image version 6.0.5 or 6.0.9

[phospi]

@jean-christophe-manciot I think, I found the problem. Could you check if SELINUX is enabled and then disable it? This should do the trick...

[jean-christophe-manciot]

• I don't use SELinux, but apparmor.
• SELinux/apparmor offer important security features; you can't disable them just to work around a bug
• the error messages come from within the container; unless I'm missing something here, selinux/apparmor should not have any impact regarding this specific behavior.

[jitujk]

I'm facing this issue with AWX 13/14.1 & 15.0 too ?
RHEL 7 / SELinux disabled / installed under root ? Can anybody please help me ? Is there a workaround for this bug ?
awx_install_logs.txt Uploading the installation log for AWX 13/14 & 15 for reference.

All variables except directories are same, these were kept in their awx[version] dir i.e. awx15 / awx14 / awx13

localhost ansible_connection=local ansible_python_interpreter="/usr/bin/env python3"
[all:vars]
dockerhub_base=ansible
awx_task_hostname=awx
awx_web_hostname=awxweb
postgres_data_dir="/data01/awx15/pgdocker"
host_port=80
host_port_ssl=443
docker_compose_dir="/data01/awx15/awxcompose"
pg_username=awx
pg_password=awxpass
pg_database=awx
pg_port=5432
admin_user=admin
admin_password=password
create_preload_data=True
secret_key=awxsecret
project_data_dir=/data01/awx15/projects

[root@awxlab installer]# docker logs --details --tail -10000 awx_redis 1:C 07 Dec 2020 07:49:55.760 # Fatal error, can't open config file '/usr/local/etc/redis/redis.conf': Permission denied 1:C 07 Dec 2020 07:49:58.535 # Fatal error, can't open config file '/usr/local/etc/redis/redis.conf': Permission denied 1:C 07 Dec 2020 07:50:01.905 # Fatal error, can't open config file '/usr/local/etc/redis/redis.conf': Permission denied 1:C 07 Dec 2020 07:50:04.759 # Fatal error, can't open config file '/usr/local/etc/redis/redis.conf': Permission denied 1:C 07 Dec 2020 07:50:10.207 # Fatal error, can't open config file '/usr/local/etc/redis/redis.conf': Permission denied 1:C 07 Dec 2020 07:50:12.537 # Fatal error, can't open config file '/usr/local/etc/redis/redis.conf': Permission denied 1:C 07 Dec 2020 07:50:14.606 # Fatal error, can't open config file '/usr/local/etc/redis/redis.conf': Permission denied 1:C 07 Dec 2020 07:50:17.041 # Fatal error, can't open config file '/usr/local/etc/redis/redis.conf': Permission denied 1:C 07 Dec 2020 07:50:19.420 # Fatal error, can't open config file '/usr/local/etc/redis/redis.conf': Permission denied 1:C 07 Dec 2020 07:50:22.614 # Fatal error, can't open config file '/usr/local/etc/redis/redis.conf': Permission denied 1:C 07 Dec 2020 07:50:27.243 # Fatal error, can't open config file '/usr/local/etc/redis/redis.conf': Permission denied 1:C 07 Dec 2020 07:50:35.168 # Fatal error, can't open config file '/usr/local/etc/redis/redis.conf': Permission denied 1:C 07 Dec 2020 07:50:49.441 # Fatal error, can't open config file '/usr/local/etc/redis/redis.conf': Permission denied 1:C 07 Dec 2020 07:51:16.562 # Fatal error, can't open config file '/usr/local/etc/redis/redis.conf': Permission denied 1:C 07 Dec 2020 07:52:09.256 # Fatal error, can't open config file '/usr/local/etc/redis/redis.conf': Permission denied 1:C 07 Dec 2020 07:53:53.159 # Fatal error, can't open config file '/usr/local/etc/redis/redis.conf': Permission denied 1:C 07 Dec 2020 07:57:19.231 # Fatal error, can't open config file '/usr/local/etc/redis/redis.conf': Permission denied 1:C 07 Dec 2020 08:04:10.262 # Fatal error, can't open config file '/usr/local/etc/redis/redis.conf': Permission denied 1:C 07 Dec 2020 08:17:50.938 # Fatal error, can't open config file '/usr/local/etc/redis/redis.conf': Permission denied 1:C 07 Dec 2020 08:45:10.800 # Fatal error, can't open config file '/usr/local/etc/redis/redis.conf': Permission denied

[jean-christophe-manciot]

I've looked into the Fatal error, can't open config file '/usr/local/etc/redis/redis.conf': Permission denied issue and it seems to be linked to improper file owner/group permissions:

docker-jump-into-container.sh awx_redis
root@a5e1be5cfc26:/data# ls -al /usr/local/etc/redis/redis.conf
-rw-rw---- 1 1000 904 78 Dec  9 12:20 /usr/local/etc/redis/redis.conf

root@a5e1be5cfc26:/data# more /usr/local/etc/redis/redis.conf
unixsocket /var/run/redis/redis.sock
unixsocketperm 660
port 0
bind 127.0.0.1

root@a5e1be5cfc26:/data# _l="/etc/login.defs"
root@a5e1be5cfc26:/data# _p="/etc/passwd"
root@a5e1be5cfc26:/data#  
root@a5e1be5cfc26:/data# ## get mini UID limit ##
root@a5e1be5cfc26:/data# l=$(grep "^UID_MIN" $_l)
root@a5e1be5cfc26:/data#  
root@a5e1be5cfc26:/data# ## get max UID limit ##
root@a5e1be5cfc26:/data# l1=$(grep "^UID_MAX" $_l)
root@a5e1be5cfc26:/data#  
root@a5e1be5cfc26:/data# echo "----------[ Normal User Accounts ]---------------"
----------[ Normal User Accounts ]---------------
root@a5e1be5cfc26:/data# awk -F':' -v "min=${l##UID_MIN}" -v "max=${l1##UID_MAX}" '{ if ( $3 >= min && $3 <= max  && $7 != "/sbin/nologin" ) print $0 }' "$_p" | sort
root@a5e1be5cfc26:/data#  

root@a5e1be5cfc26:/data# echo "----------[ System User Accounts ]---------------"
----------[ System User Accounts ]---------------
root@a5e1be5cfc26:/data# awk -F':' -v "min=${l##UID_MIN}" -v "max=${l1##UID_MAX}" '{ if ( !($3 >= min && $3 <= max  && $7 != "/sbin/nologin")) print $0 }' "$_p" | sort
_apt:x:100:65534::/nonexistent:/usr/sbin/nologin
backup:x:34:34:backup:/var/backups:/usr/sbin/nologin
bin:x:2:2:bin:/bin:/usr/sbin/nologin
daemon:x:1:1:daemon:/usr/sbin:/usr/sbin/nologin
games:x:5:60:games:/usr/games:/usr/sbin/nologin
gnats:x:41:41:Gnats Bug-Reporting System (admin):/var/lib/gnats:/usr/sbin/nologin
irc:x:39:39:ircd:/var/run/ircd:/usr/sbin/nologin
list:x:38:38:Mailing List Manager:/var/list:/usr/sbin/nologin
lp:x:7:7:lp:/var/spool/lpd:/usr/sbin/nologin
mail:x:8:8:mail:/var/mail:/usr/sbin/nologin
man:x:6:12:man:/var/cache/man:/usr/sbin/nologin
news:x:9:9:news:/var/spool/news:/usr/sbin/nologin
nobody:x:65534:65534:nobody:/nonexistent:/usr/sbin/nologin
proxy:x:13:13:proxy:/bin:/usr/sbin/nologin
redis:x:999:999::/home/redis:/bin/sh
root:x:0:0:root:/root:/bin/bash
sync:x:4:65534:sync:/bin:/bin/sync
sys:x:3:3:sys:/dev:/usr/sbin/nologin
uucp:x:10:10:uucp:/var/spool/uucp:/usr/sbin/nologin
www-data:x:33:33:www-data:/var/www:/usr/sbin/nologin

root@a48dd398ee63:/data# getent group | sort
adm:x:4:
audio:x:29:
backup:x:34:
bin:x:2:
cdrom:x:24:
daemon:x:1:
dialout:x:20:
dip:x:30:
disk:x:6:
fax:x:21:
floppy:x:25:
games:x:60:
gnats:x:41:
irc:x:39:
kmem:x:15:
list:x:38:
lp:x:7:
mail:x:8:
man:x:12:
news:x:9:
nogroup:x:65534:
operator:x:37:
plugdev:x:46:
proxy:x:13:
redis:x:999:
root:x:0:
sasl:x:45:
shadow:x:42:
src:x:40:
staff:x:50:
sudo:x:27:
sys:x:3:
tape:x:26:
tty:x:5:
users:x:100:
utmp:x:43:
uucp:x:10:
video:x:44:
voice:x:22:
www-data:x:33:
root@a48dd398ee63:/data# getent group | grep 904
root@a48dd398ee63:/data#

root@a5e1be5cfc26:/data# chown redis: /usr/local/etc/redis/redis.conf
chown: changing ownership of '/usr/local/etc/redis/redis.conf': Read-only file system

root@a48dd398ee63:/data# ls -al /var/run/redis/redis.sock
srw-rw---- 1 1000 904 0 Dec 11 09:52 /var/run/redis/redis.sock

As a summary,

• the /usr/local/etc/redis/redis.conf file owner and group 1000:904 are unknown
• it is not world-readable despite the ansible configuration in git-awx/installer/roles/local_docker/tasks/compose.yml:

    - file: redis.conf
      mode: "0664"

• the owner/group of the file cannot be changed while running.
• they could be redis:redis, i.e 999:999 here
• same issue with /var/run/redis/redis.sock

[jean-christophe-manciot]

Confirming that redis is started as redis:redis inside the awx_redis container:

# docker-jump-into-container.sh awx_redis
root@d4041a4cf7a0:/data# shopt -s extglob
root@d4041a4cf7a0:/data# for p in $(cd /proc && echo +([0-9]))
> do
>         ls -al /proc/${p}/exe  2>/dev/null| grep -P redis
> done
lrwxrwxrwx 1 redis redis 0 Dec 16 12:32 /proc/1/exe
root@d4041a4cf7a0:/data# 

[jean-christophe-manciot]

I have just realized that the permissions of /usr/local/etc/redis/redis.conf inside awx_redis container mirror exactly the ones of the docker_compose_dir (from installer/inventory) local folder where the redis socket is created.
They are "sucked in" the container and used somehow to define the permissions of:

• /usr/local/etc/redis/redis.conf
• /run/redis
• /run/redis/redis.sock

# docker-jump-into-container.sh awx_redis
root@6c9e49fce8c3:/data# ls -al /usr/local/etc/redis/redis.conf
-rw-rw-rw- 1 1000 redis 78 Dec 17 11:22 /usr/local/etc/redis/redis.conf
root@6c9e49fce8c3:/data# find / -u 1000
find: unknown predicate `-u'
root@6c9e49fce8c3:/data# find / -uid 1000
/usr/local/etc/redis/redis.conf
/run/redis
/run/redis/redis.sock

So, a temporary workaround is to change the owner and/or group of docker_compose_dir folder to 999 right after the playbook is launched with:

cd installer
ansible-playbook -i inventory install.yml
sudo chown --no-dereference -R ${user}:999 "$docker_compose_dir"
find -L "$docker_compose_dir" -type d -exec sudo chmod 770 {} \;
find -L "$docker_compose_dir" -type f -exec sudo chmod 660 {} \;

With that dirty trick, I was able to work around this issue.

[jean-christophe-manciot]

As a summary:

• all docker_compose_dir permissions are sucked into awx_redis container and used for redis configuration and socket files
• those permissions are not extended to the redis daemon itself
• those permissions are also sucked into the other AWX containers
• this issue happens only when the permissions of the docker_compose_dir are changed right after AWX installation to prevent the folder from being world-accessible, for instance from 777 to 770.
• a workaround exists as described in the previous post

My proposal to improve the security of AWX when using docker compose is to:

• change in installer/roles/local_docker/tasks/compose.yml mode: 0777 ---> mode: 0770
• change the owner/group of the redis daemon to match docker_compose_dir ones
• are other adjustments necessary?