Awx: AWX does not start after reboot using Docker

[Techdox]

ISSUE TYPE

• Bug Report

COMPONENT NAME

Docker Containers

SUMMARY

When AWX is installed via docker containers using the install.yaml file AWX is installed and runs fine, but when the server (Ubuntu) is restarted I get the following errors when trying to start up the 2 / 5 docker containers that turned off.
Error response from daemon: OCI runtime create failed: container_linux.go:344: starting container process caused "process_linux.go:424: container init caused \"rootfs_linux.go:58: mounting \\\"/tmp/awxcompose/SECRET_KEY\\\" to rootfs \\\"/var/lib/docker/overlay2/ba75a051fdae455c34a77e607dbb1c52ad5f03a8eb3f46cf02577ff4e89da0aa/merged\\\" at \\\"/var/lib/docker/overlay2/ba75a051fdae455c34a77e607dbb1c52ad5f03a8eb3f46cf02577ff4e89da0aa/merged/etc/tower/SECRET_KEY\\\" caused \\\"not a directory\\\"\"": unknown: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type Error response from daemon: OCI runtime create failed: container_linux.go:344: starting container process caused "process_linux.go:424: container init caused \"rootfs_linux.go:58: mounting \\\"/tmp/awxcompose/SECRET_KEY\\\" to rootfs \\\"/var/lib/docker/overlay2/9d6a0c7a4909b70ff47f4c75abd44b968277267af21409761ed7c24fc25d6264/merged\\\" at \\\"/var/lib/docker/overlay2/9d6a0c7a4909b70ff47f4c75abd44b968277267af21409761ed7c24fc25d6264/merged/etc/tower/SECRET_KEY\\\" caused \\\"not a directory\\\"\"": unknown: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type

ENVIRONMENT

• AWX version: 4.0.0
• AWX install method: Docker
• Ansible version: 2.7.9
• Operating System: Ubuntu 18.04.2 LTS
• Web Browser: Firefox

STEPS TO REPRODUCE

Install AWX via the install.yaml script using docker, and the reboot the system.

EXPECTED RESULTS

The docker containers to be still running after a reboot, or at least able to be started.

ACTUAL RESULTS

After reboot 2 out of 5 docker containers have stopped and can not be restarted

ADDITIONAL INFORMATION

SCREENSHOTS:

[jakemcdermott]

@Techdox
Make sure you define a value for postgres_data_dir in your install inventory.

https://github.com/ansible/awx/blob/devel/installer/inventory#L54

This is where your database files are supposed to live. We have the default for this and other variables tuned for short-lived development environments that don't stick around for long. It uses /tmp/pgdocker which won't persist across reboots for some systems.

[Techdox]

@jakemcdermott I have made that change now, but even after the change it still does not launch.

[jakemcdermott]

you'll need to delete your containers and completely reinstall after changing the config

[Techdox]

@jakemcdermott Sorry I should of said I did delete the old ones, recreated and then rebooted. No luck

[ryan012880]

@Techdox the error is caused by container not able to mount the volume bindings in /tmp/awxcompose so make sure that those files are available. Installer will create docker-compose files by default in /tmp/awxcompose if docker_compose_dir is not defined in your inventory file.

FYI, if you move this directory to another location after installation you need to remove the container first and edit the file docker-compose.yml to make sure the volumes are pointing to the correct path for task and web service then create the container again.

[jakemcdermott]

@ryan012880 makes a great point here - there are probably _other_ inventory variables defaulting to /tmp/* that you'll want to set as well. Have a look at the inventory and installer playbooks to make sure you have all the settings you need.

[Techdox]

@ryan012880 Cheers! I made the docker_compose_dir linking to a directory I created and it's surviving a reboot now. Thanks

[mwiora]

@jakemcdermott I've realized that meanwhile there is another line beside of the postgres-one that causes:

"Error": "OCI runtime create failed: container_linux.go:345: starting container process caused \"process_linux.go:424: container init caused \\\"rootfs_linux.go:58: mounting \\\\\\\"/tmp/awxcompose/SECRET_KEY\\\\\\\" to rootfs \\\\\\\"/var/lib/docker/overlay2/.../merged\\\\\\\" at \\\\\\\"/var/lib/docker/overlay2/.../merged/etc/tower/SECRET_KEY\\\\\\\" caused \\\\\\\"not a directory\\\\\\\"\\\"\": unknown: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type"

Is it really a good idea to place the two parts to /tmp? As most of the users will install their environment - luckily use it and afterwards - e.g. after and update or something will reboot the device..... and then will be upset if the data is missing and their containers not running.....

Maybe it would be possible to place it where it belongs... or to at least bring up a warning that a /tmp/ folder has been used during the compose script....

just my 2cents....
// matthias

[LaurentDumont]

Just got bit by this. I'm not sure why the default location would be somewhere that is deleted after a reboot. Seems a bit counterintuitive.

[mwiora]

just created a PR to solve this issue permanently

[shoguevara]

3705

[eholz1]

Hello All - we have this issue still - but I am confused (still!)
We want a local install. I have modified the inventory file to put pg_docker and awxcompose in their own folder, and NOT in the root/.awx path. The inventory file I am using is 3 weeks old, and has NO refs for any variable referencing the /tmp directory. It does ref the /root/.awx dir. I changed those directories to /AWX/ etc - When the intall.yml file runs (for the first time) it creates those directories.

Also I have uncommented the /var/lib/awx/projects directory and put a directory there to hold the yaml files - that works fine.

I am confused by this line at the top of the inventory file: dockerhub_base=ansible, is this supposed to be commented out for a local install?

Also as a "safety" measure I have commented out the awxcompose and pgdocker directory creation if for some reason I have to re-run the install.yml file. If I do not do this, the install will crash with the "read only file system" error (since the directories are already there).

I would like some clarification on what it really takes to prevent the system from crashing on a reboot - our system is Ubuntu 19.10m and the sympton is the awx_postgres container is stuck/sticks in the "restarting" mode.

Thanks for a good source of info.
ewholz