ISSUE TYPE
- Bug Report
SUMMARY
AWX Installation failed with unknow error
ENVIRONMENT
- AWX version: 4.0.0
- AWX install method: docker on linux
- Ansible version: 2.8.0
- Operating System: CentOS 7.6.1810
- Web Browser: Chrome
STEPS TO REPRODUCE
I had AWX 3.0.1 running and then I tried to install AWX 4.0.0 on top, the same procedure worked fine to upgrade AWX 3.0.0. to AWX 3.0.1
`# ansible-playbook -i inventory install.yml -vvvv
.
.
TASK [local_docker : Start the containers] **************************************************************************************************************************************************************************************************
task path: /root/awx-4/installer/roles/local_docker/tasks/compose.yml:25
<localhost> ESTABLISH LOCAL CONNECTION FOR USER: root
<localhost> EXEC /bin/sh -c 'echo ~root && sleep 0'
<localhost> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /root/.ansible/tmp/ansible-tmp-1559049469.25-220943567830274 `" && echo ansible-tmp-1559049469.25-220943567830274="` echo /root/.ansible/tmp/ansible-tmp-1559049469.25-220943567830274 `" ) && sleep 0'
Using module file /usr/lib/python2.7/site-packages/ansible/modules/cloud/docker/_docker_service.py
<localhost> PUT /root/.ansible/tmp/ansible-local-60293Nj0BSi/tmpiVJnh7 TO /root/.ansible/tmp/ansible-tmp-1559049469.25-220943567830274/AnsiballZ__docker_service.py
<localhost> EXEC /bin/sh -c 'chmod u+x /root/.ansible/tmp/ansible-tmp-1559049469.25-220943567830274/ /root/.ansible/tmp/ansible-tmp-1559049469.25-220943567830274/AnsiballZ__docker_service.py && sleep 0'
<localhost> EXEC /bin/sh -c '/usr/bin/env python /root/.ansible/tmp/ansible-tmp-1559049469.25-220943567830274/AnsiballZ__docker_service.py && sleep 0'
<localhost> EXEC /bin/sh -c 'rm -f -r /root/.ansible/tmp/ansible-tmp-1559049469.25-220943567830274/ > /dev/null 2>&1 && sleep 0'
The full traceback is:
WARNING: The below traceback may *not* be related to the actual failure.
File "/tmp/ansible_docker_service_payload_Dj9vLs/__main__.py", line 764, in cmd_up
timeout=self.timeout)
File "/usr/lib/python2.7/site-packages/compose/project.py", line 559, in up
'Encountered errors while bringing up the project.'
[DEPRECATION WARNING]: The 'docker_service' module has been renamed to 'docker_compose'.. This feature will be removed in version 2.12. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.
fatal: [localhost]: FAILED! => {
"changed": false,
"errors": [],
"invocation": {
"module_args": {
"api_version": "auto",
"build": false,
"ca_cert": null,
"client_cert": null,
"client_key": null,
"debug": false,
"definition": null,
"dependencies": true,
"docker_host": "unix://var/run/docker.sock",
"files": null,
"hostname_check": false,
"nocache": false,
"project_name": null,
"project_src": "/var/lib/awx",
"pull": false,
"recreate": "smart",
"remove_images": null,
"remove_orphans": false,
"remove_volumes": false,
"restarted": false,
"scale": null,
"services": null,
"ssl_version": null,
"state": "present",
"stopped": false,
"timeout": 10,
"tls": false,
"tls_hostname": "localhost",
"validate_certs": false
}
},
"module_stderr": "Creating awx_web ... \r\n\r\u001b[1B",
"module_stdout": "",
"msg": "Error starting project unknown cause"
}
PLAY RECAP **********************************************************************************************************************************************************************************************************************************
localhost : ok=8 changed=0 unreachable=0 failed=1 skipped=77 rescued=0 ignored=0
`
more dockers are created:
`# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
6b20e23a0be7 ansible/awx_rabbitmq:3.7.4 "docker-entrypoint.s…" 7 minutes ago Up 6 seconds 4369/tcp, 5671-5672/tcp, 15671-15672/tcp, 25672/tcp awx_rabbitmq
5d25e036cd96 memcached:alpine "docker-entrypoint.s…" 7 minutes ago Up 6 seconds 11211/tcp awx_memcached
695c8aa6d2b7 postgres:9.6 "docker-entrypoint.s…" 7 minutes ago Up 5 seconds 5432/tcp awx_postgres
8246270599a5 ansible/awx_task:3.0.1 "/tini -- /bin/sh -c…" About an hour ago Up 4 seconds 8052/tcp awx_task
cb18997147ae ansible/awx_web:3.0.1 "/tini -- /bin/sh -c…" About an hour ago Up 4 seconds 0.0.0.0:8052->8052/tcp awx_web
1580ef93f47d memcached:alpine "docker-entrypoint.s…" 21 hours ago Up 6 seconds 11211/tcp memcached
97c10bb0b7ac ansible/awx_rabbitmq:3.7.4 "docker-entrypoint.s…" 21 hours ago Up 6 seconds 4369/tcp, 5671-5672/tcp, 15671-15672/tcp, 25672/tcp rabbitmq
4c527a00794b postgres:9.6 "docker-entrypoint.s…" 21 hours ago Up 6 seconds 5432/tcp postgres`
EXPECTED RESULTS
AWX Installed successfully
ACTUAL RESULTS
"module_stderr": "Creating awx_web ... \r\n\r\u001b[1B",
"module_stdout": "",
"msg": "Error starting project unknown cause"
ADDITIONAL INFORMATION
`# cat inventory
localhost ansible_connection=local ansible_python_interpreter="/usr/bin/env python"
[all:vars]
# Remove these lines if you want to run a local image build
# Otherwise the setup playbook will install the official Ansible images. Versions may
# be selected based on: latest, 1, 1.0, 1.0.0, 1.0.0.123
# by default the base will be used to search for ansible/awx_web and ansible/awx_task
dockerhub_base=ansible
# Openshift Install
# Will need to set -e openshift_password=developer -e docker_registry_password=$(oc whoami -t)
# or set -e openshift_token=TOKEN
# openshift_host=127.0.0.1:8443
# openshift_project=awx
# openshift_user=developer
# openshift_skip_tls_verify=False
# openshift_pg_emptydir=True
# Kubernetes Install
# kubernetes_context=test-cluster
# kubernetes_namespace=awx
# tiller_namespace=kube-system
# Optional Kubernetes Variables
# pg_persistence_storageClass=StorageClassName
# pg_cpu_limit=1000
# pg_mem_limit=2
# Kubernetes and Openshift Install Resource Requests
# This are the request and limit values for a pod's container for task/web/rabbitmq/memcached/management.
# The total amount of requested resources for a pod is the sum of all
# resources requested by all containers in the pod
# A cpu_request of 1500 is 1.5 cores for the container to start out with.
# A cpu_limit defines the maximum cores that that container can reserve.
# A mem_request of 2 is for 2 gigabytes of memory for the container
# A mem_limit defines the maximum memory that that container can reserve.
# Default values for these entries can be found in ./roles/kubernetes/defaults/main.yml
# task_cpu_request=1500
# task_mem_request=2
# task_cpu_limit=2000
# task_mem_limit=4
# web_cpu_limit=1000
# web_mem_limit=2
# rabbitmq_cpu_limit=1000
# rabbitmq_mem_limit=3
# memcached_cpu_limit=1000
# memcached_mem_limit=2
# management_cpu_limit=2000
# management_mem_limit=2
# Common Docker parameters
awx_task_hostname=awx
awx_web_hostname=awxweb
postgres_data_dir=/ansible/pgdocker
host_port=8052
# Docker Compose Install
# use_docker_compose=false
# The docker_compose.yml file will be created in this directory
# The name of the directory (here "awx") will be the prefix of the docker containers
docker_compose_dir=/var/lib/awx
# Required for Openshift when building the image on your own
# Optional for Openshift if using Dockerhub or another prebuilt registry
# Required for Standalone Docker Install if building the image on your own
# Optional for Standalone Docker Install if using Dockerhub or another prebuilt registry
# Required for Docker Compose Install if building the image on your own
# Optional for Docker Compose Install if using Dockerhub or another prebuilt registry
# Define if you want the image pushed to a registry. The container definition will also use these images
# docker_registry=172.30.1.1:5000
# docker_registry_repository=awx
# docker_registry_username=developer
# Docker_image will not attempt to push to remote if the image already exists locally
# Set this to true to delete images from docker on the build host so that they are pushed to the remote repository
# docker_remove_local_images=False
# Set pg_hostname if you have an external postgres server, otherwise
# a new postgres service will be created
# pg_hostname=postgresql
pg_username=xxxx
# pg_password should be random 10 character alphanumeric string, when postgresql is running on kubernetes
# NB: it's a limitation of the "official" postgres helm chart
pg_password=xxxx$$
pg_database=xxxx_db
pg_port="5432"
# RabbitMQ Configuration
rabbitmq_password=XXXX
rabbitmq_erlang_cookie=XXXX
# Use a local distribution build container image for building the AWX package
# This is helpful if you don't want to bother installing the build-time dependencies as
# it is taken care of already.
# NOTE: IMPORTANT: If you are running a mininshift install, using this container might not work
# if you are using certain drivers like KVM where the source tree can't be mapped
# into the build container.
# Thus this setting must be set to False which will trigger a local build. To view the
# typical dependencies that you might need to install see:
# installer/image_build/files/Dockerfile.sdist
# use_container_for_build=true
# This will create or update a default admin (superuser) account in AWX, if not provided
# then these default values are used
admin_user=admin
admin_password=XXXXX
# Whether or not to create preload data for demonstration purposes
create_preload_data=True
# AWX Secret key
# It's *very* important that this stay the same between upgrades or you will lose the ability to decrypt
# your credentials
secret_key=XXXXX
# Build AWX with official logos
# Requires cloning awx-logos repo into the project root.
# Review the trademark guidelines at https://github.com/ansible/awx-logos/blob/master/TRADEMARKS.md
# awx_official=false
# Proxy
#http_proxy=http://proxy:3128
#https_proxy=http://proxy:3128
#no_proxy=mycorp.org
# Container networking configuration
# Set the awx_task and awx_web containers' search domain(s)
#awx_container_search_domains=example.com,ansible.com
# Alternate DNS servers
#awx_alternate_dns_servers="10.1.2.3,10.2.3.4"
# AWX project data folder. If you need access to the location where AWX stores the projects
# it manages from the docker host, you can set this to turn it into a volume for the container.
#project_data_dir=/var/lib/awx/projects
project_data_dir=/ansible/projects
# CA Trust directory. If you need to provide custom CA certificates, supplying
# this variable causes this directory on the host to be bind mounted over
# /etc/pki/ca-trust in the awx_task and awx_web containers.
# NOTE: only obeyed in local_docker install
#ca_trust_dir=/etc/pki/ca-trust/source/anchors
# Include /etc/nginx/awx_extra.conf
# Note the use of glob pattern for nginx
# which makes include "optional" - i.e. not fail
# if file is absent
#extra_nginx_include="/etc/nginx/awx_extra[.]conf"`
ssaammoo
All 30 comments
@ssaammoo I've seen errors like this with older docker-compose versions. What version of docker-compose are you using? Try updating to the latest?
ryanpetrello
on 28 May 2019
@ryanpetrello
here we go:
# docker-compose --version
/usr/lib/python2.7/site-packages/requests/__init__.py:91: RequestsDependencyWarning: urllib3 (1.24.1) or chardet (2.2.1) doesn't match a supported version!
RequestsDependencyWarning)
docker-compose version 1.24.0, build 0aa5906
@ssaammoo this version is working for me, maybe give it a shot?
docker-compose --version
docker-compose version 1.23.2, build 1110ad0
ryanpetrello
on 28 May 2019
Unfortunately didn't work either :-(
TASK [local_docker : Start the containers] **************************************************************************************************************************************************************************************************
task path: /root/awx-4/installer/roles/local_docker/tasks/compose.yml:25
<localhost> ESTABLISH LOCAL CONNECTION FOR USER: root
<localhost> EXEC /bin/sh -c 'echo ~root && sleep 0'
<localhost> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /root/.ansible/tmp/ansible-tmp-1559123810.83-41538014110879 `" && echo ansible-tmp-1559123810.83-41538014110879="` echo /root/.ansible/tmp/ansible-tmp-1559123810.83-41538014110879 `" ) && sleep 0'
Using module file /usr/lib/python2.7/site-packages/ansible/modules/cloud/docker/_docker_service.py
<localhost> PUT /root/.ansible/tmp/ansible-local-72810XBXq4/tmpTRtFv8 TO /root/.ansible/tmp/ansible-tmp-1559123810.83-41538014110879/AnsiballZ__docker_service.py
<localhost> EXEC /bin/sh -c 'chmod u+x /root/.ansible/tmp/ansible-tmp-1559123810.83-41538014110879/ /root/.ansible/tmp/ansible-tmp-1559123810.83-41538014110879/AnsiballZ__docker_service.py && sleep 0'
<localhost> EXEC /bin/sh -c '/usr/bin/env python /root/.ansible/tmp/ansible-tmp-1559123810.83-41538014110879/AnsiballZ__docker_service.py && sleep 0'
<localhost> EXEC /bin/sh -c 'rm -f -r /root/.ansible/tmp/ansible-tmp-1559123810.83-41538014110879/ > /dev/null 2>&1 && sleep 0'
The full traceback is:
WARNING: The below traceback may *not* be related to the actual failure.
File "/tmp/ansible_docker_service_payload_JsXP6k/__main__.py", line 764, in cmd_up
timeout=self.timeout)
File "/usr/lib/python2.7/site-packages/compose/project.py", line 507, in up
'Encountered errors while bringing up the project.'
[DEPRECATION WARNING]: The 'docker_service' module has been renamed to 'docker_compose'.. This feature will be removed in version 2.12. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.
fatal: [localhost]: FAILED! => {
"changed": false,
"errors": [],
"invocation": {
"module_args": {
"api_version": "auto",
"build": false,
"ca_cert": null,
"client_cert": null,
"client_key": null,
"debug": false,
"definition": null,
"dependencies": true,
"docker_host": "unix://var/run/docker.sock",
"files": null,
"hostname_check": false,
"nocache": false,
"project_name": null,
"project_src": "/var/lib/awx",
"pull": false,
"recreate": "smart",
"remove_images": null,
"remove_orphans": false,
"remove_volumes": false,
"restarted": false,
"scale": null,
"services": null,
"ssl_version": null,
"state": "present",
"stopped": false,
"timeout": 10,
"tls": false,
"tls_hostname": "localhost",
"validate_certs": false
}
},
"module_stderr": "Creating awx_web ... \r\n\r\u001b[1B",
"module_stdout": "",
"msg": "Error starting project unknown cause"
}
PLAY RECAP **********************************************************************************************************************************************************************************************************************************
localhost : ok=8 changed=0 unreachable=0 failed=1 skipped=77 rescued=0 ignored=0
and got multiple dockers
[root@alva-atmans01 installer]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
8acb85229d6c memcached:alpine "docker-entrypoint.s…" 24 seconds ago Up 23 seconds 11211/tcp awx_memcached
1465ade16aba ansible/awx_rabbitmq:3.7.4 "docker-entrypoint.s…" 24 seconds ago Up 23 seconds 4369/tcp, 5671-5672/tcp, 15671-15672/tcp, 25672/tcp awx_rabbitmq
adef4b3ff855 postgres:9.6 "docker-entrypoint.s…" 24 seconds ago Up 23 seconds 5432/tcp awx_postgres
8246270599a5 ansible/awx_task:3.0.1 "/tini -- /bin/sh -c…" 22 hours ago Up 18 hours 8052/tcp awx_task
cb18997147ae ansible/awx_web:3.0.1 "/tini -- /bin/sh -c…" 22 hours ago Up 18 hours 0.0.0.0:8052->8052/tcp awx_web
1580ef93f47d memcached:alpine "docker-entrypoint.s…" 42 hours ago Up 18 hours 11211/tcp memcached
97c10bb0b7ac ansible/awx_rabbitmq:3.7.4 "docker-entrypoint.s…" 42 hours ago Up 18 hours 4369/tcp, 5671-5672/tcp, 15671-15672/tcp, 25672/tcp rabbitmq
4c527a00794b postgres:9.6 "docker-entrypoint.s…" 42 hours ago Up 18 hours 5432/tcp postgres
[root@alva-atmans01 installer]#
Regretfully I'm running into the same thing. If you attempt to start the containers using docker-compose or docker directly, you'll get the following:
Error response from daemon: OCI runtime create failed: container_linux.go:345: starting container process caused "process_linux.go:424: container init caused \"rootfs_linux.go:58: mounting \\\"/tmp/awxcompose/SECRET_KEY\\\" to rootfs \\\"/var/lib/docker/overlay2/dcb48bab9cd37651132f72a37c17d1450940272cff5cec20c85544cb04d33c48/merged\\\" at \\\"/var/lib/docker/overlay2/dcb48bab9cd37651132f72a37c17d1450940272cff5cec20c85544cb04d33c48/merged/etc/tower/SECRET_KEY\\\" caused \\\"not a directory\\\"\"": unknown: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type
If you attempt the following, you'll get the error specified by the OP
$ sudo ansible-playbook -i inventory install.yml
Looking at where the installer is pulling the files, SECRET_KEY exists and is a directory:
$ ls -lah /tmp/awxcompose/
total 28K
drwxr-xr-x 5 root root 4.0K May 29 12:00 .
drwxrwxrwt 10 root root 4.0K May 29 12:21 ..
drwxr-xr-x 2 root root 4.0K May 29 08:56 credentials.py
-rw------- 1 root root 1.8K May 29 08:56 docker-compose.yml
drwxr-xr-x 2 root root 4.0K May 29 08:56 environment.sh
drwxr-xr-x 2 root root 4.0K May 29 08:50 SECRET_KEY
Ubuntu 16.04
docker-compose 1.24.0
docker 18.09
picapdjoe
on 29 May 2019
FWIW I encountered the same error when starting awx-web which wants to listen on port 80, turned out a separate install of nginx was using it (confirm via netstat -anpo | grep 80). After stopping nginx, the install playbook completed successfully.
kernelzilla
on 4 Jun 2019
👍12
I've opened a pull request (#4002) to check and give the user a clearer message regarding the port conflict instead of having to check awx_web log status for the error.
kernelzilla
on 4 Jun 2019
@ssaammoo @picapdjoe any chance either of you are encountering something similar?
ryanpetrello
on 4 Jun 2019
I had the same issue and it is port conflict.
jomach
on 5 Jul 2019
👍1
@ryanpetrello
Hi Ryan,
I am redoing the test now with a new version of AWX (5.0), I stopped nginx that was running as reverse proxy so I won't get a conflict, however I am still getting the same issue:
TASK [local_docker : Start the containers] **************************************************************************************************************************************************************************************************
task path: /root/awx/installer/roles/local_docker/tasks/compose.yml:25
<localhost> ESTABLISH LOCAL CONNECTION FOR USER: root
<localhost> EXEC /bin/sh -c 'echo ~root && sleep 0'
<localhost> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /root/.ansible/tmp/ansible-tmp-1562668950.98-226850187534945 `" && echo ansible-tmp-1562668950.98-226850187534945="` echo /root/.ansible/tmp/ansible-tmp-1562668950.98-226850187534945 `" ) && sleep 0'
Using module file /usr/lib/python2.7/site-packages/ansible/modules/cloud/docker/_docker_service.py
<localhost> PUT /root/.ansible/tmp/ansible-local-48857My9OcO/tmpMYu8nE TO /root/.ansible/tmp/ansible-tmp-1562668950.98-226850187534945/AnsiballZ__docker_service.py
<localhost> EXEC /bin/sh -c 'chmod u+x /root/.ansible/tmp/ansible-tmp-1562668950.98-226850187534945/ /root/.ansible/tmp/ansible-tmp-1562668950.98-226850187534945/AnsiballZ__docker_service.py && sleep 0'
<localhost> EXEC /bin/sh -c '/usr/bin/env python /root/.ansible/tmp/ansible-tmp-1562668950.98-226850187534945/AnsiballZ__docker_service.py && sleep 0'
<localhost> EXEC /bin/sh -c 'rm -f -r /root/.ansible/tmp/ansible-tmp-1562668950.98-226850187534945/ > /dev/null 2>&1 && sleep 0'
The full traceback is:
WARNING: The below traceback may *not* be related to the actual failure.
File "/tmp/ansible_docker_service_payload_XzUYXM/__main__.py", line 764, in cmd_up
timeout=self.timeout)
File "/usr/lib/python2.7/site-packages/compose/project.py", line 559, in up
'Encountered errors while bringing up the project.'
[DEPRECATION WARNING]: The 'docker_service' module has been renamed to 'docker_compose'.. This feature will be removed in version 2.12. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.
fatal: [localhost]: FAILED! => {
"changed": false,
"errors": [],
"invocation": {
"module_args": {
"api_version": "auto",
"build": false,
"ca_cert": null,
"client_cert": null,
"client_key": null,
"debug": false,
"definition": null,
"dependencies": true,
"docker_host": "unix://var/run/docker.sock",
"files": null,
"hostname_check": false,
"nocache": false,
"project_name": null,
"project_src": "/tmp/awxcompose",
"pull": false,
"recreate": "smart",
"remove_images": null,
"remove_orphans": false,
"remove_volumes": false,
"restarted": false,
"scale": null,
"services": null,
"ssl_version": null,
"state": "present",
"stopped": false,
"timeout": 10,
"tls": false,
"tls_hostname": "localhost",
"validate_certs": false
}
},
"module_stderr": "Creating awx_web ... \r\n\r\u001b[1B",
"module_stdout": "",
"msg": "Error starting project unknown cause"
}
PLAY RECAP **********************************************************************************************************************************************************************************************************************************
localhost : ok=8 changed=0 unreachable=0 failed=1 skipped=77 rescued=0 ignored=0
I have just tried with AWX 6.0 and got the same result :-(
TASK [local_docker : Start the containers] **************************************************************************************************************************************************************************************************
task path: /root/awx/installer/roles/local_docker/tasks/compose.yml:25
<localhost> ESTABLISH LOCAL CONNECTION FOR USER: root
<localhost> EXEC /bin/sh -c 'echo ~root && sleep 0'
<localhost> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /root/.ansible/tmp/ansible-tmp-1562670134.55-11038997918105 `" && echo ansible-tmp-1562670134.55-11038997918105="` echo /root/.ansible/tmp/ansible-tmp-1562670134.55-11038997918105 `" ) && sleep 0'
Using module file /usr/lib/python2.7/site-packages/ansible/modules/cloud/docker/_docker_service.py
<localhost> PUT /root/.ansible/tmp/ansible-local-20508258sWX/tmpTqiJI7 TO /root/.ansible/tmp/ansible-tmp-1562670134.55-11038997918105/AnsiballZ__docker_service.py
<localhost> EXEC /bin/sh -c 'chmod u+x /root/.ansible/tmp/ansible-tmp-1562670134.55-11038997918105/ /root/.ansible/tmp/ansible-tmp-1562670134.55-11038997918105/AnsiballZ__docker_service.py && sleep 0'
<localhost> EXEC /bin/sh -c '/usr/bin/env python /root/.ansible/tmp/ansible-tmp-1562670134.55-11038997918105/AnsiballZ__docker_service.py && sleep 0'
<localhost> EXEC /bin/sh -c 'rm -f -r /root/.ansible/tmp/ansible-tmp-1562670134.55-11038997918105/ > /dev/null 2>&1 && sleep 0'
The full traceback is:
WARNING: The below traceback may *not* be related to the actual failure.
File "/tmp/ansible_docker_service_payload_xgDsOe/__main__.py", line 764, in cmd_up
timeout=self.timeout)
File "/usr/lib/python2.7/site-packages/compose/project.py", line 559, in up
'Encountered errors while bringing up the project.'
[DEPRECATION WARNING]: The 'docker_service' module has been renamed to 'docker_compose'.. This feature will be removed in version 2.12. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.
fatal: [localhost]: FAILED! => {
"changed": false,
"errors": [],
"invocation": {
"module_args": {
"api_version": "auto",
"build": false,
"ca_cert": null,
"client_cert": null,
"client_key": null,
"debug": false,
"definition": null,
"dependencies": true,
"docker_host": "unix://var/run/docker.sock",
"files": null,
"hostname_check": false,
"nocache": false,
"project_name": null,
"project_src": "/tmp/awxcompose",
"pull": false,
"recreate": "smart",
"remove_images": null,
"remove_orphans": false,
"remove_volumes": false,
"restarted": false,
"scale": null,
"services": null,
"ssl_version": null,
"state": "present",
"stopped": false,
"timeout": 10,
"tls": false,
"tls_hostname": "localhost",
"validate_certs": false
}
},
"module_stderr": "Creating awx_web ... \r\n\r\u001b[1B",
"module_stdout": "",
"msg": "Error starting project unknown cause"
}
PLAY RECAP **********************************************************************************************************************************************************************************************************************************
localhost : ok=8 changed=0 unreachable=0 failed=1 skipped=77 rescued=0 ignored=0
I think my problem is not related to the port (at least now), I have add check port to my playbook:
https://github.com/kernelzilla/awx/commit/949e37e809513adf66ba868d9c3bf5088696ec2e
And I am getting still generic error.
TASK [local_docker : Start the containers] **************************************************************************************************************************************************************************************************
[DEPRECATION WARNING]: The 'docker_service' module has been renamed to 'docker_compose'.. This feature will be removed in version 2.12. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.
fatal: [localhost]: FAILED! => {"changed": false, "errors": [], "module_stderr": "Creating awx_web ... \r\n\r\u001b[1B", "module_stdout": "", "msg": "Error starting project unknown cause"}
PLAY RECAP **********************************************************************************************************************************************************************************************************************************
localhost : ok=10 changed=1 unreachable=0 failed=1 skipped=79 rescued=0 ignored=0
If I start the nginx I get error about the port, so my problem now is something else (I am trying with AWX 6.0)
# systemctl start nginx
# ansible-playbook -i inventory install.yml | tail -25
TASK [check_vars : include_tasks] ***********************************************************************************************************************************************************************************************************
skipping: [localhost]
TASK [check_vars : include_tasks] ***********************************************************************************************************************************************************************************************************
included: /root/awx/installer/roles/check_vars/tasks/check_docker.yml for localhost
TASK [check_vars : postgres_data_dir should be defined] *************************************************************************************************************************************************************************************
ok: [localhost] => {
"changed": false,
"msg": "All assertions passed"
}
TASK [check_vars : host_port should be defined] *********************************************************************************************************************************************************************************************
ok: [localhost] => {
"changed": false,
"msg": "All assertions passed"
}
TASK [check_vars : host_port should be available] *******************************************************************************************************************************************************************************************
fatal: [localhost]: FAILED! => {"changed": false, "elapsed": 2, "msg": "host_port is currently in use"}
PLAY RECAP **********************************************************************************************************************************************************************************************************************************
localhost : ok=4 changed=0 unreachable=0 failed=1 skipped=1 rescued=0 ignored=0
Can someone please tell me how can I check the logs of the installation
ssaammoo
on 10 Jul 2019
anyone can help me please :-)
ssaammoo
on 15 Jul 2019
So I uninstall NGINX from the machine, rebooted it and installed available CentOS updates, I got the AWX ver 6 again and I tried to update again, I got same error:
TASK [local_docker : Start the containers] **************************************************************************************************************************************************************************************************
task path: /root/awx/installer/roles/local_docker/tasks/compose.yml:25
<localhost> ESTABLISH LOCAL CONNECTION FOR USER: root
<localhost> EXEC /bin/sh -c 'echo ~root && sleep 0'
<localhost> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /root/.ansible/tmp/ansible-tmp-1563197952.57-243929466938101 `" && echo ansible-tmp-1563197952.57-243929466938101="` echo /root/.ansible/tmp/ansible-tmp-1563197952.57-243929466938101 `" ) && sleep 0'
Using module file /usr/lib/python2.7/site-packages/ansible/modules/cloud/docker/_docker_service.py
<localhost> PUT /root/.ansible/tmp/ansible-local-33608Frek_n/tmpLy2Mno TO /root/.ansible/tmp/ansible-tmp-1563197952.57-243929466938101/AnsiballZ__docker_service.py
<localhost> EXEC /bin/sh -c 'chmod u+x /root/.ansible/tmp/ansible-tmp-1563197952.57-243929466938101/ /root/.ansible/tmp/ansible-tmp-1563197952.57-243929466938101/AnsiballZ__docker_service.py && sleep 0'
<localhost> EXEC /bin/sh -c '/usr/bin/env python /root/.ansible/tmp/ansible-tmp-1563197952.57-243929466938101/AnsiballZ__docker_service.py && sleep 0'
<localhost> EXEC /bin/sh -c 'rm -f -r /root/.ansible/tmp/ansible-tmp-1563197952.57-243929466938101/ > /dev/null 2>&1 && sleep 0'
The full traceback is:
WARNING: The below traceback may *not* be related to the actual failure.
File "/tmp/ansible_docker_service_payload_rbsc6r/__main__.py", line 764, in cmd_up
timeout=self.timeout)
File "/usr/lib/python2.7/site-packages/compose/project.py", line 559, in up
'Encountered errors while bringing up the project.'
[DEPRECATION WARNING]: The 'docker_service' module has been renamed to 'docker_compose'.. This feature will be removed in version 2.12. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.
fatal: [localhost]: FAILED! => {
"changed": false,
"errors": [],
"invocation": {
"module_args": {
"api_version": "auto",
"build": false,
"ca_cert": null,
"client_cert": null,
"client_key": null,
"debug": false,
"definition": null,
"dependencies": true,
"docker_host": "unix://var/run/docker.sock",
"files": null,
"hostname_check": false,
"nocache": false,
"project_name": null,
"project_src": "/tmp/awxcompose",
"pull": false,
"recreate": "smart",
"remove_images": null,
"remove_orphans": false,
"remove_volumes": false,
"restarted": false,
"scale": null,
"services": null,
"ssl_version": null,
"state": "present",
"stopped": false,
"timeout": 10,
"tls": false,
"tls_hostname": "localhost",
"validate_certs": false
}
},
"module_stderr": "Creating awx_web ... \r\n\r\u001b[1B",
"module_stdout": "",
"msg": "Error starting project unknown cause"
}
I disable nginx and I tried to upgrade from 3.1 to 6.1 and still getting the same error ...the logs are not guiding to certain error and doesn't seem to be port thing:
TASK [local_docker : Start the containers] **************************************************************************************************************************************************************************************************
[DEPRECATION WARNING]: The 'docker_service' module has been renamed to 'docker_compose'.. This feature will be removed in version 2.12. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.
fatal: [localhost]: FAILED! => {"changed": false, "errors": [], "module_stderr": "Creating awx_postgres ... \r\nCreating awx_rabbitmq ... \r\nCreating awx_memcached ... \r\n\r\u001b[2BCreating awx_web ... \r\n\r\u001b[1B", "module_stdout": "6.1.0: Pulling from ansible/awx_web\nDigest: sha256:51a4f11acc4b1f47a1ed233f866035246925781b997937709f64c69863f993f4\nStatus: Downloaded newer image for ansible/awx_web:6.1.0\n6.1.0: Pulling from ansible/awx_task\nDigest: sha256:3044edfcaf9924b488632750c65d41b8fb54058aee0018291b9aac0fdb7fcc67\nStatus: Downloaded newer image for ansible/awx_task:6.1.0\n", "msg": "Error starting project unknown cause"}
PLAY RECAP **********************************************************************************************************************************************************************************************************************************
localhost : ok=8 changed=1 unreachable=0 failed=1 skipped=77 rescued=0 ignored=0
# netstat -an | grep "80\|443"
tcp 0 0 10.14.15.60:51836 10.14.11.2:8080 TIME_WAIT
tcp 0 0 10.14.15.60:51842 10.14.11.2:8080 TIME_WAIT
tcp 0 0 10.14.15.60:51844 10.14.11.2:8080 TIME_WAIT
tcp 0 0 10.14.15.60:51838 10.14.11.2:8080 TIME_WAIT
tcp 0 0 10.14.15.60:51840 10.14.11.2:8080 TIME_WAIT
tcp 0 0 10.14.15.60:48098 10.14.15.194:58547 ESTABLISHED
tcp 0 0 10.14.15.60:51846 10.14.11.2:8080 TIME_WAIT
tcp 0 0 10.14.15.60:51848 10.14.11.2:8080 TIME_WAIT
tcp6 0 0 :::8052 :::* LISTEN
udp6 0 0 fe80::c485:21ff:fec:123 :::*
udp6 0 0 fe80::42:e1ff:feb2::123 :::*
udp6 0 0 fe80::706a:29ff:fe3:123 :::*
udp6 0 0 fe80::c0ae:63ff:fef:123 :::*
udp6 0 0 fe80::6c2a:80ff:feb:123 :::*
udp6 0 0 fe80::48d2:69ff:fe5:123 :::*
udp6 0 0 fe80::783b:7bff:fe7:123 :::*
udp6 0 0 fe80::60de:43ff:fe2:123 :::*
udp6 0 0 fe80::f402:40ff:fe8:123 :::*
udp6 0 0 fe80::42:55ff:fe71::123 :::*
udp6 0 0 fe80::250:56ff:fe92:123 :::*
unix 2 [ ACC ] STREAM LISTENING 34080 private/bounce
unix 2 [ ACC ] STREAM LISTENING 13307 @/containerd-shim/moby/4c527a00794bd9fc0ad498a50d458c81828082714ee4f4329e9a635caa511b99/shim.sock
unix 2 [ ACC ] STREAM LISTENING 91455 @/containerd-shim/moby/7007a14577d1ac0d20053787c08237f6d3c1dbf223f8f80c69ec53c693a7fc22/shim.sock
unix 2 [ ACC ] STREAM LISTENING 38950 @/containerd-shim/moby/1580ef93f47d99a68cc7d2abd13f302db9360a7d80b7f1436578065795d7ba30/shim.sock
unix 3 [ ] STREAM CONNECTED 13180
unix 3 [ ] STREAM CONNECTED 980 /run/systemd/journal/stdout
unix 3 [ ] STREAM CONNECTED 36010 @/containerd-shim/moby/1580ef93f47d99a68cc7d2abd13f302db9360a7d80b7f1436578065795d7ba30/shim.sock
unix 3 [ ] STREAM CONNECTED 36027 @/containerd-shim/moby/4c527a00794bd9fc0ad498a50d458c81828082714ee4f4329e9a635caa511b99/shim.sock
unix 3 [ ] STREAM CONNECTED 92938 @/containerd-shim/moby/7007a14577d1ac0d20053787c08237f6d3c1dbf223f8f80c69ec53c693a7fc22/shim.sock
unix 2 [ ] DGRAM 80797
Are there any stopped containers after the task fails whose logs you can see? If there are, doing a docker logs awx_web might have more information on what's going wrong.
File "/tmp/ansible_docker_service_payload_rbsc6r/__main__.py", line 764, in cmd_up
timeout=self.timeout)
File "/usr/lib/python2.7/site-packages/compose/project.py", line 559, in up
'Encountered errors while bringing up the project.'
This makes me think something's up with your Docker daemon. Does AWX start if you do a docker-compose up -d in the /tmp/awxcompose/ directory?
konkolorado
on 23 Jul 2019
I'm having the same issue.
When I do the docker-compose up -d I get the following message:
awx_postgres is up-to-date
awx_rabbitmq is up-to-date
awx_memcached is up-to-dateStarting 02c59e00ac79_awx_web ... error
ERROR: for 02c59e00ac79_awx_web Cannot start service web: driver failed programming >external connectivity on endpoint 02c59e00ac79_awx_web >(0944c6ee2f7faecd88f5c38804b022087830a28c6daa846ab11d0fcf502a30ae): (iptables failed: i>
iptables --wait -t filter -A DOCKER ! -i br-XX -o br-XX -p tcp -d 172.18.0.5 >--dport 8052 -j ACCEPT: iptables: No chain/target/match by that name.
(exit status 1))ERROR: for web Cannot start service web: driver failed programming external connectivity on >endpoint 02c59e00ac79_awx_web >(0944c6ee2f7faecd88f5c38804b022087830a28c6daa846ab11d0fcf502a30ae): (iptables failed: >iptables --wait -t filter -A DOCKER ! -i br-XX -o br-XX -p tcp -d 172.18.0.5 >--dport 8052 -j ACCEPT: iptables: No chain/target/match by that name.
(exit status 1))
ERROR: Encountered errors while bringing up the project.
I've added the following rule to my iptables script:
-A INPUT -s x.x.x.x/xx -p tcp --dport 8052 -j ACCEPT
But still I can't start the project.
zbenta
on 23 Aug 2019
The Strangest thing happened, I updated my OS, rebooted the machine and it started correctly.
Go figure...
zbenta
on 26 Aug 2019
Have you tried, removing the docker images and re-running the install command?. Hope you backed up the postgres/playbooks.
doves99
on 26 Aug 2019
When I try to create the docker images manually I get the following error, I've already killed any existing containers before running:
administrator@admin-awx-ubuntu:/tmp/awxcompose$ docker-compose up -d
Starting awx_memcached ... done
Starting awx_postgres ... done
Starting awx_rabbitmq ... done
Creating awx_web ... error
ERROR: for awx_web Cannot create container for service web: Conflict. The container name "/awx_web" is already in use by container "187153c1ad90f43db84544cf343c87148b94557d238637529deac95136ec9e43". You have to remove (or rename) that container to be able to reuse that name.
ERROR: for web Cannot create container for service web: Conflict. The container name "/awx_web" is already in use by container "187153c1ad90f43db84544cf343c87148b94557d238637529deac95136ec9e43". You have to remove (or rename) that container to be able to reuse that name.
ERROR: Encountered errors while bringing up the project.
administrator@admin-awx-ubuntu:/tmp/awxcompose$
tchiapuziowong
on 5 Sep 2019
I am facing the same confusing error in a totally unrelated place. This bug is caused by https://github.com/docker/compose/issues/6899 which prevents API users (like ansible module) from getting the real error that caused the failure.
Based on previous experience it will take a lot ok poking to make Docker fix this bug, but the sooner we start the faster we are likely to address it.
ssbarnea
on 18 Sep 2019
Just ran into this. I had to remove and deploy from scratch. Why this happens originally, I don't know.
- Stop all AWX containers
docker stop awx_web awx_task awx_postgres awx_memcached awx_rabbitmq - Delete all AWX containers
docker rm awx_web awx_task awx_postgres awx_memcached awx_rabbitmq - Remove the awxcompose directory
rm -rf /tmp/awxcompose - Run the installer
ansible-playbook -i inventory install.yml
konkolorado
on 18 Sep 2019
I see this error if any containers of the same still exists (ie, docker ps -a). Manually removing the containers gets around the "error".
memelet
on 29 Sep 2019
What most of you are encountering here _does_ look to me like a symptom of a bug in docker-compose itself: https://github.com/docker/compose/issues/6899
ryanpetrello
on 30 Sep 2019
Even after following steps mentioned by @konkolorado I am facing the same issue.
ssbarnea
on 3 Oct 2019
I'm getting this error as well. A temporary workaround that has been working for me is to checkout commit 44907b33dc17b528dfd1d6dfa2c990c2d31d307a before installing. any commit more recent than this one will fail with unknown error on my installation attempts.
gcgoncalves
on 14 Oct 2019
I was facing the same problem.
Started from scrath and isntalled ansible using PIP to force it to use python3
I created this tutorial.
brbrow
on 27 Oct 2019
@zbenta thanks: When I do the docker-compose up -d
helped me to determine the issue:
ERROR: for awx_web Cannot create container for service web: b'invalid volume specification:
\'/Users/vincent/Desktop/AWX/opt/my-envs:~/Desktop/AWX/opt/my-envs:rw\': invalid mount
config for type "bind": invalid mount path: \'~/Desktop/AWX/opt/my-envs\' mount path must be
absolute'
setting the custom_venv_dir="/Users/vincent/Desktop/AWX/opt/my-envs/" as an absolute path in the file install/inventory and not "'~/Desktop/AWX/opt/my-envs" made all the issues go away :)
vincentkersten
on 29 Oct 2019
change it in the inventory file
ansible/awx_tasksud
dockerhub_base=ansible
will work
sivabgl
on 5 Jun 2020
Was this page helpful?
0 / 5 - 0 ratings
Related issues
agaffney
·
3Comments
augabet
·
3Comments
FloThinksPi
·
3Comments
kakkotetsu
·
3Comments
worsco
·
3Comments
Most helpful comment
FWIW I encountered the same error when starting awx-web which wants to listen on port 80, turned out a separate install of nginx was using it (confirm via netstat -anpo | grep 80). After stopping nginx, the install playbook completed successfully.