Awx: Cannot checkout ansible galaxy roles after upgrade to 9.2.0

[ilijamt]

ISSUE TYPE

• Bug Report

SUMMARY

Cannot check out the ansible galxy roles on a project run. Running this locally works. The roles are on a private git server, and the same SSH key that is used to checkout the project is also used to download the roles

Additional information: Killing the container awx-celery will make it work only the first run only. After that the error bellow from the run log shows up.

ENVIRONMENT

• AWX version: 9.2.0
• AWX install method: k8s
• Ansible version: 2.9.3
• Operating System:
• Web Browser:

STEPS TO REPRODUCE

EXPECTED RESULTS

Being able to check out the ansible galaxy roles

ACTUAL RESULTS

Hangs indefinitely for a checkout of the ansible galaxy roles.

ADDITIONAL INFORMATION

Run log

Identity added: /tmp/awx_2550_0fu1yi5d/artifacts/2550/ssh_key_data (awx)
ansible-playbook 2.9.3
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/var/lib/awx/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3.6/site-packages/ansible
  executable location = /usr/bin/ansible-playbook
  python version = 3.6.8 (default, Nov 21 2019, 19:31:34) [GCC 8.3.1 20190507 (Red Hat 8.3.1-4)]
Using /etc/ansible/ansible.cfg as config file
host_list declined parsing /tmp/awx_2550_0fu1yi5d/inventory/hosts as it did not pass its verify_file() method
script declined parsing /tmp/awx_2550_0fu1yi5d/inventory/hosts as it did not pass its verify_file() method
auto declined parsing /tmp/awx_2550_0fu1yi5d/inventory/hosts as it did not pass its verify_file() method
Parsed /tmp/awx_2550_0fu1yi5d/inventory/hosts inventory source with ini plugin
PLAYBOOK: project_update.yml ***************************************************
2 plays in project_update.yml
PLAY [Update source tree if necessary] *****************************************
META: ran handlers
META: ran handlers
META: ran handlers
PLAY [Install content with ansible-galaxy command if necessary] ****************
META: ran handlers
TASK [detect requirements.yml] *************************************************
task path: /var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/playbooks/project_update.yml:133
<127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: awx
<127.0.0.1> EXEC /bin/sh -c 'echo ~awx && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /var/lib/awx/.ansible/tmp/ansible-tmp-1581719618.0153997-274054373534382 `" && echo ansible-tmp-1581719618.0153997-274054373534382="` echo /var/lib/awx/.ansible/tmp/ansible-tmp-1581719618.0153997-274054373534382 `" ) && sleep 0'
Using module file /usr/lib/python3.6/site-packages/ansible/modules/files/stat.py
<127.0.0.1> PUT /var/lib/awx/.ansible/tmp/ansible-local-3zvuarjl6/tmp08hc4g9h TO /var/lib/awx/.ansible/tmp/ansible-tmp-1581719618.0153997-274054373534382/AnsiballZ_stat.py
<127.0.0.1> EXEC /bin/sh -c 'chmod u+x /var/lib/awx/.ansible/tmp/ansible-tmp-1581719618.0153997-274054373534382/ /var/lib/awx/.ansible/tmp/ansible-tmp-1581719618.0153997-274054373534382/AnsiballZ_stat.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '/usr/bin/python3.6 /var/lib/awx/.ansible/tmp/ansible-tmp-1581719618.0153997-274054373534382/AnsiballZ_stat.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c 'rm -f -r /var/lib/awx/.ansible/tmp/ansible-tmp-1581719618.0153997-274054373534382/ > /dev/null 2>&1 && sleep 0'
ok: [localhost] => {
    "changed": false,
    "invocation": {
        "module_args": {
            "checksum_algorithm": "sha1",
            "follow": false,
            "get_attributes": true,
            "get_checksum": true,
            "get_md5": false,
            "get_mime": true,
            "path": "/var/lib/awx/projects/_8__baseline/roles/requirements.yml"
        }
    },
    "stat": {
        "atime": 1581719306.717563,
        "attr_flags": "",
        "attributes": [],
        "block_size": 4096,
        "blocks": 8,
        "charset": "unknown",
        "checksum": "873ac04b1592376ef928d565cc2ce5155883d558",
        "ctime": 1581719306.2855692,
        "dev": 228,
        "device_type": 0,
        "execu…
TASK [fetch galaxy roles from requirements.yml] ********************************
task path: /var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/playbooks/project_update.yml:138
<127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: awx
<127.0.0.1> EXEC /bin/sh -c 'echo ~awx && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /var/lib/awx/.ansible/tmp/ansible-tmp-1581719618.5172336-42218225795671 `" && echo ansible-tmp-1581719618.5172336-42218225795671="` echo /var/lib/awx/.ansible/tmp/ansible-tmp-1581719618.5172336-42218225795671 `" ) && sleep 0'
Using module file /usr/lib/python3.6/site-packages/ansible/modules/commands/command.py
<127.0.0.1> PUT /var/lib/awx/.ansible/tmp/ansible-local-3zvuarjl6/tmpjax4m5ei TO /var/lib/awx/.ansible/tmp/ansible-tmp-1581719618.5172336-42218225795671/AnsiballZ_command.py
<127.0.0.1> EXEC /bin/sh -c 'chmod u+x /var/lib/awx/.ansible/tmp/ansible-tmp-1581719618.5172336-42218225795671/ /var/lib/awx/.ansible/tmp/ansible-tmp-1581719618.5172336-42218225795671/AnsiballZ_command.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c 'ANSIBLE_FORCE_COLOR=False /usr/bin/python3.6 /var/lib/awx/.ansible/tmp/ansible-tmp-1581719618.5172336-42218225795671/AnsiballZ_command.py && sleep 0'
The authenticity of host 'git.<redacted>.com (176.9.x.x)' can't be established.
ECDSA key fingerprint is SHA256:qPurwghZ7CHDtzfDJ2JfNVfj8LEAnHpprxDy+3P23JY.

ps -faux

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
awx       2645  0.0  0.0  14948  3604 pts/1    Ss   22:40   0:00 /bin/bash
awx       2656  0.0  0.0  49276  3660 pts/1    R+   22:43   0:00  \_ ps -faux
awx       2657  0.0  0.0   9672  2116 pts/1    S+   22:43   0:00  \_ less
awx          1  0.0  0.0  37344  4160 ?        Ss   Feb13   0:00 bash /usr/bin/launch_awx_task.sh
awx        132  0.0  0.2 105860 24128 ?        S    Feb13   0:20 /usr/bin/python3.6 /usr/local/bin/supervisord -c /supervisor_task.conf
awx        135  0.0  0.2  83264 17876 ?        S    Feb13   0:00  \_ python3 /usr/bin/config-watcher
awx        136  0.1  1.8 770312 152220 ?       Sl   Feb13   2:22  \_ /var/lib/awx/venv/awx/bin/python3 /usr/bin/awx-manage run_dispatcher
awx        160  0.0  0.0      0     0 ?        Z    Feb13   0:35  |   \_ [awx-manage] <defunct>
awx        161  0.0  1.6 473996 136804 ?       Sl   Feb13   0:41  |   \_ /var/lib/awx/venv/awx/bin/python3 /usr/bin/awx-manage run_dispatcher
awx        162  0.0  1.6 474252 137220 ?       Sl   Feb13   0:42  |   \_ /var/lib/awx/venv/awx/bin/python3 /usr/bin/awx-manage run_dispatcher
awx       2578  0.0  0.0  23060  4504 ?        S    22:33   0:00  |   |   \_ git cat-file --batch-check
root      2581  0.0  0.0  17652  1264 pts/0    Ss+  22:33   0:00  |   |   \_ /usr/bin/bwrap --die-with-parent --unshare-pid --dev-bind / / --proc /proc --bind /tmp/ansible_runner_pi_445pegtp/ansible_runner_pi_rv6c_wj8/tmp_twvbhc6 /etc/ssh --bind /tmp/ansible_runner_pi_445pegtp/ansible_runner_pi_rv6c_wj8/tmpr_on7l8s /etc/tower --bind /tmp/ansible_runner_pi_445pegtp/ansible_runner_pi_rv6c_wj8/tmpgbjqe0a0 /tmp --bind /tmp/ansible_runner_pi_445pegtp/ansible_runner_pi_rv6c_wj8/tmpf6a20bcu /var/lib/awx --bind /tmp/ansible_runner_pi_445pegtp/ansible_runner_pi_rv6c_wj8/tmp369ttbmc /var/lib/awx/projects --bind /tmp/ansible_runner_pi_445pegtp/ansible_runner_pi_rv6c_wj8/tmpwl9a_sbj /var/log --ro-bind /var/lib/awx/venv/ansible /var/lib/awx/venv/ansible --ro-bind /var/lib/awx/venv/awx /var/lib/awx/venv/awx --bind /tmp/awx_2547_imikqrpl /tmp/awx_2547_imikqrpl --bind /tmp/awx_2550_0fu1yi5d /tmp/awx_2550_0fu1yi5d --bind /var/lib/awx/projects /var/lib/awx/projects --bind /var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/playbooks /var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/playbooks --chdir /var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/playbooks ssh-agent sh -c ssh-add /tmp/awx_2550_0fu1yi5d/artifacts/2550/ssh_key_data && rm -f /tmp/awx_2550_0fu1yi5d/artifacts/2550/ssh_key_data && ansible-playbook -vvv -t install_roles -i /tmp/awx_2550_0fu1yi5d/inventory/hosts -e @/tmp/awx_2550_0fu1yi5d/env/extravars project_update.yml
awx       2582  0.0  0.0  17748  1316 pts/0    S+   22:33   0:00  |   |       \_ /usr/bin/bwrap --die-with-parent --unshare-pid --dev-bind / / --proc /proc --bind /tmp/ansible_runner_pi_445pegtp/ansible_runner_pi_rv6c_wj8/tmp_twvbhc6 /etc/ssh --bind /tmp/ansible_runner_pi_445pegtp/ansible_runner_pi_rv6c_wj8/tmpr_on7l8s /etc/tower --bind /tmp/ansible_runner_pi_445pegtp/ansible_runner_pi_rv6c_wj8/tmpgbjqe0a0 /tmp --bind /tmp/ansible_runner_pi_445pegtp/ansible_runner_pi_rv6c_wj8/tmpf6a20bcu /var/lib/awx --bind /tmp/ansible_runner_pi_445pegtp/ansible_runner_pi_rv6c_wj8/tmp369ttbmc /var/lib/awx/projects --bind /tmp/ansible_runner_pi_445pegtp/ansible_runner_pi_rv6c_wj8/tmpwl9a_sbj /var/log --ro-bind /var/lib/awx/venv/ansible /var/lib/awx/venv/ansible --ro-bind /var/lib/awx/venv/awx /var/lib/awx/venv/awx --bind /tmp/awx_2547_imikqrpl /tmp/awx_2547_imikqrpl --bind /tmp/awx_2550_0fu1yi5d /tmp/awx_2550_0fu1yi5d --bind /var/lib/awx/projects /var/lib/awx/projects --bind /var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/playbooks /var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/playbooks --chdir /var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/playbooks ssh-agent sh -c ssh-add /tmp/awx_2550_0fu1yi5d/artifacts/2550/ssh_key_data && rm -f /tmp/awx_2550_0fu1yi5d/artifacts/2550/ssh_key_data && ansible-playbook -vvv -t install_roles -i /tmp/awx_2550_0fu1yi5d/inventory/hosts -e @/tmp/awx_2550_0fu1yi5d/env/extravars project_update.yml
awx       2584  6.5  0.8 388624 70520 pts/0    Sl+  22:33   0:39  |   |           \_ /usr/bin/python3.6 -s /usr/bin/ansible-playbook -vvv -t install_roles -i /tmp/awx_2550_0fu1yi5d/inventory/hosts -e @/tmp/awx_2550_0fu1yi5d/env/extravars project_update.yml
awx       2585  0.0  0.0  29440   548 ?        Ss   22:33   0:00  |   |               \_ ssh-agent sh -c ssh-add /tmp/awx_2550_0fu1yi5d/artifacts/2550/ssh_key_data && rm -f /tmp/awx_2550_0fu1yi5d/artifacts/2550/ssh_key_data && ansible-playbook -vvv -t install_roles -i /tmp/awx_2550_0fu1yi5d/inventory/hosts -e @/tmp/awx_2550_0fu1yi5d/env/extravars project_update.yml
awx       2609  0.0  0.7 396804 64140 pts/0    S+   22:33   0:00  |   |               \_ /usr/bin/python3.6 -s /usr/bin/ansible-playbook -vvv -t install_roles -i /tmp/awx_2550_0fu1yi5d/inventory/hosts -e @/tmp/awx_2550_0fu1yi5d/env/extravars project_update.yml
awx       2618  0.0  0.0  14816  3216 pts/0    S+   22:33   0:00  |   |                   \_ /bin/sh -c ANSIBLE_FORCE_COLOR=False /usr/bin/python3.6 /var/lib/awx/.ansible/tmp/ansible-tmp-1581719618.5172336-42218225795671/AnsiballZ_command.py && sleep 0
awx       2619  0.0  0.2  92212 18136 pts/0    S+   22:33   0:00  |   |                       \_ /usr/bin/python3.6 /var/lib/awx/.ansible/tmp/ansible-tmp-1581719618.5172336-42218225795671/AnsiballZ_command.py
awx       2620  0.0  0.5 166468 42588 pts/0    S+   22:33   0:00  |   |                           \_ /usr/bin/python3.6 -s /usr/bin/ansible-galaxy install -r requirements.yml -p /tmp/awx_2547_imikqrpl/requirements_roles -vvv
awx       2624  0.0  0.0  23420  4668 pts/0    S+   22:33   0:00  |   |                               \_ /usr/bin/git clone git@git.<redacted>.com:msp/ansible/roles/msp.sshd.git msp.sshd
awx       2625  0.0  0.0  48216  6296 pts/0    S+   22:33   0:00  |   |                                   \_ /usr/bin/ssh git@git.<redacted>.com git-upload-pack 'msp/ansible/roles/msp.sshd.git'
awx        930  0.0  1.7 694040 145176 ?       Sl   21:11   0:04  |   \_ /var/lib/awx/venv/awx/bin/python3 /usr/bin/awx-manage run_dispatcher
awx       1008  0.0  1.8 762644 149716 ?       Sl   21:12   0:05  |   \_ /var/lib/awx/venv/awx/bin/python3 /usr/bin/awx-manage run_dispatcher
awx        137  0.0  1.5 605312 125328 ?       Sl   Feb13   0:14  \_ /var/lib/awx/venv/awx/bin/python3 /usr/bin/awx-manage run_callback_receiver
awx        154  0.1  1.5 389792 125400 ?       Sl   Feb13   2:30  |   \_ /var/lib/awx/venv/awx/bin/python3 /usr/bin/awx-manage run_callback_receiver
awx        155  0.1  1.5 389512 125264 ?       Sl   Feb13   2:29  |   \_ /var/lib/awx/venv/awx/bin/python3 /usr/bin/awx-manage run_callback_receiver
awx        156  0.1  1.5 390112 125148 ?       Sl   Feb13   2:29  |   \_ /var/lib/awx/venv/awx/bin/python3 /usr/bin/awx-manage run_callback_receiver
awx        157  0.1  1.5 389592 125316 ?       Sl   Feb13   2:30  |   \_ /var/lib/awx/venv/awx/bin/python3 /usr/bin/awx-manage run_callback_receiver
awx        138  0.0  1.6 320288 132104 ?       S    Feb13   0:23  \_ /var/lib/awx/venv/awx/bin/python3 /usr/bin/awx-manage runworker --only-channels websocket.*

ansible.cfg

[defaults]
host_key_checking = False
timeout = 30
remote_tmp = ~/.ansible/tmp
remote_user = root
interpreter_python = auto_silent
roles_path = ./roles
strategy = linear

[ssh_connection]
pipelining = yes

[ilijamt]

This is ran from inside the awx-celery pod

By finding the ssh-agent socket and running

$ SSH_AUTH_SOCK=/tmp/ansible_runner_pi_445pegtp/ansible_runner_pi_rv6c_wj8/tmpgbjqe0a0/ssh-LK1nNJS7K20T/agent.3; export SSH_AUTH_SOCK
$ SSH_AGENT_PID=2585; export SSH_AGENT_PID
$ ssh-add -l
256 SHA256:XQNhU8QRDSgwFKaodGvmHUlLcaJfa11H7H6KVKzYDSw awx (ED25519)
$ /usr/bin/git clone git@git.<redacted>.com:msp/ansible/roles/msp.sshd.git msp.sshd
Cloning into 'msp.sshd'...
remote: Enumerating objects: 41, done.
remote: Counting objects: 100% (41/41), done.
remote: Compressing objects: 100% (26/26), done.
Receiving objects: 100% (41/41), 5.15 KiB | 5.15 MiB/s, done.
remote: Total 41 (delta 10), reused 0 (delta 0), pack-reused 0
Resolving deltas: 100% (10/10), done.
$ ls -lah msp.sshd/
total 4.0K
drwxr-xr-x 10 awx  root 134 Feb 14 22:51 .
drwxrwxrwt  1 root root 153 Feb 14 22:51 ..
drwxr-xr-x  2 awx  root  22 Feb 14 22:51 defaults
drwxr-xr-x  8 awx  root 163 Feb 14 22:51 .git
drwxr-xr-x  2 awx  root  22 Feb 14 22:51 handlers
drwxr-xr-x  2 awx  root  22 Feb 14 22:51 meta
-rw-r--r--  1 awx  root 246 Feb 14 22:51 README.md
drwxr-xr-x  2 awx  root  22 Feb 14 22:51 tasks
drwxr-xr-x  2 awx  root  28 Feb 14 22:51 templates
drwxr-xr-x  2 awx  root  39 Feb 14 22:51 tests
drwxr-xr-x  2 awx  root  22 Feb 14 22:51 vars

Based on this the exported ssh key is correct.

[AlanCoding]

If you still have access to that awx-celery pod, can you help further triage the issue by determining if the ansible-galaxy command succeeds or fails manually? You could run it like:

mkdir /tmp/roles_test
ansible-galaxy install git+https://github.com/geerlingguy/ansible-role-apache.git -p /tmp/roles_test -vvv

where you're using the same thing in your requirements.yml file

based on

https://github.com/ansible/awx/blob/e131e8c15127d7f994b8a25814a721308c74bccf/awx/playbooks/project_update.yml#L139

thanks

[ilijamt]

I do have access to the awx-celery pod, so this is my test that I ran

Found the ssh-agent socket and added it

$ export SSH_AUTH_SOCK=/tmp/ansible_runner_pi_qj6tp_vm/ansible_runner_pi_khvpnk39/tmpp_hyuzri/ssh-jniaNuG1U0Kc/agent.3
$ export SSH_AGENT_PID=1282
$ ssh-agent -l
256 SHA256:XQNhU8QRDSgwFKaodGvmHUlLcaJfa11H7H6KVKzYDSw awx (ED25519)

From public repo:

$ mkdir /tmp/roles_test_public_repo
$ ansible-galaxy install git+https://github.com/geerlingguy/ansible-role-apache.git -p /tmp/roles_test_public_repo -vvv
ansible-galaxy 2.9.3
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/home/awx/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3.6/site-packages/ansible
  executable location = /usr/bin/ansible-galaxy
  python version = 3.6.8 (default, Nov 21 2019, 19:31:34) [GCC 8.3.1 20190507 (Red Hat 8.3.1-4)]
Using /etc/ansible/ansible.cfg as config file
Processing role ansible-role-apache 
archiving ['/usr/bin/git', 'archive', '--prefix=ansible-role-apache/', '--output=/home/awx/.ansible/tmp/ansible-local-8746fsbouycu/tmpd889o6n8.tar', 'HEAD']
- extracting ansible-role-apache to /tmp/roles_test_public_repo/ansible-role-apache
- ansible-role-apache was installed successfully

From private gitlab instance

$ mkdir -p /tmp/roles_test_private_repo
$ /usr/bin/ansible-galaxy install -r ./var/lib/awx/projects/_8__baseline/roles/requirements.yml -p /tmp/roles_test_private_repo/ -vvv
ansible-galaxy 2.9.3
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/home/awx/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3.6/site-packages/ansible
  executable location = /usr/bin/ansible-galaxy
  python version = 3.6.8 (default, Nov 21 2019, 19:31:34) [GCC 8.3.1 20190507 (Red Hat 8.3.1-4)]
Using /etc/ansible/ansible.cfg as config file
Reading requirement file at './var/lib/awx/projects/_8__baseline/roles/requirements.yml'
found role {'src': 'jnv.debian-backports', 'name': 'jnv.debian-backports', 'version': '', 'scm': None} in yaml file
found role {'src': 'elastic.elasticsearch', 'version': '7.4.1', 'name': 'elastic.elasticsearch', 'scm': None} in yaml file
found role {'src': 'git@git.<redacted>.com:msp/ansible/roles/msp.sshd.git', 'scm': 'git', 'name': 'msp.sshd', 'version': ''} in yaml file
found role {'src': 'git@git.<redacted>.com:msp/ansible/roles/msp.timezone.git', 'scm': 'git', 'name': 'msp.timezone', 'version': ''} in yaml file
found role {'src': 'git@git.<redacted>.com:msp/ansible/roles/msp.user-password.git', 'scm': 'git', 'name': 'msp.user-password', 'version': ''} in yaml file
found role {'src': 'git@git.<redacted>.com:msp/ansible/roles/msp.user-authorized-key.git', 'scm': 'git', 'name': 'msp.user-authorized-key', 'version': ''} in yaml file
found role {'src': 'git@git.<redacted>.com:msp/ansible/roles/msp.haveged.git', 'scm': 'git', 'name': 'msp.haveged', 'version': ''} in yaml file
found role {'src': 'git@git.<redacted>.com:msp/ansible/roles/msp.locales.git', 'scm': 'git', 'name': 'msp.locales', 'version': ''} in yaml file
found role {'src': 'git@git.<redacted>.com:msp/ansible/roles/msp.packages.git', 'scm': 'git', 'name': 'msp.packages', 'version': ''} in yaml file
found role {'src': 'git@git.<redacted>.com:msp/ansible/roles/msp.git-msp-deploy-key.git', 'scm': 'git', 'name': 'msp.git-msp-deploy-key', 'version': ''} in yaml file
found role {'src': 'git@git.<redacted>.com:msp/ansible/roles/msp.golang.git', 'scm': 'git', 'name': 'msp.golang', 'version': ''} in yaml file
found role {'src': 'git@git.<redacted>.com:msp/ansible/roles/msp.hostname.git', 'scm': 'git', 'name': 'msp.hostname', 'version': ''} in yaml file
found role {'src': 'git@git.<redacted>.com:msp/ansible/roles/msp.hostfile.git', 'scm': 'git', 'name': 'msp.hostfile', 'version': ''} in yaml file
found role {'src': 'git@git.<redacted>.com:msp/ansible/roles/msp.common.git', 'scm': 'git', 'name': 'msp.common', 'version': ''} in yaml file
found role {'src': 'git@git.<redacted>.com:msp/ansible/roles/msp.network.git', 'scm': 'git', 'name': 'msp.network', 'version': ''} in yaml file
found role {'src': 'git@git.<redacted>.com:msp/ansible/roles/msp.resolv.git', 'scm': 'git', 'name': 'msp.resolv', 'version': ''} in yaml file
found role {'src': 'git@git.<redacted>.com:msp/ansible/roles/msp.firewalld.git', 'scm': 'git', 'name': 'msp.firewalld', 'version': ''} in yaml file
found role {'src': 'git@git.<redacted>.com:msp/ansible/roles/msp.haproxy-lb4.git', 'scm': 'git', 'name': 'msp.haproxy-lb4', 'version': ''} in yaml file
found role {'src': 'git@git.<redacted>.com:msp/ansible/roles/msp.redis.git', 'scm': 'git', 'name': 'msp.redis', 'version': ''} in yaml file
found role {'src': 'git@git.<redacted>.com:msp/ansible/roles/msp.php.git', 'scm': 'git', 'name': 'msp.php', 'version': ''} in yaml file
found role {'src': 'git@git.<redacted>.com:msp/ansible/roles/msp.apache2.git', 'scm': 'git', 'name': 'msp.apache2', 'version': ''} in yaml file
found role {'src': 'git@git.<redacted>.com:msp/ansible/roles/msp.nginx.git', 'scm': 'git', 'name': 'msp.nginx', 'version': ''} in yaml file
found role {'src': 'git@git.<redacted>.com:msp/ansible/roles/msp.proftpd.git', 'scm': 'git', 'name': 'msp.proftpd', 'version': ''} in yaml file
found role {'src': 'git@git.<redacted>.com:msp/ansible/roles/msp.webmin.git', 'scm': 'git', 'name': 'msp.webmin', 'version': ''} in yaml file
found role {'src': 'git@git.<redacted>.com:msp/ansible/roles/msp.virtualmin.git', 'scm': 'git', 'name': 'msp.virtualmin', 'version': ''} in yaml file
found role {'src': 'git@git.<redacted>.com:msp/ansible/roles/msp.mysql.git', 'scm': 'git', 'name': 'msp.mysql', 'version': ''} in yaml file
found role {'src': 'git@git.<redacted>.com:msp/ansible/roles/msp.mysql-percona.git', 'scm': 'git', 'name': 'msp.mysql-percona', 'version': ''} in yaml file
found role {'src': 'git@git.<redacted>.com:msp/ansible/roles/msp.monit.git', 'scm': 'git', 'name': 'msp.monit', 'version': ''} in yaml file
Processing role jnv.debian-backports 
Opened /home/awx/.ansible/galaxy_token
- downloading role 'debian-backports', owned by jnv
Opened /home/awx/.ansible/galaxy_token
Opened /home/awx/.ansible/galaxy_token
- downloading role from https://github.com/jnv/ansible-role-debian-backports/archive/v0.4.1.tar.gz
- extracting jnv.debian-backports to /tmp/roles_test_private_repo/jnv.debian-backports
- jnv.debian-backports (v0.4.1) was installed successfully
Processing role elastic.elasticsearch 
- downloading role 'elasticsearch', owned by elastic
Opened /home/awx/.ansible/galaxy_token
Opened /home/awx/.ansible/galaxy_token
- downloading role from https://github.com/elastic/ansible-elasticsearch/archive/7.4.1.tar.gz
- extracting elastic.elasticsearch to /tmp/roles_test_private_repo/elastic.elasticsearch
- elastic.elasticsearch (7.4.1) was installed successfully
Processing role msp.sshd 
archiving ['/usr/bin/git', 'archive', '--prefix=msp.sshd/', '--output=/home/awx/.ansible/tmp/ansible-local-8774svrzva4z/tmp_soxq3n_.tar', 'HEAD']
- extracting msp.sshd to /tmp/roles_test_private_repo/msp.sshd
- msp.sshd was installed successfully
Processing role msp.timezone 
archiving ['/usr/bin/git', 'archive', '--prefix=msp.timezone/', '--output=/home/awx/.ansible/tmp/ansible-local-8774svrzva4z/tmpllds8lyj.tar', 'HEAD']
- extracting msp.timezone to /tmp/roles_test_private_repo/msp.timezone
- msp.timezone was installed successfully
Processing role msp.user-password 
archiving ['/usr/bin/git', 'archive', '--prefix=msp.user-password/', '--output=/home/awx/.ansible/tmp/ansible-local-8774svrzva4z/tmpgqr_gner.tar', 'HEAD']
- extracting msp.user-password to /tmp/roles_test_private_repo/msp.user-password
- msp.user-password was installed successfully
Processing role msp.user-authorized-key 
archiving ['/usr/bin/git', 'archive', '--prefix=msp.user-authorized-key/', '--output=/home/awx/.ansible/tmp/ansible-local-8774svrzva4z/tmpea3kvl9j.tar', 'HEAD']
- extracting msp.user-authorized-key to /tmp/roles_test_private_repo/msp.user-authorized-key
- msp.user-authorized-key was installed successfully
Processing role msp.haveged 
archiving ['/usr/bin/git', 'archive', '--prefix=msp.haveged/', '--output=/home/awx/.ansible/tmp/ansible-local-8774svrzva4z/tmpr_t0s4gf.tar', 'HEAD']
- extracting msp.haveged to /tmp/roles_test_private_repo/msp.haveged
- msp.haveged was installed successfully
Processing role msp.locales 
archiving ['/usr/bin/git', 'archive', '--prefix=msp.locales/', '--output=/home/awx/.ansible/tmp/ansible-local-8774svrzva4z/tmpyn812v40.tar', 'HEAD']
- extracting msp.locales to /tmp/roles_test_private_repo/msp.locales
- msp.locales was installed successfully
Processing role msp.packages 
archiving ['/usr/bin/git', 'archive', '--prefix=msp.packages/', '--output=/home/awx/.ansible/tmp/ansible-local-8774svrzva4z/tmprgimcnqn.tar', 'HEAD']
- extracting msp.packages to /tmp/roles_test_private_repo/msp.packages
- msp.packages was installed successfully
Processing role msp.git-msp-deploy-key 
archiving ['/usr/bin/git', 'archive', '--prefix=msp.git-msp-deploy-key/', '--output=/home/awx/.ansible/tmp/ansible-local-8774svrzva4z/tmpe7o4vlg3.tar', 'HEAD']
- extracting msp.git-msp-deploy-key to /tmp/roles_test_private_repo/msp.git-msp-deploy-key
- msp.git-msp-deploy-key was installed successfully
Processing role msp.golang 
archiving ['/usr/bin/git', 'archive', '--prefix=msp.golang/', '--output=/home/awx/.ansible/tmp/ansible-local-8774svrzva4z/tmpq8hcm01u.tar', 'HEAD']
- extracting msp.golang to /tmp/roles_test_private_repo/msp.golang
- msp.golang was installed successfully
Processing role msp.hostname 
archiving ['/usr/bin/git', 'archive', '--prefix=msp.hostname/', '--output=/home/awx/.ansible/tmp/ansible-local-8774svrzva4z/tmp156meqoe.tar', 'HEAD']
- extracting msp.hostname to /tmp/roles_test_private_repo/msp.hostname
- msp.hostname was installed successfully
Processing role msp.hostfile 
archiving ['/usr/bin/git', 'archive', '--prefix=msp.hostfile/', '--output=/home/awx/.ansible/tmp/ansible-local-8774svrzva4z/tmp74ihi6c6.tar', 'HEAD']
- extracting msp.hostfile to /tmp/roles_test_private_repo/msp.hostfile
- msp.hostfile was installed successfully
Processing role msp.common 
archiving ['/usr/bin/git', 'archive', '--prefix=msp.common/', '--output=/home/awx/.ansible/tmp/ansible-local-8774svrzva4z/tmplwwmyz14.tar', 'HEAD']
- extracting msp.common to /tmp/roles_test_private_repo/msp.common
- msp.common was installed successfully
Processing role msp.network 
archiving ['/usr/bin/git', 'archive', '--prefix=msp.network/', '--output=/home/awx/.ansible/tmp/ansible-local-8774svrzva4z/tmpltq5y3fv.tar', 'HEAD']
- extracting msp.network to /tmp/roles_test_private_repo/msp.network
- msp.network was installed successfully
Processing role msp.resolv 
archiving ['/usr/bin/git', 'archive', '--prefix=msp.resolv/', '--output=/home/awx/.ansible/tmp/ansible-local-8774svrzva4z/tmp4xudur16.tar', 'HEAD']
- extracting msp.resolv to /tmp/roles_test_private_repo/msp.resolv
- msp.resolv was installed successfully
Processing role msp.firewalld 
archiving ['/usr/bin/git', 'archive', '--prefix=msp.firewalld/', '--output=/home/awx/.ansible/tmp/ansible-local-8774svrzva4z/tmpyk__jzfs.tar', 'HEAD']
- extracting msp.firewalld to /tmp/roles_test_private_repo/msp.firewalld
- msp.firewalld was installed successfully
Processing role msp.haproxy-lb4 
archiving ['/usr/bin/git', 'archive', '--prefix=msp.haproxy-lb4/', '--output=/home/awx/.ansible/tmp/ansible-local-8774svrzva4z/tmp4p3kgna3.tar', 'HEAD']
- extracting msp.haproxy-lb4 to /tmp/roles_test_private_repo/msp.haproxy-lb4
- msp.haproxy-lb4 was installed successfully
Processing role msp.redis 
archiving ['/usr/bin/git', 'archive', '--prefix=msp.redis/', '--output=/home/awx/.ansible/tmp/ansible-local-8774svrzva4z/tmp5zrdnpc0.tar', 'HEAD']
- extracting msp.redis to /tmp/roles_test_private_repo/msp.redis
- msp.redis was installed successfully
Processing role msp.php 
archiving ['/usr/bin/git', 'archive', '--prefix=msp.php/', '--output=/home/awx/.ansible/tmp/ansible-local-8774svrzva4z/tmp6wkgzjv_.tar', 'HEAD']
- extracting msp.php to /tmp/roles_test_private_repo/msp.php
- msp.php was installed successfully
Processing role msp.apache2 
archiving ['/usr/bin/git', 'archive', '--prefix=msp.apache2/', '--output=/home/awx/.ansible/tmp/ansible-local-8774svrzva4z/tmp32ebawbk.tar', 'HEAD']
- extracting msp.apache2 to /tmp/roles_test_private_repo/msp.apache2
- msp.apache2 was installed successfully
Processing role msp.nginx 
archiving ['/usr/bin/git', 'archive', '--prefix=msp.nginx/', '--output=/home/awx/.ansible/tmp/ansible-local-8774svrzva4z/tmpz1zao3x6.tar', 'HEAD']
- extracting msp.nginx to /tmp/roles_test_private_repo/msp.nginx
- msp.nginx was installed successfully
Processing role msp.proftpd 
archiving ['/usr/bin/git', 'archive', '--prefix=msp.proftpd/', '--output=/home/awx/.ansible/tmp/ansible-local-8774svrzva4z/tmp6tnctlny.tar', 'HEAD']
- extracting msp.proftpd to /tmp/roles_test_private_repo/msp.proftpd
- msp.proftpd was installed successfully
Processing role msp.webmin 
archiving ['/usr/bin/git', 'archive', '--prefix=msp.webmin/', '--output=/home/awx/.ansible/tmp/ansible-local-8774svrzva4z/tmpvlg5e15e.tar', 'HEAD']
- extracting msp.webmin to /tmp/roles_test_private_repo/msp.webmin
- msp.webmin was installed successfully
Processing role msp.virtualmin 
archiving ['/usr/bin/git', 'archive', '--prefix=msp.virtualmin/', '--output=/home/awx/.ansible/tmp/ansible-local-8774svrzva4z/tmp_d1fvt36.tar', 'HEAD']
- extracting msp.virtualmin to /tmp/roles_test_private_repo/msp.virtualmin
- msp.virtualmin was installed successfully
Processing role msp.mysql 
archiving ['/usr/bin/git', 'archive', '--prefix=msp.mysql/', '--output=/home/awx/.ansible/tmp/ansible-local-8774svrzva4z/tmp4e1_xiib.tar', 'HEAD']
- extracting msp.mysql to /tmp/roles_test_private_repo/msp.mysql
- msp.mysql was installed successfully
Processing role msp.mysql-percona 
archiving ['/usr/bin/git', 'archive', '--prefix=msp.mysql-percona/', '--output=/home/awx/.ansible/tmp/ansible-local-8774svrzva4z/tmphwab_tpm.tar', 'HEAD']
- extracting msp.mysql-percona to /tmp/roles_test_private_repo/msp.mysql-percona
- msp.mysql-percona was installed successfully
Processing role msp.monit 
archiving ['/usr/bin/git', 'archive', '--prefix=msp.monit/', '--output=/home/awx/.ansible/tmp/ansible-local-8774svrzva4z/tmp7pwtp_ur.tar', 'HEAD']
- extracting msp.monit to /tmp/roles_test_private_repo/msp.monit
- msp.monit was installed successfully

[ilijamt]

@AlanCoding

1. Freshly stared container runs successfully the first time.
2. Next time it just hangs with
   The authenticity of host 'git..com (176.9.x.x)' can't be established.
   ECDSA key fingerprint is SHA256:qPurwghZ7CHDtzfDJ2JfNVfj8LEAnHpprxDy+3P23JY.

Modifying the awx/awx/playbooks/project_update.yml, by just adding GIT_SSH_COMMAND: "ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no" now it works every time.

File in the container: /var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/playbooks/project_update.yml

    - block:
        - name: detect requirements.yml
          stat:
            path: '{{project_path|quote}}/roles/requirements.yml'
          register: doesRequirementsExist

        - name: fetch galaxy roles from requirements.yml
          command: ansible-galaxy install -r requirements.yml -p {{roles_destination|quote}}{{ ' -' + 'v' * ansible_verbosity if ansible_verbosity else '' }}
          args:
            chdir: "{{project_path|quote}}/roles"
          register: galaxy_result
          when: doesRequirementsExist.stat.exists
          changed_when: "'was installed successfully' in galaxy_result.stdout"
          environment:
            ANSIBLE_FORCE_COLOR: false
            GIT_SSH_COMMAND: "ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no"

[clammers]

Ran into same issue and mapped known_hosts file to docker container to fix issue with Host Key Checking

[Grekkor]

I ran into the same issue. This ought to be fixed right away.

[ilijamt]

Same issue affects AWX 9.3.0

[ilijamt]

After adding to Settings > Jobs > Extra Environment Variables

{
 "HOME": "/var/lib/awx",
 "ANSIBLE_HOST_KEY_CHECKING": "false",
 "GIT_SSH_COMMAND": "ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no"
}

This works without any modifications to the code.

[chrismeyersfsu]

Unable to recreate.

Steps to recreate

• Added account-wide github ssh key
• Add private github repo ansible_private_project
• Add private github role ansible_private_role
• Put roles/requirements.yml file in ansible_private_project that references private git repo ansible_private_role
• Call ansible_private_role from main.yml to verify
• Setup Credential --> Source Control in AWX w/ github cred
• Setup Project to point to ansible_private_project
• Create Job Template to run main.yml
• Run Job Template

The above job runs successfully.
Tried the above on both AWX docker deploy 9.3.0 and 9.2.0

[ryanpetrello]

@chrismeyersfsu what if you:

1. Set up a Project that is public (requires no SSH key auth to clone).
2. Give that project a roles/requirements.yml that references a _private_ git repo (that requires SSH auth).
3. Don't specify a Source Control credential on the project

[ryanpetrello]

I'm able to get this to hang on devel

bash-4.4$ cat /var/lib/awx/projects/_6__demo_project/roles/requirements.yml
- src: [email protected]:ryanpetrello/private-repo.git
  scm: git

...but this makes sense, because I've specified an SSH clone in my requirements.txt (and by default, ansible-galaxy install seems to be performing host key checking).

[ryanpetrello]

Okay, I'm only able to reproduce this in the ansible-2.10.0.dev0 of ansible-galaxy. If I downgrade to ansible==2.9.3, the host key checking warning goes away.

[ryanpetrello]

It seems to me like the issue here is that ansible-galaxy doesn't actually respect ANSIBLE_HOST_KEY_CHECKING:

[chrismeyersfsu]

Was able to recreate.

I was not able to recreate at first because my private tower project was causing the host key to be known ad added to the known_hosts file. Later, when the ansible-galaxy command w/ a private role, living on the same github.com host; the remote host was known because of the project update.

private project on project update + private role <--- SUCCESS
public project (no ssh) + private role ansible-galaxy <--- FAIL

[ilijamt]

Confirmed that the patch works on AWX 10.0.0 deployed using k8s

[ryanpetrello]

Thanks @ilijamt,

This fix will go out in the next release of AWX in the coming weeks.

[kdelee]

Going to close this as has been verified by community users and has been out in a released version of AWX for a few weeks now.