Awx: Can't authenticate with pubkey to remote hosts

[AdrianDutu]

ISSUE TYPE

• Bug Report
  
  
  
  • Feature Idea
  
  

SUMMARY

I have upgraded from 6.1.0 to 9.0.1. Now I can't authenticate to my remote hosts with a DSA key. After troubleshooting, I discovered that the ssh-client in the new awx-celery image was upgraded to 7.8 from 6.x. This version doesn't like DSA keys. I went inside the container and edited the ssh_config file and added:

Host * User s.a.awx HostKeyAlgorithms +ssh-dss PubkeyAcceptedKeyTypes +ssh-dss

Now from the shell, I can ssh to my remote hosts. So I tried to add these to the ansible config file in the ssh_args variable.

ssh_args = -C -o ControlMaster=auto -o ControlPersist=60s -o Host=* -o User=s.a.awx -o HostKeyAlrogithms=+ssh-dss -o PubkeyAcceptedTypes=+ssh-dss

The problem is I'm not sure which ansible.cfg file I need to modify since there are a bunch. I tried:
/etc/ansible/ansible.cfg
/var/lib/awx/projects/_10__automation_ansible/ansible.cfg (_10_automation_ansible is my project name so I assumed this would be a cfg just for that project).

It didn't work. If I debug my job in awx I can't see that the args I've added are sent. This is what it sends:

SSH: EXEC ssh -vvv -o ControlMaster=auto -o ControlPersist=60s -o PreferredAuthentications=publickey -o StrictHostKeyChecking=no -o Port=22 -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="s.a.awx"' -o ConnectTimeout=10 -o ControlPath=/tmp/ansible-ssh-%h-%p-%r

Where is the ansible.cfg file that I need to modify in order for this to work?
Also, please expose the ansible related configurations at installation because digging through the venvs in the container is pretty cumbersome as you can see. So the ansible.cfg and vault-pw files would be nice.

ENVIRONMENT

• AWX version: 9.0.1
• AWX install method: kubernetes
• Ansible version: 2.8.5
• Operating System: CentOS7

STEPS TO REPRODUCE

1. Configure a Machine Type credential in AWX and use a DSA key
2. Run any job that uses that credential on any host

EXPECTED RESULTS

Connection succeeding and running the play.

ACTUAL RESULTS

The public key is refused during authentication.

[ryanpetrello]

This version doesn't like DSA keys.

You should stop using DSA keys:

http://www.openssh.com/legacy.html

OpenSSH 7.0 and greater similarly disables the ssh-dss (DSA) public key algorithm. It too is weak and we recommend against its use.

That said, this _can_ be pretty tricky to get right:

The problem is I'm not sure which ansible.cfg file I need to modify since there are a bunch. I tried:
/etc/ansible/ansible.cfg
/var/lib/awx/projects/_10__automation_ansible/ansible.cfg (_10_automation_ansible is my project name so I assumed this would be a cfg just for that project).

Are you managing manual projects, or pulling your playbooks from SCM (i.e., GitHub)? If it's the latter, have you tried putting the ansible.cfg in the same directory as our playbooks (in source control)?

[AdrianDutu]

We plan to change that key but it is very widely used so it will take some time.
We do use SCM but I can't put an ansible.cfg in the playbook dir since the same repo is used outside of AWX.

[ryanpetrello]

Have you tried adding ANSIBLE_SSH_ARGS to AWX_TASK_ENV in /api/v2/settings/jobs/? That might be the easiest path instead of managing config files across containers.

[AdrianDutu]

I tried it after you've recommended it. Doesn't work. First, it tells me that "Host directive not supported as a command-line option", then if I remove that it says "Bad configuration option: hostkeyalrogithms". If I remove that then it says "Bad configuration option: pubkeyacceptedtypes".

[ryanpetrello]

ssh_args = -C -o ControlMaster=auto -o ControlPersist=60s -o Host=* -o User=s.a.awx -o HostKeyAlrogithms=+ssh-dss -o PubkeyAcceptedTypes=+ssh-dss

Bad configuration option: hostkeyalrogithms

You have a typo.

[AdrianDutu]

Oops, my bad. Anyways, PubkeyAcceptedTypes and Host are still not accepted. And just with HostKeyAlgorithms it's not working. It looks like it tries to find private keys id_rsa, id_dsa etc in /var/lib/awx/.ssh and does not find them. Isn't the key stored in the db? Why is it looking for it on the disk?

Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password\r\ndebug3: start over, passed a different list publickey,gssapi-keyex,gssapi-with-mic,password\r\ndebug3: preferred gssapi-with-mic,gssapi-keyex,hostbased,publickey\r\ndebug3: authmethod_lookup gssapi-with-mic\r\ndebug3: remaining preferred: gssapi-keyex,hostbased,publickey\r\ndebug3: authmethod_lookup gssapi-keyex\r\ndebug3: remaining preferred: hostbased,publickey\r\ndebug3: authmethod_lookup publickey\r\ndebug3: remaining preferred: ,publickey\r\ndebug3: authmethod_is_enabled publickey\r\ndebug1: Next authentication method: publickey\r\ndebug1: Trying private key: /var/lib/awx/.ssh/id_rsa\r\ndebug3: no such identity: /var/lib/awx/.ssh/id_rsa: No such file or directory\r\ndebug1: Trying private key: /var/lib/awx/.ssh/id_dsa\r\ndebug3: no such identity: /var/lib/awx/.ssh/id_dsa: No such file or directory\r\ndebug1: Trying private key: /var/lib/awx/.ssh/id_ecdsa\r\ndebug3: no such identity: /var/lib/awx/.ssh/id_ecdsa: No such file or directory\r\ndebug1: Trying private key: /var/lib/awx/.ssh/id_ed25519\r\ndebug3: no such identity: /var/lib/awx/.ssh/id_ed25519: No such file or directory\r\ndebug1: Trying private key: /var/lib/awx/.ssh/id_xmss\r\ndebug3: no such identity: /var/lib/awx/.ssh/id_xmss: No such file or directory\r\ndebug2: we did not send a packet, disable method\r\ndebug1: No more authentication methods to try.\r\ns.a.awx@atvp1xdcnd121: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).

[ryanpetrello]

AWX doesn't place SSH keys in /var/lib/awx/.ssh/. When jobs are launched, they're pulled from the database and written to ssh-agent.

If you're seeing No more authentication methods I suspect you might have something misconfigured.

Can you share AWX_TASK_ENV value at /api/v2/settings/jobs/?

[AdrianDutu]

Yes, I knew it should be taken from the DB. That's why I was very confused when that error popped up.

"AWX_TASK_ENV": {
"HOME": "/var/lib/awx",
"ANSIBLE_SSH_ARGS": "-C -o ControlMaster=auto -o ControlPersist=60s -o User=s.a.awx -o HostKeyAlgorithms=+ssh-dss"
}

[ryanpetrello]

Have you tried running a job with higher verbosity (2 or 3)? When you do so, can you see the ssh args specified properly in the output?

[AdrianDutu]

SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o HostKeyAlgorithms=+ssh-dss -o StrictHostKeyChecking=no -o Port=22 -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="s.a.awx"' -o ConnectTimeout=10 -o ControlPath=/tmp/ansible-ssh-%h-%p-%r atvp1xdcnd121 '/bin/sh -c '"'"'echo ~s.a.awx && sleep 0'"'"''

[ryanpetrello]

Yep, that looks right to me 😞

There must be some sort of configuration missing here that's preventing SSH from trying the DSA key that AWX writes into ssh-agent.

Any way you can share the entire job output with verbosity at 3?

[AdrianDutu]

> Identity added: /tmp/awx_2079_hmtdci9l/artifacts/2079/ssh_key_data (/tmp/awx_2079_hmtdci9l/artifacts/2079/ssh_key_data)
> ansible-playbook 2.8.5
>   config file = /tmp/awx_2079_hmtdci9l/project/ansible.cfg
>   configured module search path = ['/tmp/awx_2079_hmtdci9l/project/library']
>   ansible python module location = /usr/lib/python3.6/site-packages/ansible
>   executable location = /usr/bin/ansible-playbook
>   python version = 3.6.8 (default, Oct  7 2019, 17:58:22) [GCC 8.2.1 20180905 (Red Hat 8.2.1-3)]
> Using /tmp/awx_2079_hmtdci9l/project/ansible.cfg as config file
> host_list declined parsing /tmp/awx_2079_hmtdci9l/tmpwgcx8i5z as it did not pass it's verify_file() method
> Parsed /tmp/awx_2079_hmtdci9l/tmpwgcx8i5z inventory source with script plugin
> PLAYBOOK: backend-versions.yml *************************************************
> 2 plays in plays/backend-versions.yml
> PLAY [cassandra] ***************************************************************
> TASK [Gathering Facts] *********************************************************
> task path: /tmp/awx_2079_hmtdci9l/project/plays/backend-versions.yml:2
> <atvp1xdcnd121> ESTABLISH SSH CONNECTION FOR USER: s.a.awx
> <atvp1xdcnd121> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o HostKeyAlgorithms=+ssh-dss -o StrictHostKeyChecking=no -o Port=22 -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="s.a.awx"' -o ConnectTimeout=10 -o ControlPath=/tmp/ansible-ssh-%h-%p-%r atvp1xdcnd121 '/bin/sh -c '"'"'echo ~s.a.awx && sleep 0'"'"''
> <atvp1xdcnd121> (255, b'', b"Warning: Permanently added 'atvp1xdcnd121,10.130.209.171' (ECDSA) to the list of known hosts.\r\ns.a.awx@atvp1xdcnd121: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).\r\n")
> fatal: [atvp1xdcnd121]: UNREACHABLE! => {
>     "changed": false,
>     "msg": "Failed to connect to the host via ssh: Warning: Permanently added 'atvp1xdcnd121,10.130.209.171' (ECDSA) to the list of known hosts.\r\ns.a.awx@atvp1xdcnd121: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).",
>     "unreachable": true
> }
> PLAY RECAP *********************************************************************
> atvp1xdcnd121              : ok=0    changed=0    unreachable=1    failed=0    skipped=0    rescued=0    ignored=0  

[ryanpetrello]

@AdrianDutu are you certain your key is correct?

Identity added: /tmp/awx_2079_hmtdci9l/artifacts/2079/ssh_key_data

You can see where it's attempted to use the key:

Failed to connect to the host via ssh: Warning: Permanently added 'atvp1xdcnd121,10.130.209.171' (ECDSA) to the list of known hosts.\r\ns.a.awx@atvp1xdcnd121: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).

[AdrianDutu]

Yes, the key is correct.

[bhavyanshu]

I am having the same issue. We recently wanted to try out awx. So I cloned the current master. Set it up using docker-compose. We use vault signed certificate (RSA SHA256). The signed cert works via SSH command line. I am able to ssh but not able to do it with AWX Machine type credential. I do see the identity added and certificate lines at the beginning when the job is executed. I noticed it says no mutual signature algorithm while providing the public cert. Could it be the version of openssh in the container? How can I try downgrading it?

[AdrianDutu]

@bhavyanshu run update-ca-certs in the container, might work. I had some issues with certificates but seems different than yours. Worth a try though.

[bhavyanshu]

@AdrianDutu Hi, thanks for the tip. I tried it and sadly, it didn't work. However, I noticed a bug related to openssh 7.8 : https://bugs.launchpad.net/ubuntu/+source/openssh/+bug/1790963

I manually upgraded to OpenSSH 8.0 in the awx_task container. Basically getting this https://cdn.openbsd.org/pub/OpenBSD/OpenSSH/portable/openssh-8.0p1.tar.gz and installing all the deps using yum. Restarted the ssh-agent in the task container and it worked.

It might be worth trying for you as well. Ref this for how to manually update: https://www.tecmint.com/install-openssh-server-from-source-in-linux/

[AdrianDutu]

@bhavyanshu I read the bug. It's not the same issue. But that's a nasty bug, maybe they will upgrade to OpenSSH 8 with the next AWX version.

[BDworak]

Any update on this? Will this be fixed in the next awx release?

[chinochao]

If you are still using ssh-dss keys you best solution is to add the following to your host level, group leve or inventory level extra variables in AWX.

ansible_ssh_common_args: '-o PubkeyAcceptedKeyTypes=+ssh-dss'

Instead of making any changes to your container files. I was able to get my inventory working after adding this. It is also recommended to upgrade your keys to RSA for security purpose.