Awx: Kerberos Setup on Task Container needs to be privileged

[mkayontour]

ISSUE TYPE

• Bug Report

COMPONENT NAME

• Container
• Installer

SUMMARY

As I tried to setup kerberos on the docker container I came across this error:

invalid uid in persistent keyring name while getting default ccache

In result I needed to give the tasks container privileged access to make it work.

Question: How would be it the right way to do it? I had to modify the installer and so I have troubles updating the environment.

ENVIRONMENT

• AWX version: 1.0.1
• AWX install method: docker on linux
• Ansible version: 2.4.x
• Operating System: Debian Jessie
• Web Browser: Chrome

STEPS TO REPRODUCE

Setup Winrm like explained in the documentation.
https://docs.ansible.com/ansible/devel/windows_winrm.html#kerberos

If you try to test your connection, you will get this error

kinit -C [email protected] 
invalid uid in persistent keyring name while getting default ccache

ADDITIONAL INFORMATION

Manual Fix: add privileged: true to your Task container.
vim awx/installer/local_docker/tasks/main.yml

- name: Activate AWX Task Container
  docker_container:
    name: awx_task
    state: started
>  privileged: true
    restart_policy: unless-stopped
    image: "{{ awx_task_docker_actual_image }}"
    volumes:
      - "{{ awx_lib_dir }}:/var/lib/awx/projects"
    links: "{{ awx_task_container_links|list }}"
    user: root

[TheBigBear]

@mkayontour thank you so much for this. I have added this manually to my local docker deploy and all I had to to now, was (also manually) edit the /etc/krb5.conf file in the awx_tasks docker container and voila I can use ms ad and winrm kerberos transport. thank you so much fro finding this. Now there is hoping that it will get merged quickly, so we can all use ansible windows automation in AWX.

[matburt]

I think I'd like to figure out a way to do this without privileged mode.

[deepan10]

Thank you @TheBigBear Post manually changed the awx_task docker container, It worked!. Now able to connect winrm.

[steaksauce-]

I took a whack at it :)

CORRECTION: I missed where @matburt mentioned trying to do it without priviledged mode :( Back to the drawing board!

[steaksauce-]

I've done some research into this, and I've found some evidence (in the form of other containers) that this should be able to work without privileged containers. I am still looking into the matter and setting up my test environment now and will keep this updated with my findings

[steaksauce-]

@matburt @mkayontour
I was able to get this working without a privileged container under the following conditions:

• Utilize docker exec -i -t awx_task /bin/bash to get into the container and modify it.
• Install gcc within the container (python-pip needs it to install pywinrm[kerberos])
• Follow the documentation on setting up Kerberos for Ansible
• BE SURE that the domain name is uppercase in /etc/krb5.conf
• Inside /etc/krb5.conf, comment out the line default_ccache_name = KEYRING:persistent:%{uid}

The commands to verify functionality once configured:

kinit [email protected] # notice that domain is still uppercase
klist -l
Principal name                 Cache name
--------------                 ----------
[email protected]      FILE:/tmp/krb5cc_0

FINDINGS: The default_ccache_name "specifies the name of the default credential cache". The credential cache holds credentials while they remain valid. As you can see from my output above, when this is not specified, it still created a cache location, just in my /tmp folder. Unless someone _very knowledgeable_ in the dryness of the Kerberos workings has a good reason why we should not comment out this line, I don't see a problem in doing this. Just for reference, per MIT, the default_ccache_name is determined in order:

1. The KRB5CCNAME environment variable. For example, KRB5CCNAME=DIR:/mydir/.
2. The default_ccache_name profile variable in [libdefaults].
3. The hardcoded default, DEFCCNAME.

RECOMMENDATIONS MOVING FORWARD:
To simply the awx deployment for environments that need Ansible to use Kerberos to authenticate to Windows, FreeIPA, or anything like that, a user should be able to specify an option for Kerberos while installing AWX via playbook. The playbook can either configure the docker image for kerberos based off the presence of this option (also allowing the user to specify the variables to setup krb5.conf via templates or something), or less-than-likely, check out a kerberos-ready version of the awx-task container. But who wants to keep up with two task containers?

Any thoughts on this going forward?

[matburt]

This is going to be pretty tricky, but probably doable on a number of fronts.

We can build in whatever dependencies are necessary from the image build: https://github.com/ansible/awx/blob/devel/installer/image_build/templates/Dockerfile.j2

As opposed to providing kerberos credentials at install time it might be better to define a new credential to perform some of these actions at job launch: https://github.com/ansible/awx/blob/devel/awx/main/expect/run.py#L30-L42

On the Tower side the playbook process runs under bubblewrap. Which is something I'd like to do here but will completely isolate the playbook run from the rest of the system.

There's also Isolated instances which tower has installer level support for and those will come to AWX pretty soon... that's based around taking that expect module and loading it onto external instances.

tldr... it needs to be baked into the job execution semantics rather than the container itself.

[steaksauce-]

When you say

it needs to be baked into the job execution semantics rather than the container itself

do you mean that when a user executes the job (job template or workflow), the job will handle the Kerberos connection rather than the system (ie, task container) being configured for Kerberos?

Trying to wrap my head around this. It seems like we will need to build in the Kerberos deps regardless (per pywinm[kerberos] reqs), but I can't find anything about it requiring that Kerberos actually being configured (just installed).

I'm not a developer, but it seems like something similar the function wrap_args_with_ssh_agent from the link you had can be written for Kerberos connections (again, not a developer!)..

Granted that any of my above assumptions are correct, I've hacked together a proof of concept, based on one of the pywinrm examples:

# code may not be accurate
# only meant to be demonstration of pywinrm
from winrm.protocol import Protocol
ps_script = "/some/powershell/script.ps1"

p = Protocol(
    endpoint='https://windows-host:5986/wsman',
    transport='kerberos',
    username=r'somedomain\someuser',
    password='secret',
    server_cert_validation='ignore')
shell_id = p.open_shell()
command_id = p.run_command(shell_id, ps_script)
std_out, std_err, status_code = p.get_command_output(shell_id, command_id)
p.cleanup_command(shell_id, command_id)
p.close_shell(shell_id)

[matburt]

Yep, we are on the same page.

The key to it is support it on our Credential objects as another Machine credential.

[steaksauce-]

From a UX/UI standpoint, a user goes to create a new credential, specifies machine credential, and then checks a box to specify to use kerberos when trying the credentials sounds like an easy way to expose the feature when/if it is created

[wenottingham]

Given we have behind-the-scenes kinit already for Windows, why wouldn't we push this down to Ansible to also work for Linux/Unix?

[falencastro]

But if it's added as a new credential option, wouldn't it be necessary to inform kerberos realm, kdc (there can be multiple) and admin servers for every new kerberos credential? Since there can be multiple users for the same kerberos domain, and multiple domains, I imagine it would be better if there was a "kerberos domain" object, with all its particularities described above, and when you create a new machine credential you would have the ability to select a pre-created domain and associate the credential with it.

[chancerollins]

I wanted to leave a comment here to help the community as I just fought the battle of getting WinRM Kerberos authentication with AWX and had to spend a lot of time trying and failing and reading loads of github issue posts to get this. I also had to do several additional packages on my side. @Steaksauce and @mkayontour were very helpful however it is unfortunate the difficulty in getting this setup and obviously doing upgrades of AWX is going to be a major pain the way this is currently.

Changes I did

• Modified AWX Task Container to privileged and re-deployed AWX - @mkayontour
• Entered into container - docker exec -i -t awx_task /bin/bash - @Steaksauce
• yum install krb5-workstation, krb5-devel, krb5-libs, gcc, python-devel
• pip install kerberos
• Edited /etc/krb5.conf as described in AWX documentation - http://docs.ansible.com/ansible/latest/intro_windows.html#kerberos

Then with the combination of following I was successful:

• ConfigureRemotingForAnsible.ps1 script Run
• The following variables in my inventory in AWX
  ansible_port: 5986
  ansible_connection: winrm
  ansible_winrm_server_cert_validation: ignore
• Machine credentials with username like '[email protected]'

[mkayontour]

as I had troubles updating awx on the customer side, I started with a fresh installation on 1.0.4 and tried the kerberos connection again without using privileged containers. And it worked out!

We still wondering why because all those options where set before..
The only things we need to setup after an update will be installing dependencies and setup the configfile.

@matburt do you think it might be helpful for people to have the kerberos dependencies preinstalled and let them only setup the configuration file?

Otherwise I'm good with this issue and the community is informed what do to about it.

[matburt]

I think that sounds like a good idea. Would want to see this in a PR and documented. The config setup might should be its own credential?

[riponbanik]

@mkayontour I have followed the process and configured krb5.conf but still shows the error below -

"kerberos: authGSSClientStep() failed: (('Unspecified GSS failure. Minor code may provide more information', 851968), ('Server not found in Kerberos database', -1765328377)), plaintext: the specified credentials were rejected by the server",

klist shows my cred that I created using kinit successfully
klist -l
Principal name Cache name
-------------- ----------
[email protected] FILE:/tmp/krb5cc_0

[riponbanik]

I have resolved the above issue by using host fqdn instead of ip address in the inventory.