Awx: AWX ignores parameters in ansible.cfg

[infestonn]

ISSUE TYPE

• Bug Report

SUMMARY

AWX seems to read right ansible.cfg file but uses default _roles_path_ value.

ENVIRONMENT

• AWX version: 7.0.0
• AWX install method: docker on linux
• Ansible version: 2.8.4
• Operating System: Ubuntu 18.04

STEPS TO REPRODUCE

Directory tree:

├── ansible.cfg
├── hosts
├── inventory
├── main.yml
├── playbooks
│   ├── consul
│   │   ├── files
│   │   ├── group_vars
│   │   ├── host_vars
│   │   ├── main.yml
│   │   └── vars
├── roles
│   ├── brianshumate.consul
...
# cat ansible.cfg

[defaults]
roles_path = roles

Place roles directory inside project root.
Set _roles_path = roles_ in ansible.cfg
Run playbook which requires any role e.g.: _playbooks/consul/main.yml_

EXPECTED RESULTS

Ansible Tower runs ansible-playbook from the root of the project directory, where it will then find the custom ansible.cfg file. An ansible.cfg anywhere else in the project will be ignored.

ACTUAL RESULTS

Identity added: /tmp/awx_1162_3sgq_dhy/artifacts/1162/ssh_key_data (/tmp/awx_1162_3sgq_dhy/artifacts/1162/ssh_key_data) ansible-playbook 2.8.4 config file = /tmp/awx_1162_3sgq_dhy/project/ansible.cfg configured module search path = [u'/var/lib/awx/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules'] ansible python module location = /usr/lib/python2.7/site-packages/ansible executable location = /usr/bin/ansible-playbook python version = 2.7.5 (default, Jun 20 2019, 20:27:34) [GCC 4.8.5 20150623 (Red Hat 4.8.5-36)] Using /tmp/awx_1162_3sgq_dhy/project/ansible.cfg as config file host_list declined parsing /tmp/awx_1162_3sgq_dhy/tmpmjfgzc8z as it did not pass it's verify_file() method ... ERROR! the role 'brianshumate.consul' was not found in /tmp/awx_1162_3sgq_dhy/project/playbooks/consul/roles:/tmp/awx_1162_3sgq_dhy/requirements_roles:/tmp/awx_1162_3sgq_dhy/project/playbooks/consul

ADDITIONAL INFORMATION

Worked in AWX 6.1.0 with ansible 2.8.2

[AlanCoding]

This is related to a bunch of stuff going on, see https://github.com/ansible/ansible/issues/61778, as that was raised to document what is a known limitation. Your error mentioned:

ERROR! the role 'brianshumate.consul' was not found in

• /tmp/awx_1162_3sgq_dhy/project/playbooks/consul/roles
• /tmp/awx_1162_3sgq_dhy/requirements_roles
• /tmp/awx_1162_3sgq_dhy/project/playbooks/consul

The Ansible default for roles_path is ~/.ansible/roles:/usr/share/ansible/roles:/etc/ansible/roles. So I think it would be fair to say that it clobbers the Ansible defaults, not that it _uses_ the defaults. It also clobbers your own setting (which is the issue here).

I can confirm that the relative path is expected to work via ANSIBLE_ROLES_PATH=alan ansible-config dump --only-changed.

This is unfortunate. We do not have a means of combining user config values with the values that the system needs to apply (in this case, the location that ansible-galaxy role install installs to.

In other words, resolving this issue may require a general solution to the long-standing problem of combining project settings with system settings.

[AlanCoding]

Putting the blocked label on this because it is the exact thing that the linked Ansible core issue is looking for. We can also solve it by parsing the config file ourselves, but this would be dicey. We would need to confer about the architecture before embarking on something like that.

[AlanCoding]

On the question, "could we parse the config file?"

https://github.com/ansible/ansible/blob/7b28960ebe435fd4eaea23abe918a28a112843a0/lib/ansible/config/manager.py#L317-L337

I think we could, given the set of what Ansible core currently supports right now. Unlike YAML data, there seem to be no special syntax allowances beyond the python config parser. This still says that it would like to support YAML in the future (I'm sure the custom types would come into play if they did), but that has not yet happened.

[AlanCoding]

Ping @wenottingham, since https://github.com/ansible/awx/pull/4656 addressed a larger problem (global defaults) whereas this is a narrower limitation, but still related to the same architectural problem with config settings.

[wenottingham]

I think parsing the config file in Tower/runner is not the road we want to go down.

[ryanpetrello]

I think parsing the config file in Tower/runner is not the road we want to go down.

I agree with this statement.

[ryanpetrello]

@AlanCoding is this new, or has it always been this way?

[AlanCoding]

is this new, or has it always been this way?

It's not that simple. Both.

It was always the case that Ansible config variables set to AWX / Tower clobbered both the defaults and the user's value.

Whenever we set new config variables (as demanded by feature requests) we clobber a new thing, this was discussed in the original development of the collections path. This issue is hashing out the same stuff as before, only applied to the roles path as opposed to the collections path, along with the 1 other minor addendum that we hard code the defaults, so Ansible defaults are not clobbered. But user values are.

The reason users are noticing this is because the roles_path config has been around for a long time, so it stands to reason that users were already using it, and after the upgrade, their values got clobbered.

Why did we start installing the roles to a new place? Because of the change to copy the project folder on each run. If we wanted to address this concern, what would that take? ~It would require changing the documented job directory structure. Would that introduce other problems? Maybe.~ After looking back at the code, I realize complications I forgot earlier. I cannot think of a of a way we could do this right now.

[sumkincpp]

As a possible workaround, moving all project roles to requirements_roles doesn't help either.

Are requirements_roles require some specific role structure, i.e. like for ansible-galaxy?

[AlanCoding]

That folder is not inside the project folder (adjacent to it), so I can't imagine any practical means of the user putting content in it.

[badstreff]

While not really a good solution, a fix is to add the roles/ path back into the roles_path at the time it is overwritten. We already expect requirements.yml to be in this directory, why not other roles? In my testing this has worked for me.

[AlanCoding]

While not really a good solution, a fix is to add the roles/ path back into the roles_path at the time it is overwritten.

This issue is about Ansible ignoring roles_path entry in ansible.cfg file in the root of the project directory. If the relative path roles/ adjacent to the playbook is not respected, that is a different matter. If you are seeing these roles ignored, I would appreciate any more information you have about the layout of your project, where the playbook is, where this folder is, and how the roles are referenced in the playbook.

[zx1986]

I got this issue too.
My roles is sitting in my project root in AWX.
But it shows:

ERROR! the role 'geerlingguy.docker' was not found in
/tmp/awx_88_wtn43n4z/project/playbooks/roles:
/tmp/awx_88_wtn43n4z/requirements_roles:
/tmp/awx_88_wtn43n4z/project/playbooks

I am expect it will look for roles in /tmp/awx_88_wtn43n4z/project/roles,
but it didn't :-(

AWX web 7.0.0
AWX task 7.0.0

[hlavacek87]

Same problem, temporary workaround with soft link in git repository fixed the issue.

[stherien]

Could you describe the steps used for your temporary workaround ? How did you manage to create soft link in git repository ?

[AlanCoding]

@zx1986 Could you give more information, specifically why it is expected to be looking in /tmp/awx_88_wtn43n4z/project/roles? Is that because of your settings in ansible.cfg? Or is it playbook-adjacent?

[hlavacek87]

@AlanCoding I have the same problem, ansible.cfg in project dir includes: "roles_path = ./ansible/roles" which is not where the playbooks are stored and this config is ignored by AWX.

@stherien git will handle standard ln -s, but you need to use relative paths, not absolute .. furthermore both roles dir and playbooks must be in the same repo:
# ll plays_iaas/roles lrwxrwxrwx. 1 root root 14 Sep 25 15:20 plays_iaas/roles -> ../../../roles

[AlanCoding]

My current reading right now is that:

https://github.com/ansible/awx/issues/4675#issuecomment-537888508

Is a different issue from the problem that most other comments here have reported. I may re-file this as a new issue, and I expect interplay with the ansible-runner package.

[badstreff]

@AlanCoding My current directory structure looks like the following

├── ansible.cfg
├── inventory/
├── site.yml
├── playbooks/
    ├── app/
        ├── main.yml  # includes the my_app role
├── roles/
    ├── my_app  # not installed via galaxy

I believe throwing the roles adjacent to the main.yml playbook under the app folder would resolve the issue, however I would prefer they remain in the root

[AlanCoding]

After looking further into it, I am trending toward thinking that the roles folder would have never worked _without_ the previously mentioned ansible.cfg (things like the working directory do not factor in).

[defaults]
roles_path = roles

in which case there's not some other issue related to this. Just what has already been established here.

[zx1986]

@AlanCoding My ansible.cfg in my project repository is

[defaults]
roles_path = ./roles

/tmp/awx_88_wtn43n4z/project/ is the AWX project I run with, I believe it was a random directory created by AWX task worker, and it was the root directory of the AWX project.

I use symbolic link as @hlavacek87 said, and it works.

[AlanCoding]

Hmm... roles_path = ./roles versus roles_path = roles. The latter I have confidence about, and will use the "roles" directory relative _to the config file_. The working directory should probably be the project directory, so I expect that either solution should probably be the same, but either ansible-runner or Ansible could introduce further complications in its processing.

Adding another complication to this, I put up https://github.com/ansible/test-playbooks/pull/79 to express the case where there is a requirements file (which installs roles) and a playbook using that role which is not in the root project directory. I think we will all agree that that _should_ work.

[zx1986]

My project structure is like:

.
├── Dockerfile
├── Makefile
├── README.md
├── ansible.cfg      <=== here
├── filter_plugins
├── inventories
├── library
├── module_utils
├── playbooks   <=== and I put a symbolic link in here ( roles -> ../roles )
├── roles      <=== here
└── scripts

[ondravondra]

I used this workaround:

ansible-playbook-awx-workaround:

#!/bin/bash
# https://github.com/ansible/awx/issues/4675

unset ANSIBLE_ROLES_PATH
/usr/bin/ansible-playbook-2 $@

Dockerfile:

FROM ansible/awx_task:7.0.0
USER 0
COPY ansible-playbook-awx-workaround /usr/bin/
RUN \
  rm /usr/bin/ansible-playbook &&\
  ln -s /usr/bin/ansible-playbook-awx-workaround /usr/bin/ansible-playbook
USER 1000

[merou]

Just to clarify and for those struggling with this issue:
1/ In AWX 7.0.0 roles_path entry in ansible.cfg is not read.
2/ For people having a playbooks folder at the root.
3/ While waiting for an elegant solution (agree Tower parsing this config will lead to further issues)

workaround is have the roles inside the playbooks folder via a symlink:
ln -s ../roles playbooks/roles

[AlanCoding]

ansible.cfg is still read and other parameters you define in that file will take effect (now, while the patch is yet to land). This affects 2 parameters related to search paths for roles & collections, where the value in ansible.cfg is ignored.

And the patch will read the config file(s).

[josemgom]

Just to clarify and for those struggling with this issue:
1/ In AWX 7.0.0 roles_path entry in ansible.cfg is not read.
2/ For people having a playbooks folder at the root.
3/ While waiting for an elegant solution (agree Tower parsing this config will lead to further issues)

workaround is have the roles inside the playbooks folder via a symlink:
ln -s ../roles playbooks/roles

Agree, is the only workaround that I see here until we have a solution.

[josemgom]

It seems that this issue has been fixed in AWX 8.0.0 which was released 14 hours ago.

If someone can confirm that it works fine now..

commit 31bdde00c9a415e2083fef285f189b31caca32b4
Author: Bill Nottingham [e-mail]
Date: Thu Oct 10 15:08:20 2019 -0400

Check the user's ansible.cfg for role/collection paths.

There's no other way to add our new paths reliably without breaking things.

[hluaces]

@josemgom I've just updated to 8.0.0 and the issue seems to be solved.

As there seems to be a lot going on in this issue I'll give you some details on my use case to make sure that I'm not misunderstood.

I was using this folder structure:

├── ansible.cfg -> config/ansible.cfg
├── config
│   └── ansible.cfg
├── playbooks
│   ├── files
│   ├── folder
│   │   ├── playbook.yml
│   │   └── playbook2.yml
│   └── playbook3.yml
└── roles
    ├── role_1
    └── role_2

My roles_path was set to /etc/ansible/roles:/usr/share/ansible/roles:roles/.

Upong trying to execute anything on the previous version roles couldn't be found, now they are running properly.

[kdelee]

I agree, just tried this out and it is fixed for me on latest awx. Going to close! Please try updating and confirm that it works for you.

[vvxxvvxx]

The same issue happened in AWX 9.2.0.

Using /tmp/awx_110_0u9nob3i/project/ansible/ansible.cfg as config file
Vault password: 
ERROR! the role 'security' was not found in /tmp/awx_110_0u9nob3i/project/ansible/playbook/xx/security/roles:/tmp/awx_110_0u9nob3i/requirements_roles:/var/lib/awx/.ansible/roles:/usr/share/ansible/roles:/etc/ansible/roles:/tmp/awx_110_0u9nob3i/project/ansible/playbook/xx/security

but if using ansible-playbook to run the playbook which doesn't have the role defined/called, the playbook runs successfully.

the ansible.cfg can be read, but the roles_path defined in the ansible.cfg seems being ignored.

[AlanCoding]

^ check if you have a global config, which could mean you're hitting https://github.com/ansible/awx/pull/6038

[zendritic]

I just ran into this in AWX 7.0.0 for the first time. We're a bit slow to upgrade. Is there some problem I'm not seeing with fixing this by adding an environmental variable in AWX jobs settings? "ANSIBLE_ROLES_PATH": "roles" seems to work for me so far in my test environment.

[sumkincpp]

I am no more facing this issue with AWX 9.0.0, 10.0.0 and 11.2.0.

[AlanCoding]

For future information, my final regression test case was:

https://github.com/ansible/test-playbooks/pull/124/files