Awx: Allow for different role requirements file names/locations

[geerlingguy]

Summary

I was trying to figure out a way for AWX to automatically install role requirements from Ansible Galaxy prior to running my playbook in a new Project Template, but didn't see any way to do it.

My current project structure is:

project/
  requirements.yml #role requirements
  main.yml # playbook

But it looks like according to the documentation, Ansible Galaxy requirements will _only_ be installed if there is a file named requirements.yml inside a specifically-named roles directory.

Environment

• AWX version: 1.0.0-[latest as of yesterday]
• Ansible version: 2.3.1.0
• Operating System: macOS 10.12
• Web Browser: Safari

Steps To Reproduce:

1. Create a project with the structure as shown above.
2. Add a Template in AWX for this project.
3. Run a new Job with the Template.

Expected Results:

AWX should find my role requirements file in any location inside the repository, maybe as specified in the template itself.

Actual Results:

ERROR! the role 'geerlingguy.firewall' was not found in /var/lib/awx/projects/_6__my_project/roles:/etc/ansible/roles:/var/lib/awx/projects/_6__my_project

My root requirements.yml file was not used to install Galaxy roles.

Additional Information:

Another note—for some projects, I have an ansible.cfg in the project root that defines a specific roles path (which may not always be ./roles/). It would be especially nice if AWX could figure out a way to parse that setting if present and download the roles into that location... though that could prove fruitless since that setting could be a set of many paths :-/

[matburt]

We have definitely been opinionated in the past about where things like this get picked up. I'm fine with someone tackling this but it's not a priority for us.

We do see some extremely large directory trees in projects on user's systems so an exhaustive traversal of the entire project hierarchy will be rejected.

[wenottingham]

To be fair, extremely large project directory trees are going to break with isolated nodes anyway.

[geerlingguy]

To avoid the large directory tree issue, it would be nice to maybe add a directive to ansible.cfg for 'role_requirements_file' or something like that, or to add a field in the template form to allow setting a specific path. Automatically discovering it would definitely be a pain.

[wenottingham]

If you do need the naive version of this, it's something like this:

diff --git a/awx/playbooks/project_update.yml b/awx/playbooks/project_update.yml
index 20c84b666..fb739ac30 100644
--- a/awx/playbooks/project_update.yml
+++ b/awx/playbooks/project_update.yml
@@ -127,15 +127,28 @@
         scm_version: "{{results.version}}"
       when: scm_type == 'insights' and results is defined

-    - name: detect requirements.yml
-      stat: path={{project_path|quote}}/roles/requirements.yml
-      register: doesRequirementsExist
+    - name: find role directories
+      find:
+        paths: "{{ project_path|quote }}"
+        patterns: "roles"
+        file_type: "directory"
+        recurse: True
+      register: findResults
+      when: scm_full_checkout|bool
+
+    - name: find requirements.yml in role directories
+      stat:
+        path: "{{ item.path }}/requirements.yml"
+      with_items: "{{ findResults.files }}"
+      register: requirementsFiles
+      when: scm_full_checkout|bool

     - name: fetch galaxy roles from requirements.yml
-      command: ansible-galaxy install -r requirements.yml -p {{project_path|quote}}/roles/ --force
+      command: ansible-galaxy install -r requirements.yml -p {{item.item.path|quote}}/roles/ --force
       args:
-        chdir: "{{project_path|quote}}/roles"
-      when: doesRequirementsExist.stat.exists and scm_full_checkout|bool
+        chdir: "{{ item.item.path }}"
+      when: scm_full_checkout|bool
+      with_items: "{{ requirementsFiles.results }}"

       # format provided by ansible is ["Revision: 12345", "URL: ..."]
     - name: parse subversion version string properly

[wenottingham]

The non-naive version of this likely involves only pulling from where the playbook for the current JT that's being checked out. That may require API support.

[fredleger]

that is definitely something we are looking to also. to make it efficient i don't see any problem to precise the path or the requirements.yml in the UI.

[victorock]

Ran on this few times when trying to avoid git submodules.
Well, by default ansible looks for _current_playbook_path_/roles in addition to roles_path defined in ansible.cfg.

Perhaps the proper behaviour would be:

1. When the update on launch is defined for the project:
2. project_update considers the _current_playbook_path_/roles/requirements.yml.

3. project_update considers the roles_path defined in _current_playbook_path_/ansible.cfg.

4. When update on launch is not defined for the project:

5. project_update considers the _project_root_/roles/requirements.yml
6. project_update considers the roles_path defined in _project_root_/ansible.cfg.

[wenottingham]

That's not what 'update_on_launch' means. 'update_on_launch' means "update the SCM revision to the latest for the configured branch/tag/etc before running the playbook; it's entirely orthogonal to checking out roles.

[wenottingham]

... that being said, a change so that it also parses roles/requirements.yml relative to the playbook directory would make sense. This would require both a change to project_update.yml to do this, and a change to awx/main/tasks.py to pass that path as an extra var to the project update.

[victorock]

It has been so painful for me that i can try to work on it.

[akcrisp]

just wondering if there's been any further progress on this. We would like the ability to reference a different requirements.yml file from awx.

We've been heavily using ansible-galaxy as part of our build automation work. That is parsing a parameter to the playbook {{ playbook }}-requirements.yml so that we can pull in specific app team git repos for installing specific application code.

This currently comes out of a central git repo owned by our build team (who have defined different playbooks for installing different teams applications, so within the roles directory there is a requirements.yml named after the calling playbook.

This works perfectly. However trying to now move this into AWX fails because of the hard coded use of requirements.yml. Otherwise we will have to re-engineer all of these into individual git repos and then in awx we will have to create a project per repo and thus a playbook etc to orchestrate the app install. Where as if we could simply pass a in the param to awx to tell it the requirements.yml file we could happily use the one job template / project.

Andy

[colinodell]

In a similar vein, we typically version our Ansible playbooks as part of a larger project by placing it within a subdirectory of the git repository:

project/
    ansible/
        main.yml # main playbook
        requirements.yml # requirements file

Being able to somehow configure project_path and the location of requirements.yml on a per-project basis would be a huge help!

[00Asgaroth00]

Hi, new to AWX :)

AWX Version: AWX 6.1.0.0 (docker)
Ansible Version: 2.8.2

Perhaps the proper behaviour would be:

1. When the update on launch is defined for the project:


* project_update considers the _current_playbook_path_/roles/requirements.yml.

* project_update considers the roles_path defined in  _current_playbook_path_/ansible.cfg.

I'm encountering this issue as well, and the above suggestion would probably correct it.

An example layout that I'm currently testing is as follows:

project/
    playbook_1/
        ansible.cfg
        playbook_1.yaml
        roles/
            requirements.yml
    playbook_2/
        ansible.cfg
        playbook_2.yaml
        roles/
            requirements.yml

The ansible.cfg defines roles_path as

roles_path = roles

The roles_path may not be required, but I tried it to test.

It would be great if AWX was able to scan for requirements.yml (and/or, incedently requirements.yaml) relative to the playbook's path.

Is there another way to achieve the above?

[matburt]

This sounds like it would make a great pull request!

[AlanCoding]

https://github.com/ansible/awx/pull/3887

[momcilo78]

What is the present state of this issue?
With AWX 11.2 I can not get it to to download the roles even when placed in: roles/requirements.yml
If I got it right I should see this being downloaded with SCMUpdate job, but it looks like this every time:

PLAY [Update source tree if necessary] *****************************************

TASK [update project using git] ************************************************
changed: [localhost]

TASK [Set the git repository version] ******************************************
ok: [localhost]

TASK [Repository Version] ******************************************************
ok: [localhost] => {
    "msg": "Repository Version b9fe0e436da0ad3d7a9a1ec40a929f3468385292"
}

PLAY [Install content with ansible-galaxy command if necessary] ****************

PLAY RECAP *********************************************************************
localhost                  : ok=3    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   

[AlanCoding]

The roles should be installed before a job run that uses that project, this is a UX headache, but issue for that is https://github.com/ansible/awx/issues/5518

[momcilo78]

So, if I got it right, placing collections/roles inside git repository would actually yield a faster job startup time?

[geerlingguy]

@momcilo78 - Technically, yes. Additionally, you would not have a dependency on Galaxy/Automation Hub being up and responsive during your job runs if you committed your dependencies.

However, there are caveats to doing that (for example developers getting lazy about updating the dependencies), and you have to make sure you have a good process for managing the committed dependencies—and not being tempted to monkey patch the committed dependencies, since an update (which should be done regularly) could wipe out that change.

[bimmerkiev]

Is there any updates about this ticket?

[momcilo78]

As far as know:
awx will look into:

• roles/requirements.yml/yaml
• collections/requirements.yml/yaml

I am not sure if it will lookup relative paths to playbook or obey the ansible.cfg.

If your awx contains ansible 2.10.x it will also be able to fetch the collections stored in git repositories. Note that the default awx image still contains 2.9.16 version, and I believe the 2.9.x branch is missing the git source support.

[bimmerkiev]

As far as know:
awx will look into:

• roles/requirements.yml/yaml
• collections/requirements.yml/yaml

I am not sure if it will lookup relative paths to playbook or obey the ansible.cfg.

Hi. Thanks for your reply, but this ticket about allowing different requirements file paths.
We really need it especially in AWX.

[momcilo78]

@bimmerkiev 15.0.1 does not do this - I did test it since my team originally did not place playbooks on the root. For 16.0.0 I did not check, so best suggestion to you to simply try it.

[bimmerkiev]

@bimmerkiev 15.0.1 does not do this - I did test it since my team originally did not place playbooks on the root. For 16.0.0 I did not check, so best suggestion to you to simply try it.

I tried in 16.0.0 and it work the same. You can't define the path to custom requirements.yml file