Molecule: Read dependencies from role meta.

[brucellino]

Issue Type

Feature request

Currently, molecule relies on the developer expressing the role dependencies in a requirements.yml, when Galaxy is used as a dependency manager.
However, this information is redundant, since in production use the role should express it's dependencies in meta/main.yml

I consider this sub-optimal on two fronts:

1. There is duplication of effort - the developer has to write the dependencies both in requirements.yml as well as in meta/main.yml
2. There is no consistency check between meta/main.yml['dependencies'] and requirements.yml meaning that tests could pass in dev but fail in prod.

I suggest that an option could be added to read dependencies from meta/main.yml

It looks like it could go in

https://github.com/metacloud/molecule/blob/6404e938b2aec7b244c63fdbfa4153c84629934b/molecule/dependency/ansible_galaxy.py#L100

perhaps with a check on whether options['read_from_meta'] is set?

I would love to send a PR for this, but alas, my :snake: is weak. I will try, but it'll take a while. In the meantime, I wanted raise the issue for discussion.

Thanks!

Molecule and Ansible details

ansible 2.5.0
  config file = None
  configured module search path = [u'/home/becker/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python2.7/dist-packages/ansible
  executable location = /usr/local/bin/ansible
  python version = 2.7.15rc1 (default, Apr 15 2018, 21:51:34) [GCC 7.3.0]

molecule --version
molecule, version 2.14.0

• Molecule installation method : pip
• Ansible installation method: pip

[retr0h]

However, this information is redundant, since in production use the role should express it's dependencies in meta/main.yml

We actually consider using meta/main.yml an anti-pattern, and prefer users continue to describe their dependencies in requirements.yml. We have run into far too many problems in production roles using meta dependencies, that we feel its in the best interest people just stay away from them.

We feel requirements.yml provides the most flexibility for users.

[retr0h]

Appreciate the issue but not something we will support at this time.

[brucellino]

I would be lying if I said I had never run into this before (meta-hell). I'm willing to accept this invalid tag - thanks for that!

[manics]

@brucellino FYI I brought up this issue a while ago: https://github.com/metacloud/molecule/pull/717
The conclusion was that ansible-galaxy should support reading from meta/main.yml: https://github.com/ansible/ansible/issues/20537

[brucellino]

ah thanks @manics - I tried to search the issues, but didn't find it.

This seems to contradict what @retr0h said in a previous comment? Nothing wrong with that per-se, I just want to have a fluid experience when adopting molecule.

For me, the important thing is having common practice. Since I work in a federation, and am working on the style guide for the developers of roles. We want the roles to be developed in a very robust manner to try and encourage re-use. So, dependencies are very important to us, and we want a common way for roles to express them - even if that way is "there are no explicit dependencies".

This is necessary for us to speed up code review (avoid arguments about how to compose services, etc), respect the infrastructure spec that we're given - but most of all avoid repetition of work. If roles are properly described in meta, they will show up in Galaxy.

[manics]

I think it comes down to whether you automatically load dependencies from meta/main.yml (potential for ending up in "meta-hell") or whether you explicitly use it as your requirements.yml to save duplicating the content, so effectively it's another format to support but the developer is still responsible for ensuring the dependencies are consistent.

[artburkart]

Perhaps the wrong forum, but what is an example of "meta-hell"? What leads to "meta-hell"? Also, is there a better way than defining role variables in the meta/main.yml that other people follow? I don't think I can enforce variables in the requirements.yml, right?

[retr0h]

Rather than having role dependencies live in meta/main.yml, and having ansible automatically execute those dependencies prior to executing the role which defined the dependencies. We prefer to be explicit in our playbooks, and simply define the role deps there.

[punkstarman]

@retr0h, this view runs counter to what almost every package manager I know does (pip, pipenv, npm, Ruby gems, apt, rpm, yum, Maven, Gradle, Ivy, Nuget, Haskell stack, Ansible Galaxy, ...). The only one I know of that forces a project to specify the full transitive graph of external dependencies is Bazel. This last tool does this because it assumes a Workspace view of the build which appears to be pervasive at Google. Bazel's view is a deal breaker from some projects.

While we can all begrudge dependency hell and downloading the Internet, the matter of the fact is that it is desirable to only have to specify immediate dependencies and have the system infer transitive dependencies. This allows abstraction.

If none of this has been convincing, my last argument would be that your view isn't aligned with the behaviour of ansible-galaxy -- which will read meta/main.yml and download any dependencies specified there -- or ansible-playbook -- which also will read meta/main.yml in roles and execute dependent roles.

[peschmae]

Not implementing this results in extremely confusing and confusing behaviour for molecule.

On one hand it relies on the developer specifying the role dependencies a second time (so you can't keep your code DRY), but since it then triggers a normal playbook the role will still be executed.

While I totally agree that the whole meta/main.yml setup for dependencies is not ideal, it would be preferable to have that behaviour be consistent with what ansible-galaxy and ansible-playbook do.

[artburkart]

I hadn't noticed this project is now managed by Red Hat. @gundalow, I see you made the updates to point to the https://github.com/ansible org ~5 months ago. Do you know if there's anyone who would be open to discussing this issue again and the possibility of bringing it more in line with ansible-galaxy and ansible-playbook? It seems it was given a pretty hard no from the beginning.

[dhoer]

@retr0h is there any plans to work with ansible-galaxy to fix this? I would be nice if meta/main.yml would allow for the following:

requirements:
  - requirement: requirements.yml

And it would only download the dependencies. I don't like the fact that requirements yaml today requires such tight coupling, where I need to know what is the name of yaml is.

[lopf]

I came up with this rather hacky workaround in prepare.yml the other day:

---
- name: prepare roles
  hosts: all
  tasks:
    - include_vars: ../../meta/main.yml

    - name: install roles from git
      shell: "ansible-galaxy install {{ item.scm |default('git')}}+{{ item.src }},{{ item.version }} --force"
      loop: "{{ dependencies|flatten(levels=1) }}"
      when:
        - item.src is defined
        - item.scm is defined
        - item.scm == 'git'
      delegate_to: localhost
      run_once: true

    - name: install roles from galaxy
      shell: "ansible-galaxy install {{ item.role }},{{ item.version }} --force"
      loop: "{{ dependencies|flatten(levels=1) }}"
      when:
        - item.role is defined
        - item.src is undefined
      delegate_to: localhost
      run_once: true

Still wondering what "meta-hell" people are referring to. Any hints?

[punkstarman]

Still wondering what "meta-hell" people are referring to. Any hints?

It is another term for "dependency-hell". Some people seem to be allergic to dependencies, especially transitive ones.