Molecule: Extending existing sequences / add custom sequence

[danihodovic]

I would like to extend the default create.yml/destroy.yml sequence to have additional code. Is it possible to add a custom sequence in a scenario or to extend an existing scenario? I don't want to copy the code from the default scenario and add my own code towards the bottom of the file as this is unmaintainable in the long run.

[AlexisLessard]

I could see that. Maybe the playbooks could have an include task condition with a task file specified as an option? Therefore, you could just specify the tasks you want to add and maintain them yourself. Feels like the easiest way to implement this to me.

[ragingpastry]

I also have a usecase for this feature I believe.

Specifically, I want to run a "cleanup" playbook before the destroy step. Currently I don't think that is possible.

[AlexisLessard]

@ragingpastry I'm curious. Could you tell me a use case where you need a cleanup? Aren't roles tested in docker, and therefore, deleted at the end of the test suite?

[wilmardo]

You can extend the create.yml with a prepare.yml playbook. If I remember correctly the destroy.yml has no such thing but I could be wrong.
Might be an idea to add this?

[ragingpastry]

@AlexisLessard some of my tests reach out and touch things which are difficult to replicate in the molecule environment. These are things like AD servers or databases. Admittedly this probably isn't the most "correct" way to run these tests as ideally everything should be self contained, but it can be difficult to cram some of these systems into test. This is something I am working on but currently don't have a solution for.

One example would be a new user creation in ActiveDirectory. I have a development AD server which I hit with my new user molecule role. I'd like to have the molecule run delete the user on a cleanup step before deleting the server. Ideally I'd spin up an entire AD server as a part of the molecule test to make this more self contained, but that also has its drawbacks.

If you have any other suggestions I'd love to hear them!

EDIT:
I am almost done with a PR which would add functionality for a cleanup phase. This would be used as an opposite to the "prepare" phase. Is this something that would be accepted?

[decentral1se]

Excuse my copy/pasta but here's a relevant discussion from IRC:

14:47 < nwilburn> I'm looking at submitting a PR very soon to add a "cleanup" step which was discussed here: https://github.com/ansible/molecule/issues/1638. I was hoping to get some feedback first. Do the maintainers of molecule think that this has a chance of being accepted?
14:51 < decentral1se> nwilburn: having some custom step has typically solved by 'ejecting' and writing your own create/prepare/destroy.yml in your scenario configuration folder
14:51 < decentral1se> I recognize this is not ideal but this is how issues like this have been solved in the past
14:52 < nwilburn> Yea that is my current workaround. I think a cleanup step would mirror the prepare step nicely though. I understand that the number of steps could get out of hand pretty quickly though
14:54 < decentral1se> right, that's true and it is not set in stone that all these steps should exist in the future (but working nicely so far)
14:55 < decentral1se> What Can we get a import_* hook into the playbooks? Is this an easy way out?
14:55 < decentral1se> Molecule supports back to Ansible 2.4 and we have that https://docs.ansible.com/ansible/2.4/playbooks_reuse_includes.html
14:56 < decentral1se> We could add pre_/post task hooks on all existing steps even ...
14:59 < nwilburn> Sorry I got pulled away for a moment. I'm not sure I'm entirely tracking. Are you saying that instead of breaking out individual steps Molecule could instead move to a process where most steps are combined, and then
playbooks are dynamically included?
15:03 < decentral1se> combining, no but the dynamic part, yes. It's just an idea
15:03 < decentral1se> Taking the default docker create play: https://github.com/ansible/molecule/blob/d7397ca3ff09272c7918e78776b7dbc5e970251e/molecule/provisioner/ansible/playbooks/docker/create.yml
15:04 < decentral1se> it's just a play. Can we not add pre/post_tasks in this default, which try to dynamically include some name-by-convention plays?
15:05 < decentral1se> If molecule finds a pre-create.yml in the scenario folder, then it will import the tasks and run them otherwise, do nothing
15:05 < decentral1se> Is that clearer?
15:07 < nwilburn> Yes I think so. My initial concern with that would be how you would call those pre-create playbooks without running the create step. Or how you would call the pre-destroy playbook without running the full destroy. I'd
like for there to be a nice interface for this. Currently that interface is molecule command, so molecule prepare/whatever.
15:07 < nwilburn> It just makes it easier to iterate while building out your molecule testing. Once its done you don't gain that much benefit from the "nice interface" but its still nice to have =P
15:17 < decentral1se> nwilburn: makes sense!
15:18 < decentral1se> Looking at https://molecule.readthedocs.io/en/latest/configuration.html#scenario, there are probably a number of ways to solve this.
15:19 < decentral1se> It would be nice to not have sequences tied to commands which are hard-coded into the command line interface
15:19 < decentral1se> which would be a start to allow free form steps but would be a bit of work ...
15:21 < decentral1se> For now, I think this addition is fine but we should investigate other methods in the near future so we can avoid the necessity of adding entirely new commands when we want to customise
15:21 < decentral1se> I say, submit that PR! :)

[tace]

I need similar functionality. I need to do additional cleanup/destroy after bundled destroy.yml run. How to do that? I don't want to overwrite bundled destroy.yml or copy-paste it to my own destroy.

[decentral1se]

@tace, can you give https://github.com/ansible/molecule/pull/1676 a review pass and weigh in on if it fits your needs?

[tace]

I checked that review and looks good to me as such. Also it may suite to my needs.

However would it be easier to allow user to use bundled playbooks code as they wish?
Currently you either override create or destroy (by placing create.yml or destroy.yml to scenario dir) or use the default.

But I would like to place my own destroy.yml on (or create) to scenario dir and inside that playbook have access to the default destroy.yml playbook and run it where I want.
E.g. in my own destroy.yml

do my staff
- import_playbook: {{ lookup('env', 'MOLECULE_BUNDLED_PLAYBOOKS')/destroy.yml }}
do my other staff

So basically I just want to know the path where I can find molecule bundled default create/delete playbooks to able to utilize them.

[decentral1se]

Hey @tace thanks for the review.

I would recommend to review the available config env vars:

https://github.com/ansible/molecule/blob/d7e209fe63ffb33bf9e8a24cbd482e069bc75fac/molecule/config.py#L187-L206

Defaults are provided in https://github.com/ansible/molecule/tree/master/molecule/provisioner/ansible/playbooks. Maybe it's not easy to access those right now but I'd happily accept a PR or take guidance on the solution for that (and I could work on it at some point ...). I think your use case is interesting and something I'd like to see supported. Perhaps open another issue?

[Lirt]

Removed... Thank you for implementation.

I didn't notice this is only merged into master. Will this be out in v2.20? I'm +1 for this in 2.20 release.

[decentral1se]

@Lirt, yep! Pre-release is already out, see:

https://www.reddit.com/r/ansible/comments/ayetme/molecule_v220_prerelease_announcement/

[fourstepper]

@Lirt @decentral1se I suppose this issue could be closed?

[Lirt]

@fourstepper Yes, thank you.

[fourstepper]

@ssbarnea This issue can be closed :)