Molecule: Molecule test by default runs cleanup before destroy on test

[JoelBChapmanII]

Issue Type

• Bug report

Molecule and Ansible details

ansible 2.8.1
  config file = None
  configured module search path = ['/Users/user/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /Users/user/Envs/project/lib/python3.7/site-packages/ansible
  executable location = /Users/user/Envs/project/bin/ansible
  python version = 3.7.3 (default, Jun 25 2019, 11:10:25) [Clang 10.0.1 (clang-1001.0.46.4)]
molecule, version 2.20.1

Molecule installation method (one of):

• pip

Ansible installation method (one of):

• pip

Detail any linters or test runners used:

Desired Behavior

Molecule test runs like the following:

test_sequence:
    - lint
    - destroy
    - dependency
    - syntax
    - create
    - prepare
    - converge
    - idempotence
    - side_effect
    - verify
    - cleanup
    - destroy

Actual Behaviour

Molecule test is being ran like the following:

test_sequence:
    - lint
    - cleanup
    - destroy
    - dependency
    - syntax
    - create
    - prepare
    - converge
    - idempotence
    - side_effect
    - verify
    - cleanup
    - destroy

Molecule test should not run cleanup before VMs are created and converged. Molecule test fails because ansible can't run on vms that do not exist yet. This is using Vagrant/Virtualbox, I have not tested with Docker.

Molecule Test Log

```
--> Validating schema /Users/user/git/project/roles/common/molecule/default/molecule.yml.
Validation completed successfully.
--> Test matrix

└── default
├── lint
├── cleanup
├── destroy
├── dependency
├── syntax
├── create
├── prepare
├── converge
├── idempotence
├── side_effect
├── verify
├── cleanup
└── destroy

--> Inventory /Users/user/git/project/roles/common/molecule/default/../../../../development.yml linked to /var/folders/pl/rxj8k5v54017g9nnb34yxkr4tg3p52/T/molecule/common/default/inventory/hosts
--> Inventory /Users/user/git/project/roles/common/molecule/default/../../../../group_vars/ linked to /var/folders/pl/rxj8k5v54017g9nnb34yxkr4tg3p52/T/molecule/common/default/inventory/group_vars
--> Inventory /Users/user/git/project/roles/common/molecule/default/../../../../host_vars/ linked to /var/folders/pl/rxj8k5v54017g9nnb34yxkr4tg3p52/T/molecule/common/default/inventory/host_vars
--> Scenario: 'default'
--> Action: 'lint'
--> Executing Yamllint on files found in /Users/user/git/project/roles/common/...
Lint completed successfully.
--> Executing Flake8 on files found in /Users/user/git/project/roles/common/molecule/default/tests/...
Lint completed successfully.
--> Executing Ansible Lint on /Users/user/git/project/roles/common/molecule/default/playbook.yml...

Lint completed successfully.
--> Inventory /Users/user/git/project/roles/common/molecule/default/../../../../development.yml linked to /var/folders/pl/rxj8k5v54017g9nnb34yxkr4tg3p52/T/molecule/common/default/inventory/hosts
--> Inventory /Users/user/git/project/roles/common/molecule/default/../../../../group_vars/ linked to /var/folders/pl/rxj8k5v54017g9nnb34yxkr4tg3p52/T/molecule/common/default/inventory/group_vars
--> Inventory /Users/user/git/project/roles/common/molecule/default/../../../../host_vars/ linked to /var/folders/pl/rxj8k5v54017g9nnb34yxkr4tg3p52/T/molecule/common/default/inventory/host_vars
--> Scenario: 'default'
--> Action: 'cleanup'

PLAY [Converge] ****************************************************************

TASK [Gathering Facts] *********************************************************
fatal: [cos-7]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: ssh: Could not resolve hostname cos-7: nodename nor servname provided, or not known", "unreachable": true}
fatal: [u-1604]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: ssh: connect to host u-1604 port 22: Operation timed out", "unreachable": true}
fatal: [u-1804]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: ssh: connect to host u-1804 port 22: Operation timed out", "unreachable": true}

PLAY RECAP *********************************************************************
cos-7             : ok=0    changed=0    unreachable=1    failed=0    skipped=0    rescued=0    ignored=0
u-1604            : ok=0    changed=0    unreachable=1    failed=0    skipped=0    rescued=0    ignored=0
u-1804            : ok=0    changed=0    unreachable=1    failed=0    skipped=0    rescued=0    ignored=0

ERROR:
An error occurred during the test sequence action: 'cleanup'. Cleaning up.
--> Inventory /Users/user/git/project/roles/common/molecule/default/../../../../development.yml linked to /var/folders/pl/rxj8k5v54017g9nnb34yxkr4tg3p52/T/molecule/common/default/inventory/hosts
--> Inventory /Users/user/git/project/roles/common/molecule/default/../../../../group_vars/ linked to /var/folders/pl/rxj8k5v54017g9nnb34yxkr4tg3p52/T/molecule/common/default/inventory/group_vars
--> Inventory /Users/user/git/project/roles/common/molecule/default/../../../../host_vars/ linked to /var/folders/pl/rxj8k5v54017g9nnb34yxkr4tg3p52/T/molecule/common/default/inventory/host_vars
--> Scenario: 'default'
--> Action: 'destroy'

PLAY [Destroy] *****************************************************************

TASK [Destroy molecule instance(s)] ********************************************
ok: [localhost] => (item=None)
ok: [localhost] => (item=None)
ok: [localhost] => (item=None)
ok: [localhost]

TASK [Populate instance config] ************************************************
ok: [localhost]

TASK [Dump instance config] ****************************************************
skipping: [localhost]

PLAY RECAP *********************************************************************
localhost                  : ok=2    changed=0    unreachable=0    failed=0    skipped=1    rescued=0    ignored=0

[decentral1se]

From https://molecule.readthedocs.io/en/latest/configuration.html:

The cleanup step is executed directly before every destroy step. Just like the destroy step, it will be run twice. An initial clean before converge and then a clean before the last destroy step. This means that the cleanup playbook must handle failures to cleanup resources which have not been created yet.

Our documentation isn't so discoverable but yes, this is all by design.

[JoelBChapmanII]

I've read that, but the way this is designed makes molecule test impossible to run with a cleanup playbook without modification to the molecule.yml file.

[decentral1se]

the way this is designed makes molecule test impossible to run with a cleanup playbook without modification to the molecule.yml file.

Explain please.

[JoelBChapmanII]

If molecule has not created my VMs yet it can't cleanup, causing the error

    TASK [Gathering Facts] *********************************************************
    fatal: [cos-7]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: ssh: Could not resolve hostname cos-7: nodename nor servname provided, or not known", "unreachable": true}
    fatal: [u-1604]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: ssh: connect to host u-1604 port 22: Operation timed out", "unreachable": true}
    fatal: [u-1804]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: ssh: connect to host u-1804 port 22: Operation timed out", "unreachable": true}

If I for instance ran molecule create then molecule test test would work since it would be able to connect to my existing vms, run cleanup, run destroy, then run the rest of the stack, but if I have not created my vms molecule cleanup will try to connect to non-existant vms and fail.

[decentral1se]

The documentation states that your cleanup.yml should handle the case when test resources are not available. Cleanup should not assume molecule managed resources. It's not for that. If you want to propose a design change, please raise another issue in a proposal form.

[JoelBChapmanII]

If this is the way this is designed to work that's fine, but are there examples of how to handle the case where resources aren't available?

[decentral1se]

Happily accepting documention PRs!

[Zempashi]

I end up with something like this:

- hosts: all
  gather_facts: false
  tasks:
    - wait_for_connection:
        timeout: 10
      register: wait_for_instances
      failed_when: false

    - name: 'cleanup'
      command: losetup -d /dev/loop0
      when: wait_for_instances.elapsed < 10

Sadly I have to sacrifice 10 seconds to know whether instances has been spawned yet or not.

[JoelBChapmanII]

Thanks @Zempashi , I think for now I will just force test to skip the first cleanup by using the test_scenario override here.

[jghal]

I'm presently using when: ansible_host != inventory_hostname on tasks in cleanup.yml and that seems to working as ansible_host is populated in the inventory file when the host is created.

[Fixmetal]

I landed here because had the same problem, but with RHEL images on Docker.

My scenario is as follows:

• I use the docker driver
• Role must be tested on RHEL
• Preparation needs some packages thus needing a subscription
• I didn't want to manually remove subscriptions from RH portal

My previous setup was to subscribe the container with the prepare.yml and remove the subscription at the end of converge.yml in a block/always fashion.
This way I was losing idempotence test though. Since both cleanup/destroy are done BEFORE AND AFTER everything it would just not be enough to unsubscribe in those steps.

TL;DR:

EDIT
A better way which doesn't imply using docker_container_info (which is not available in Ansible 2.7 - and couldn't find a way to handle with 2.7) is to use the wait_for_connection module, which I wasn't aware of.

$ cat molecule/default/destroy.yml 
---
- name: Destroy
  hosts: all
  gather_facts: no
  tasks:
    - name: Block
      block:
        - name: Wait for containers to be up
          wait_for_connection:
            delay: 1
            timeout: 2
          register: connection
          ignore_errors: yes

        - name: Containers are not up, quit from here
          fail:
          when: connection['failed']

        - name: Gather facts
          setup:
            gather_subset:
              - '!all'
              - '!any'
              - distribution

        - name: Unregister system
          redhat_subscription:
            state: absent
          when: ansible_facts['distribution'] == "RedHat"
      rescue:
        - name: It's ok we're at startup
          meta: noop

ORIGINAL POST

$ cat molecule/default/prepare.yml 
---
- name: Prepare
  hosts: all
  tasks:
    - name: Register container
      redhat_subscription:
         state: present
         username: "{{ rhn_username | default('default-username-if-any', true) }}"
         password: "{{ rhn_password | default('default-password-if-any', true) }}"
         auto_attach: true
      when: ansible_facts['distribution'] == "RedHat"
      vars:
        rhn_username: overridden-username
        rhn_password: overridden-username

$ cat molecule/default/destroy.yml 
---
- name: Destroy
  hosts: all
  gather_facts: no
  tasks:
    - name: Block
      block:
        - name: Check if containers are up
          docker_container_info:
            name: "{{ item['name'] }}"
          register: molecule_test
          until:
            - molecule_test['exists']
            - molecule_test['container']['State']['Running']
          retries: 1
          delay: 1
          loop: "{{ molecule_yml['platforms'] | flatten(levels=1) }}"
          delegate_to: localhost

        - name: Gather facts
          setup:
            gather_subset:
              - '!all'
              - '!any'
              - distribution

        - name: Unregister system
          redhat_subscription:
            state: absent
          when: ansible_facts['distribution'] == "RedHat"
      rescue:
        - name: It's ok we're at startup
          meta: noop

I even made a Dockerfile.j2 template to have an RHEL container image with custom packages made only during tests:

$ cat molecule/default/Dockerfile.j2 
# Molecule managed

{% if item.registry is defined %}
FROM {{ item.registry.url }}/{{ item.image }}
{% else %}
FROM {{ item.image }}
{% endif %}

{% if item.env is defined %}
{% for var, value in item.env.items() %}
{% if value %}
ENV {{ var }} {{ value }}
{% endif %}
{% endfor %}
{% endif %}

ARG RHN_USER="YOUR USER"
ARG RHN_PWD="YOUR PASSWORD"

RUN if [ $(command -v apt-get) ]; then apt-get update && apt-get install -y python sudo bash ca-certificates iproute2 && apt-get clean; \
    elif [ $(command -v dnf) ]; then dnf makecache && dnf --assumeyes install python sudo python-devel python*-dnf bash iproute && dnf clean all; \
    elif [ $(command -v yum) ]; then grep -w "Red Hat Enterprise Linux" /etc/redhat-release  && \
      if [ $? -eq 0 ]; then \
        subscription-manager register --username="${RHN_USER}" --password="${RHN_PWD}" && \
        subscription-manager attach --auto && \
        yum makecache fast && \
        yum install -y \
          <needed-packages> && \
        yum clean all && \
        subscription-manager unregister && \
      else \
        yum makecache fast && yum install -y python sudo yum-plugin-ovl bash iproute && sed -i 's/plugins=0/plugins=1/g' /etc/yum.conf && yum clean all; fi; \
    elif [ $(command -v yum) ]; then yum makecache fast && yum install -y python sudo yum-plugin-ovl bash iproute && sed -i 's/plugins=0/plugins=1/g' /etc/yum.conf && yum clean all; \
    elif [ $(command -v zypper) ]; then zypper refresh && zypper install -y python sudo bash python-xml iproute2 && zypper clean -a; \
    elif [ $(command -v apk) ]; then apk update && apk add --no-cache python sudo bash ca-certificates; \
    elif [ $(command -v xbps-install) ]; then xbps-install -Syu && xbps-install -y python sudo bash ca-certificates iproute2 && xbps-remove -O; fi

All of this because I wanted to use the ansible_facts['distribution'] == "RedHat" and not checking for the content of /etc/redhat-release.

Hope this will help somebody else.