Molecule: Test running on the wrong VM with multiple-instance and Vagrant

[robsonpeixoto]

Issue Type

• Bug report

Molecule and Ansible details

ansible --version

ansible 2.6.1.post0 (stable-2.6 8c829778d1) last updated 2018/07/17 21:18:01 (GMT -300)
  config file = /Users/robinho/.ansible.cfg
  configured module search path = [u'/Users/robinho/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /Users/robinho/app/ansible/src/lib/ansible
  executable location = /Users/robinho/app/ansible/venv/bin/ansible
  python version = 2.7.15 (default, Jun 17 2018, 12:46:58) [GCC 4.2.1 Compatible Apple LLVM 9.1.0 (clang-902.0.39.2)]

molecule --version

molecule, version 2.16.0

Molecule installation method (one of):

• pip

Ansible installation method (one of):

• source

Desired Behavior

Run the test on the right virtual machine

Actual Behaviour (Bug report only)

Hi guys!

I'm using molecule + vagrant to test a ansible module.
I'd like to test it on Ubuntu 16.04 and 18.04.

So, my molecule.yml is:

---
dependency:
  name: galaxy
driver:
  name: vagrant
  provider:
    name: virtualbox
lint:
  name: yamllint
platforms:
  - name: ubuntu-xenial64
    box: ubuntu/xenial64
  - name: ubuntu-bionic64
    box: ubuntu/bionic64
provisioner:
  name: ansible
  lint:
    name: ansible-lint
scenario:
  name: default
verifier:
  name: testinfra
  lint:
    name: flake8

I got a fail on the test and I try to debug. As pdb doesn't work,
I add an assert forcing the failure after run the command.

def test_hostname_is_resolved(host):
    hostname_cmd = host.run('hostname -s')
    assert hostname_cmd.rc == 0
    assert hostname_cmd.stdout == ''

The output is:

___
_ test_hostname_is_resolved[ansible://ubuntu-bionic64] ____

    host = <testinfra.host.Host object at 0x109bbb550>, Ansible = <ansible>

        def test_hostname_is_resolved(host, Ansible):
            hostname_cmd = host.run('hostname -s')
            assert hostname_cmd.rc == 0
    >       assert hostname_cmd.stdout == ''
    E       AssertionError: assert 'ubuntu-xenial64' == ''
    E         - ubuntu-xenial64

    tests/test_default.py:40: AssertionError
    ____ test_hostname_is_resolved[ansible://ubuntu-xenial64] ____

    host = <testinfra.host.Host object at 0x109bbb8d0>, Ansible = <ansible>

        def test_hostname_is_resolved(host, Ansible):
            hostname_cmd = host.run('hostname -s')
            assert hostname_cmd.rc == 0
    >       assert hostname_cmd.stdout == ''
    E       AssertionError: assert 'ubuntu-bionic64' == ''
    E         - ubuntu-bionic64

So, the bionic64 tests are running on xenial64.
And the xenial64 tests are running on bionic64.

But the login works well:

$ molecule login --host ubuntu-xenial64
--> Validating schema .../molecule/default/molecule.yml.
Validation completed successfully.
Warning: Permanently added '[127.0.0.1]:2222' (ECDSA) to the list of known hosts.
...
vagrant@ubuntu-xenial64:~$ hostname -s
ubuntu-xenial64

$ molecule login --host ubuntu-bionic64
--> Validating schema .../molecule/default/molecule.yml.
Validation completed successfully.
Warning: Permanently added '[127.0.0.1]:2200' (ECDSA) to the list of known hosts.
vagrant@ubuntu-bionic64:~$ hostname -s
ubuntu-bionic64

Has anyone got this error? How to solve it?

Thanks

Below has the output of molecule --debug test for the toy project https://github.com/robsonpeixoto/molecule-vagrant-bug.

https://gist.github.com/robsonpeixoto/bab49bb305b865e839f2b780b6a400f2

[retr0h]

Can you gist your test files.

[robsonpeixoto]

@retr0h the module are here https://github.com/robsonpeixoto/molecule-vagrant-bug with all tests

[retr0h]

Hi @robsonpeixoto the repo is empty.

[robsonpeixoto]

My bad. I’ll push it.

[robsonpeixoto]

Done @retr0h

[robsonpeixoto]

I tested with python 2.7 and 3.6. Booth with the same bug

[retr0h]

Your tests are configured to run across all systems.

Your test then asserts the hostname is an empty string, which of course will fail. Your tests also make no distinction between nodes.

You want to do something similar to the following. Where the tests target a particular instance, so that you can assert the expected outcome.

[robsonpeixoto]

Your test then asserts the hostname is an empty string, which of course will fail.

I did it just to show that, for some reason, the host.run('hostname -s')is running on the "wrong" VM.

Your tests also make no distinction between nodes.

I didn't like to make a distinction between nodes. My real role is configuring a consul role with dnsmasq and I'd like to test the name resolution. I saw a weird behalf and I'm reporting it here.

Your tests also make no distinction between nodes.

IMHO the ansible://ubuntu-bionic64 should return address the right virtual machine.
In the example below the test hostname -s are returning ubuntu-xenial64 but should return ubuntu-bionic64.

Or does molecule not guarantee it? Did I use it wrong?

    __________________ test_hosts_file[ansible://ubuntu-bionic64] __________________

    host = <testinfra.host.Host object at 0x10ae2f250>

        def test_hosts_file(host):
            hostname = host.run('hostname -s').stdout
    >       assert hostname == ''
    E       AssertionError: assert 'ubuntu-xenial64' == ''
    E         - ubuntu-xenial64

[retr0h]

I receive the following, which seems to make sense. 1 of 2 failures where ubuntu-xenial64 does not match ubuntu-bionic64.

    =================================== FAILURES ===================================
    __________________ test_hosts_file[ansible://ubuntu-xenial64] __________________

    host = <testinfra.host.Host object at 0x1119bde90>

        def test_hosts_file(host):
            hostname = host.run('hostname -s').stdout
    >       assert 'ubuntu-xenial64' == hostname
    E       AssertionError: assert 'ubuntu-xenial64' == 'ubuntu-bionic64'
    E         - ubuntu-xenial64
    E         + ubuntu-bionic64

    tests/test_default.py:20: AssertionError
    ====================== 1 failed, 1 passed in 2.82 seconds ======================```

tests/test_default.py::test_hosts_file[ansible://ubuntu-bionic64] PASSED [ 50%]
tests/test_default.py::test_hosts_file[ansible://ubuntu-xenial64] FAILED [100%]```

[retr0h]

IMHO the ansible://ubuntu-bionic64 should return address the right virtual machine.
In the example below the test hostname -s are returning ubuntu-xenial64 but should return ubuntu-bionic64.

How can it? You are running the same exact test against 2 different hosts. You will always have 50% failure.

[robsonpeixoto]

@retr0h Could it be a testinfra bug, as the molecule just give the inventory file to the testinfra? I started to read the molecule code today.

Why on ansible://ubuntu-bionic64 the command hostname -s are returning ubuntu-xenial64?

[retr0h]

@retr0h Could it be a testinfra bug, as the molecule just give the inventory file to the testinfra? I started to read the molecule code today.

It might be, I'll look further to confirm.

[retr0h]

Looks to be a testinfra 1.13.1 bug. Just opened.

[robsonpeixoto]

@retr0h And looks be a bug with python2. I tested here with

• 2.7.15: the bug happens
• 3.6.6: works fine

[robsonpeixoto]

I tested here and is working =D
Thanks @retr0h

[retr0h]

Will close when testinfra 1.14.1 lands.