Molecule: Molecule error : UNREACHABLE && Authentication or permission failure on tmp folder

[gabyfulchic]

Molecule error : UNREACHABLE && Authentication or permission failure on tmp folder

I've wrote all tests molecule for my Ansible roles using a local docker host.
And recently when i decided to add a scenario named remote/ in my molecule folders
to use a remote docker host, i encountered an error.
(i decided to use a remote docker host to centralize all docker activity)

Versionning :

python 2.7.16
ansible 2.7.10
ansible-lint 4.1.0
molecule 2.20.0

In my remote scenario, the molecule.yml looks like this :

---
dependency:
  name: galaxy
driver:
  name: docker
lint:
  name: yamllint
  options:
    config-file: ../../.yamllint
platforms:
  - name: molecule-bash-deb
    image: jrei/systemd-debian:9
    dockerfile: ../shared_files/Dockerfile.j2
    tty: true
    command: /lib/systemd/systemd
    docker_host: HIDDEN
    tls_verify: true
    ssl_version: TLSv1_2
    tls_hostname: HIDDEN
    tls_client_cert: HIDDEN
    tls_client_key: HIDDEN
    tls_ca_cert: HIDDEN
    userns_mode: host
    env:
      container: docker
  - name: molecule-bash-rhel
    image: jrei/systemd-centos:7
    dockerfile: ../shared_files/Dockerfile.j2
    tty: true
    command: /lib/systemd/systemd
    docker_host: HIDDEN
    tls_verify: true
    ssl_version: TLSv1_2
    tls_hostname: HIDDEN 
    tls_client_cert: HIDDEN
    tls_client_key: HIDDEN
    tls_ca_cert: HIDDEN
    userns_mode: host
    env:
      container: docker
provisioner:
  name: ansible
  log: true
  config_options:
    defaults:
      local_tmp: /tmp/.ansible-$USER/tmp
      remote_tmp: /tmp/.ansible-$USER/tmp
  playbooks:
    converge: ../shared_files/converge.yml
    create: ../../../../test/molecule/create.yml
    destroy: ../../../../test/molecule/destroy.yml
  # prepare: ../../../../test/molecule/prepare.yml
  # side_effect: ../../../../test/molecule/side_effect.yml
  inventory:
    links:
      host_vars: ../../../../inventories/molecule/host_vars
  #   group_vars: ../../../../inventories/molecule/group_vars
  lint:
    name: ansible-lint
scenario:
  name: remote
  test_sequence:
    - lint
    - destroy
    - dependency
    - syntax
    - create
    - prepare
    - converge
    - idempotence
    - side_effect
    - verify
    - destroy
verifier:
  name: testinfra
  directory: ../shared_files/tests/
  options:
    vvv: true
  lint:
    name: flake8

The error is the following : "molecule converge -s remote"

fatal: [molecule-bash-rhel]: UNREACHABLE! => {
        "changed": false, 
        "msg": "Authentication or permission failure. In some cases, you may have been able to authenticate and did not have permissions on the target directory. Consider changing the remote tmp path in ansible.cfg to a path rooted in \"/tmp\". Failed command was: ( umask 77 && mkdir -p \"` echo /tmp/.ansible-gabyf/tmp/ansible-tmp-1576674115.03-57049687540370 `\" && echo ansible-tmp-1576674115.03-57049687540370=\"` echo /tmp/.ansible-gabyf/tmp/ansible-tmp-1576674115.03-57049687540370 `\" ), exited with result 1", 
        "unreachable": true
    }
    fatal: [molecule-bash-deb]: UNREACHABLE! => {
        "changed": false, 
        "msg": "Authentication or permission failure. In some cases, you may have been able to authenticate and did not have permissions on the target directory. Consider changing the remote tmp path in ansible.cfg to a path rooted in \"/tmp\". Failed command was: ( umask 77 && mkdir -p \"` echo /tmp/.ansible-gabyf/tmp/ansible-tmp-1576674115.04-254934152569179 `\" && echo ansible-tmp-1576674115.04-254934152569179=\"` echo /tmp/.ansible-gabyf/tmp/ansible-tmp-1576674115.04-254934152569179 `\" ), exited with result 1", 
        "unreachable": true
    }

Some details
I'm using 2 containers in the same scenario to make my test accross differents distribution,
and when the bug appears i well see the containers running on the remote host.
When i'm trying the command returning non-zero code on the molecule error message above, it well works. It works in local for local tmp, and in remote on the containers side.
the error command working for me when i do it in container :

umask 77 && mkdir -p "echo /tmp/.ansible-gabyf/tmp/ansible-tmp-1576674115.04-254934152569179"

I tried to molecule destroy -s remote before to molecule converge again, and i've got the same result.

i searched a lot on github issues, and i've followed this links without any solution for my use case.

• https://github.com/ansible/molecule/issues/1412
• https://github.com/ansible/molecule/issues/1043
• https://github.com/ansible/molecule/issues/1690

[gabyfulchic]

Ok i've found a provisional solution.
Using ENV_VARS and taking molecule ENV_VARS, all works good :

#!/bin/bash

# ./molecule.sh role

export DOCKER_HOST=${MOLECULE_DOCKER_HOST:-"tcp://toto.com:2376"}
export DOCKER_TLS_VERIFY=${MOLECULE_DOCKER_TLS_VERIFY:-1}
export DOCKER_TLS_HOSTNAME=${MOLECULE_DOCKER_TLS_HOSTNAME:-"toto.com"}
export DOCKER_CERT_PATH=${MOLECULE_DOCKER_CERT_PATH:-"/etc/docker/ssl"}

cd "roles/$1" && molecule test

I think that the problem is only a scope problem. The variables are surely overidden somewhere or maybe just forgotten.
In the platforms step the vars are well set, so the containers are well created on the remote docker host, and when you reach the provisioner step, the vars take their defaults values.

[triwats]

Jut an FYI I found this happens to me when I have a bad state, either I've restarted or Docker has crashed (macOS). The remedy has been to do:

molecule destroy