Packer: Ansible remote provisioner error when used with Docker builder

[paolodedios]

Provisioning a docker image via the new ansible remote provisioner on packer 0.9.0 results in a fatal docker builder error. Ansible reports that the remote host is "UNREACHABLE", but the initial Ansible command to create temp directories reports back a syntax error.

docker: fatal: [default]: UNREACHABLE! => {"changed": false, "msg": "ERROR! Authentication or permission failure. In some cases, you may have been able to authenticate and did not have permissions on the remote directory. Consider changing the remote temp path in ansible.cfg to a path rooted in \"/tmp\". Failed command was: ( umask 22 && mkdir -p \"$( echo $HOME/.ansible/tmp/ansible-tmp-1456343039.36-36812161460300 )\" && echo \"$( echo $HOME/.ansible/tmp/ansible-tmp-1456343039.36-36812161460300 )\" ), exited with result 1: /bin/sh: line 1: ((: umask 22 && mkdir -p /root/.ansible/tmp/ansible-tmp-1456343039.36-36812161460300 && echo /root/.ansible/tmp/ansible-tmp-1456343039.36-36812161460300 : syntax error in expression (error token is \"22 && mkdir -p /root/.ansible/tmp/ansible-tmp-1456343039.36-36812161460300 && echo /root/.ansible/tmp/ansible-tmp-1456343039.36-36812161460300 \")\n", "unreachable": true}

The Ansible remote provisioner is configured as follows and the docker builder was run through packer 0.9.0 on CentOS 7.

    {
            "type" : "ansible",
            "ansible_env_vars" : [
                "ANSIBLE_HOST_KEY_CHECKING=False",
                "ANSIBLE_SSH_ARGS='-v -o ControlMaster=auto -o ControlPersist=15m'"
            ],
            "extra_arguments" : [
                "-vvv",
                "--tags='base'"
            ],
            "playbook_file" : "provisioners/ansible/playbooks/images.yml"
        }

The entire Ansible remote provisioning step with PACKER_LOG=1 is included below.

2016/02/24 19:43:58 ui: ==> docker: Provisioning with Ansible...
==> docker: Provisioning with Ansible...
==> docker: SSH proxy: serving on 127.0.0.1:58134
2016/02/24 19:43:58 ui: ==> docker: SSH proxy: serving on 127.0.0.1:58134
2016/02/24 19:43:58 ui: ==> docker: Executing Ansible: ansible-playbook /shared/packer/provisioners/ansible/playbooks/images.yml -i /tmp/packer-provisioner-ansible434615222 --private-key /tmp/ansible-key385111795 -vvv --tags='base'
==> docker: Executing Ansible: ansible-playbook /shared/packer/provisioners/ansible/playbooks/images.yml -i /tmp/packer-provisioner-ansible434615222 --private-key /tmp/ansible-key385111795 -vvv --tags='base'
    docker: No config file found; using defaults
2016/02/24 19:43:59 ui:     docker: No config file found; using defaults
2016/02/24 19:43:59 ui:     docker: 1 plays in /shared/packer/provisioners/ansible/playbooks/images.yml
    docker: 1 plays in /shared/packer/provisioners/ansible/playbooks/images.yml
2016/02/24 19:43:59 ui:     docker:
    docker:
    docker: PLAY [Base image containerizer] ************************************************
2016/02/24 19:43:59 ui:     docker: PLAY [Base image containerizer] ************************************************
2016/02/24 19:43:59 ui:     docker:
    docker:
2016/02/24 19:43:59 ui:     docker: TASK [setup] *******************************************************************
    docker: TASK [setup] *******************************************************************
2016/02/24 19:43:59 ui:     docker: <127.0.0.1> ESTABLISH SSH CONNECTION FOR USER: packer-ansible
    docker: <127.0.0.1> ESTABLISH SSH CONNECTION FOR USER: packer-ansible
2016/02/24 19:43:59 ui:     docker: <127.0.0.1> SSH: EXEC ssh -C -q -v -o ControlMaster=auto -o ControlPersist=15m -o StrictHostKeyChecking=no -o Port=58134 -o 'IdentityFile="/tmp/ansible-key385111795"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=packer-ansible -o ConnectTimeout=10 -o ControlPath=/home/vagrant/.ansible/cp/ansible-ssh-%h-%p-%r -tt 127.0.0.1 '( umask 22 && mkdir -p "$( echo $HOME/.ansible/tmp/ansible-tmp-1456343039.36-36812161460300 )" && echo "$( echo $HOME/.ansible/tmp/ansible-tmp-1456343039.36-36812161460300 )" )'
    docker: <127.0.0.1> SSH: EXEC ssh -C -q -v -o ControlMaster=auto -o ControlPersist=15m -o StrictHostKeyChecking=no -o Port=58134 -o 'IdentityFile="/tmp/ansible-key385111795"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=packer-ansible -o ConnectTimeout=10 -o ControlPath=/home/vagrant/.ansible/cp/ansible-ssh-%h-%p-%r -tt 127.0.0.1 '( umask 22 && mkdir -p "$( echo $HOME/.ansible/tmp/ansible-tmp-1456343039.36-36812161460300 )" && echo "$( echo $HOME/.ansible/tmp/ansible-tmp-1456343039.36-36812161460300 )" )'
2016/02/24 19:43:59 ui:     docker: SSH proxy: accepted connection
    docker: SSH proxy: accepted connection
==> docker: authentication attempt from 127.0.0.1:50288 to 127.0.0.1:58134 as packer-ansible using none
2016/02/24 19:43:59 ui: ==> docker: authentication attempt from 127.0.0.1:50288 to 127.0.0.1:58134 as packer-ansible using none
2016/02/24 19:43:59 ui: ==> docker: authentication attempt from 127.0.0.1:50288 to 127.0.0.1:58134 as packer-ansible using publickey
==> docker: authentication attempt from 127.0.0.1:50288 to 127.0.0.1:58134 as packer-ansible using publickey
2016/02/24 19:43:59 packer: 2016/02/24 19:43:59 Executing in container 1ff272335e43a27dc2aec79009d2cd21351d214ecb314f5c23078b7dc43a390c: "(( umask 22 && mkdir -p \"$( echo $HOME/.ansible/tmp/ansible-tmp-1456343039.36-36812161460300 )\" && echo \"$( echo $HOME/.ansible/tmp/ansible-tmp-1456343039.36-36812161460300 )\" )) >/packer-files/cmd565453115 2>&1; echo $? >/packer-files/cmd565453115-exit"
2016/02/24 19:44:01 packer: 2016/02/24 19:44:01 Waiting for exit code to appear for remote command...
2016/02/24 19:44:01 packer: 2016/02/24 19:44:01 Executed command exit status: 1
2016/02/24 19:44:03 [INFO] RPC client: Communicator ended with: 1
2016/02/24 19:44:03 [INFO] RPC endpoint: Communicator ended with: 1
2016/02/24 19:44:03 [INFO] 0 bytes written for 'stdin'
2016/02/24 19:44:03 packer: 2016/02/24 19:44:03 [INFO] RPC endpoint: Communicator ended with: 1
2016/02/24 19:44:03 [INFO] 362 bytes written for 'stdout'
2016/02/24 19:44:03 [INFO] 0 bytes written for 'stderr'
2016/02/24 19:44:03 packer: 2016/02/24 19:44:03 [INFO] 362 bytes written for 'stdout'
2016/02/24 19:44:03 packer: 2016/02/24 19:44:03 [INFO] 0 bytes written for 'stderr'
2016/02/24 19:44:03 packer: 2016/02/24 19:44:03 [INFO] RPC client: Communicator ended with: 1
2016/02/24 19:44:03 packer: 2016/02/24 19:44:03 [INFO] 0 bytes written for 'stdin'
2016/02/24 19:44:03 ui:     docker: fatal: [default]: UNREACHABLE! => {"changed": false, "msg": "ERROR! Authentication or permission failure. In some cases, you may have been able to authenticate and did not have permissions on the remote directory. Consider changing the remote temp path in ansible.cfg to a path rooted in \"/tmp\". Failed command was: ( umask 22 && mkdir -p \"$( echo $HOME/.ansible/tmp/ansible-tmp-1456343039.36-36812161460300 )\" && echo \"$( echo $HOME/.ansible/tmp/ansible-tmp-1456343039.36-36812161460300 )\" ), exited with result 1: /bin/sh: line 1: ((: umask 22 && mkdir -p /root/.ansible/tmp/ansible-tmp-1456343039.36-36812161460300 && echo /root/.ansible/tmp/ansible-tmp-1456343039.36-36812161460300 : syntax error in expression (error token is \"22 && mkdir -p /root/.ansible/tmp/ansible-tmp-1456343039.36-36812161460300 && echo /root/.ansible/tmp/ansible-tmp-1456343039.36-36812161460300 \")\n", "unreachable": true}
2016/02/24 19:44:03 ui:     docker:
2016/02/24 19:44:03 ui:     docker: PLAY RECAP *********************************************************************
    docker: fatal: [default]: UNREACHABLE! => {"changed": false, "msg": "ERROR! Authentication or permission failure. In some cases, you may have been able to authenticate and did not have permissions on the remote directory. Consider changing the remote temp path in ansible.cfg to a path rooted in \"/tmp\". Failed command was: ( umask 22 && mkdir -p \"$( echo $HOME/.ansible/tmp/ansible-tmp-1456343039.36-36812161460300 )\" && echo \"$( echo $HOME/.ansible/tmp/ansible-tmp-1456343039.36-36812161460300 )\" ), exited with result 1: /bin/sh: line 1: ((: umask 22 && mkdir -p /root/.ansible/tmp/ansible-tmp-1456343039.36-36812161460300 && echo /root/.ansible/tmp/ansible-tmp-1456343039.36-36812161460300 : syntax error in expression (error token is \"22 && mkdir -p /root/.ansible/tmp/ansible-tmp-1456343039.36-36812161460300 && echo /root/.ansible/tmp/ansible-tmp-1456343039.36-36812161460300 \")\n", "unreachable": true}
    docker:
    docker: PLAY RECAP *********************************************************************
    docker: default                    : ok=0    changed=0    unreachable=1    failed=0
    docker:
2016/02/24 19:44:03 ui:     docker: default                    : ok=0    changed=0    unreachable=1    failed=0
2016/02/24 19:44:03 ui:     docker:
2016/02/24 19:44:03 ui: ==> docker: shutting down the SSH proxy
==> docker: shutting down the SSH proxy
2016/02/24 19:44:03 ui: ==> docker: Killing the container: 1ff272335e43a27dc2aec79009d2cd21351d214ecb314f5c23078b7dc43a390c
==> docker: Killing the container: 1ff272335e43a27dc2aec79009d2cd21351d214ecb314f5c23078b7dc43a390c
2016/02/24 19:44:04 ui error: Build 'docker' errored: Error executing Ansible: Non-zero exit status: exit status 3
2016/02/24 19:44:04 Builds completed. Waiting on interrupt barrier...
2016/02/24 19:44:04 machine readable: error-count []string{"1"}
2016/02/24 19:44:04 ui error:
==> Some builds didn't complete successfully and had errors:
2016/02/24 19:44:04 machine readable: docker,error []string{"Error executing Ansible: Non-zero exit status: exit status 3"}
2016/02/24 19:44:04 ui error: --> docker: Error executing Ansible: Non-zero exit status: exit status 3
2016/02/24 19:44:04 ui:
==> Builds finished but no artifacts were created.
2016/02/24 19:44:04 waiting for all plugin processes to complete...
Build 'docker' errored: Error executing Ansible: Non-zero exit status: exit status 3
2016/02/24 19:44:04 /usr/local/bin/packer: plugin process exited

==> Some builds didn't complete successfully and had errors:
--> docker: Error executing Ansible: Non-zero exit status: exit status 3

==> Builds finished but no artifacts were created.
2016/02/24 19:44:04 /usr/local/bin/packer: plugin process exited
2016/02/24 19:44:04 /usr/local/bin/packer: plugin process exited
2016/02/24 19:44:04 /usr/local/bin/packer: plugin process exited
2016/02/24 19:44:04 /usr/local/bin/packer: plugin process exited

[cbednarski]

Thanks for reporting this. Can you provide a minimal repro (i.e. templates and configs?). I can't see whether you are running docker on a remote host or locally on linux, or what's inside your images playbook for example.

[paolodedios]

Attached is a test case that includes a packer configuration file and ansible playbook files that reproduce the issue I detailed above.

Extract the archive and just run:

packer build centos-7-x86_64-docker-base-image.json

using packer 0.9.0 and docker-1.10.0 on CentOS 7.2 to duplicate my test environment. The results are the same whether I run this on VMware or AWS.

gh-issue-3260-test-case.tar.gz

[rickard-von-essen]

@cbednarski I think when should just warn or fail if there is a ansible provisioner configured for docker builder since there is no reason that a docker container should be accessible on SSH and add some docs.

[cbednarski]

@rickard-von-essen I see your point and I think turning this into a validation error is a good fix for now.

I think people will expect this to work. I have no plans to work on this, though.

[jstnlef]

I just ran into this myself. I'm not sure I agree with the conclusion, however. Technically speaking, isn't it possible to do this? I ask because I'm not sure. It seems to me, as someone who appreciates build automation with Ansible and also appreciates smaller images which don't include downloading ansible into the running container to build the image, this should be possible but I appreciate if there are stumbling blocks preventing such a thing. Thanks.

[jstnlef]

I'm willing to contribute to a fix if possible.

[rickard-von-essen]

@jstnlef I think it's only partially possible (and maybe not very useful) since ansible talks SSH and Docker of course speaks docker API, so it would only work if you start a container running sshd.

[jstnlef]

Hrm. So it would seem that the fix should probably be on ansible's end if this were to happen, with minimal change here to use the new functionality.

[b11z]

Is this fixable by changing which connection driver ansible uses? Packer would have to activate the docker connection driver any time the builder is docker. This is a new feature in ansible since 2.0: the ability to tunnel its commands over docker exec instead of SSH. Ref: http://blog.oddbit.com/2015/10/13/ansible-20-the-docker-connection-driver/

[rickard-von-essen]

@b11z Interesting, the strange thin is that there is zero mentions about this in the official docs.

[bhcleek]

This should be technically possible now. I suspect the problem is either that a ptty is being allocated or else the docker container doesn't have an sftp subsystem available where the ansible provisioner expects it to be.

Just for everyone's clarification; the ansible provisioner does not require that sshd be run on the machine being provisioned; SSH is only used to communicate between Ansible and Packer, the ansible provisioner then translates SSH commands to Packer communicator commands to actually talk to the node.

[yveslaroche]

The Ansible remote provisioner with the Docker builder already works. You will need to change the ansible_connection from "ssh" to "docker" and set a Docker container name via the --name option. On a CI server you probably want to overwrite ansible_host with a random name.

Example Packer template:

{
  "variables": {
    "ansible_host": "default",
    "ansible_connection": "docker"
  },
  "builders":[
    {
      "type": "docker",
      "image": "centos:7",
      "commit": true,
      "run_command": [ "-d", "-i", "-t", "--name", "{{user `ansible_host`}}", "{{.Image}}", "/bin/bash" ]
    }
  ],
  "provisioners": [
    {
      "type": "ansible",
      "groups": [ "webserver" ],
      "playbook_file": "./webserver.yml",
      "extra_arguments": [
        "--extra-vars",
        "ansible_host={{user `ansible_host`}} ansible_connection={{user `ansible_connection`}}"
      ]
    }
  ]
}

Example playbook:

- name: configure webserver
  hosts: webserver
  tasks:
    - name: install Apache
      yum:
        name: httpd

Build:

❯ packer build centos-7.json
docker output will be in this color.

==> docker: Creating a temporary directory for sharing data...
==> docker: Pulling Docker image: centos:7
    docker: 7: Pulling from library/centos
    docker: a3ed95caeb02: Already exists
    docker: 196355c4b639: Already exists
    docker: a3ed95caeb02: Already exists
    docker: a3ed95caeb02: Already exists
    docker: Digest: sha256:3cdc0670fe9130ab3741b126cfac6d7720492dd2c1c8ae033dcd77d32855bab2
    docker: Status: Image is up to date for centos:7
==> docker: Starting docker container...
    docker: Run command: docker run -v /Users/yves.laroche/.packer.d/tmp/packer-docker381956357:/packer-files -d -i -t --name default centos:7 /bin/bash
    docker: Container ID: eb1901cbd823dd49971724f1949098635e203dd0c8acdc3683f263505789a458
==> docker: Provisioning with Ansible...
==> docker: SSH proxy: serving on 127.0.0.1:49810
==> docker: Executing Ansible: ansible-playbook /Users/yves.laroche/Desktop/packer-docker-test/webserver.yml -i /var/folders/yp/cfy1wf5x5pj7vmt67pk4_9wc0000gp/T/packer-provisioner-ansible342080785 --private-key /var/folders/yp/cfy1wf5x5pj7vmt67pk4_9wc0000gp/T/ansible-key877420026 --extra-vars ansible_host=default ansible_connection=docker
    docker:
    docker: PLAY [configure webserver] *****************************************************
    docker:
    docker: TASK [setup] *******************************************************************
    docker: ok: [default]
    docker:
    docker: TASK [install Apache] **********************************************************
    docker: changed: [default]
    docker:
    docker: PLAY RECAP *********************************************************************
    docker: default                    : ok=2    changed=1    unreachable=0    failed=0
    docker:
==> docker: shutting down the SSH proxy
==> docker: Committing the container
    docker: Image ID: sha256:072e3084834e626e9cf80d33c41cb58357db8e2aa7a644aa8e8e6581bc5b86e1
==> docker: Killing the container: eb1901cbd823dd49971724f1949098635e203dd0c8acdc3683f263505789a458
Build 'docker' finished.

==> Builds finished. The artifacts of successful builds are:
--> docker: Imported Docker image: sha256:072e3084834e626e9cf80d33c41cb58357db8e2aa7a644aa8e8e6581bc5b86e1

[jstnlef]

:+1: Excellent for this. It'd be great if this could end up making its way to the documentation :). Thanks.

[paolodedios]

Using the docker connection that ships with Ansible 2.0 works for me. Thank you @yveslaroche.

One thing to note, though, is that Ansible's docker connection implementation does not pass the "-t" flag to allocate a pseudo tty when translating ansible directives to execute via docker exec. Commands like sudo end up failing. I had to disable the "requiretty" setting in /etc/sudoers as a workaround. Hopefully they address this at some point.

[bhcleek]

I can confirm that there are problems with the ansible-remote provisioner and docker builder. I am working on a solution and hope to submit a PR in the coming days.

In the meantime, though, you should know that using Ansible's docker connection will work in some scenarios with Packer, but not all. Thankfully, the best supported scenario is also the primary use case; when the Docker host is accessible from the Packer host. But if you're using the amazon-instance builder to create a Docker image in coordination with the Artifice post-provisioner, I think you'll run into problems trying to use Ansible's docker connection.

[gamename]

A simple, elegant, explanation. Better than any I've seen @yveslaroche

Well done, sir

[AndydeCleyre]

I am getting the same problem, but specifying the docker connection and an image name as @yveslaroche suggests does not solve the problem. Here's my template.json:

{
  "builders": [
    {
      "type": "docker",
      "image": "ubuntu:xenial",
      "run_command": ["-d", "-i", "-t", "--name", "Project-{{timestamp}}", "{{.Image}}", "/bin/bash"],
      "export_path": "image.tar"
    }
  ],
  "provisioners": [
    {
      "type": "ansible",
      "playbook_file": "./playbook.yml",
      "extra_arguments": [
        "--vault-password-file", "vault_password",
        "--connection", "docker"
      ]
    }
  ]
}

packer-io 0.10.1
ansible 2.1.0.0
docker 1.11.1

[rickard-von-essen]

@AndydeCleyre What's your OS and if you run docker-machine which version and driver?

[AndydeCleyre]

@rickard-von-essen I'm using Arch Linux, and don't run docker-machine.

[emoshaya]

I have the same issue.. I get the following message:

Docker-Image: fatal: [updatesvc]: UNREACHABLE! => {"changed": false, "msg": "Authentication or permission failure. In some cases, you may have been able to authenticate and did not have permissions on the remote directory. Consider changing the remote temp path in ansible.cfg to a path rooted in "/tmp". Failed command was: ( umask 22 && mkdir -p "echo $HOME/.ansible/tmp/ansible-tmp-1464869785.18-101205323886806" && echo "echo $HOME/.ansible/tmp/ansible-tmp-1464869785.18-101205323886806" ), exited with result 1", "unreachable": true}

[benjumanji]

I have tried runnin _exactly_ the the packer template + playbook as suggested by @yveslaroche, unfortunately I get the same UNREACHABLE as @emoshaya-cognito.

docker: 1.11.1
packer: 0.10.1
ansible: 2.1.0.0

Any help on this would be hugely appreciated, or even some ideas about how to go about debugging it.

EDIT: I note that I am running an identical setup to @AndydeCleyre

[benjumanji]

I have found that adding ansible_user=root to extra-args seems to do the trick

[nullobject]

This issue seems to have regressed with packer 1.0.2

[rickard-von-essen]

Hi @nullobject, thanks for reaching out.

This ticket is really old so it's probably not relevant to the issue you're seeing. If you're still having trouble, would you please open a new issue with the debug log and a json that reproduces the issue? Thanks!

[phreddrick]

This is happening to me too, only not with the docker connect, I'm building a vmware-vmx machine from an ubuntu host with packer 1.0.3, it hangs on the ssh to the client machine. If I use packer 1.0.0, it works fine.

[SwampDragons]

@phreddrick there was a bug with ansible hanging that was introduced in 1.0.3, but 1.0.4 was released today and should fix it.

[phreddrick]

LOL.. I literally downloaded 1.0.3 today.. Thanks!

[nullobject]

1.0.4 doesn't make any difference for me 😿

[lucasyvas]

I found that everything is fine when running this from my droplet on DigitalOcean, but running Packer (1.0.2)/Ansible/Docker within a container (on CircleCI 2) was a nightmare. Probably because I don't understand it.

This post worked though:

https://github.com/hashicorp/packer/issues/3260#issuecomment-199233230

I did have to add the "user": "root" property to the provisioner though. I wish I understood more about to figure out if I could correct something in the environment to have it just work like it did in the droplet.

[rickard-von-essen]

@lucasyvas I created #5287 to address this. Feel free to open an issue if the are missing information in the docs or ask a question on the mailing list.

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.