Packer: Packer hangs when I run Ansible Provisioner or ansible-playbook via shell provisioner.

[upodroid]

SUMMARY

Packer hangs when I run Ansible in verbose mode. This happens with ansible provisioner and shell provisioner.

ANSIBLE + Packer VERSION

==> googlecompute: Provisioning with shell script: /tmp/packer-shell165541510
    googlecompute: ansible 2.7.10
    googlecompute:   config file = None
    googlecompute:   configured module search path = ['/home/packer/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
    googlecompute:   ansible python module location = /usr/local/lib/python3.6/dist-packages/ansible
    googlecompute:   executable location = /usr/local/bin/ansible
    googlecompute:   python version = 3.6.7 (default, Oct 22 2018, 11:32:17) [GCC 8.2.0]

upo@helper-instance:~/ops-infra/packer$ packer --version
1.4.0

CONFIGURATION

• ansible playbook

- hosts: localhost
  become: yes
  tasks:

  - name: create Jenkins user
    user: 
      name: jenkins
      state: present
      shell: /bin/bash

  - name: Jenkins | Add Jenkins APT KEY
    apt_key:
      url: https://pkg.jenkins.io/debian/jenkins.io.key 
      state: present

  - name: Jenkins | Add repo to APT sources
    shell: "echo deb https://pkg.jenkins.io/debian-stable binary/ > /etc/apt/sources.list.d/jenkins.list"

  - name: Manual Apt update
    apt: 
      update_cache: yes

  - name: Install dependencies
    package:  
      name:
          - nginx
          - openjdk-8-jdk
          - make
          - git

  - name: Install Jenkins
    package:
      name: jenkins
      state: present

• packer.json

{
    "variables": {
        "zone": "europe-west2-b",
        "project_id": "PID",
        "region": "europe-west2"
      },

    "builders": [
      {
        "type": "googlecompute",
        "region": "{{user `region`}}",
        "zone": "{{user `zone`}}",
        "project_id" : "{{user `project_id`}}",
        "ssh_username": "packer",
        "source_image_family": "ubuntu-1804-lts",
        "preemptible": "true",
        "image_description": "Jenkins with (Java + Docker)",
        "disk_type": "pd-ssd",
        "disk_size": "10",
        "metadata": [
              {

              "block-project-ssh-keys":"false",
              "startup-script-url": "gs://pid-artefacts/init.sh"

              }],
        "image_name": "ubuntu1804-jenkins-{{isotime | clean_image_name}}",
          "image_family": "ubuntu1804-jenkins"

      }
    ],


    "provisioners": [
        {
          "type": "file",
          "source": "../ansible/jenkins.yml",
          "destination": "/tmp/jenkins.yml"
        },

        {
          "type": "shell",
          "inline": [" ansible --version && cd /tmp && ansible-playbook jenkins.yml -vvv"],
          "pause_before": "140s"
        }
      ]
  } 

OS / ENVIRONMENT

Google Compute Engine + Ubuntu 18.04 LTS

STEPS TO REPRODUCE

Build a packer image.

EXPECTED RESULTS

Here is a log without the verbose flag.

googlecompute output will be in this color.

==> googlecompute: Checking image does not exist...
==> googlecompute: Creating temporary SSH key for instance...
==> googlecompute: Using image: ubuntu-1804-bionic-v20190404
==> googlecompute: Creating instance...
    googlecompute: Loading zone: europe-west2-b
    googlecompute: Loading machine type: n1-standard-1
    googlecompute: Requesting instance creation...
    googlecompute: Waiting for creation operation to complete...
    googlecompute: Instance has been created!
==> googlecompute: Waiting for the instance to become running...
    googlecompute: IP: 35.230.134.102
==> googlecompute: Using ssh communicator to connect: 35.230.134.102
==> googlecompute: Waiting for SSH to become available...
==> googlecompute: Connected to SSH!
==> googlecompute: Uploading ../ansible/jenkins.yml => /tmp/jenkins.yml
jenkins.yml 741 B / 741 B [=============================================================================================================================================================================================================] 100.00% 0s
==> googlecompute: Pausing 2m20s before the next provisioner...
==> googlecompute: Provisioning with shell script: /tmp/packer-shell487567343
    googlecompute: ansible 2.7.10
    googlecompute:   config file = None
    googlecompute:   configured module search path = ['/home/packer/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
    googlecompute:   ansible python module location = /usr/local/lib/python3.6/dist-packages/ansible
    googlecompute:   executable location = /usr/local/bin/ansible
    googlecompute:   python version = 3.6.7 (default, Oct 22 2018, 11:32:17) [GCC 8.2.0]
==> googlecompute:  [WARNING]: Unable to parse /etc/ansible/hosts as an inventory source
==> googlecompute:  [WARNING]: No inventory was parsed, only implicit localhost is available
==> googlecompute:  [WARNING]: provided hosts list is empty, only localhost is available. Note
==> googlecompute: that the implicit localhost does not match 'all'
    googlecompute: PLAY [localhost] ***************************************************************
    googlecompute: TASK [Gathering Facts] *********************************************************
    googlecompute: ok: [localhost]
    googlecompute: TASK [create Jenkins user] *****************************************************
    googlecompute: changed: [localhost]
    googlecompute: TASK [Jenkins | Add Jenkins APT KEY] *******************************************
    googlecompute: changed: [localhost]
    googlecompute: TASK [Jenkins | Add repo to APT sources] ***************************************
    googlecompute: changed: [localhost]
    googlecompute: TASK [Manual Apt update] *******************************************************
    googlecompute: changed: [localhost]
    googlecompute: TASK [Install dependencies] ****************************************************
    googlecompute: changed: [localhost]
    googlecompute: TASK [Install Jenkins] *********************************************************
    googlecompute: changed: [localhost]
    googlecompute: PLAY RECAP *********************************************************************
    googlecompute: localhost                  : ok=7    changed=6    unreachable=0    failed=0
==> googlecompute: Deleting instance...
    googlecompute: Instance has been deleted!
==> googlecompute: Creating image...
==> googlecompute: Deleting disk...
    googlecompute: Disk has been deleted!
Build 'googlecompute' finished.

==> Builds finished. The artifacts of successful builds are:
--> googlecompute: A disk image was created: ubuntu1804-jenkins-2019-04-18t09-19-32z

I filed an issue with ansible but it works fine in with a startup script as I demonstrated in the case .
https://github.com/ansible/ansible/issues/55510

[upodroid]

https://gist.github.com/upodroid/1da201c7928cd78a43c847d52b72fb33

A Packer run with verbose flag. It is stuck on APT step and I had to cancel the build.

[SwampDragons]

This may be related to @AndrewCi's comment on #7506: https://github.com/hashicorp/packer/issues/7506#issuecomment-484535475

[upodroid]

Hi

Let me know if you have a binary with a potential fix and I'll give it a try.

[SwampDragons]

I don't yet.

[SwampDragons]

Did this happen in version 1.3.5, or is it a new bug in 1.4.0?

[upodroid]

I used 1.4.0.

I'll re run it with 1.3.5 and post the logs here.

[SwampDragons]

thanks -- also, how are you installing ansible? I want to try to reproduce exactly. Looks like you're calling packer on a linux instance?

[upodroid]

Yeah so I'm running Packer on a GCE Instance and i basically run the json file above and it spins up new GCE instance. There is an startup script I run before the provisioners are ran hence the 140s delay.

startup-script

#!/bin/bash
apt-get update
apt-get upgrade -y
unattended-upgrade -v
apt install unzip make git wget curl software-properties-common python-minimal python3-pip -y
pip3 install ansible
curl -L get.docker.com | bash

BTW, the build ran fine with Packer 1.3.5 so it is a bug that was introduced recently.

https://gist.github.com/upodroid/1da201c7928cd78a43c847d52b72fb33 The second file.

Thanks

[nferch]

I'm having a similar problem, with both 1.4.0 and 1.3.5.

running the Ansible provisioner with "-vvb" produces a hang on a lengthy apt-get upgrade task that produces a lot of output. Works fine with just "-b".

SSH'ing into the machine after the hang shows a stale sshd:

"""
321 ? Ss 0:00 /usr/sbin/sshd -D
398 ? Ss 0:00 _ sshd: packer [priv]
422 ? S 0:00 | _ sshd: packer@notty
"""

It does appear that the Ansible task successfully completed, but the Ansible Playbook and Packer are stuck and must be kill -9'd.

[SwampDragons]

@nferch your issue is probably different since the previous is a regression new in 1.4.0. Do you have an issue open for it already?

[SwampDragons]

Just an update here -- I am able to reproduce the hang but am still trying to find the specific source of the deadlock.

[upodroid]

Thanks for the update.

[SwampDragons]

I think this is closed in #7579; there are binaries available on that PR if you want to test one out.

[ChrisGWarp]

Has this issue re-emerged?
Packer 1.4.3 and Ansible 2.8.4 hang.
Downgrading to Ansible 2.7.10 works perfectly.
Anyone else see this behaviour, or am I special? :)

[SwampDragons]

@ChrisGWarp See https://github.com/hashicorp/packer/issues/7667 for details on the issue you're facing.

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.