Packer: Option for builder to wait on cloud-init to complete

[steve-jansen]

Hi Mitchell and team!

Similar to how Packer's AMI builder waits on an sshd in the target instance to become accessible, it would be awesome to have an option for the builder to also wait on cloud-init to complete inside the builder's EC2 instance.

Why? I had a Packer build in Atlas that would intermittently fail on apt-get commands, even when using the exact same Packer code and config (I repeatedly queued the same build in Atlas to prove it). I eventually realized I had a race condition, with Packer sometimes starting my provisioning script while cloud-init was still in flight. The official Ubuntu images use cloud-init to update the apt sources to local mirrors.

I added this bash hack to make it stable. Might be worth adding as an option to the builder(s):

#!/bin/bash -e
echo "waiting 180 seconds for cloud-init to update /etc/apt/sources.list"
timeout 180 /bin/bash -c \
  'until stat /var/lib/cloud/instance/boot-finished 2>/dev/null; do echo waiting ...; sleep 1; done'
echo "running apt-get update ..."
cat /etc/apt/sources.list
sudo -E apt-get update

Hope this helps someone else!
Steve

[cbednarski]

Thanks for opening a ticket. I'm not super familiar with cloud-init so I might get some of this wrong, but I think there are several approaches you could take to check for this. For instance:

• Make your first provisioner block until cloud-init is complete
• Use pause_before on your first provisioner
• Do the apt configuration in a base image or in a provisioner; don't do it in cloud-init on each build

If one of these options works for you I'd be happy to include it in the docs. I think it's hard to support as a feature since I have no idea what might be running in your cloud-init. I think in this case you as the operator are in the best position to assess when the machine is ready to provision.

[carlossg]

this is what I run as a provisioner in ubuntu

while ! grep "Cloud-init .* finished" /var/log/cloud-init.log; do
    echo "$(date -Ins) Waiting for cloud-init to finish"
    sleep 2
done

[grubernaut]

I've added documentation for this issue to the website in the following commit, https://github.com/mitchellh/packer/commit/6af2fd5bd08826e8e1a9d78afc017853db58150b.

There are also some sporadic issues with using amazon's apt mirror, but that's a separate issue entirely, and this issue can be closed

[steve-jansen]

:+1:

[jblachly]

Piling on to this closed ticket because I hit the same issue building an image for DigitalOcean.

Google does not rank the relevant documentation page ( https://www.packer.io/docs/other/debugging.html ) for the types of search terms I (others?) would use when trying to figure how why only a subset of packages got installed.

The suggestion to wait on /var/lib/cloud/instance/boot-finished indeed fixes it, but I would still advocate (as did the original issue opener) for a generic builder option to wait on cloud-init.

[gjenkins8]

To help peeps who find this issue, a direct solution for waiting for cloud-init looks like:

provisioners:
  - type: shell
    inline:
    - /usr/bin/cloud-init status --wait

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.