Packer: Digitalocean builder timeout

[jsaletan]

I'm using packer 0.10.1 on Ubuntu 16.04 x86_64. I'm able to create a snapshot using the digitalocean builder, but the build times out and returns an error. No artifact ID is returned for the created snapshot.

==> digitalocean: Error waiting for snapshot to complete: Timeout while waiting to for droplet to become 'active'.

Example template and debug log:
https://gist.github.com/jsaletan/b2bc8bd266d163de07e86ea0f222d3fd

Build with:
packer build -var 'do_api_token=<API_TOKEN>' build.json

[rickard-von-essen]

Try to increase the state timeout, see
https://www.packer.io/docs/builders/digitalocean.html#state_timeout

On Sep 2, 2016 23:34, "jsaletan" [email protected] wrote:

I'm using packer 0.10.1 on Ubuntu 16.04 x86_64. I'm able to create a
snapshot using the digitalocean builder, but the build times out and
returns an error. No artifact ID is returned for the created snapshot.

==> digitalocean: Error waiting for snapshot to complete: Timeout while
waiting to for droplet to become 'active'.

Example template and debug log:
https://gist.github.com/jsaletan/b2bc8bd266d163de07e86ea0f222d3fd

Build with:
packer build -var 'do_api_token=' build.json

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/mitchellh/packer/issues/3853, or mute the thread
https://github.com/notifications/unsubscribe-auth/AAiCg5J6b3eTT5Q3X9_Fy0Y3MVcTGXw6ks5qmJZggaJpZM4J0GuB
.

[jsaletan]

I tried increasing state_timeout from the default of 6m to 15m with the same result. I really don't think it's a problem with slow image creation, as it seems to complete in under a minute. Both the Digital Ocean web site and the API show successful image creation, and I've been able to create a droplet from an image right away. It would be nice if packer's logging exposed the API requests it was making, as that might pinpoint the issue.

[rickard-von-essen]

It would be great if you could check in the web ui what is the state of the snapshot or the cli. Just to verify that the error really is in Packer.

[jsaletan]

That's what I meant to convey in my last comment. Both the web UI and the API show successful image (snapshot) creation within 90 seconds of starting the build. The droplet from which the snapshot was created has a state of "off" according to the API, and the UI confirms that it is not powered on. Snapshots don't appear to have state. They simply exist or they don't. Once the snapshot has been created, packer begins checking the droplet's state according to the log. It seems packer expects it to be "active", but instead it is "off". I just tried manually powering on the new droplet before packer timed out and that caused the build to complete successfully. So it seems that packer expects the droplet to be active before it will destroy it and complete the build, but it is never made active.

[rickard-von-essen]

Thank you for your analysis. Seems like packer misses the active state.

[csainty]

This is biting me too.
I think there have been changes on the DigitalOcean side.

Previously they would restart a droplet after the snapshot was finished - https://www.digitalocean.com/community/questions/why-does-powered-off-droplet-get-started-after-snapshot

Packer seems to use this fact to know when the snapshot is done so it can clean up.

DigitalOcean recently enabled "live snapshots" so you do not need to power-down first - https://twitter.com/digitalocean/status/751140236298706947

My hypothesis is that they turned off the auto-restart feature now it isn't needed to get around the lack of online snapshots.

As mentioned a workaround at the moment is to manually start the droplet once you see the snapshot is complete in the UI then Packer continues.

[rickard-von-essen]

Thanks for that insight. Then it should be fairly easily reproducible.

[csainty]

I don't have time right now to make a PR of it, but I think I can see what needs doing.

Take the Action returned from starting the snapshot - https://github.com/mitchellh/packer/blob/master/builder/digitalocean/step_snapshot.go#L23

Add a new waitForActionState like the ones already defined in wait.go
But poll the Action status api instead - https://github.com/digitalocean/godo/blob/master/action.go#L19

[rickard-von-essen]

@csainty I blindley followed your instructions and it seems to fix the issue. Can you please review the changes in #3868. Maybe there are redundant waits or steps now.?

[kapaggar]

@csainty Your hypothesis was correct. manually starting packer instance pushes progress in waiting prompt of packer.

@rickard-von-essen Thanks for the fix. waiting for merge to come in packer release.
right now I am on Packer v0.10.2 and this bug is there.

[wmhilton]

I love it when I google my error ("Timeout while waiting to for droplet to become 'active'") and find the exact Github issue where my error gets fixed. Hahaha! (does happy it-wasn't-my-fault dance)

[jsaletan]

Confirmed working in 0.11.0. Thanks!