Packer: Make the --debug pausing after each step optional

[bobziuchkovski]

It would be wonderful if, by default, --debug only paused after an error is encountered. I can see where there are times it would be useful to pause after every step, but more often than not I'm only interested in pausing at the point packer fails.

Proposed:

--debug behaves exactly as it does now, except it only pauses when an error is encountered
--stepwise switch added to pause after every step, switch valid only if --debug is enabled

[joshuacox]

I think this can be accomplished by simply export PACKER_LOG=1, you can also set the path:

https://www.packer.io/docs/other/debugging.html

I think this provides most if not all of the information that debug does in the log, without the pauses.

[bobziuchkovski]

It provides the info, but it exits and cleans-up / removes the VMs on
error. The --debug flag allows access to the VM to troubleshoot before
clean up, but requires that you sit there to hit enter after each step.
It's this delayed cleanup I'm looking for, but without needing to stay
present and hit enter at each step. I think the --debug flag also saves
the temporary SSH keys used to spin up VMs on cloud providers, which is
also useful when trying to interactively debug a failed build.

The --debug flag does a number of useful things, and overall it works
great. However, the forced pausing means I have to babysit a build when I'm
only interested in accessing it after failure, prior to cleanup.
On Dec 3, 2014 7:25 AM, "Josh Cox" [email protected] wrote:

I think this can be accomplished by simply export PACKER_LOG=1, you can
also set the path:

https://www.packer.io/docs/other/debugging.html

I think this provides most if not all of the information that debug does
in the log, without the pauses.

—
Reply to this email directly or view it on GitHub
https://github.com/mitchellh/packer/issues/1687#issuecomment-65414392.

[joshuacox]

a quick and dirty alternative would be to add a shell script provisioner at the end with something like:

read -rsp $'Press any key to continue...\n' -n1 key

to cause it to hang.

[bobziuchkovski]

I don't think it would make it to that provisioner if it failed on a prior
provisioner, which would defeat the purpose.
On Dec 3, 2014 12:03 PM, "Josh Cox" [email protected] wrote:

a quick and dirty alternative would be to add a shell script provisioner
at the end with something like:

read -rsp $'Press any key to continue...n' -n1 key

to cause it to hang.

—
Reply to this email directly or view it on GitHub
https://github.com/mitchellh/packer/issues/1687#issuecomment-65467319.

[joshuacox]

Did you try this?

the shell provisioner specifically states that it respects order:

https://packer.io/docs/provisioners/shell.html

While I am not certain of whether or not packer respects order in provisioners [i.e. if you specify a shell provisioner after other provisioners will it be last?], I would suspect that it does. [looking at the source now, I think this should work]

[bobziuchkovski]

Yes, packer respects order. It also respects exit status of all of the
provisioners. If any of them exit non-zero, packer aborts the build and
cleans up. That is wonderful behavior. I suspect you might be using shell
provisioners exclusively and not returning non-zero on failure. If you did
that, your solution would work, but it would also fail to abort if and when
your scripts encountered error, which is not what I want.
On Dec 3, 2014 6:44 PM, "Josh Cox" [email protected] wrote:

Did you try this?

the shell provisioner specifically states that it respects order:

https://packer.io/docs/provisioners/shell.html

While I am not certain of whether or not packer respects order in
provisioners, I would suspect that it does. [looking at the source now, I
think this should work]

—
Reply to this email directly or view it on GitHub
https://github.com/mitchellh/packer/issues/1687#issuecomment-65524985.

[bobziuchkovski]

I also feel we're getting off topic. I think making the built-in --debug
pausing optional makes sense as a feature suggestion.

If nobody wants to make this change, then that's fine, but I think it's a
valid suggestion.
On Dec 3, 2014 9:07 PM, "Bob Ziuchkovski" bob.[email protected] wrote:

Yes, packer respects order. It also respects exit status of all of the
provisioners. If any of them exit non-zero, packer aborts the build and
cleans up. That is wonderful behavior. I suspect you might be using shell
provisioners exclusively and not returning non-zero on failure. If you did
that, your solution would work, but it would also fail to abort if and when
your scripts encountered error, which is not what I want.
On Dec 3, 2014 6:44 PM, "Josh Cox" [email protected] wrote:

Did you try this?

the shell provisioner specifically states that it respects order:

https://packer.io/docs/provisioners/shell.html

While I am not certain of whether or not packer respects order in
provisioners, I would suspect that it does. [looking at the source now, I
think this should work]

—
Reply to this email directly or view it on GitHub
https://github.com/mitchellh/packer/issues/1687#issuecomment-65524985.

[Mil0dV]

+1 for @ziuchkovski suggestion. Our packer builds often take 5-10mins, and launching an ec2 instance with the resulting AMI isn't that fast either. To improve our debug-workflow I would really like a 'fire and forget' debug mode which allows me to switch to something else and when I come back I can see the output and log in to the instance to determine the steps needed to get the desired result.

(Ideally I would have our rake task that fires packer log in to the instance for me, but that's also somewhat off topic. It also seems near-impossible to use the 'hang' @joshuacox suggested in this use-case)

[zettabyte]

I've been watching feature request #409 for close to two years now (not long after it was created). As many people have commented on that issue and as @Mil0dV and @ziuchkovski have requested here, we really do need a way to fire a build, come back later and if there's an error, have the opportunity to inspect the instance. So, here's another +1 for this (or #409).

[joshuacox]

Not to detract from your +1, but don't forget about setting
PACKER_LOG=1
I believe this gives all the detailed output you are going to see in debug mode without it halting at each of the steps, you just have to set an env var rather than supply an option. What would that option be named out of curiosity? -debug-nowait perhaps? I kind of prefer the method of just appending an enormous amount v's in a flag (i.e. -vvvvvvvvvvvvvvvvvvvvvvvv)

[joshuacox]

just a suggestion on perhaps a name and implied functionality:

--rm=false <--- perhaps would be useful, like in docker, and might take care @zettabyte and @ziuchkovski 's needs to inspect them

[zettabyte]

Yeah, bringing up PACKER_LOG is a red herring as far as I'm concerned. I specifically mentioned, "have the opportunity to inspect the instance" for a reason. The second comment on this issue is a suggestion to set PACKER_LOG followed up by a response explaining why that's not sufficient.

[mattolenik]

I'm also having difficulty debugging Packer provisioning failures. The debug logs do not provide enough useful information...I really need a way to have the VM left intact to inspect.

[ebelew]

Situation where I can't think of an alternative to a way to pause in the middle: Using packer:vmware-iso builder, I had added the tools_upload options on a windows guest. The upload appeared to be working, but 7zip failed to extract the archive.

I had to enable the head, log in on the head and open explorer while other provisioning was being done. Then opened explorer to watch the folder, and see that the file was 0bytes. However, this wasted a lot of time because I had to baby-sit the VM creation from start to finish.

The first suggestion is definitely what I was looking for when I found this problem. The log gave no real info except that 7zip failed due to a bad archive file.

[orivej]

https://github.com/orivej/multistep/commit/e02bce9811c65138ea2e84c7162cd8769f35858f implements @ziuchkovski's suggestion regarding behaviour of --debug, but it requires https://github.com/mitchellh/multistep/pull/5 to be useful.

[orivej]

Oops, wrong link, the patch for packer is at https://github.com/orivej/packer/tree/debug-on-error

[absolutejam]

And the ability to re-run a step if it errors!

It pains me to start a build, wait 20 minutes to see if the error that happened last time is still present (Which it usually is) and re-start the process 5 times (I'm in the midst of doing that now!).

[orivej]

https://github.com/orivej/packer/tree/debug-on-error-2 allows you to retry the failed step, although without further modifications the overall build is likely to fail even if retry succeeds.

Currently build steps are coarse-grained; in particular, the whole provisioners sections is a single step.

This branch is recent enough to incorporate #2608 (packer with built-in plugins), and to use it you have to delete previously installed plugin binaries (with names starting with packer-).

[mjperrone]

dropping into an ssh session on error would be super helpful to make my debug loop shorter.

[amosshapira]

+1 for any way to make it possible to ssh into an instance after a failed build. "--debug" is annoying not only because it requires pressing "Enter" every time but mainly because it sometimes silently requires more than one "Enter", requiring almost random pressing extra "Enter"'s until I see a response.

[orivej]

@amosshapira, reliable input handling can be restored by removing the call to setupStdin() from main.go. (https://github.com/mitchellh/packer/issues/409#issuecomment-161530252)

[jmound]

An option that would help would be continue-through-step $stepname such that packer build -debug would not pause until it reached that particular step. eg. packer build -debug --continue-through-step StepConnect would run packer in debug mode but not pause until hitting the StepConnect step.

I believe I saw this idea mentioned elsewhere, but can't find it at the moment.

[munjeli]

Well, I'd be a fan of don't destroy, like vagrant. I set the log var and the detailed output does help a lot, but for now my workaround is simply to put a long sleep in a shell provisioner before the failure and ssh into the box to run the failure by hand, which makes me feel silly, but works. I name the ssh key for the amazon-ebs builder so packer doesn't make its own.

[PeterMika]

For those who share my goal of compiling the latest packer dev release while also integrating orivej earlier fix that pauses on first fail of packer build here are the steps I took that worked for me.

1. Complete "Setting up Go to work on Parker" steps 1-4 . ( see https://github.com/mitchellh/packer/blob/master/CONTRIBUTING.md )
2. git checkout master
3. git remote add fork https://github.com/orivej/packer
4. git fetch fork
5. git checkout -b debug-on-error fork/debug-on-error
6. git merge debug-on-error
7. make
8. run ./bin/packer build -debug template.json

I can confirm that this worked for me and provisioning only paused when there was an error.

I was not able to successfully merge https://github.com/orivej/packer/tree/debug-on-error-2.

I'm curious, I'm fairly new to packer and git and this issue; is there some other way people have been implementing orivej's fixes then how I have described?

[jpza]

It can't merge because debug-on-error-2 is so far behind origin/master and there's bound to be conflicts. Also this fix needs to become a PR based on a current dev branch of packer. I'm taking a look now and will try to submit something useful. This is a much needed feature.

[rickard-von-essen]

Closing this since the --on-error flag added in 0.11.0 (#3885) should cover this use case.

[lklepner]

HI @rickard-von-essen

I don't think the --on-error flag addresses the issue the OP raised, which is to enter debug mode with a keyfile and IP address when an error is encountered.

As things are currently stand omitting the -debug flag skips the output of the IP address and download of the key file early on.. so even though the partial build is saved, accessing it is difficult if not impossible.

[rickard-von-essen]

Finding the IP is simple and for getting access I recommend to using a non temporary ssh key pair.

[erichs]

Hi, @rickard-von-essen!

With respect, using a non-temporary ssh key is a workaround. If setting the key to a known, probably shared key is a requirement to get tighter feedback loops during development, developers will do so, and are incentivized to check this key into the repo.

At this point, one must ask: what is the purpose of supporting temporary ssh keys if we can't use them effectively?

[rickard-von-essen]

The point of ephemeral temporary keys is that the are secure, they only exist in memory for the duration of the build.

In small organizations it's easy to have personal keypairs in AWS that you can reference in a user variable. In larger organizations centralized auth is inevitable. So getting access is not handled by Packer/AWS at all.

[erichs]

@rickard-von-essen, thanks for the response!

Granted: it's on me and my organization to provision SSH keys or other means of access. Possibly this provisioning happens at Packer build time, possibly it happens via some other bootstrapping with Chef or Ansible post-launch.

As I understand the OP, the request here is for behavior that, on error during Packer provisioning, uses the in-memory, ephemeral credentials to drop me in an SSH session automatically. As I understand it, the new --on-error flag does not support this. Please correct me if I'm wrong here, I'd love it if that were the case!

If that can't be arranged, the secondary ask is that the ephemeral credentials can be written out to disk, without having to manually step through all steps, as the -debug flag does currently. Basically, there's a common workflow that would really benefit from --on-error=ssh (preferred) or --on-error=write-pemfile (workable). As I see it, this common workflow isn't being supported, and this is contrary to the Tao of Hashicorp.

[rickard-von-essen]

As I understand the OP, the request here is for behavior that, on error during Packer provisioning, uses the in-memory, ephemeral credentials to drop me in an SSH session automatically. As I understand it, the new --on-error flag does not support this.

That is correct. That exact solution requires allot of work. I don't say we object it but it's not a priority for us currently.

Basically, there's a common workflow that would really benefit from --on-error=ssh (preferred) or --on-error=write-pemfile (workable).

Feel free to open a PR.

As I see it, this common workflow isn't being supported, and this is contrary to the Tao of Hashicorp.

I don't see how the current solution contradict Tao. There is basically three types of debugging/developing of a Packer template. 1) Builder part where dropping to SSH doesn't help since the problem is not in that area. 2) Connectivity issues, requires some sort of out of band trouble shooting since SSH is (obviously) not working. 3) Provisioner problem/issue, these are best to troubleshoot with a null builder.

[mooperd]

+1 In order to speed up debugging I would also like to be able to log in to a virtual machine without having to press enter throughout the build process. I'm quite confused why this issue has been closed.

[rickard-von-essen]

See #5107