Packer: amazon-ebssurrogate fails with missing packer-fleet-launch-template

[nergdron]

packer version: Packer v1.4.2
host: ubuntu 19.04 amd64

when attempting to use spot instance configuration with amazon-ebssurrogate, it fails, saying "InvalidLaunchTemplateName.NotFoundException: The specified launch template, with template name packer-fleet-launch-template, does not exist."

gist of error message and example config:
https://gist.github.com/nergdron/cdfbf493958f1eec8132028201da22a3

it looks like it isn't actually creating the spot instance request launch template, and so everything else fails.

[rickard-von-essen]

Can you try the latest version?

[nergdron]

@rickard-von-essen: I'm confused, I just installed this on Friday, and the releases page says v1.4.2 is the latest version.

[rickard-von-essen]

Sorry it's me that is confused. I thaught that 1.4.3 was released already.

[nergdron]

just to see, I tried building packer from this repo/master just now, and it still errored out, but with different errors:

amazon-ebssurrogate output will be in this color.

Warnings for build 'amazon-ebssurrogate':

* spot_price_auto_product is deprecated and no longer necessary for Packer builds. In future versions of Packer, inclusion of spot_price_auto_product will error your builds. Please take a look at our current documentation to understand how Packer requests Spot instances.

==> amazon-ebssurrogate: Prevalidating AMI Name: ubuntu-18.04-1565631710
    amazon-ebssurrogate: Found Image ID: ami-something
    amazon-ebssurrogate: Found Subnet ID: subnet-something
==> amazon-ebssurrogate: Using SSH Agent for existing key pair my.key
==> amazon-ebssurrogate: Creating temporary security group for this instance: packer_5d51a4e1-f13a-0d51-75a0-26aa84d7ae43
==> amazon-ebssurrogate: Authorizing access to port 22 from [0.0.0.0/0] in the temporary security groups...
==> amazon-ebssurrogate: Launching a spot AWS instance...
==> amazon-ebssurrogate: Interpolating tags for spot instance...
    amazon-ebssurrogate: Adding tag: "packer": "1"
    amazon-ebssurrogate: Adding tag: "salt": "1"
    amazon-ebssurrogate: Adding tag: "Name": "Packer Builder"
    amazon-ebssurrogate: Loading User Data File...
    amazon-ebssurrogate: Creating Spot Fleet launch template...
    amazon-ebssurrogate: Sending spot request ()...
    amazon-ebssurrogate: Adding tag: "packer": "1"
    amazon-ebssurrogate: Adding tag: "src": "salt/packer"
    amazon-ebssurrogate: Adding tag: "salt": "1"
    amazon-ebssurrogate: Adding tag: "Name": "packer-ebssurrogate"
==> amazon-ebssurrogate: Error tagging spot request: InvalidParameterValue: Value ( null ) for parameter resourceId is invalid. Null/empty value for resourceId is invalid
==> amazon-ebssurrogate:    status code: 400, request id: cf3bdce5-b346-4964-addc-0ce350f06b5c
==> amazon-ebssurrogate: No volumes to clean up, skipping
==> amazon-ebssurrogate: Deleting temporary security group...
Build 'amazon-ebssurrogate' errored: Error tagging spot request: InvalidParameterValue: Value ( null ) for parameter resourceId is invalid. Null/empty value for resourceId is invalid
    status code: 400, request id: cf3bdce5-b346-4964-addc-0ce350f06b5c

so it looks like the spot instance code is in a state of flux? or I don't know the right configuration to get it to actually work.

[SwampDragons]

What values did you supply in your template?

[nergdron]

@SwampDragons ok first off great profile name. second, if you look at the gist, you'll see the config I'm using to test.

[SwampDragons]

Hah, Thanks! :D

Oops, I see it now, thanks. I was responding via email and didn't see the gist. I'll take a look as soon as I can.

[rsluzhynskyy]

I am having the same error when using "spot_instance_types" with the latest packer v1.4.2.

[rnhurt]

This looks like the same issue as #7941.

[SwampDragons]

Yeah, I think you're right.

[SwampDragons]

Gonna close this as a duplicate and track on 7941.

[nergdron]

I don't actually believe these are the same error. when I run with debug logging, I don't get any errors about tags, it doesn't get that far. and removing the spot_tags key doesn't fix the problem, I still get this error.

[SwampDragons]

Looks like we get a different error from your repro case when we remove spot_tags:

amazon-ebssurrogate: Found Image ID: ami-07d0cf3af28718ef8
==> amazon-ebssurrogate: Creating temporary keypair: packer_5d5335d7-e391-f0eb-06c5-89c9eff9330a
==> amazon-ebssurrogate: Creating temporary security group for this instance: packer_5d5335da-fb6a-0197-5dac-e6850241cf65
2019/08/13 15:12:42 packer: 2019/08/13 15:12:42 [DEBUG] Waiting for temporary security group: sg-070903e2b540d5bcc
2019/08/13 15:12:43 packer: 2019/08/13 15:12:43 [DEBUG] Found security group sg-070903e2b540d5bcc
==> amazon-ebssurrogate: Authorizing access to port 22 from [0.0.0.0/0] in the temporary security groups...
==> amazon-ebssurrogate: Launching a spot AWS instance...
==> amazon-ebssurrogate: Interpolating tags for spot instance...
    amazon-ebssurrogate: Adding tag: "packer": "1"
    amazon-ebssurrogate: Adding tag: "Name": "Packer Builder"
    amazon-ebssurrogate: Loading User Data File...
    amazon-ebssurrogate: Creating Spot Fleet launch template...
    amazon-ebssurrogate: Sending spot request ()...
==> amazon-ebssurrogate:    status code: 400, request id: 5f291595-cbe6-4092-bb9a-9c3e0878f79e
==> amazon-ebssurrogate: Error waiting for spot request () to become ready: InvalidFleetConfig: The fleet configuration contains duplicate instance pools.
==> amazon-ebssurrogate:    status code: 400, request id: 5f291595-cbe6-4092-bb9a-9c3e0878f79e
==> amazon-ebssurrogate: No volumes to clean up, skipping
==> amazon-ebssurrogate: Deleting temporary security group...

[SwampDragons]

Okay, dug into this a bit more. If I remove the duplicate pools from the spot_instance_types as the error message indicates, I get another error: "Your requested instance type (t2a.nano) is not supported in your requested Availability Zone (us-east-1f)." . So, I remove that instance and I get similar error messages for t2.nano, t2a.micro, and a bunch more. Final working spot_instance_types that work for my zone:

    "spot_instance_types": [
      "t3.nano",
      "t3.micro",
      "t3.small",
      "t2.micro"
    ],

Wen I run with the above, I get an error about an invalid device name. "Invalid device name /dev/root_target",, which is unrelated to being a spot instance at all. I get the same error if I launch your template on an ondemand instance.

I didn't dig much further because I think based on some quick googling that this last error is also a configuration issue. Amazon has some docs on naming here which may be helpful. I'd do more tinkering to get this to work for you but I'm not really an expert on aws launch devices either. I am confident though that at least through this point, we're looking at first the spot_tags error that we labeled this as a duplicate of, and then a handful of configuration errors that are amazon-related, not Packer issues.

I'll leave this issue open for now, but I'm mired pretty deeply into investigating a mitigation for the spot_tags error at the moment and probably won't get to working on this further until after the 1.4.3 release.

[nergdron]

yeah, I've stripped the spot instance stuff from my actual code for the time being so I can move forward, since the rest of it seems to work. discovered that if you don't create a device name with the name AWS is expecting, it just doesn't show up, which is a kind of hidden gotcha it might be nice to call out in the docs.

the thing that's confusing me here is I can't replicate the errors you're getting, and you're not getting the error I'm seeing. I'm also confused as to why you'd have to pair down the list of instance types, I'd expect that given a list of types, packer would choose the first type that is available in the specified az/subnet. if that's not the case, then it makes providing a list a lot less useful. some regions (cough us-east-1) seem to fluctuate what's available on a day-by-day basis, so it'd mean manually evaluating what's available on practically every run.

in any case, I'll keep working at testing this and seeing if I can at least replicate the error you see here. good luck with the other 1.4.3 stuff!

[SwampDragons]

The device name requirements are called out in Amazon's docs, which I'll admit aren't the most readable or intuitive. But this is an eternal question I face. How much work do we put in to cleaning up Amazon's bad UX? Most of the Packer options are just passthroughs to Amazon options, so we generally assume users have an understanding of the underlying Amazon mechanisms. Otherwise the Packer docs just turn into a replica of the AWS docs.

We probably can edit the list of types behind the scenes to remove duplicates and retry the query on a loop, removing invalid platforms, until one goes through, but currently we're just acting as a passthrough and expecting you to know what's available to you on Amazon. Helpfully "fixing" things is normally something I don't like to do because it could blindside users with unexpected behavior, but in this case you're right that it would be a big improvement to user experience.

As to why you aren't getting the invalid device name error, I don't know -- but if you want to share logs from a run that has the spot_tags option removed that could help me figure it out.

[SwampDragons]

the spot_tags issue has been resolved on the master branch -- if you are still having issues, can you update the details for steps to reproduce so we can keep digging into this?

[nergdron]

ok, just got a chance to test this. with the same test config I used before, packer built from master today crashes hard. so something is definitely different, but not necessarily in a good way.

amazon-ebssurrogate output will be in this color.

==> amazon-ebssurrogate: Prevalidating AMI Name: ubuntu-18.04-1566238275
    amazon-ebssurrogate: Found Image ID: ami-06f2f779464715dc5
==> amazon-ebssurrogate: Creating temporary keypair: packer_5d5ae643-e41e-8068-e845-ce0bed32217f
==> amazon-ebssurrogate: Creating temporary security group for this instance: packer_5d5ae644-dc49-6103-1d3c-54f3f85d4e6f
==> amazon-ebssurrogate: Authorizing access to port 22 from [0.0.0.0/0] in the temporary security groups...
==> amazon-ebssurrogate: Launching a spot AWS instance...
==> amazon-ebssurrogate: Interpolating tags for spot instance...
    amazon-ebssurrogate: Adding tag: "packer": "1"
    amazon-ebssurrogate: Adding tag: "Name": "Packer Builder"
    amazon-ebssurrogate: Loading User Data File...
    amazon-ebssurrogate: Creating Spot Fleet launch template...
    amazon-ebssurrogate: Sending spot request ()...
==> amazon-ebssurrogate: No volumes to clean up, skipping
==> amazon-ebssurrogate: Deleting temporary security group...
==> amazon-ebssurrogate: Deleting temporary keypair...
Build 'amazon-ebssurrogate' errored: unexpected EOF

==> Some builds didn't complete successfully and had errors:
--> amazon-ebssurrogate: unexpected EOF

==> Builds finished but no artifacts were created.
panic: runtime error: invalid memory address or nil pointer dereference
2019/08/19 11:11:18 packer: [signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x14e677a]
2019/08/19 11:11:18 packer: 
2019/08/19 11:11:18 packer: goroutine 166 [running]:
2019/08/19 11:11:18 packer: github.com/hashicorp/packer/builder/amazon/common.(*StepRunSpotInstance).Run(0xc0003449c0, 0x3fb93c0, 0xc000066400, 0x3f9ef00, 0xc0005b2480, 0x0)
2019/08/19 11:11:18 packer:     /home/tessa/code/git/packer/builder/amazon/common/step_run_spot_instance.go:263 +0x2d0a
2019/08/19 11:11:18 packer: github.com/hashicorp/packer/helper/multistep.(*BasicRunner).Run(0xc0005b2630, 0x3fb93c0, 0xc000066400, 0x3f9ef00, 0xc0005b2480)
2019/08/19 11:11:18 packer:     /home/tessa/code/git/packer/helper/multistep/basic_runner.go:67 +0x1ec
2019/08/19 11:11:18 packer: github.com/hashicorp/packer/builder/amazon/ebssurrogate.(*Builder).Run(0xc0003d9c00, 0x3fb93c0, 0xc000066400, 0x3fd5020, 0xc0003aa2c0, 0x3f52360, 0xc000394010, 0x40d5d9, 0xc00052a4b0, 0x30, ...)
2019/08/19 11:11:18 packer:     /home/tessa/code/git/packer/builder/amazon/ebssurrogate/builder.go:305 +0x1572
2019/08/19 11:11:18 packer: github.com/hashicorp/packer/packer/rpc.(*BuilderServer).Run(0xc000142cc0, 0x1, 0xc000053ec0, 0x0, 0x0)
2019/08/19 11:11:18 packer:     /home/tessa/code/git/packer/packer/rpc/builder.go:109 +0x231
2019/08/19 11:11:18 packer: reflect.Value.call(0xc000550c00, 0xc000554100, 0x13, 0x39a321c, 0x4, 0xc00009cf18, 0x3, 0x3, 0xc000066740, 0x0, ...)
2019/08/19 11:11:18 packer:     /usr/local/go/src/reflect/value.go:447 +0x461
2019/08/19 11:11:18 packer: reflect.Value.Call(0xc000550c00, 0xc000554100, 0x13, 0xc000013f18, 0x3, 0x3, 0x0, 0x0, 0x0)
2019/08/19 11:11:18 packer:     /usr/local/go/src/reflect/value.go:308 +0xa4
2019/08/19 11:11:18 packer: net/rpc.(*service).call(0xc000066480, 0xc00055e780, 0xc0004c8008, 0xc0004c8010, 0xc000580680, 0xc0002dc1c0, 0x2ed6400, 0xc000053ebc, 0x18a, 0x2e28a60, ...)
2019/08/19 11:11:18 packer:     /usr/local/go/src/net/rpc/server.go:384 +0x14e
2019/08/19 11:11:18 packer: created by net/rpc.(*Server).ServeCodec
2019/08/19 11:11:18 packer:     /usr/local/go/src/net/rpc/server.go:481 +0x42b
2019/08/19 11:11:18 [INFO] (telemetry) ending amazon-ebssurrogate
2019/08/19 11:11:18 ui error: Build 'amazon-ebssurrogate' errored: unexpected EOF
2019/08/19 11:11:18 machine readable: error-count []string{"1"}
2019/08/19 11:11:18 ui error: 
==> Some builds didn't complete successfully and had errors:
2019/08/19 11:11:18 machine readable: amazon-ebssurrogate,error []string{"unexpected EOF"}
2019/08/19 11:11:18 /home/tessa/bin/packer: plugin process exited
2019/08/19 11:11:18 ui error: --> amazon-ebssurrogate: unexpected EOF
2019/08/19 11:11:18 ui: 
==> Builds finished but no artifacts were created.
2019/08/19 11:11:18 [INFO] (telemetry) Finalizing.
2019/08/19 11:11:18 waiting for all plugin processes to complete...
2019/08/19 11:11:18 /home/tessa/bin/packer: plugin process exited

[nergdron]

from this I'm guessing the config format for the ebssurrogate builder has changed and I'm missing something required, but it definitely shouldn't be hard crashing on that. I'd expect to see some nice guidance on what's wrong with my config.

[nergdron]

it's also interesting to see it happens after apparently failing the spot instance allocation, Sending spot request ()... seems suspicious.

[SwampDragons]

Ack sorry. Pushed a fix for that: https://github.com/hashicorp/packer/pull/8013

[SwampDragons]

build can be downloaded at https://circleci.com/gh/hashicorp/packer/9141#artifacts/containers/0

[nergdron]

Nice! that build works great vs 1.4.3 and older, looks like you nailed it down at some point, closing!

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.