Terraform: Policy doesn't allow compute_extension:volumes to be performed

[winstonsimon]

Hi there,

I am trying to spin up a new ostk baremtal host using terraform and I am getting following error while executing terraform appy.

Terraform Version

Terraform v0.8.7

Affected Resource(s)

openstack_compute_instance_v2

Terraform Configuration Files

provider "openstack" {
  user_name  = "xyz"
  tenant_name = "xyx"
  auth_url  = "https://xyz:4443/keystone_service/v2.0"
}

resource "openstack_compute_instance_v2" "xyz" {
  name = "xyz"
  image_name = "${var.image}"
  flavor_name = "${var.flavor}"
  availability_zone = "${var.availability_zone}"
  metadata = {
    hostname = "xyz.com"
    num_ipv6 = 1
    roles = "[]"
  }
}

Debug Output

openstack_compute_instance_v2.be01: Expected HTTP response code [200 204] when accessing [GET https://xyz:4443/nova_api/v2/ac6072321219443387c8eb924cab2c71/servers/bd8bce1a-bb0d-4eed-bd45-0696e707943d/os-volume_attachments], but got 403 instead
{"forbidden": {"message": "Policy doesn't allow compute_extension:volumes to be performed.", "code": 403}}

Expected Behavior

I should run refresh/apply/destroy with no error.

Actual Behavior

Failed with os-volume_attachments

We are using image_name but looks like terraform is try to look in cinder for a boot-from-volume image. At this point, openstack instance that we have doesn't expose the cinder API, or support boot from volume.

Is there a way for us to skip 'os-volume_attachments' and move forward?

Even I have tried following configuration:

resource "openstack_compute_instance_v2" "be01" {
  name = "xyz"
  image_name = "${var.image}"
  flavor_name = "${var.flavor}"
  availability_zone = "${var.availability_zone}"
  metadata = {
    hostname = "xyz"
    num_ipv6 = 1
    roles = "[]"
  }

  block_device {
    uuid = "${var.image_id}"
    source_type = "image"
    destination_type = "local"
    boot_index = 0
    delete_on_termination = true
  }
}

Above configuration is also failed due to following errors:

openstack_compute_instance_v2.be01: Creating...
  access_ip_v4:                         "" => "<computed>"
  access_ip_v6:                         "" => "<computed>"
  availability_zone:                    "" => "ip_type:priv,backplane:pda,security_zone:prod_main"
  block_device.#:                       "" => "1"
  block_device.0.boot_index:            "" => "0"
  block_device.0.delete_on_termination: "" => "true"
  block_device.0.destination_type:      "" => "local"
  block_device.0.source_type:           "" => "image"
  block_device.0.uuid:                  "" => "9ed32453-a00c-43e3-bf29-548e4fbc677d"
  flavor_id:                            "" => "<computed>"
  flavor_name:                          "" => "xyz"
  image_id:                             "" => "<computed>"
  image_name:                           "" => "xyz"
  metadata.%:                           "" => "3"
  metadata.hostname:                    "" => "xyz"
  metadata.num_ipv6:                    "" => "1"
  metadata.yroles:                      "" => "[]"
  name:                                 "" => "xyz"
  network.#:                            "" => "<computed>"
  security_groups.#:                    "" => "<computed>"
  stop_before_destroy:                  "" => "false"
2017/02/27 18:48:02 [DEBUG] plugin: terraform: openstack-provider (internal) 2017/02/27 18:48:02 [DEBUG] networks: []
2017/02/27 18:48:02 [DEBUG] plugin: terraform: openstack-provider (internal) 2017/02/27 18:48:02 [DEBUG] Block Device Options: [{SourceType:image UUID:9ed32453-a00c-43e3-bf29-548e4fbc677d BootIndex:0 DeleteOnTermination:true DestinationType:local GuestFormat: VolumeSize:0}]
2017/02/27 18:48:02 [DEBUG] plugin: terraform: openstack-provider (internal) 2017/02/27 18:48:02 [DEBUG] Create Options: &bootfromvolume.CreateOptsExt{CreateOptsBuilder:(*servers.CreateOpts)(0xc4205e07e0), BlockDevice:[]bootfromvolume.BlockDevice{bootfromvolume.BlockDevice{SourceType:"image", UUID:"9ed32453-a00c-43e3-bf29-548e4fbc677d", BootIndex:0, DeleteOnTermination:true, DestinationType:"local", GuestFormat:"", VolumeSize:0}}}
2017/02/27 18:48:03 [DEBUG] vertex "meta.count-boundary (count boundary fixup)", waiting for: "openstack_compute_instance_v2.be01"
2017/02/27 18:48:03 [DEBUG] root: eval: *terraform.EvalWriteState
2017/02/27 18:48:03 [DEBUG] root: eval: *terraform.EvalApplyProvisioners
2017/02/27 18:48:03 [DEBUG] root: eval: *terraform.EvalIf
2017/02/27 18:48:03 [DEBUG] root: eval: *terraform.EvalWriteState
2017/02/27 18:48:03 [DEBUG] root: eval: *terraform.EvalWriteDiff
2017/02/27 18:48:03 [DEBUG] root: eval: *terraform.EvalApplyPost
2017/02/27 18:48:03 [ERROR] root: eval: *terraform.EvalApplyPost, err: 1 error(s) occurred:

* openstack_compute_instance_v2.be01: Error creating OpenStack server: Invalid request due to incorrect syntax or missing required parameters.
2017/02/27 18:48:03 [ERROR] root: eval: *terraform.EvalSequence, err: 1 error(s) occurred:

* openstack_compute_instance_v2.be01: Error creating OpenStack server: Invalid request due to incorrect syntax or missing required parameters.
2017/02/27 18:48:03 [TRACE] [walkApply] Exiting eval tree: openstack_compute_instance_v2.be01
2017/02/27 18:48:03 [DEBUG] vertex "meta.count-boundary (count boundary fixup)", got dep: "openstack_compute_instance_v2.be01"
2017/02/27 18:48:03 [TRACE] Preserving existing state lineage "0f85695f-8f15-424f-abe0-5eae79d71df1"
2017/02/27 18:48:03 [TRACE] Preserving existing state lineage "0f85695f-8f15-424f-abe0-5eae79d71df1"
Error applying plan:

2017/02/27 18:48:03 [DEBUG] plugin: waiting for all plugin processes to complete...
1 error(s) occurred:

2017/02/27 18:48:03 [DEBUG] plugin: terraform: openstack-provider (internal) 2017/02/27 18:48:03 [DEBUG] plugin: waiting for all plugin processes to complete...
* openstack_compute_instance_v2.be01: Error creating OpenStack server: Invalid request due to incorrect syntax or missing required parameters.

[jtopjian]

@winstonsimon Thanks for reporting this.

For some background: the openstack_compute_instance_v2 resource was initially designed very "thick". It's a hub for an instance, its floating ips, volume attachments, and networks. I've been slowly working towards thinning this resource out to account for scenarios like you've reported.

At first glance, I don't believe this is a Boot From Volume error, but the instance resource doing a general query to see what volumes it has attached to it. Recently an openstack_compute_volume_attachment_v2 resource was created dedicated to handle this specific task with the ultimate goal of removing this code from the instance resource down the road.

I'll take a look at this in more detail. I'll modify Nova's policy in a test environment to prohibit access to os-volume_attachments to see if I can reproduce the error. If I can, I'll modify the instance resource to appropriately handle the 403 error.

Additionally, the next release of Terraform will include an extra debug argument. This will help pin down exactly where the 403 is being triggered.

Let me know if you have any questions :)

[winstonsimon]

Thank you! I'll wait for your update :)

[winstonsimon]

Have you able to reproduce this problem?

[jtopjian]

@winstonsimon Yep! #12374 should fix this issue. Can you try it out just to make sure? Let me know if you need help compiling Terraform from source.

[winstonsimon]

Awesome @jtopjian. That was quick :)

Yeah. Could you please point me the doc how I can get the latest code and compile it to test?

[jtopjian]

Yep. The main README has a section on developing, which is helpful. You'll want to do a variation of that.

First you'll need to have Go installed on your workstation (or development virtual machine or similar). I personally use gimme for this and then do something like:

$ sudo wget -O /usr/local/bin/gimme https://raw.githubusercontent.com/travis-ci/gimme/master/gimme
$ sudo chmod +x /usr/local/bin/gimme
$ mkdir ~/go
$ /usr/local/bin/gimme 1.7 >> ~/.bashrc
$ echo 'export GOPATH=$HOME/go' >> ~/.bashrc
$ source ~/.bashrc

Makefile and git are also required.

Once that's set up, do:

$ go get -u github.com/hashicorp/terraform
$ cd ~/go/src/github.com/hashicorp/terraform
$ git remote add jtopjian https://github.com/jtopjian/terraform
$ git fetch jtopjian
$ git checkout --track jtopjian/openstack-instance-volumes-403

At this point, you'll have my branch with the fix included. Now compile it to a binary:

$ make dev

Let me know if you run into any problems. If this gets too hairy, I can compile a binary for you. :)

[winstonsimon]

Sorry compilation failed

~ ➤ source ~/.bashrc                                                                                                                                                             
go version go1.7 darwin/amd64

terraform ➤ git checkout --track jtopjian/openstack-instance-volumes-403                                                                                               git:master
Branch openstack-instance-volumes-403 set up to track remote branch openstack-instance-volumes-403 from jtopjian.
Switched to a new branch 'openstack-instance-volumes-403'
terraform ➤ make dev                                                                                                                           git:openstack-instance-volumes-403
==> Checking that code complies with gofmt requirements...
go generate $(go list ./... | grep -v /terraform/vendor/)
2017/03/02 11:09:43 Generated command/internal_plugin_list.go
backend/operation_type.go:3: running "stringer": exec: "stringer": executable file not found in $PATH
backend/local/hook_count_action.go:3: running "stringer": exec: "stringer": executable file not found in $PATH
command/hook_count_action.go:3: running "stringer": exec: "stringer": executable file not found in $PATH
config/resource_mode.go:3: running "stringer": exec: "stringer": executable file not found in $PATH
helper/schema/resource_data_get_source.go:3: running "stringer": exec: "stringer": executable file not found in $PATH
terraform/context_graph_type.go:3: running "stringer": exec: "stringer": executable file not found in $PATH
make: *** [generate] Error 1
terraform ➤ pwd                                                                                                                               git:openstack-instance-volumes-403*
/Users/winston/go/src/github.com/hashicorp/terraform

[jtopjian]

d'oh, I'm sorry... try:

$ echo 'export PATH=$PATH:$HOME/go/bin' >> ~/.bashrc
$ source ~/.bashrc

Edit: with your path now looking at $HOME/go/bin, make sure you run the new terraform binary in $HOME/go/bin/terraform:

$ which terraform

to make sure :)

[winstonsimon]

Awesome it worked!

terraform ➤ ~/go/bin/terraform apply                                                                                                                                  
openstack_compute_instance_v2.xxx: Refreshing state... (ID: db920868-6cf0-4873-a699-266d75ac8cbc)
openstack_compute_instance_v2.xxx: Creating...
  access_ip_v4:        "" => "<computed>"
  access_ip_v6:        "" => "<computed>"
  availability_zone:   "" => "ip_type:priv,backplane:pda,security_zone:prod_main"
  flavor_id:           "" => "<computed>"
  flavor_name:         "" => "xxxx"
  image_id:            "" => "<computed>"
  image_name:          "" => "xxxx"
  metadata.%:          "" => "3"
  metadata.hostname:   "" => "xyz.com"
  metadata.num_ipv6:   "" => "1"
  metadata.yroles:     "" => "[]"
  name:                "" => "xxx"
  network.#:           "" => "<computed>"
  security_groups.#:   "" => "<computed>"
  stop_before_destroy: "" => "false"
openstack_compute_instance_v2.xxx: Still creating... (10s elapsed)
openstack_compute_instance_v2.xxx: Still creating... (7m0s elapsed)
openstack_compute_instance_v2.xxx: Creation complete

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

[winstonsimon]

@jtopjian When can expect this patch for our production usecase?

[jtopjian]

It's pretty safe to say this will be in the next release. :)

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.