Terraform: provider/openstack: access_ip_v4 is not used for provision despite access_network

[Fodoj]

Given:

resource "openstack_compute_instance_v2" "my_node" {
  name = "my-node}"
  availability_zone = "${var.availability_zone}"
  image_id = "${var.image_id}"
  flavor_name = "${var.machine_flavor}"
  floating_ip = "${openstack_compute_floatingip_v2.my_node_floating_ip.address}"
  key_pair = "${var.key_pair}"
  network {
    uuid = "${var.network_id}"
    access_network = true
  }
  provisioner "remote-exec" {
    inline = [
      "echo hello"
    ]
    connection {
      user = "my-user"
      bastion_host = "${var.bastion_ip}"
      bastion_user = "cloud-user"
      private_key = "${file("${var.private_key}")}"
    }
  }
}

Terraform tries to connect through bastion ip to _floating_ip_ of the node, despite access_network specified to true.

What I expected in this case is to pick auto generated by OpenStack access_ip_v4.

From what I can see, it picks ip from network only if fixed ip is specified: https://github.com/hashicorp/terraform/blob/master/builtin/providers/openstack/resource_openstack_compute_instance_v2.go#L1003

But if there is an auto-generated private ip for that network, I believe this one should be used for provisioning.

[Fodoj]

After going through source code, the issue seems to be a bit more complicated.

Following method: https://github.com/hashicorp/terraform/blob/master/builtin/providers/openstack/resource_openstack_compute_instance_v2.go#L857 iterates through all networks and then merges all instance networking info into first one.

That means that in example I showed above Terraform will generate following in the state file:

 "network.#": "1",
"network.0.access_network": "true",
"network.0.fixed_ip_v4": "192.168.2.24",
"network.0.fixed_ip_v6": "",
"network.0.floating_ip": "172.30.15.83",
"network.0.mac": "fa:16:3e:39:db:4f",
"network.0.name": "fj-network",
"network.0.port": "",
"network.0.uuid": "deb0f9-af1b-4b3a-b36d-866b970",

But floating ip belongs to other network. Nevertheless, terraform will pick floating ip because it is access network and floating ips are picked first.

There are two solutions I can see:

1. Always generate additional "default" network
2. Do not merge floating ip from external networks to network info of an instnace

[hingstarne]

@Fodoj This only harms you, when you don`t specify host parameter in your connection. If you have a working DNS in your stack, you could ....

(A change like this would ease at least the pain unless a solution is found.)

resource "openstack_compute_instance_v2" "my_node" {
  name = "my-node}"
  availability_zone = "${var.availability_zone}"
  image_id = "${var.image_id}"
  flavor_name = "${var.machine_flavor}"
  floating_ip = "${openstack_compute_floatingip_v2.my_node_floating_ip.address}"
  key_pair = "${var.key_pair}"
  network {
    uuid = "${var.network_id}"
    access_network = true
  }
  provisioner "remote-exec" {
    inline = [
      "echo hello"
    ]
    connection {
      user = "my-user"
      host =  "${self.name}"
      bastion_host = "${var.bastion_ip}"
      bastion_user = "cloud-user"
      private_key = "${file("${var.private_key}")}"
    }
  }
}

[jtopjian]

@Fodoj I'm working out some kinks with the access IPs. There's some additional info and possible ways of specifying networks and access in #5695 including a patch I have in progress to fix some logic of the access IPs.

In general, you're correct in what you see: a floating IP will always trump the access network. This is under the assumption that in most OpenStack clouds, a floating IP is also used as the public access IP. This is technically an incorrect dual-use of floating IPs, but it's become common in OpenStack usage.

Things got really complicated a few months ago when someone requested per-network floating IPs. This is where the concept of the access_network came in. The patch I mention in #5695 should make things a little more intuitive. Determining network access is done by:

1. First looping through all networks and checking for a floating IP, fixed v4 and fixed v6. If found, the floating IP takes precedence.
2. Loop again and look for an access network. If found, check for a floating IP, fixed v4 and fixed v6. If found, the floating IP of the access network takes precedence. If not, then the fixed IPs are used as the access IPs.

Thoughts? And let me know if you try out the patch and what you think.

[Fodoj]

i wil try it ouy.. but does it still merge top level floating ip into existing network with private ip?

[jtopjian]

It does, yes:

https://github.com/jtopjian/terraform/blob/c0c718cd9459f166a275abf0947f7052505d1906/builtin/providers/openstack/resource_openstack_compute_instance_v2.go#L998-L1002

So you're saying given a situation where someone has specified a "global" / top-level floating IP but has specified one or more network blocks, one of which is the access_network, the access_network should still be recognized for host access? This makes sense to me since it's possible you're running Terraform from a private management network.

[Fodoj]

@jtopjian yes, exactly. Because currently it will always prefer top level floating IP :( How about I will make a PR that avoids this merging?

[jtopjian]

@Fodoj I can just roll it into the patch I'm working on 😄

[Fodoj]

Great, looking forward to it then :)

[jtopjian]

@Fodoj Looks like my patch already had that covered :)

It does not allow someone to specify both a top-level and network-level floating IP, so because of that, your use-case is implicitly satisfied.

Check out #6181 and let me know if you have any problems!

[jtopjian]

@Fodoj I just merged #6181 which also included an acceptance test specifically for the scenario you described. Please let me know if something seems off.

[Fodoj]

Looks like it still uses floating ip in fixed private ip is not specified for access network, so bug still persists. :(

[Fodoj]

My best guess is that it happens because fixed_ip_v4 is computed in case its not specified in template, thus during getINstanceAccessAddresses method execution it is empty?..

[jtopjian]

@Fodoj Thanks for checking. I'm re-opening this and will look into it.

[Fodoj]

Any luck with this issue? Do you need my support?

[jtopjian]

Sorry for the delay on this -- I'll try to spend some time this week. If you're able to find a solution, I'd gladly take a PR :)

[jtopjian]

@Fodoj I understand why this is happening.

Take the following configuration:

resource "openstack_compute_floatingip_v2" "myip" {
}

resource "openstack_compute_instance_v2" "foo" {
  name = "terraform-test"
  security_groups = ["default"]
  floating_ip = "${openstack_compute_floatingip_v2.myip.address}"

  network {
    uuid = "dc3a7ceb-8b11-4488-99d3-b11f23b17595"
    access_network = true
  }
}

The access_ip_v4 attribute will be the floating IP because there is only one network specified. A floating IP, logically, must be attached to a network and since there's only one, it is implicitly attached to that first network. This is for two reasons:

1. Backwards compatibility from before access_network existed (for a while, implicitly attaching a floating IP to the _first_ network in this way was the only way to attach a floating IP. This was due to a limitation in Gophercloud which was fixed earlier this year).
2. This is how the Nova API handles floating IP attachment requests that do not have a corresponding fixed IP to attach to.

Running with TF_LOG=DEBUG, you can see the following:

Addresses: map[private:[map[OS-EXT-IPS-MAC:mac_addr:fa:16:3e:43:ef:5a version:6 addr:fddb:dea:340e:0:f816:3eff:fe43:ef5a  OS-EXT-IPS:type:fixed] map[version:4 addr:10.0.0.59 OS-EXT-IPS:type:fixed OS-EXT-IPS-MAC:mac_addr:fa:16:3e:43:ef:5a] map[addr:172.24.4.37 OS-EXT-IPS:type:floating OS-EXT-IPS-MAC:mac_addr:fa:16:3e:43:ef:5a version:4]]]

You can see the same thing by running nova --debug show terraform-test:

"addresses": {"private": [{"OS-EXT-IPS-MAC:mac_addr": "fa:16
:3e:43:ef:5a", "version": 6, "addr": "fddb:dea:340e:0:f816:3eff:fe43:ef5a", "OS-EXT-IPS:type": "fixed"}, {"OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:43:ef:5a", "version": 4, "addr": "10.0.0.59", "OS-EXT-IPS:type":
"fixed"}, {"OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:43:ef:5a", "version": 4, "addr": "172.24.4.37", "OS-EXT-IPS:type": "floating"}]}

Because of the above, this will always record the floating IP and not the fixed IP.

The same will happen even if you have multiple networks and the first network is to be the access network. This is because a floating IP will always be attached to the first network if a specific network was not specified.

If you modify the above configuration to:

resource "openstack_compute_floatingip_v2" "myip" {
}

resource "openstack_networking_network_v2" "foo" {
  name = "network_1"
  admin_state_up = "true"
}

resource "openstack_networking_subnet_v2" "foo" {
  name = "subnet_1"
  network_id = "${openstack_networking_network_v2.foo.id}"
  cidr = "192.168.10.0/24"
  ip_version = 4
}

resource "openstack_compute_instance_v2" "foo" {
  name = "terraform-test"
  security_groups = ["default"]
  floating_ip = "${openstack_compute_floatingip_v2.myip.address}"

  network {
    uuid = "dc3a7ceb-8b11-4488-99d3-b11f23b17595"
  }

  network {
    uuid = "${openstack_networking_network_v2.foo.id}"
    access_network = true
  }
}

Then access_ip_v4 will be the private/fixed_ip of the second network -- not the floating IP.

So with the above in mind, I think we can narrow down the issue to the following:

I want to be able to specify a top-level floating IP, but also have the ability to ensure that access_ip_v4 will be the private IP in cases where the first or only network is set as the access network.

Does that sound correct? If so, I think that's a pretty complicated use-case (but I might be incorrect -- please do let me know!). In this situation, I'd be inclined to just do the following:

    connection {
      user = "my-user"
      host = "${self.network.0.fixed_ip_v4}"
      bastion_host = "${var.bastion_ip}"
      bastion_user = "cloud-user"
      private_key = "${file("${var.private_key}")}"
    }

Thoughts?

[jtopjian]

@Fodoj I'm going to go ahead and close this one since we found the core issue. Please feel free to re-open if I'm incorrect :smile:

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.