Terraform-provider-libvirt: <host> not added to network when creating a domain

[ghtyrant]

Version Reports:

Distro version of host:

Debian GNU/Linux 9.5 (stretch)

Terraform Version Report

terraform v0.11.8

• provider.libvirt (unversioned)

I built this provider from source since Debian ships with libvirt 3.0.0, which doesn't work with the pre-built binaries.

Libvirt version

3.0.0

terraform-provider-libvirt plugin version (git-hash)

4cc6d3ba0180c5c1afcbd2f37cbe8f6f8d93ccae

---

Description of Issue/Question

Setup

main.tf:

provider "libvirt" {
  uri = "qemu:///system"
}

# Default VM Network
resource "libvirt_network" "default" {
  name = "default"
  addresses = ["192.168.100.0/24"]
  domain = "vm"
  bridge = "virbr0"
  autostart = true
  dhcp {
    enabled = true
  }
}

module "cloud_vm" {
  source = "./vm"

  name      = "cloud"
  cores     = 2
  memory    = 1024
  address   = "192.168.100.4"
  os_disk   = 4294967296 # 4 GB
  data_disk = 1024

  network_name = "${libvirt_network.default.name}"
}

Module vm (vm/main.tf):

variable "network_name" {
  type = "string"
  description = "Name of the network this machine should be in"
}

variable "name" {
  type = "string"
  description = "Name of the virtual machine"
}

variable "cores" {
  type = "string"
  description = "Number of cores"
}

variable "memory" {
  type = "string"
  description = "RAM in Megabytes"
}

variable "address" {
  type = "string"
  description = "IP address"
}

variable "os_disk" {
  type = "string"
  description = "Size of the OS disk in bytes"
}

variable "data_disk" {
  type = "string"
  description = "Size of the data disk in bytes"
}

variable "data_disk" {                                                                                                                                                                                                                        
  type = "string"                                                                                                                                                                                                                             
  description = "Size of the data disk in bytes"                                                                                                                                                                                              
}               

resource "libvirt_volume" "os_disk" {
  name = "${var.name}_os"
  pool = "vg0"
  size = "${var.os_disk}"
}

resource "libvirt_volume" "data_disk" {
  name = "${var.name}_data"
  pool = "data"
  size = "${var.data_disk}"
  format = "qcow2"

  lifecycle {
    prevent_destroy = true
  }
}

resource "libvirt_domain" "domain" {
  name = "${var.name}"

  running   = false
  autostart = false

  disk {
    file = "/home/tyrant/salt-minion-debian-preseed/images/debian-9.5-amd64-CD-1.iso"
  }

  disk {
    volume_id = "${libvirt_volume.os_disk.id}"
  }

  #disk {
  #  volume_id = "${libvirt_volume.data_disk.id}"
  #}

  console {
    type        = "pty"
    target_type = "serial"
    target_port = "0"
  }

  console {
    type        = "pty"
    target_type = "virtio"
    target_port = "1"
  }

  graphics {
    type        = "spice"
    listen_type = "address"
    autoport    = true
  }

  network_interface {
    network_name = "${var.network_name}"
    addresses = ["${var.address}"]
    hostname  = "${var.name}"
  }

  vcpu  = "${var.cores}"
  memory  = "${var.memory}"

  boot_device = {
    dev = ["hd", "cdrom"]
  }
}

Steps to Reproduce Issue

• Run terraform apply. This will create the network "default" and a domain called "cloud"
• Run sudo virsh net-dumpxml default:

<network>
  <name>default</name>
  <uuid>c0f7fabf-69fc-4499-a3ab-a949925655c3</uuid>
  <forward mode='nat'>
    <nat>
      <port start='1024' end='65535'/>
    </nat>
  </forward>
  <bridge name='virbr0' stp='on' delay='0'/>
  <mac address='52:54:00:6d:80:38'/>
  <domain name='vm'/>
  <ip family='ipv4' address='192.168.100.1' prefix='24'>
    <dhcp>
      <range start='192.168.100.2' end='192.168.100.254'/>
    </dhcp>
  </ip>
</network>

• Run terraform apply again
• Run sudo virsh net-dumpxml default again:

<network>
  <name>default</name>
  <uuid>c0f7fabf-69fc-4499-a3ab-a949925655c3</uuid>
  <forward mode='nat'>
    <nat>
      <port start='1024' end='65535'/>
    </nat>
  </forward>
  <bridge name='virbr0' stp='on' delay='0'/>
  <mac address='52:54:00:6d:80:38'/>
  <domain name='vm'/>
  <ip family='ipv4' address='192.168.100.1' prefix='24'>
    <dhcp>
      <range start='192.168.100.2' end='192.168.100.254'/>
      <host mac='2A:40:25:3F:0E:CE' name='cloud' ip='192.168.100.4'/>
    </dhcp>
  </ip>
</network>

• Observe that the line <host mac='2A:40:25:3F:0E:CE' name='cloud' ip='192.168.100.4'/> was added at the second run, but not at the first

---

I expect to be added on the first run.
This could be related to #389 or #402. I tried adding a dependency on the network to the VM but the result was the same.

[MalloZup]

Try out the wait for lease param

[MalloZup]

https://github.com/dmacvicar/terraform-provider-libvirt/blob/master/website/docs/r/domain.html.markdown#handling-network-interfaces

[ghtyrant]

I tried that, but it doesn't fix my problem. The line is still missing, and will only be added when I run terraform apply a second time.

[ghtyrant]

Let me try to make it more clear: The issue is not, that the domain does not get a DHCP lease - it gets one, but not the one I defined. And that's because the entry is not being added to the network's XML definition. When I run it a second time, it will be added and the domain will get the correct IP and hostname over DHCP.

[MalloZup]

Ok thx i will check i dont have terminal now ;)

[MalloZup]

have the feeling this was more o less related to what I experiment there but not sure, need to check

https://github.com/dmacvicar/terraform-provider-libvirt/pull/422

[MalloZup]

@ghtyrant once you have time can you attach logs with TF_LOG=debug terraform apply? thx

[ghtyrant]

Here you go, sorry for the delay:

net-dumpxml before running terraform apply:

<network>
  <name>default</name>
  <uuid>3706f7f6-ef9b-474b-9cb5-5e4ed926df4e</uuid>
  <forward mode='nat'>
    <nat>
      <port start='1024' end='65535'/>
    </nat>
  </forward>
  <bridge name='virbr0' stp='on' delay='0'/>
  <mac address='52:54:00:05:53:9d'/>
  <domain name='vm' localOnly='yes'/>
  <ip family='ipv4' address='192.168.100.1' prefix='24'>
    <dhcp>
      <range start='192.168.100.2' end='192.168.100.254'/>
      <host mac='86:6A:9A:00:1A:6B' name='mail' ip='192.168.100.9'/>
      <host mac='66:D4:98:7F:CE:B1' name='auth' ip='192.168.100.5'/>
      <host mac='86:A5:DB:1C:78:71' name='git' ip='192.168.100.8'/>
      <host mac='86:38:60:96:C4:C3' name='web' ip='192.168.100.7'/>
      <host mac='1A:BA:63:F2:50:35' name='cloud' ip='192.168.100.4'/>
      <host mac='86:63:2d:10:86:0a' name='database' ip='192.168.100.10'/>
    </dhcp>
  </ip>
</network>

Terraform output on the first run: https://gist.github.com/ghtyrant/118dca9360596b8807212780fb719c16

net-dumpxml does not change after the first run.

Terraform output on the second run: https://gist.github.com/ghtyrant/3ebc5cb6e522f79ad8fbe4dbd234bb15

net-dumpxml after running terraform apply a second time:

<network>
  <name>default</name>
  <uuid>3706f7f6-ef9b-474b-9cb5-5e4ed926df4e</uuid>
  <forward mode='nat'>
    <nat>
      <port start='1024' end='65535'/>
    </nat>
  </forward>
  <bridge name='virbr0' stp='on' delay='0'/>
  <mac address='52:54:00:05:53:9d'/>
  <domain name='vm' localOnly='yes'/>
  <ip family='ipv4' address='192.168.100.1' prefix='24'>
    <dhcp>
      <range start='192.168.100.2' end='192.168.100.254'/>
      <host mac='86:6A:9A:00:1A:6B' name='mail' ip='192.168.100.9'/>
      <host mac='66:D4:98:7F:CE:B1' name='auth' ip='192.168.100.5'/>
      <host mac='86:A5:DB:1C:78:71' name='git' ip='192.168.100.8'/>
      <host mac='86:38:60:96:C4:C3' name='web' ip='192.168.100.7'/>
      <host mac='1A:BA:63:F2:50:35' name='cloud' ip='192.168.100.4'/>
      <host mac='86:63:2d:10:86:0a' name='database' ip='192.168.100.10'/>
      <host mac='86:5A:1C:E5:85:4E' name='test' ip='192.168.100.11'/>
    </dhcp>
  </ip>
</network>

[tommyknows]

Can reproduce this issue too - executing apply again solves this.
However, I see a slighty different output of the second apply command; the only thing it changes is the hostname:

libvirt_network.kube_network: Refreshing state... (ID: f89ea3fc-49d4-4293-a381-ea1f285793c6)
libvirt_ignition.ignition: Refreshing state... (ID: /var/lib/libvirt/images/cluster.ign;5be04342-124e-c7b7-4041-5b92aa2e0560)
libvirt_network.test: Refreshing state... (ID: 9484debe-3f33-4dae-bc59-db59cc025e39)
data.template_file.network_config: Refreshing state...
libvirt_volume.coreos: Refreshing state... (ID: /var/lib/libvirt/images/coreos)
data.template_file.user_data: Refreshing state...
libvirt_cloudinit_disk.commoninit: Refreshing state... (ID: /var/lib/libvirt/images/commoninit.iso;5be04342-62e3-72eb-0965-eca6dfbd4c6d)
libvirt_domain.kubernetes: Refreshing state... (ID: 3769d589-6b84-4850-b67c-7f38fcb31781)

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  ~ libvirt_domain.kubernetes
      network_interface.0.hostname: "" => "test1"


Plan: 0 to add, 1 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

libvirt_domain.kubernetes: Modifying... (ID: 3769d589-6b84-4850-b67c-7f38fcb31781)
  network_interface.0.hostname: "" => "test1"
libvirt_domain.kubernetes: Modifications complete after 0s (ID: 3769d589-6b84-4850-b67c-7f38fcb31781)

Apply complete! Resources: 0 added, 1 changed, 0 destroyed.

After that, the host is visible when dumping the XML.

[clanig]

After encountering the same issue, I have done some research.

One might expect Terraform to resolve the name of the network to it's ID automatically. But currently it doesn't.
Specifying the network name in the network_interface currently means that all the specified IPs are ignored:

// when using a "network_name" we do not try to do anything: we just
// connect to that network

https://github.com/dmacvicar/terraform-provider-libvirt/blob/master/libvirt/domain.go#L612
According to the code the network does also have to have DHCP enabled, otherwise the host table will not be updated.

Using the name as specifier also did not work reliably in my case. It only worked for the first network interface of each machine.

As a result, there is a difference between the plan and what is done. According to the plan specifying an IP should result in looking for the network ID by the name, enabling DHCP and adding the hosts to the table of the network.
Following the Code instead, it should vanish the IP in the plan as long as the network's name is given.

Edit: Using the network ID to specify the network instead, avoids all these issues.

[arbulu89]

@MalloZup Same issue here. Hosts entries are only created in the second apply (even using the network ID)

Edit:
As @clanig mentioned, this issue only happens when the network name is used or the network mode is "none". Here, I would expect that even for "none" mode the same behaviour as libvirt supports that:
https://libvirt.org/formatnetwork.html (isolated network config).

Maybe instead of checking if HasDHCP only looking the network modes, it should check if some address is provided by the user too.

[zeenix]

I failed to reproduce the issue with this:

provider "libvirt" {
  uri = "qemu:///system"
}

# Default VM Network
resource "libvirt_network" "bah" {
  name = "bah"
  addresses = ["192.168.100.0/24"]
  domain = "vm"
  bridge = "bah0"
  autostart = true
  dhcp {
    enabled = true
  }
}

The 2nd run of terraform apply says:

ibvirt_network.bah: Refreshing state... (ID: 3024dd47-4bbd-4dd5-9896-96f03f4e906a)

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

and XML dump of the bah network remains exactly the same. @MalloZup So unless I missed something, this can be closed.

[MalloZup]

closing thx