Terraform-provider-libvirt: wait_for_lease loop indefinitely although the guest ip is ok

[gaetanquentin]

System Information

ubuntu 19.10
Terraform v0.12.21
terraform-provider-libvirt 0.6.1+git.1578064534.db13b678
Compiled against library: libvirt 4.0.0
Using library: libvirt 5.4.0
Running hypervisor: QEMU 4.0.0
Running against daemon: 5.4.0

guest vm: redhat 8.1 with qemu-guest-agent

Description of Issue/Question

my guest vm get ip well from dhcp but wait_for_lease do not detect it

Setup

provider "libvirt" {
    uri = "qemu:///system"
}


resource "libvirt_volume" "my-vm-disk-os-qcow2" {
  name   = "my-vm-disk-os-qcow2"
  pool   = "vms"
  source = "/var/lib/libvirt/vms/packer-centos8-base-img/packer-centos8-base"
}

resource "libvirt_volume" "my-vm-disk-data-qcow2" {

  name = "my-vm-disk-data-qcow2"
  pool = "vms"
  size = "20971520"
  format = "qcow2"

  lifecycle {
    prevent_destroy = false
  }
}


resource "libvirt_domain" "my-vm" {
  name = "my-vm"
  memory = "1024"
  vcpu = 1

  disk { volume_id =  libvirt_volume.my-vm-disk-os-qcow2.id }
  disk { volume_id =  libvirt_volume.my-vm-disk-data-qcow2.id }

  network_interface {
     bridge = "br0"
     hostname = "principal"
     wait_for_lease = true
   }


  console {
    type = "pty"
    target_port = "0"
    target_type = "serial"
  }

  graphics {
    type = "vnc"
    listen_type = "address"
  }
}

Steps to Reproduce Issue

(Include debug logs if possible and relevant).

---

Additional information:

Logs:
2020-03-01T16:07:38.692+0100 [DEBUG] plugin.terraform-provider-libvirt: 2020/03/01 16:07:38 [DEBUG] ifaces with addresses: []
2020-03-01T16:07:38.692+0100 [DEBUG] plugin.terraform-provider-libvirt: 2020/03/01 16:07:38 [DEBUG] 52:54:00:62:F9:5A doesn't have IP address(es) yet...
2020-03-01T16:07:38.692+0100 [DEBUG] plugin.terraform-provider-libvirt: 2020/03/01 16:07:38 [DEBUG] IP address not found for iface=52:54:00:62:F9:5A: will try in a while
2020-03-01T16:07:38.692+0100 [DEBUG] plugin.terraform-provider-libvirt: 2020/03/01 16:07:38 [TRACE] Waiting 10s before next try
2020-03-01T16:07:43.633+0100 [DEBUG] plugin.terraform-provider-libvirt: 2020/03/01 16:07:43 [WARN] WaitForState timeout after 5m0s
2020-03-01T16:07:43.633+0100 [DEBUG] plugin.terraform-provider-libvirt: 2020/03/01 16:07:43 [WARN] WaitForState starting 30s refresh grace period
2020-03-01T16:07:43.633+0100 [DEBUG] plugin.terraform-provider-libvirt: 2020/03/01 16:07:43 [DEBUG] wait-for-leases was successful
2020/03/01 16:07:43 [DEBUG] libvirt_domain.nitk-front: apply errored, but we're indicating that via the Error pointer rather than returning it: Error: couldn't retrieve IP address of domain.Please check following:
1) is the domain running proplerly?
2) has the network interface an IP address?
3) Networking issues on your libvirt setup?
4) is DHCP enabled on this Domain's network?
5) if you use bridge network, the domain should have the pkg qemu-agent installed
IMPORTANT: This error is not a terraform libvirt-provider error, but an error caused by your KVM/libvirt infrastructure configuration/setup

Guest infos

the ip is well got by the guest (dhcp):
ens3: flags=4163 mtu 1500
inet 172.16.10.247 netmask 255.255.255.0 broadcast 172.16.10.255
inet6 fe80::5054:ff:fe62:f95a prefixlen 64 scopeid 0x20
ether 52:54:00:62:f9:5a txqueuelen 1000 (Ethernet)
RX packets 8975 bytes 12680520 (12.0 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 3448 bytes 224826 (219.5 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

[gaetanquentin]

additionnal info:
this:
output "ip" {
value = libvirt_domain.vm-template.network_interface[0]
}

will show empty address

and:
virsh domifaddr my-dom --full --source lease won't show anything

and
virsh domifaddr nitk-data --full --source agent will show ip and mac

maybe it doesn't use agent to get ip?

[MalloZup]

@gaetanquentin I don't have time to dive much on this, but as potential hint you could verify if your vms host have qemu-agent pkg installed https://wiki.libvirt.org/page/Qemu_guest_agent

Afaik, when you have bridge, you need to have qemu-agent as potential method to get the Ip adress from leases.

[gaetanquentin]

hi @MalloZup . as said in my first post qemu-ga is well installed on guests . but how to verify that terraform uses the agent to retreive ip/mac information?

[MalloZup]

hi @gaetanquentin please run

TF_LOG=DEBUG terraform apply

and post the output. But I know for sure that terraform use the qemu-agent if is there, since I remeber the code there.

[frenos]

I would like to progress this issue with more information. I think I run into the same problem or at least it looks the same.
I'm trying to create a VM with terraform and have it configured with cloud-init. The qemu-agent is also installed with the cloud-init script.

Running on Ubuntu 18.04 with libvirt etc installed from default package sources

2020/07/16 11:28:54 [INFO] Terraform version: 0.12.28  
2020/07/16 11:28:54 [INFO] Go runtime version: go1.12.13
2020/07/16 11:28:54 [DEBUG] found valid plugin: "libvirt", "0.0.0", "/home/frederic/.terraform.d/plugins/terraform-provider-libvirt" (master commit b88a9dbb4d11b6c0af7b2324a48016a71a061855 (from 15.07.20))

I thought I maybe ran into the timeout problem described in #731 but it just waits longer and fails then :(.
The VM configuration is pretty basic, I use the Ubuntu 18.04 cloud image and create a default drive.

# Create the machine
resource "libvirt_domain" "domain-ubuntu" {
  name = var.hostname
  memory = var.memoryMB
  vcpu = var.cpu

  disk {
       volume_id = libvirt_volume.os_image_resized.id
  }
  network_interface {
       vepa = "eno1"
       wait_for_lease = true
  }

  cloudinit = libvirt_cloudinit_disk.commoninit.id

  # IMPORTANT
  # Ubuntu can hang is a isa-serial is not present at boot time.
  # If you find your CPU 100% and never is available this is why
  console {
    type        = "pty"
    target_port = "0"
    target_type = "serial"
  }

  graphics {
    type = "spice"
    listen_type = "address"
    autoport = "true"
  }

}

If I run virsh qemu-agent-command runner '{"execute": "guest-network-get-interfaces"}' multiple times while the process runs it works as soon as the agent is installed in the cloud-init process and gives

{
    "return": [{
        "name": "lo",
        "ip-addresses": [{
            "ip-address-type": "ipv4",
            "ip-address": "127.0.0.1",
            "prefix": 8
        }, {
            "ip-address-type": "ipv6",
            "ip-address": "::1",
            "prefix": 128
        }],
        "statistics": {
            "tx-packets": 176,
            "tx-errs": 0,
            "rx-bytes": 15340,
            "rx-dropped": 0,
            "rx-packets": 176,
            "rx-errs": 0,
            "tx-bytes": 15340,
            "tx-dropped": 0
        },
        "hardware-address": "00:00:00:00:00:00"
    }, {
        "name": "ens3",
        "ip-addresses": [{
            "ip-address-type": "ipv4",
            "ip-address": "192.168.1.24",
            "prefix": 24
        }, {
            "ip-address-type": "ipv6",
            "ip-address": "fe80::5054:ff:fe2e:66f",
            "prefix": 64
        }],
        "statistics": {
            "tx-packets": 88054,
            "tx-errs": 0,
            "rx-bytes": 308616688,
            "rx-dropped": 0,
            "rx-packets": 121612,
            "rx-errs": 0,
            "tx-bytes": 7824138,
            "tx-dropped": 0
        },
        "hardware-address": "52:54:00:2e:06:6f"
    }, {
        "name": "docker0",
        "ip-addresses": [{
            "ip-address-type": "ipv4",
            "ip-address": "172.17.0.1",
            "prefix": 16
        }],
        "statistics": {
            "tx-packets": 0,
            "tx-errs": 0,
            "rx-bytes": 0,
            "rx-dropped": 0,
            "rx-packets": 0,
            "rx-errs": 0,
            "tx-bytes": 0,
            "tx-dropped": 0
        },
        "hardware-address": "02:42:18:11:86:e8"
    }]
}

The debug log reapeats this message until the 5min timeout. After about 2min the agent installation is finished in the VM and the virsh command (see above) works.

2020-07-16T11:33:45.138+0200 [DEBUG] plugin.terraform-provider-libvirt: 2020/07/16 11:33:45 [DEBUG] waiting for network address for iface=52:54:00:2E:06:6F
2020-07-16T11:33:45.138+0200 [DEBUG] plugin.terraform-provider-libvirt: 2020/07/16 11:33:45 [DEBUG] qemu-agent is not used
2020-07-16T11:33:45.138+0200 [DEBUG] plugin.terraform-provider-libvirt: 2020/07/16 11:33:45 [DEBUG] no interfaces could be obtained with qemu-agent: falling back to the libvirt API
2020-07-16T11:33:45.138+0200 [DEBUG] plugin.terraform-provider-libvirt: 2020/07/16 11:33:45 [DEBUG] Interfaces info obtained with libvirt API:
2020-07-16T11:33:45.139+0200 [DEBUG] plugin.terraform-provider-libvirt: ([]libvirt.DomainInterface) {
2020-07-16T11:33:45.139+0200 [DEBUG] plugin.terraform-provider-libvirt: }

debug_log.txt

It seems like the plugin does not pick up an agent correctly when it's installed while trying?
If you need any more data or have ideas what I could try I would be very happy to help.

/update:
I investigated further and tried an Ubuntu image with qemu-agent preinstalled. Terraform apply still has the same error until timeout, virsh can get the ip directly after boot of vm.

[aaannz]

I have this issue when using terraform agains MicroOS (aka tumbleweed) based libvirt (libvirt-6.5.0-1.1.x86_64), qemu (qemu-5.0.0-3.1.x86_64). In the log file of libvirt I see:

libvirtd[19101]: internal error: unable to handle agent type: pty
libvirtd[19101]: g_main_context_unref: assertion 'context != NULL' failed
libvirtd[19101]: Cannot connect to QEMU guest agent for slepos-virt-28

I looked how qemu-agent is being setup and I changed its channel from using pty to use unix socket. This fixed it for me, but it was done in hacky way.

[aaannz]

This change fixed it for me:

diff --git a/libvirt/domain_def.go b/libvirt/domain_def.go
index f68b7abd..04e55339 100644
--- a/libvirt/domain_def.go
+++ b/libvirt/domain_def.go
@@ -62,6 +62,9 @@ func newDomainDef() libvirtxml.Domain {
                        },
                        Channels: []libvirtxml.DomainChannel{
                                {
+                                       Source: &libvirtxml.DomainChardevSource{
+                                               UNIX: &libvirtxml.DomainChardevSourceUNIX{},
+                                       },
                                        Target: &libvirtxml.DomainChannelTarget{
                                                VirtIO: &libvirtxml.DomainChannelTargetVirtIO{
                                                        Name: "org.qemu.guest_agent.0",

It changes default to use guest agent over unix. I'm not sure if this is acceptable in general.

[MalloZup]

thx @aaannz, Yeah good question. It might be something need research/tests to see this change.

A good solution would be perhaps to let the provider deterimine intelligently which source need to be used. But it will cost some efforts to implement

[gaetanquentin]

FYI qemu_agent = true was missing in the libvirt_domain resource declaration. It worked fine then.

[aaannz]

FYI qemu_agent = true was missing in the libvirt_domain resource declaration. It worked fine then.

Then I was solving different issue with the same symptoms. In my case terraform is controlled by sumaform and there is qemu_agent = true

[ajilty]

FYI qemu_agent = true was missing in the libvirt_domain resource declaration. It worked fine then.

Then I was solving different issue with the same symptoms. In my case terraform is controlled by sumaform and there is qemu_agent = true

@aaannz I think you need something like this in your cloud-init. I don't think the agent is starting at first boot

package_update: true
packages:
    - qemu-guest-agent
runcmd:
   - systemctl start qemu-guest-agent.service 

[MalloZup]

closing this issue as duplicated of #765 @aaannz I guess the problem is same in that issue. I guess something has changed on the qemu-agent new package

[dvzrv]

@MalloZup I think the issues are actually two separate ones! In #765 the issue is, that the channel type for the qemu-guest-agent is not generated correctly.

However, this issue is about terraform timing out although the network is properly setup, the domains are reachable, the domains have qemu-guest-agent running and reachable and have a working network configuration (and I can still reproduce this, after fixing #765).

Fixing #765 is a prerequisite to fixing this issue it seems.

[dvzrv]

@MalloZup I think I have to revise my previous comment. My pull request (which is actually @aaannz 's suggestion) fixes the issue also for this case (where wait_for_lease = true and qemu_agent = true on a bridge interface that is handled externally).

Due to the various bug reports tracking this got a bit confusing, but I'm also positive now, that this here issue is just a symptom of #765. No re-opening required!