Terraform-provider-libvirt: Unable to create multiple cloudinit_disks in parallel when connecting to remote libvirt

[2stacks]

System Information

Linux distribution

Ubuntu 16.04

Terraform version

0.11.13

Provider and libvirt versions

terraform-provider-libvirt was not built correctly
Compiled against library: libvirt 1.3.1
Using library: libvirt 1.3.1
Running hypervisor: QEMU 2.5.0
Running against daemon: 1.3.1

If that gives you "was not built correctly", get the Git commit hash from your local provider repository:

46b0ba055a47587a8c0b4b2c76f5e47165b7c6c4-dirty

---

Description of Issue/Question

Not sure if this is a bug or if the libvirt_cloudinit_disk resource was not intended to be used this way however, when trying to create multiple instances of cloudinit_disk with count, creation fails when disks are created in parallel.

This only happens when the provider resource is over ssh as shown below. I can not reproduce this issue when Terraform and libvirt are running on the same system.

Additionally if I run Terraform with -parallelism=1 all resources are created as expected on both local and remote libvirt systems.

Setup

provider "libvirt" {
  uri = "qemu+ssh://<remote_ip>/system"
  #uri = "qemu:///system"
}

data "template_file" "user_data" {
  template = "${file("${path.module}/cloud_init.yml")}"

  vars {
    user_name          = "test"
    ssh_authorized-key = "test"
  }
}

data "template_file" "meta_data" {
  count    = 3
  template = "${file("${path.module}/meta_data.yml")}"

  vars {
    hostname = "${format("test-%02d", count.index + 1)}"
  }
}

data "template_file" "network_config" {
  template = "${file("${path.module}/network_config.yml")}"
}

resource "libvirt_cloudinit_disk" "commoninit" {
  count          = 3
  name           = "${format("test-seed-%01d.iso", count.index + 1)}"
  pool           = "default"
  user_data      = "${data.template_file.user_data.rendered}"
  meta_data      = "${data.template_file.meta_data.*.rendered[count.index]}"
  network_config = "${data.template_file.network_config.rendered}"
}

Steps to Reproduce Issue

export TF_LOG=TRACE
terraform plan
terraform apply

Terraform will loop trying to create the second disk and never complete.

libvirt_cloudinit_disk.commoninit[1]: Creation complete after 1s (ID: /ds01/test-seed-2.iso;5cd48747-22f9-11de-7618-680f963ee281)
2019/05/09 16:02:15 [TRACE] root: eval: *terraform.EvalUpdateStateHook
2019/05/09 16:02:15 [TRACE] Preserving existing state lineage "b86357de-6b2b-58cc-b306-d811eb71e7e8"
2019/05/09 16:02:15 [TRACE] Preserving existing state lineage "b86357de-6b2b-58cc-b306-d811eb71e7e8"
2019/05/09 16:02:15 [TRACE] Preserving existing state lineage "b86357de-6b2b-58cc-b306-d811eb71e7e8"
2019/05/09 16:02:15 [TRACE] Preserving existing state lineage "b86357de-6b2b-58cc-b306-d811eb71e7e8"
2019/05/09 16:02:15 [TRACE] [walkApply] Exiting eval tree: libvirt_cloudinit_disk.commoninit[1]
2019-05-09T16:02:15.545-0400 [DEBUG] plugin.terraform-provider-libvirt: 2019/05/09 16:02:15 376832 bytes uploaded
2019/05/09 16:02:19 [TRACE] dag/walk: vertex "provider.libvirt (close)", waiting for: "libvirt_cloudinit_disk.commoninit[0]"
2019/05/09 16:02:19 [TRACE] dag/walk: vertex "meta.count-boundary (count boundary fixup)", waiting for: "libvirt_cloudinit_disk.commoninit[0]"
2019/05/09 16:02:19 [TRACE] dag/walk: vertex "root", waiting for: "provider.libvirt (close)"
2019/05/09 16:02:24 [TRACE] dag/walk: vertex "provider.libvirt (close)", waiting for: "libvirt_cloudinit_disk.commoninit[0]"
2019/05/09 16:02:24 [TRACE] dag/walk: vertex "root", waiting for: "provider.libvirt (close)"
2019/05/09 16:02:24 [TRACE] dag/walk: vertex "meta.count-boundary (count boundary fixup)", waiting for: "libvirt_cloudinit_disk.commoninit[0]"
libvirt_cloudinit_disk.commoninit.2: Still creating... (10s elapsed)
libvirt_cloudinit_disk.commoninit.0: Still creating... (10s elapsed)
2019/05/09 16:02:29 [TRACE] dag/walk: vertex "root", waiting for: "provider.libvirt (close)"
2019/05/09 16:02:29 [TRACE] dag/walk: vertex "meta.count-boundary (count boundary fixup)", waiting for: "libvirt_cloudinit_disk.commoninit[0]"
2019/05/09 16:02:29 [TRACE] dag/walk: vertex "provider.libvirt (close)", waiting for: "libvirt_cloudinit_disk.commoninit[0]"
2019/05/09 16:02:34 [TRACE] dag/walk: vertex "root", waiting for: "provider.libvirt (close)"
2019/05/09 16:02:34 [TRACE] dag/walk: vertex "meta.count-boundary (count boundary fixup)", waiting for: "libvirt_cloudinit_disk.commoninit[0]"
2019/05/09 16:02:34 [TRACE] dag/walk: vertex "provider.libvirt (close)", waiting for: "libvirt_cloudinit_disk.commoninit[0]"

---

Additional information:

The reason I'm creating multiple cloudinit disks is to be able to pass different hostnames to the guest domains via cloud-init.

[MalloZup]

@2stacks thx for this. Yes i can confirm this bug, and i was experimenting it time ago this same issue when using a remote.

At that time I came at same conclusion as you, that there is a deadlock somewhere even if the disks are created.

[2stacks]

Unfortunately I don't know enough Go to investigate much further but Im
happy to test things as needed. Im currently using the provider to deploy
everything from Kubernetes clusters to Wifi controllers and this is the
only issue Ive come across.

Its odd that I can create multiple libvirt_volumes in parallel without
locking issues. I would have assumed both resources would behave the same.

On Mon, May 13, 2019, 4:32 AM Dario Maiocchi [e-mail]
wrote:

@2stacks https://github.com/2stacks thx for this. Yes i can confirm
this bug, and i was experimenting it time ago this same issue when using a
remote.

At that time I came at same conclusion as you, that there is a deadlock
somewhere even if the disks are created.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/dmacvicar/terraform-provider-libvirt/issues/594#issuecomment-491727443,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ADUNEHFH6X5PUTFFLJDJWXLPVERQHANCNFSM4HL5TN4A
.

[MalloZup]

@2stacks ok thx. Yop I understand and agree.

First at all, thx for your really nice report, this help us a lot.

My question is, are you able to reproduce this every time the issue or just from time to time?

Are the 2 OS systems the same? Ubuntu with same libvirt version etc?

[2stacks]

Yes I am able to reproduce this every time. The only way I can deploy multiple resources to my remote libvirt hosts is by using the -parallelism=1 option. Both my local and remote system are on (libvirtd (libvirt) 1.3.1). I thought it may be the type of storage as well so I tried a few different storage pools (default, nfs and zfs). They all failed the same except for zfs but I think that may be a different issue.

[MalloZup]

If you have a chance and time to do it, i would like to known if you upgrade to a libvirt version like major like 4 or 5 and see if you can reproduce. This would let us exclude a libvirt bug.

[2stacks]

I've compiled libvirt 5.3.0 on my host (Ubuntu 16.04) and the issue is not there. I'm able to create the cloudinit_disks in parallel. Let me know if you want me to test any other versions.

[MalloZup]

@2stacks 5.30 on both hosts would be perfect!. If it works, we have an answer :tada: And we might need to lower the version until it doesn't. Otherwise it something on the provider

[2stacks]

I was able to test successfully with 1.3.1 on the client and 5.3.0 on the
server. There doesn't appear to be any locking issues in 5.3.0. Thanks
for your help.

On Fri, May 17, 2019, 4:44 AM Dario Maiocchi [e-mail]
wrote:

@2stacks https://github.com/2stacks 5.30 on both hosts would be
perfect!. If it works, we have an answer 🎉

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/dmacvicar/terraform-provider-libvirt/issues/594?email_source=notifications&email_token=ADUNEHESFV6MFFN7KLG4R4TPVZV7NA5CNFSM4HL5TN4KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODVUEV6Q#issuecomment-493374202,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ADUNEHEHIBULHVCLZ2HQVRTPVZV7NANCNFSM4HL5TN4A
.

[kskewes]

Thanks for finding this! I was stuck with 1.3.x on server as well.
Have now been able to do this with v4.0.0 on the server which is default in Ubuntu 18.04 LTS.

[2stacks]

Thanks for the update. I may look at upgrading my OS. Compiling Libvirt
and getting it to work wasn't fun.

On Wed, May 29, 2019 at 4:24 AM Karl [e-mail] wrote:

Thanks for finding this! I was stuck with 1.3.x on server as well.
Have now been able to do this with v4.0.0 on the server which is default
in Ubuntu 18.04 LTS.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/dmacvicar/terraform-provider-libvirt/issues/594?email_source=notifications&email_token=ADUNEHBNE2VKMMWEMVEYZNDPXY4URA5CNFSM4HL5TN4KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODWOSEHY#issuecomment-496837151,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ADUNEHAHAMHHZJG6NKD5SPDPXY4URANCNFSM4HL5TN4A
.

[MalloZup]

@2stacks thx for this. I was asking my self if we might add a documentation on this or something related, like known libvirt issues

Because having an issue open isn't something correct in this sense.

What do you think it could improve the UX?

[2stacks]

Maybe something generic to warn about really old libvirt versions would
do. It was a bit of work to pinpoint the issue so I definitely see how a
mention could save someone else time. This particular issue may be so
specific that the only hope one has is to come across this post.

On Wed, Jun 5, 2019, 11:43 AM Dario Maiocchi [e-mail]
wrote:

@2stacks https://github.com/2stacks thx for this. I was asking my self
if we might add a documentation on this or something related, like known
libvirt issues

Because having an issue open isn't something correct in this sense.

What do you think it could improve the UX?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/dmacvicar/terraform-provider-libvirt/issues/594?email_source=notifications&email_token=ADUNEHHO3F5LTOTRCMLSJNDPY7NKPA5CNFSM4HL5TN4KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXAEHZI#issuecomment-499139557,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ADUNEHCNKERVSXB4SODRX73PY7NKPANCNFSM4HL5TN4A
.

[MalloZup]

closing since I need to clean up backlog .