Terraform-provider-libvirt: could not open disk image /dev/vghost/vol1: Image is not in qcow2 format

[mjiao]

Version Reports:

Distro version of host:

Centos7

Terraform Version Report

(Provided by running terraform -v.)
0.11.7

Libvirt version

virsh --version
3.9.0

terraform-provider-libvirt plugin version (git-hash)

git log
0.4.2

---

Description of Issue/Question

Setup

(Please provide the full main.tf file for reproducing the issue (Be sure to remove sensitive info)

variable "domain_name" {
  type = "string"
}

variable "hypervisor" {
  type = "string"
  default = "aaa.in"
}

provider "libvirt" {
    uri = "qemu+ssh://root@${var.hypervisor}/system"
}

resource "libvirt_volume" "base_vol" {
  name = "${var.domain_name}_base_vol"
  source = "/home/jenkins/output-images/centos7-base"
}

resource "libvirt_volume" "base_disk" {
  name = "${var.domain_name}_base_disk"
  base_volume_id = "${libvirt_volume.base_vol.id}"
}

resource "libvirt_volume" "vol1" {
  name = "vol1"
  # vghost is a lvm storage pool
  pool = "vghost"
  size = 10000000000
  format = "raw"
}

resource "libvirt_domain" "terraform_test" {
  name = "${var.domain_name}"
  disk {
    volume_id = "${libvirt_volume.base_disk.id}"
    scsi = "true"
  }
  disk {
    volume_id = "${libvirt_volume.vol1.id}"
  }
  vcpu = "1"
  memory = "2048"
  running = "true"
  network_interface = {
    bridge = "br1"
    hostname = "${var.domain_name}"
    # wait_for_lease = 1
  }
  autostart = "true"
}

Steps to Reproduce Issue

(Include debug logs if possible and relevant.)
terraform init
terraform apply -var domain_name=test

---

Additional Infos:

The storage pool used is based on lvm:
pool = "vghost"
It seems that it always uses format qcows in the disk scope in the domain definition.

Error: Error applying plan:

1 error(s) occurred:

2018/08/15 00:57:34 [DEBUG] plugin: waiting for all plugin processes to complete...
* libvirt_domain.terraform_test: 1 error(s) occurred:

* libvirt_domain.terraform_test: Error creating libvirt domain: virError(Code=1, Domain=10, Message='internal error: process exited while connecting to monitor: 2018-08-14T22:57:34.437492Z qemu-kvm: -drive file=/dev/vghost/vol1,format=qcow2,if=none,id=drive-virtio-disk1: could not open disk image /dev/vghost/vol1: Image is not in qcow2 format')

Do you have SELinux or Apparmor/Firewall enabled? Some special configuration?
Have you tried to reproduce the issue without them enabled?
Yes

[dmacvicar]

Thanks for the report. I think the provider lacks logic in terms of what format to use given storage pool types.

Can you share a working domain definition? Do you need to specify the format or will libvirt automatically detect/select the right one? If we confirm that libvirt is smarter than I think, we may actually remove setting/hardcoding the format.

_{Sent with GitHawk}

[andreashaerter]

Hello,

A bit of framing: There is a LVM volgroup /dev/vghost. A VM shall get a logical volume in there as disk. The volgroup was added as storage pool via virsh pool-define-as 'vghost' 'logical' --target '/dev/vghost'. It is running and active:

$ virsh pool-list
 Name                 Status     Automatischer Start
-------------------------------------------
 default              Aktiv      ja        
 [...]      
 vghost               Aktiv      ja

The Terraform run @mjiao mentioned created the volume:

$ lsblk
NAME                                  MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda                                     8:0    0  5,5T  0 disk 
├─sda1                                  8:1    0    1M  0 part 
├─sda2                                  8:2    0  768M  0 part /boot
└─sda3                                  8:3    0  5,5T  0 part 
  ├─[...]
  ├─vghost-vm_jenkins.netzwerk.host   253:8    0  100G  0 lvm 
  └─vghost-vm_openshift.netzwerk.host 253:15   0  100G  0 lvm

A working manual domain creation would look as follows (took a Jenkins test VM as example):

$ sudo lvcreate --name 'vm_jenkins.netzwerk.host' --size '100G' 'vghost'
$ sudo virsh pool-refresh 'vghost'
$ sudo virt-install --connect qemu:///system --hvm --graphics spice --noautoconsole \
  --name=jenkins.netzwerk.host --description 'The Jenkins' --os-type linux \
  --os-variant centos7.0 --arch=x86_64 --vcpus=2 --ram=4096 --check-cpu --cpu host \
  --network bridge=br4,model=virtio --cdrom=/var/lib/libvirt/iso/CentOS-7-x86_64-Minimal-1804.iso \
  --disk vol=vghost/vm_jenkins.netzwerk.host,cache=none,bus=virtio

Resulting in

$ virsh dumpxml jenkins.netzwerk.host | less
[...]
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native'/>
      <source dev='/dev/vghost/vm_jenkins.netzwerk.host'/>
      <backingStore/>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </disk>
[...]

Edit: Attached the whole domain definition of a working VM which was created manually: 2018-08-16_virsh-dump-jenkins.netzwerk.host.txt

[dmacvicar]

Ok, this seems to be the relevant part of the documentation:

Valid pool format types
The logical volume pool supports only the lvm2 format, although not supplying a
format value will result in automatic selection of the _lvm2_ format.

Valid volume format types
The logical volume pool does not use the volume format type element.

So I guess we could fix it by not passing any format. However, I am suspicious of the XML dump, given that the volume was created by the provider code that mostly assumes that we are dealing with a _dir_ pool and _qcow2_ files.

Reading _libvirt_ documentation I notice there is a source="volume" supported since _0.8.7_ that I suspect may help in referring to a volume in a pool in a more standardized way across pool types, instead of hardcoding something like file=. I will test once I get access to my laptop in a few days (on iPad right now).

[mjiao]

Thanks for your reply @dmacvicar . In addition to the info provided by @andreashaerter, here is info regarding the successful and failed cases:

Success case

The following is an example of a working domain terraform code and its xml dump (only a Packer baked image is used as base of volume creation).

resource "libvirt_volume" "base_vol" {
  name = "${var.domain_name}_base_vol"
  # The centos7-base is in qcow2 format
  source = "/home/jenkins/output-images/centos7-base"
}

resource "libvirt_volume" "base_disk" {
  name = "${var.domain_name}_base_disk"
  base_volume_id = "${libvirt_volume.base_vol.id}"
}
resource "libvirt_domain" "terraform_test" {
  name = "${var.domain_name}"
  disk {
    volume_id = "${libvirt_volume.base_disk.id}"
    scsi = "true"
  }
  vcpu = "1"
  memory = "2048"
  running = "true"
  network_interface = {
    bridge = "br120"
    hostname = "${var.domain_name}"
    # wait_for_lease = 1
  }
  autostart = "true"
}

The xml dump:

<domain type='kvm' id='85'>
  <name>issue358</name>
  <uuid>069fdae8-305d-46b3-a9f1-1912a0e451c5</uuid>
  ...
  <devices>
  ...
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/issue358_base_disk'/>
      <backingStore type='file' index='1'>
        <format type='qcow2'/>
        <source file='/var/lib/libvirt/images/issue358_base_vol'/>
        <backingStore/>
      </backingStore>
      <target dev='vda' bus='scsi'/>
      <wwn>05abcd97de317c75</wwn>
      <alias name='scsi0-0-0-0'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    ...
  </devices>
</domain>

Failed cases

1. Try to allocate from lvm pool in format raw/qcow2

resource "libvirt_volume" "vol1" {
  name = "vol1"
  pool = "vghost"
  size = 1000000000
  # error message is the same while format = "qcow2"
  format = "raw"
}

Error: Error applying plan:

1 error(s) occurred:

* libvirt_domain.terraform_test: 1 error(s) occurred:

* libvirt_domain.terraform_test: Error creating libvirt domain: virError(Code=1, Domain=10, Message='internal error: qemu unexpectedly closed the monitor: 2018-08-16T22:33:00.528488Z qemu-kvm: -drive file=/dev/vghost/vol1,format=qcow2,if=none,id=drive-scsi0-0-0-1: could not open disk image /dev/vghost/vol1: Image is not in qcow2 format')



md5-1d9c5ea0aa4e13953d70e5414c8435b5



resource "libvirt_volume" "vol1" {
  name = "vol1"
  pool = "default"
  size = 1000000000
  format = "raw"
}

resource "libvirt_domain" "terraform_test" {
  name = "${var.domain_name}"
  disk {
    volume_id = "${libvirt_volume.base_disk.id}"
    scsi = "true"
  }
  disk {
    volume_id = "${libvirt_volume.vol1.id}"
  }
 ...
}



md5-65b14ab0213581748bf51e67e1fe8d88



* libvirt_domain.terraform_test: 1 error(s) occurred:

* libvirt_domain.terraform_test: Error creating libvirt domain: virError(Code=1, Domain=10, Message='internal error: qemu unexpectedly closed the monitor: 2018-08-16T22:11:05.933393Z qemu-kvm: -drive file=/var/lib/libvirt/images/vol1,format=qcow2,if=none,id=drive-virtio-disk1: could not open disk image /var/lib/libvirt/images/vol1: Image is not in qcow2 format')



md5-2014792a9b4598fdc14938578b0e3700



...
resource "libvirt_volume" "vol1" {
  name = "vol1"
  pool = "default"
  size = 1000000000
  format = "qcow2"
}

resource "libvirt_domain" "terraform_test" {
...
  disk {
    volume_id = "${libvirt_volume.vol1.id}"
  }
...
}



md5-1919a1eca9a291b4dd9c3bdffaaa1c3a



...
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/vol1'/>
      <backingStore/>
      <target dev='vdb' bus='virtio'/>
      <alias name='virtio-disk1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </disk>
...

[dmacvicar]

I have started to work in this issue. I am still unsure what is the best way of fixing it.

Originally I thought it was directly related to the fact that we hardcoded the file type of volume, and I already have a branch improving this. However this was only part of it.

There are two format parameters. One is part of the volume definition, and then there is an attribute for the _driver_ that the disk uses to access the volume. And those should match.

So my current idea would be:

• For uploaded volumes (source=) use some _mime_ library to find out if the file is a _qcow2_ and auto-set the volume format to _qcow2_. Right now we rely on a default and on looking at the extension.
• For terraform created volumes, default to _raw_ instead of _qcow2_ unless the format parameter is specified.
• For the terraform disk referring to a volume, instead of hardcoding the _qcow2_ driver (which we do right now, except for _iso_ files), get the volume definition and check the format.

I believe with all these pieces, the provider will behave better. There may still be corner cases, as we don't want to introduce the complexity of having to expose the _driver_ in the disk definition.

I will post updates when I get to implement this and post the branch for testing.

[dmacvicar]

@mjiao Are you able to build a test branch yourself in order to test?

[mjiao]

@dmacvicar yes, I can do that.

[dmacvicar]

Hi @mjiao. See the PR referenced above.

[mjiao]

@dmacvicar thanks for the PR. Will test that.

[MalloZup]

Thx @mjiao

[mjiao]

Test summary: it works correctly

• [x] Volume format is set correctly for both qcow2 and raw while creating on default pool. Domain is configured correctly. Tested using the examples below:

variable "domain_name" {
  type    = "string"
  default = "test"
}

resource "libvirt_volume" "base_vol" {
  name = "${var.domain_name}_base_vol"
  # source = "/home/jenkins/output-images/test-centos7-base.qcow2"
  source = "/home/jenkins/output-images/test-centos7-base.raw"
}

resource "libvirt_domain" "terraform_test" {
  name = "${var.domain_name}"
  disk {
    volume_id = "${libvirt_volume.base_vol.id}"
    scsi = "true"
  }

  vcpu = "1"
  memory = "2048"
  running = "true"
  network_interface = {
    bridge = "br120"
    hostname = "${var.domain_name}"
    # wait_for_lease = 1
  }
  autostart = "true"
}

• [x] raw volumes can be created correctly using vghost lvm pool. Domain is configured correctly with the correct format of volume attached. Tested using the example below:

variable "domain_name" {
  type    = "string"
  default = "test"
}

resource "libvirt_volume" "base_vol" {
  name = "${var.domain_name}_base_vol"
  pool = "vghost"
  source = "/home/jenkins/output-images/test-centos7-base.raw"
}

resource "libvirt_domain" "terraform_test" {
  name = "${var.domain_name}"
  disk {
    volume_id = "${libvirt_volume.base_vol.id}"
    scsi = "true"
  }

  vcpu = "1"
  memory = "2048"
  running = "true"
  network_interface = {
    bridge = "br120"
    hostname = "${var.domain_name}"
    # wait_for_lease = 1
  }
  autostart = "true"
}

• [x] Extra volumes can also be created and attached to the domain with the correct format.

variable "domain_name" {
  type    = "string"
  default = "test"
}

resource "libvirt_volume" "base_vol" {
  name = "${var.domain_name}_base_vol"
  pool = "vghost"
  source = "/home/jenkins/output-images/test-centos7-base.raw"
}

resource "libvirt_volume" "1st_vol" {
  name = "${var.domain_name}_1st_vol"
  pool = "vghost"
  size = 1000000000
}

resource "libvirt_domain" "terraform_test" {
  name = "${var.domain_name}"
  disk {
    volume_id = "${libvirt_volume.base_vol.id}"
    scsi = "true"
  }
  disk {
    volume_id = "${libvirt_volume.1st_vol.id}"
  }

  vcpu = "1"
  memory = "2048"
  running = "true"
  network_interface = {
    bridge = "br120"
    hostname = "${var.domain_name}"
    # wait_for_lease = 1
  }
  autostart = "true"
}

Thanks a lot for your effort! @dmacvicar @MalloZup @andreashaerter

[MalloZup]

merged closing :+1: @mjiao thank you for your feedback !! :rose: