Terraform-provider-libvirt: libvirt_volume: add compressed image to "source" ?

[remoe]

I want to add a compressed image to source of libvirt_volume:

resource "libvirt_volume" "coreos-qcow2" {
  name = "coreos-qcow2"
  source = "https://stable.release.core-os.net/amd64-usr/current/coreos_production_qemu_image.img.bz2"
}

Container Linux does not provide an uncompressed qcow2-image.
Is it possible to create an image from a compressed source file ?

[dmacvicar]

@MalloZup this is not about changing the format, but supporting compressed images on the fly. So doing decompression as we upload to libvirt, so that one can continue using the compressed URLs.

Look here:
https://github.com/dmacvicar/terraform-provider-libvirt/blob/master/libvirt/utils_volume.go#L62
https://github.com/dmacvicar/terraform-provider-libvirt/blob/master/libvirt/utils_volume.go#L161

Both Import methods return use a _copier_ constructed with the libvirt volume.

The _copier_ is constructed here and takes the _reader_:
https://github.com/dmacvicar/terraform-provider-libvirt/blob/master/libvirt/utils_volume.go#L197

So it is probably a matter of wrapping the reader with another reader that does automatic decompression (doing _Peek_ internally), do a _Peek_ and see if the first two bytes are 0x1f8b for _gzip_, etc and wrap conditionally.

PRs welcome.

[MalloZup]

@dmacvicar yop i was thinking the same about the reader, but since we have some coreos users i was wondering how/why they didn"t asked for such feature before and if they were using other stuff like unzipping image and take the url from there via provider or other ways to do it i didn"t know.

yop the solution is to implement the reader with gzip in that case.

[MalloZup]

i can take it since i know already the codebase there,

[dmacvicar]

Be careful with not breaking the volume format auto/detection, which is used to configure the disk driver.

[MalloZup]

https://stackoverflow.com/questions/28309988/how-to-read-from-either-gzip-or-plain-text-reader-in-golang

• http://www.onicos.com/staff/iz/formats/gzip.html
• https://en.wikipedia.org/wiki/List_of_file_signatures

[MalloZup]

i will fix this on PR https://github.com/dmacvicar/terraform-provider-libvirt/pull/475 on wip , closing since i will track the work there thx

@remoe once that PR got merged you will be able to use compressed images ( at moment is in WIP)

[MalloZup]

@remoe can you give me a full main.tf example so i could add it to the example ?
TIA.

I did some progress and i would like to test them with a real use case ! :football:

[remoe]

@MalloZup , I haven't done much with terraform, but i hope this helps:

resource "libvirt_volume" "os_image" {
  name = "os_image-${var.name}"
  source = "https://stable.release.core-os.net/amd64-usr/current/coreos_production_qemu_image.img.bz2"
  format = "bz2"
}

resource "libvirt_volume" "volume" {
  count = "${var.instance_count}"
  name = "${var.name}-volume-${count.index + 1}"
  base_volume_id = "${libvirt_volume.os_image.id}"
}

resource "libvirt_domain" "instance" {
  count = "${var.instance_count}"
  name = "${var.instance_count > 1 ? "${var.name}${count.index + 1}" : "${var.name}"}" 
  memory = "${var.memory}"
  vcpu = "${var.vcpu}"

  coreos_ignition = "${var.ignition}"

  network_interface {
    network_name = "${var.network}"
    wait_for_lease = true
  }

  disk {
       volume_id = "${element(libvirt_volume.volume.*.id, count.index)}"
  }

  console {
    type        = "pty"
    target_port = "0"
    target_type = "serial"
  }

  console {
      type        = "pty"
      target_type = "virtio"
      target_port = "1"
  }

  graphics {
    type = "spice"
    listen_type = "address"
    autoport = true
  }
}

[MalloZup]

@remoe as temp workaround you could do this:

https://github.com/kubic-project/kubic-init/pull/77/files#diff-c929d6abc0a7fde7ff8d19acc056fcaeR16

where you download the image and bzip/gzip them via a null-resource

:sunflower: ( basically you define a resource for the download of the image, and extraction )

For the direct url support i will need to take look on other issues i have found doing this feature, like internal ones, like we set the size of the volume by reading the image but having a compreessed image we need to detect it after. :smile: because is compressed so is smaller. ( we cannot create a vol that size)

A part of this, i saw that coreos_production image has only one file zipped/bzipped this is good and reflect the way i want to do the feature because if we have like a archive with multiples files this will never work out, ( or it can be really complicated to mantain in longterm)

I will use this tf as test.example thank you and enjoy your day, i will ping you once i have something ready :)

[remoe]

another sample: https://github.com/AmpereComputing/terraform-openstack-images/blob/master/fedora/main.tf

[MalloZup]

@remoe I'm re-opening this issue for discussion. Allthought the auto-detection is lot of work and need to me thought carefully also to be maintained.

[remoe]

I don't understand why this extraction of archives is not a base functionality of terraform.