Terraform-provider-libvirt: Permission Denied

[littlehunch]

System Information

Linux distribution

 Ubuntu 18.04-lts
 ```

### Terraform version

```sh
terraform -v
Terraform v0.11.11
+ provider.libvirt (unversioned)
+ provider.template v2.0.0

Provider and libvirt versions

terraform-provider-libvirt -version
terraform-provider-libvirt 0.5.1
Compiled against library: libvirt 4.0.0
Using library: libvirt 4.0.0
Running hypervisor: QEMU 2.11.1
Running against daemon: 4.0.0

If that gives you "was not built correctly", get the Git commit hash from your local provider repository:

git describe --always --abbrev=40 --dirty

---

Checklist

• [ ] Is your issue/contribution related with enabling some setting/option exposed by libvirt that the plugin does not yet support, or requires changing/extending the provider terraform schema?

  • [ ] Make sure you explain why this option is important to you, why it should be important to everyone. Describe your use-case with detail and provide examples where possible.
  • [ ] If it is a very special case, consider using the _XSLT_ support in the provider to tweak the definition instead of opening an issue
  • [ ] Maintainers do not have expertise in every libvirt setting, so please, describe the feature and how it is used. Link to the appropriate documentation

• [ ] Is it a bug or something that does not work as expected? Please make sure you fill the version information below:

Description of Issue/Question

Setup

(Please provide the full _main.tf_ file for reproducing the issue (Be sure to remove sensitive information)

provider "libvirt" {
  uri = "qemu:///system"
}

#
## VM Varibles
#
variable "instance_count" {
  default = "1"
}

variable "disk_img" {
  default = "https://cloud-images.ubuntu.com/releases/xenial/release/ubuntu-16.04-server-cloudimg-amd64-disk1.img"
}

#
## Networking Varibles
#
variable "vm_network_addresses" {
  description = "Defines the network in the CIDR format"
  default = "10.0.1.0/24"
}

variable "vm_network_name" {
  description = "Defines the network name"
  default = "vm_network"
}

variable "domain_name" {
  default = "k8s.local"
}

#---- End Of Varibles ---#

resource "libvirt_volume" "os_image" {
  name   = "os_image"
  source = "${var.disk_img}"
}

#volume
resource "libvirt_volume" "volume" {
  name           = "volume-${count.index}"
  base_volume_id = "${libvirt_volume.os_image.id}"
  count = "${var.instance_count}"
}

# network

resource "libvirt_network" "vm_network" {

   name = "${var.vm_network_name}"
   addresses = ["${var.vm_network_addresses}"]
   domain    = "${var.domain_name}"

   mode = "nat"

   dhcp {
       enabled = true
   }

   dns {
     local_only = true
   }

   autostart = true
}

#Domain
resource "libvirt_domain" "domain" {
  name = "master-${count.index}"

  memory = "1024"
  vcpu = "1"

  network_interface {
    network_id   = "${libvirt_network.vm_network.id}"
    hostname = "master-${count.index}"
    wait_for_lease = true
  }

  console {
    type        = "pty"
    target_port = "0"
    target_type = "serial"
  }

  console {
    type        = "pty"
    target_type = "virtio"
    target_port = "1"
  }

  disk {
    volume_id = "${libvirt_volume.volume.id}"
  }

  count = "${var.instance_count}"
}

#############
## Outputs ##
#############

output "disk_id" {
  value = "${libvirt_volume.volume.*.id}"
}

output "network_id" {
  value = "${libvirt_network.vm_network.id}"
}

output "ip_addresses" {
  value = "${libvirt_domain.domain.*.network_interface.0.addresses}"
}

Steps to Reproduce Issue

(Include debug logs if possible and relevant).

Error: Error applying plan:

1 error(s) occurred:

* libvirt_domain.domain: 1 error(s) occurred:

* libvirt_domain.domain: Error creating libvirt domain: virError(Code=1, Domain=10, Message='internal error: process exited while connecting to monitor: 2019-01-28T02:29:14.861688Z qemu-system-x86_64: -drive file=/var/lib/libvirt/images/volume-0,format=qcow2,if=none,id=drive-virtio-disk0: Could not open '/var/lib/libvirt/images/volume-0': Permission denied')

---

Additional information:

Do you have SELinux or Apparmor/Firewall enabled? Some special configuration?
Have you tried to reproduce the issue without them enabled?

[rustychapin]

I am experiencing the same issue and have the same system configs. I am using the unmodified example from: https://github.com/dmacvicar/terraform-provider-libvirt/tree/master/examples/ubuntu

I have tried modifying the user:group libvirtd runs under to be root:root and i receive the same result

The image permissions output by terraform apply is as follows:
rchapin@r0-kvm-12:~/github/terraform$ sudo ls /var/lib/libvirt/images -al
total 290488
drwx--x--x 2 root root 4096 Jan 28 23:57 .
drwxr-xr-x 7 root root 4096 Jan 23 22:01 ..
-rw-r--r-- 1 libvirt-qemu kvm 374784 Jan 28 23:57 commoninit.iso
-rw-r--r-- 1 root root 297074688 Jan 28 23:58 ubuntu-0

[MalloZup]

Hi all thx for reportong this. However this is not a provider issue. Is an issue part of your libvirt configuration. Permission denied means you don't have rights to write in that dir. Maybe it is your user or other libvirt minimal conf. Try to change default pool repo or investigate on the basic libvirt conf. Needed for let libvirt running. Normally you need user on KVM/libvirt and other stuff depending on which Linux distro. Enjoy

[rustychapin]

MalloZup I am not sure that this is the case. If you take a look at the permissions, in my previous comment, on commoninit.iso and ubuntu-0 they are receiving different permissions though they are created via the libvirt provider. I have attempted modifying the qemu.conf to be root:root and still receive permission denied. My libvirt config is unmodified. I have spent hours researching solutions to this problem and have tried several different fixes and no avail. It seems to me that if the provider is responsible for creating the image file that it should be doing so with the correct permissions. Is it possible that this could be a feature request in that we could provide the user:group we need? Note that I have also enabled, as a test, enabled the dynamic permissions configuration in qemu.conf and this had no effect on the image file.

[rustychapin]

@MalloZup - sorry meant to ping you on this so it got your attention... see above.

[zacbayhan]

I'm still having issues after changing the location of the volume to my home dir, I've played around a bit with the file permissions / ownership but don't seem to making much headway. I did notice some apparmor logs in syslog

provider-provider-libvirt:

* libvirt_domain.domain: Error creating libvirt domain: virError(Code=1, Domain=10, Message='internal error: process exited while connecting to monitor: 2019-01-30T00:04:24.705596Z qemu-system-x86_64: -drive file=/home/zack/home_vm_network/iso_image/os_image,format=qcow2,if=none,id=drive-virtio-disk0: Could not open '/home/zack/home_vm_network/iso_image/os_image': Permission denied')

syslog:

Jan 29 19:09:24 zack-dt libvirtd[15221]: 2019-01-30 00:09:24.750+0000: 15223: warning : qemuDomainObjTaint:5602 : Domain id=11 name='master-0' uuid=e1ee5c06-1f3c-4fa9-baba-772dd011a8a5 is tainted: high-privileges
Jan 29 19:09:24 zack-dt kernel: [263802.885482] audit: type=1400 audit(1548806964.776:22663): apparmor="DENIED" operation="open" profile="libvirt-e1ee5c06-1f3c-4fa9-baba-772dd011a8a5" name="/home/zack/home_vm_network/iso_image/os_image" pid=10369 comm="qemu-system-x86" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
Jan 29 19:09:24 zack-dt kernel: [263802.885520] audit: type=1400 audit(1548806964.776:22664): apparmor="DENIED" operation="open" profile="libvirt-e1ee5c06-1f3c-4fa9-baba-772dd011a8a5" name="/home/zack/home_vm_network/iso_image/os_image" pid=10369 comm="qemu-system-x86" requested_mask="wr" denied_mask="wr" fsuid=0 ouid=0
Jan 29 19:09:24 zack-dt libvirtd[15221]: 2019-01-30 00:09:24.788+0000: 15223: error : qemuMonitorOpenUnix:385 : failed to connect to monitor socket: No such process
Jan 29 19:09:24 zack-dt libvirtd[15221]: 2019-01-30 00:09:24.789+0000: 15223: error : qemuProcessReportLogError:1914 : internal error: process exited while connecting to monitor: 2019-01-30T00:09:24.781838Z qemu-system-x86_64: -drive file=/home/zack/home_vm_network/iso_image/os_image,format=qcow2,if=none,id=drive-virtio-disk0: Could not open '/home/zack/home_vm_network/iso_image/os_image': Permission denied
Jan 29 19:09:24 zack-dt kernel: [263803.012808] audit: type=1400 audit(1548806964.904:22665): apparmor="STATUS" operation="profile_remove" profile="unconfined" name="libvirt-e1ee5c06-1f3c-4fa9-baba-772dd011a8a5" pid=10388 comm="apparmor_parser"
Jan 29 19:09:24 zack-dt libvirtd[15221]: 2019-01-30 00:09:24.927+0000: 15221: error : virNetSocketReadWire:1811 : End of file while reading data: Input/output error

I wouldn't be surprised to findout that I've installed or configured something wrong, I'm just having difficulty figuring out what that is. So let my say thanks any help is much appreciated

[rustychapin]

@MalloZup - After another day of research I have found some historical context that is interesting. However I DO AGREE... This is NOT an issue with terraform-provider-libvirt.

This looks like it may be a recurrence of an issue where QEMU is failing to generate the correct permissions for qcow2 formatted volumes. Either way this issue should be closed on your side. Would be great if we could get some traction with the QEMU team and see what they can find. Thanks again for supporting this provider for terraform!

[MalloZup]

@rustychapin @rustychapin @littlehunch issues with permission denied is always kind tricky to find out.

Normally in my experience, on a openSUSE machine: i have my user belong to libvirtd and kvm groups. The rest is done via configuration files ( libvirt/qemu)

I would disable apparmor and try if if works. i am closing this issue but you can either write on this issue for sync with others or you can also join the gitter chat channel for asking more help.
Have nice day enjoy :rocket: :white_flower:

[andersla]

Could very well be same as #97 see fix there, already in documentation, but very easy to miss, still an issue on ubuntu 18.04, documentation; 22f096d

[MalloZup]

Thx @andersla for pointer

[thebithead]

I fixed this problem with this:

On Ubuntu distros SELinux is enforced by qemu even if it is disabled globally, this might cause unexpected Could not open '/var/lib/libvirt/images/<FILE_NAME>': Permission denied errors. Double check that security_driver = "none" is uncommented in /etc/libvirt/qemu.conf and issue sudo systemctl restart libvirt-bin to restart the daemon.

[ulm0]

This is still true in ubuntu 20.04, thanks to @andersla that pointed to the security_driver thing, that did the trick. That may be added back to the readme.

[sneak]

Terraform seems to create the pool with owner/group set to 0:

root@lstor1:~# virsh pool-dumpxml lstor1-pool-root
<pool type='dir'>
  <name>lstor1-pool-root</name>
  <uuid>479e75d0-8442-4ca0-9bf9-5bf36ca4584c</uuid>
  <capacity unit='bytes'>23096088592384</capacity>
  <allocation unit='bytes'>525992448</allocation>
  <available unit='bytes'>23095562599936</available>
  <source>
  </source>
  <target>
    <path>/srv/lstor1/virt/root</path>
    <permissions>
      <mode>0755</mode>
      <owner>0</owner>
      <group>0</group>
    </permissions>
  </target>
</pool>

However libvirtd (which writes the uploaded files, such as the cloud init, or the base ISO) runs as non-root, writing those files as libvirt-qemu:kvm:

root@lstor1:/srv/lstor1/virt/root# ls -tla
total 513473
-rw-r--r-- 1 libvirt-qemu kvm     374784 Oct 28 17:23 testvm-cloudinit.iso
drwxr-xr-x 2 root         root         5 Oct 28 17:23 .
-rw-r--r-- 1 root         root    196912 Oct 28 17:19 testvm-root
-rw-r--r-- 1 libvirt-qemu kvm  527368192 Oct 28 00:54 lstor1-focal-baseimg
drwxr-xr-x 4 root         root         4 Oct 27 21:57 ..
root@lstor1:/srv/lstor1/virt/root#

I think there may be some XSLT that can be applied to the pool definitions that will change the owner/group to the right settings, but I will need to research them.

[marshallford]

@sneak I think you are on to something. I'm confused why the provider creates the pool with the owner/group set to root but requires different permissions for actually starting a domain with an image from such a pool.

[gdombrov]

the recipe for @thebithead helped me

[johnjameswhitman]

This seems like a known issue w/ apparmor's handling of volumes from a libvirt pool (apparmor profiles require the full path to allow file-access, but virt-aa-helper doesn't actually map the volume to its path on the filesystem). Some details:

• Open ubuntu/libvirt bug from 2017: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1677398
• Apparmor + libvirt overview: https://gitlab.com/apparmor/apparmor/-/wikis/Libvirt

The quick-fix for me was setting security = "none" in /etc/libvirt/qemu.conf until ubuntu/libvirt fixes the apparmor profiles.

[MalloZup]

Hi guys thx for sharing help each other.

The terraform libvirt is just a "consumer" of libvirt API so it doesn't change permission etc.

As many of you noticed this is more a libvirt os configuration and OS application issue.

Is not something unfortunately we can fix it here.

[marshallford]

@gdombrov and @johnjameswhitman -- That fix worked for me as well on Ubuntu 20.04.1.

@MalloZup Makes sense. Thanks for all of your work on this project!

[bozakov]

Still running into this libvirt issue on 20.04.2 regardless of the pool used.

Slightly modified workaround is setting security_driver = "none" in /etc/libvirt/qemu.conf but followed by sudo systemctl restart libvirtd.

[jsonhc]

I fixed this problem with this:

On Ubuntu distros SELinux is enforced by qemu even if it is disabled globally, this might cause unexpected Could not open '/var/lib/libvirt/images/<FILE_NAME>': Permission denied errors. Double check that security_driver = "none" is uncommented in /etc/libvirt/qemu.conf and issue sudo systemctl restart libvirt-bin to restart the daemon.

good