Nixpkgs: runInLinuxVM, test-driver: investigate virtiofs as 9p replacement

[volth]

qemu 4.2.0 (released Dec 12th 2019) has new mechanism to pass a host folder into a VM: virtiofs.

We should investigate replacing 9p which is currently using to pass host's /nix/store folder into VM in runInLinuxVM, test-driver, ... and also 3rd party projects such as declarative VMs.

One of the reason could be poor performance of 9p (once upon a time NixOS had a kernel patch to improve 9p performance which has been removed because it brought some bugs and none wanted to dig in)

cc @dezgeg

[aszlig]

This could possibly also get rid of our 9p patches for QEMU.

[aszlig]

Okay dug a bit into it and if I understand this correctly, we'd need even more patches:

• We possibly need to trim down sandbox initialisation during Nix builds to only do seccomp BPF and possibly also make sure that the FS root is correctly recognised (since we can't do pivot_root in Nix builds).
• Mapping UID to 0 might still be needed, since we do have a few programs (sudo, cups, whatnot...) that expect the store being owned by UID 0. However within Nix builds, we're already within a user namespace and so we get UID 65534 (nobody) and GID 65534 (nobody) instead.

The advantage however is that we only need to patch virtiofsd, so it might be easier to maintain.

So I guess if the performance gains really make a huge difference, I think it's worth introducing one more patch.

[JJJollyjim]

Basing my work on #86291, with a patch for virtiofsd to not write to /var/run/virtiofsd and this patch to spawn a userns, I have the store working on virtiofs, when run outside of a nix-build context. There is a >1s speedup, even without DAX (both with qemu5):

It turns out that DAX support hasn't yet been merged into upstream QEMU. I am working currently on trying it out against a dev build.

More info to come on getting it working from within a nix-build context.

[JJJollyjim]

I ran some better benchmarks against some random tests:

Only one regression without DAX enabled, and none with (on the latest virtio-fs-dev commit from gitlab.com/virtio-fs/qemu)

I'll get it cleaned up and pushed tonight or tomorrow (as well as properly investigating what is or is not broken :)

[Mic92]

I will also work for my research on an alternative virtiofsd implementation. I will consider the nix use case, maybe we could use this implementation than instead so no patches are necessary.

[volth]

Kata Containers is adopting virtiofs and patches qemu to improve virtiofs performance (https://github.com/kata-containers/kata-containers/pull/945)