Rkt: Include KVM QEMU stage 1 as a release binary

[tjdett]

Building the QEMU stage 1 is not a trivial task. It statically compiles QEMU, which brings in a number of dependencies not covered by rkt's helper scripts. For new users, its likely to be entirely too much hassle to try out this new KVM stage 1 flavour.

There are known deficiencies in the LKVM stage 1 (eg. #2576). It would be good to have both LKVM & QEMU stage 1 images pre-compiled for new releases so users can easily test hypervisor-isolated containers with rkt.

[jjlakis]

@tjdett There was an agreement for now, that we won't check 3rd party dependencies during rkt building process. But including pre-compiled QEMU stage1 in new releases is necessary. @s-urbaniak, what you think?

cc @coreos/rkt-kvm-maintainers

[jonboulle]

Yep, we should build and distribute the QEMU stage1 with our releases.

[s-urbaniak]

@jjlakis sgtm, the rkt-builder builds on debian-sid, so we shouldn't have issues with the glib static libs needed for quemu compilation.

[lucab]

I agree that there should be pre-built qemu flavors and I also agree that it would be nice to re-use rkt-builder for that.

What I'm more concerned about is:

• it seems we may be hitting bugs in qemu interaction with regards to 9p, net and devices, so I'm cautious in exposing this to users if we don't have enough resources to triage
• we need some CI story for kvm-qemu (and even the existing functional tests for lkvm are not on par with other stage1)
• there seems to be some qemu-lite work in progress, will that mean another image to be distributed soon?

Can we perhaps work on building it via rkt-builder but distributing it out-of-band until we have a proper transition plan and infrastructure for qemu? It should be feasible for @coreos/rkt-kvm-maintainers to distribute stage1-qemu (and qemu-lite, later) signed ACIs on their own, and just point --stage1-name= to them, while stabilization is being worked on.

[tjdett]

@lucab I'm certainly sympathetic in relation to the maturity question. I realise there are going to be bugs in the QEMU stage 1, and I'm happy to put the work into reporting them in as much detail as I can when I find them. Nothing gets stable overnight.

However, you can't say "KVM stage1 flavor is now a mature alternative to traditional namespace-based engines" without giving the impression that there's a KVM stage 1 that users should actually be using and submitting bugs on. I'm pretty sure from @jellonek's previous comments around LKVM that it's not the future (or else #2576 would have received more love) so by elimination it's QEMU. If you're not keen on that right now, then slap an "experimental" label on both of them for now until they or qemu-lite are ready.

I think hypervisor-level isolation using KVM stage 1 is a big _potential_ drawcard for rkt, but for it to be taken seriously the rhetoric in CoreOS presentations/blogs and the way it's treated as part of rkt releases needs to get closer together.

Apologies if that's a bit harsh. I know you can't magic resources out of thin air. I just want a better idea around what I can expect from KVM stage 1 now and in the near future so I can plan accordingly.