Runtime: Support ARM64 hosts without GIC

[srcman]

Description of problem

Apparently Raspberry Pi 3 (arm64) doesn't have a GIC, so getHostGICVersion() in qemu_arm64.go can't find the GIC version in /proc/interrupts, and giving the gic-version=host machine option to QEMU (which is the current fallback) results in an error in this case.

Not giving the gic-version option to QEMU at all if the host doesn’t have a GIC seems to fix this problem. (The kernel_irqchip=off option could be given instead, in order to explicitly request irqchip emulation in user space, but actually that doesn’t seem to be required here.)

Expected result

VM for Kata container started successfully on, e.g., a 64-bit RPi 3 with KVM

Actual result

QEMU: "Unable to determine GIC version supported by host” (since creating a VGIC device in QEMU fails)

[jodh-intel]

/cc @Pennyzct, @Weichen81.

[Weichen81]

The first time when I saw the issue, I got shocked. There are having some SoCs really don't support GIC. Then I checked the RPI code, https://github.com/torvalds/linux/blob/bebc6082da0a9f5d47a1ea2edc099bf671058bd4/arch/arm/boot/dts/bcm2837.dtsi#L12.
I found it's real. RPI is using bcm2836-l1-intc as interrupt controller. I think I can send a PR to fix it.
But what I concern is that, if we use an irqchip emulation in user space, the performance should be not good. So, I think should we need to define a minimal hardware requirement for kata-containers?

[grahamwhaley]

I think I agree - we should have the GIC case as the working default, as generally Kata is a server type thing, and you'd expect all servers to have GICs?
What we probably do want to do then is detail how to make Kata work with a GIC-less chip - detail if that needs a new kernel config/build and/or add any QEMU config options in that can be set in the kata config maybe?

[jodh-intel]

It would be rather awesome to have kata running on a Pi. Either way, the logic to specify the minimum hardware spec should be part of the kata-check command so users can run that and it will tell them clearly whether their hardware is supported (and if not, it should explain why):

• https://github.com/kata-containers/runtime#hardware-requirements

[srcman]

Thanks for considering GIC-less support. I guess the main purpose for running Kata Containers on a Raspberry Pi would be just experimenting, not any very serious use.

Actually leaving out the GIC parameter seemed to be pretty much the only code-level issue that needed to be addressed in order to create and start Kata instances on a 64-bit RPi 3 B+ host (that has Raspbian's 4.14.70 arm64 Linux kernel with KVM enabled, Debian Buster's base system etc., and QEMU 2.12). In this case I simply used Kata's default guest kernel config for arm64, as well as an Alpine-based initrd guest image, and launched containers through cri-containerd. Didn't do any more thorough testing so far, though.

[Weichen81]

@srcman Don't worry, I am willing to send a PR to fix this issue. Doesn't like x86, most of Arm developers don't have real Arm servers like x86-based PC. So various kinds of PI are their debug platforms. I certainly wouldn't refuse to pay a small price for so many potential developers or users : )
Your issue just reminds us we may have ignored some points in hardware check.

@jodh-intel @grahamwhaley Maybe we can rate the kata-check to levels:
Recommend hardware requirements(All hardware virtualization accelerate technologies are ready, kata-containers can work smoothly)
Runable hardware requirements (One or more hardware features are missed, kata-containers can run, but only can be used for debug, experience)
No-workable(One or more key hardware features are missed, kata-containers can not run)

As we are using QEMU now, it is a full-functionality emulator, even supports binary translation. But for NEMU, I think most of these features, like binary translation and user space irqchip emulator would be removed. And if we support Xen and Kvmtools later, both of them don't support irqchip emulator.
So I think a rating kata-check would be better to notify users in production environment or in debug environment.

[jodh-intel]

@Weichen81 - this sounds like an interesting idea!

By default, running sudo kata-runtime kata-check should always run all checks and fail if any expected/recommended features are missing.

However, if we recall that kata-runtime kata-check is simply an informational command for users (the runtime itself never calls the check code as part of normal operations), it seems reasonable that we could add a --permissive option. If the user specified that, kata-check would switch to checking for the minimum hardware requirements (what you've called "Runable hardware requirements"). That way, the normal flow would be:

1. Run sudo kata-runtime kata-check.
2. If this fails and running in a non-developer environment, get new hardware.
3. If running on a developer / test / low-spec system, run sudo kata-runtime kata-check --permissive.
4. If that fails, get new developer hardware :smile:

/cc @sameo, @egernst, @sboeuf, @gnawux, @bergwolf, @grahamwhaley.

[sboeuf]

@jodh-intel
--permissive sounds good to me.

[srcman]

As a side note regarding Kata on RPis, the newly released Raspberry Pi 4 actually seems to have a GIC (v2), if I read their bcm2711.dtsi and bcm2838.dtsi files for Linux 4.19 correctly.

[grahamwhaley]

Nice @srcman - yeah, saw there was a new Pi :-) /cc @Pennyzct

[Pennyzct]

Hi~@srcman really excited to see people running kata containers on Pi. Pi 4 is really a new generation, having cortex-A72 core.
If you want to check what gic version you have, another way is that: cat /proc/interrupts | grep -i 'gic'
BTW, since we have supported vNVDIMM, your host kernel must be v4.20+, since v5.0/v5.1 is the closest stable version, I'm highly recommend you to use v5.0/v5.1. ;)
If you have found any bug, just ping me, happy to help. o(////▽////)q