Pmbootstrap: Improve Qemu video performance

[PureTryOut]

Right now, Qemu doesn't have OpenGL support whatsoever, which causes some issues with Plasma Mobile (and probably future DE's and potential apps developers). I believe we should look into using the QXL video driver for the guest (instead of what we use now, vga I guess?), together with SPICE for viewing the VM.

I use this method right now for a Windows VM I need for work, and although it isn't great, there is basic video acceleration, and it might be enough for app development and Plasma. It also allows for better integration with the host by means of shared clipboard and no locked mouse cursor.

It'll require the user to have installed SPICE on his system, so maybe we can let pmbootstrap check for SPICE first and use vga as a fallback?

QXL and SPICE

[ollieparanoid]

Since this is network based, maybe it makes sense to use the qemu from within the chroot then.

Just in theory, I think it would not be impossible to use spice from the chroot too (although it needs to talk to X/wayland compositor of the host system). In theory, preloading all the required .so files and executing Alpine's spice binary from outside the chroot should be enough.

[MartijnBraam]

We could also provide a statically linked spice client to execute on the host I guess.

[ollieparanoid]

Yes, but then we would need to compile it by ourselves.

[PabloCastellano]

I did some research.

Actually pmbootstrap qemu runs the following command:

qemu-system-x86_64 -kernel /var/pmbootstrap/chroot_rootfs_qemu-amd64/boot/vmlinuz-postmarketos -initrd /var/pmbootstrap/chroot_rootfs_qemu-amd64/boot/initramfs-postmarketos -append "console=tty1 console=ttyS0 PMOS_NO_OUTPUT_REDIRECT" -m 1024 -redir tcp:2222::22 -serial stdio -drive file=/var/pmbootstrap/chroot_native/home/user/rootfs/qemu-amd64.img,format=raw -enable-kvm

Following @PureTryOut's linked article, I have tried appending the following arguments to the previous command:

-vga qxl -spice port=5900,addr=127.0.0.1,disable-ticketing

And I can successfully connect to it using spicy (from the spice-client-gtk package in Ubuntu)

spicy -h 127.0.0.1 -p 5900

First I tried without QXL support and weston didn't show up. Instead, the motd message was printed in a loop every few seconds (issue https://github.com/postmarketOS/pmbootstrap/issues/330)

After adding QXL support (CONFIG_DRM_QXL), weston runs nicely. I've also noticed that the mouse precision is way higher using spice than normal qemu (I can easily click the terminal icon in the top left corner :smile:).

Some thoughts about the implementation

• Should we run spicy (or any similar tool) for the user? (At first I thought it was a bit intrusive for pmbootstrap - then I realized that we are actually running qemu so it's the same :smile:)
  
  
  
  • No: the user would be left waiting for his VM to show up if spice works. We can show a messaging telling how to connect. In the case of the fallback, the VM would show up as it is done now.
  
  
  • Yes: The user will see a VM popping up right after running pmbootstrap qemu it doesn't matter if the connection is done in normal qemu or using spice
  
  

• How can we detect spice compatibility?
  
  
  
  • If the kernel doesn't have qxl support, it will simply not run weston as happened to me.
  
  
  • We can simply add QXL support to the qemu kernel configs and detect if spicy is installed in the host machine. If installed, use it.
  
  

So, to summarize I see two ways here:

• Detect spicy and launch qemu with -vga qxl. If spict is not found in the host machine, run qemu as we do now.
• Add a --spice (or better, --no-spice) argument so that users can choose.

What do you think?

[ollieparanoid]

Great research work! I would actually combine both:

• Add a --no-spice switch
• If spice is enabled (default) and spice is not installed, print a message that it can either be installed or disabled.

[PabloCastellano]

Good, I will give it a try

[PabloCastellano]

Said by @NotKit on Matrix:

btw virtio-gpu works as well, at least for x86_64
needed to enable relevant kernel options, install mesa-dri-virtio and start QEMU manually with -vga virtio -display gtk,gl=on
CONFIG_DRM_VIRTIO_GPU=m, CONFIG_VIRTIO_PCI=m, CONFIG_VIRTIO_BALLOON=m
it was discussed here before that virtio requires fairly new QEMU but it provides OpenGL acceleration

[PureTryOut]

As far as I know, virtio-gpu might not work with every hardware setup. I believe it literally adds a virtual GPU right? I seem to recall the AMD drivers in Mesa only getting support for it a few months ago, and it was one of the first. This will have to be verified though.

[PabloCastellano]

Quoting @PureTryOut

If we're going for virtio, check Arch's wiki. Specifically note Currently a work in progress, supporting only very recent (>= 4.4) Linux guests with mesa (>=11.2) compiled with the option --with-gallium-drivers=virgl.. TL;DR: it won't work with proprietary drivers (like NVIDIA, screw them). Could still be an option though.

EDIT: Actually, I might be misreading that. Is it only talking about the guest support? If that's true then it shouldn't be a problem.

EDIT 2: So, I've just tested virtio-gpu instead, and it seems to work.

I've set CONFIG_DRM_VIRTIO_GPU=m, CONFIG_VIRTIO_PCI=m, and CONFIG_VIRTIO_BALLOON=m in the kernel config, and launched Qemu with -vga virtio -display gtk,gl=on. In Qemu I've installed the mesa-dri-virtio and mesa-dri-swrast packages. Although I didn't get kwin running properly yet, the log reports it initializing the OpenGL backend succesfully using DRM.

That's good news and so, virtio-gpu is definitely the way to go!

Since the spice approach is giving me better performance and virtio-gpu is only supported in recent versions of qemu (I'm not sure if virtio-gpu will work for me), I would like to keep the spice functionality (which would benefit others in my situation).

We need more testers

...to learn more about the performance issues and once we get more feedback, we can decide what should be the default behaviour when virtio-gpu is not supported in qemu, that is, if it usually performs bad, then convert --no-spice to --spice. So by default people will get the most performance approach and in rare cases like mine, I can still use --spice.

[ollieparanoid]

I did a bit of research on my idea to run graphical programs from within the chroot. And we can do it without actually entering the chroot! That way we do not need to pass a X11 cookie or similar, and gtk programs for example will use the user's gtk settings. It will be like a native program.

The real advantage is, that we don't need to support multiple versions of qemu (or spice), just the one version shipped with Alpine.

Here is an example:

pmbootstrap chroot -- apk install xterm
cd ~/.local/var/pmbootstrap/chroot_native
./lib/ld-musl-x86_64.so.1 --library-path=./lib/:./usr/lib ./usr/bin/xterm

(While this works for dynamically linked programs, it will not work for programs, that depend on files, that are only present in the chroot).

[PureTryOut]

I've been testing virtio-gpu a bit more and so far I got 3D acceleration to work and Weston to launch. Right now I'm having issues with the mouse however. If I use the Qemu GTK backend (-display gtk,gl=on) the mouse events don't get send to the guest at all, and if I use the Qemu SDL2 backend (-display sdl,gl=on) the mouse jumps all over the place. I'm out of ideas so if anyone knows anything, I'm all ears.

[ollieparanoid]

This was mentioned in the chat already, but not sure if you have tried it:

Gentoo Wiki says:

-usbdevice tablet - (Recommended) Use a USB tablet instead of the default PS/2 mouse. Recommend, because the tablet sends the mouse cursor's position to match the host mouse cursor.

[PureTryOut]

I've tried it, no luck.

If you launch a demo animation (say the spinning wheel), you'll notice it slowing down while moving the mouse. This makes me think a bit more goes wrong then just the cursor position. It seems like the whole VM is actually slowing down. I really have no clue what can cause that.

[PabloCastellano]

Some more research on enabling OpenGL in QEMU.

I found that Debian enabled virtio gpu and opengl support (and switched from sdl1 to gtk3) in QEMU 2.8 but then reverted it :-(.

The reason seems to be the following:

• Recently virgl had a lot of security issues (including memleaks, heap overflows and DoS!), which spikes the maintenance effort around it
• Some people do not want qemu built with gtk support because it includes many new dependencies (see the Debian thread)
• Still, this is a very new feature and some dependencies are not packaged yet in the main repositories
  (??? - but still, they enabled it)

For more information read the full threads:

• https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1540692
• https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=839695

Still, I gave a try to get qemu with opengl support inside pmbootstrap's chroot. I failed to build it with sdl2 support but built it successfully with gtk support. I'm also able to run qemu from within the chroot using @ollieparanoid's master trick :tada: but gtk complains in different ways and falling back to sdl1 is possible, but it doesn't support OpenGL (SDL2 does!).

So maybe the solution is close, from now I see the following paths, always using our chrooted qemu:

• Headless (but with Opengl support) plus SPICE*
• Fix GTK complains
• Fix SDL2 build

(*) I'm getting

[    5.894779] [drm] pci: virtio-vga detected at 0000:00:02.0
[    5.894871] [drm] virgl 3d acceleration not supported by host

I'll try these approaches in the following days :smiley: in the specified order.

[PureTryOut]

I recommend trying to fix the SDL2 build rather than fix GTK, as right now GTK with OpenGL seems to completely break the mouse. Sure, so does SDL2, but at least it's somewhat usable.

Also, I'm not entirely sure how you want to achieve a headless setup with OpenGL support. From my testing it seems that we require -display {sdl,gtk},gl=on, otherwise we get that [drm] virgl 3d acceleration not supported by host message.

[PureTryOut]

I think we should re-look to enabling Virtio support. QXL doesn't seem to be much of an improvement after all, 3D hardware acceleration was still lacking and performance was really slow if I interacted with anything in the Plasma shell. However, after a suggestion of @craftyguy, I re-installed Virtio support, and performance is now great! Everything runs full speed and even the mouse moves properly (besides some random twitches) unlike before.

Do note that it only seems to work if I use -display sdl,gl=on, rather than -display gtk,gl=on. With the latter the window stays 800x600 and the window is just black. When running with GTK I get the following in the console:

qemu-system-x86_64:14659): Gdk-WARNING **: gdk_gl_context_set_required_version - GL context versions less than 3.2 are not supported.