Runtime: Running GUI applications within Kata Containers

[greg-hydrogen]

Description of problem

Is there a way to run GUI applications withing Kata containers? With docker exposing the X11 socket does the trick, but since this is not an option for Kata Containers is there an alternative?

I apologize if this is the incorrect area to post this question

Expected result

Ability to run GUI applications

Actual result

Display is not available as the socket is not available

[devimc]

interesting issue, I'll take a look

[mviereck]

Have a look at x11docker.
x11docker supports to run GUI application with kata-runtime.

Sharing the X unix socket as often seen elsewhere is not possible with kata. Instead, x11docker runs an additional nested X server (optionally nxagent, Xephyr or xpra) and connects the GUI applications with X over TCP.

Example: Install x11docker and nxagent and run:

x11docker --runtime=kata-runtime --desktop x11docker/xfce

[grahamwhaley]

@mviereck - oh, cool! Thanks for that link and the info. We should add this to our docs. I'll go put a link in the in-flight user guide wiki page as a start...

[mviereck]

@grahamwhaley I am glad you like it. :-)

I am the developer of x11docker. If anything comes up, issues or kata-specific feature requests, just let me know.

Thank you for kata! I was pointed to it a few weeks ago and find it quite interesting. I'll explore it a bit more in future, so far I just managed to get it to work with x11docker.
Currently i don't know if it is e.g. possible to somehow share device files in /dev/dri to support GPU hardware acceleration.

[jodh-intel]

Hi @mviereck - I think you're probably looking for https://github.com/kata-containers/documentation/blob/master/use-cases/GPU-passthrough-and-Kata.md.

[mviereck]

@jodh-intel Thanks for the link!

I have included experimental GPU support for kata-runtime in x11docker.

From what I see, the only thing x11docker itself can do is to share /dev/vfio. The host configuration setup has do be done by the user.
I cannot test myself because I don't have Intel hardware.

From the X servers supported by x11docker only Xorg will be able to provide GPU access for kata containers.

If you like to give it a try, please install latest x11docker master version from github.
Switch to a tty and run:

docker pull x11docker/check
x11docker --xorg --gpu --runtime=kata-runtime x11docker/check

The image x11docker/check provides a GUI with some check buttons for GPU access.

(To share a specific device in /dev/vfio, use e.g. --share /dev/vfio/1 instead of --gpu. Option --gpu share entire /dev/vfio, that might not be the best way, but good enough for a first test.)

---

Edit: Possible troubleshooting:

• x11docker creates an unprivileged user in container and adds it to group video. To avoid possible permission issues, try --user=root.
• /dev/dri is not shared in this setup. If that has to be done:

x11docker --xorg --gpu --runtime=kata-runtime -- --device=/dev/dri -- x11docker/check

---

Things would be easier if x11docker could share host unix sockets and host device files.
Will that ever be possible or can't that be done across different kernels?

[jodh-intel]

I have included experimental GPU support for kata-runtime in x11docker.

@mviereck - Thanks! Great news!!

@amshinde - can you tal at this maybe?

[amshinde]

Thanks @mviereck !

@eadamsintel Want to give this a spin ?

[devimc]

closing issue, @mviereck feel free to reopen it for future discussions

[mviereck]

@devimc I was in hope that someone of you with an Intel GPU would test whether hardware acceleration with --gpu --xorg works. However, I can leave it in untested/experimental stage in the code.
--runtime=kata-runtime in general (without --gpu, I have AMD hardware) works well here.

[eadamsintel]

It is on my list of things to try out. I'll work to find time to give this a spin this week. I've done both GVT-d and GVT-g pass through with Kata using Docker so this doesn't look too complex.

[eadamsintel]

I had some time this afternoon to look at this. I didn't spend a lot of time understanding x11docker but it looks like just a wrapper for docker to add in some GUI components. For the setup, I ssh into the machine to run the commands and boot into an X-org GUI. I am using ClearLinux as the OS since that is what I already had configured with kata-runtime. This was done on a Skylake Skullcanyon NUC which has a single Intel GPU (Iris Pro Graphics 580 Rev 9). I couldn't find nxagent for Clear Linux but when using x11docker with runc it appeared to work so I don't know if this is absolutely necessary for the tests I did below which couldn't boot up a similar display. To make sure my GVT-d and GVT-d setup worked I did a transcode in a kata container and it successfully transcoded using the GPU so I know device passthrough is working. I had intel_iommu=on for my kernel boot and for GVT-g had i915.enable_gvt=1.

kata-runtime version = 1.6.2

runc
./x11docker --xorg --gpu --runtime=runc x11docker/check
This appears to work. It brings up a grey background GUI with lots of options to click and 4 tabs across the top. I didn't actually test this part out. I did have it fail sometimes if I tried to launch it and was still at the login screen for the gnome desktop installed but overall is seemed to work.

kata-runtime with GVT-d
./x11docker --xorg --gpu --runtime=kata-runtime x11docker/check
If using GVT-d I lose my display when I bind it to vfio-pci. x11docker did not work for me and gave the following errors.

cp: cannot create regular file '/home/kata/.local/share/x11docker/tini-static-kata': No such file or directory
chmod: cannot access '/home/kata/.local/share/x11docker/tini-static-kata': No such file or directory
x11docker WARNING: Your tini binary is not executeable. Please run
    chmod +x /home/kata/.local/share/x11docker/tini-static-kata

x11docker WARNING: Sharing device file: /dev/vfio

x11docker note: setxkbmap not found. Need setxkbmap and xkbcomp to set keyboard layout.
  You can look for the package name of this command at:
 https://github.com/mviereck/x11docker/wiki/dependencies#table-of-all-packages


x11docker ERROR: X server --xorg not ready after 60s.

  Type 'x11docker --help' for usage information
  Debug options: '--verbose' (full log) or '--debug' (log excerpt).
  Logfile will be: /home/kata/.cache/x11docker/x11docker.log
  Please report issues at https://github.com/mviereck/x11docker

I also tried the following
./x11docker --xorg --gpu --share /dev/vfio/1 --runtime=kata-runtime x11docker/check
It also fails with similar errors

cp: cannot create regular file '/home/kata/.local/share/x11docker/tini-static-kata': No such file or directory
chmod: cannot access '/home/kata/.local/share/x11docker/tini-static-kata': No such file or directory
x11docker WARNING: Your tini binary is not executeable. Please run
    chmod +x /home/kata/.local/share/x11docker/tini-static-kata

x11docker WARNING: Sharing device file: /dev/vfio/1

x11docker WARNING: Sharing device file: /dev/vfio

x11docker note: setxkbmap not found. Need setxkbmap and xkbcomp to set keyboard layout.
  You can look for the package name of this command at:
 https://github.com/mviereck/x11docker/wiki/dependencies#table-of-all-packages


x11docker ERROR: X server --xorg not ready after 60s.

  Type 'x11docker --help' for usage information
  Debug options: '--verbose' (full log) or '--debug' (log excerpt).
  Logfile will be: /home/kata/.cache/x11docker/x11docker.log
  Please report issues at https://github.com/mviereck/x11docker

kata-runtime with GVT-g
./x11docker --xorg --gpu --runtime=kata-runtime x11docker/check
A grey screen comes up with a mouse cursor but there is no dialog box. Below is the last of the output from my ssh session where I started the kata container. This one did not time out like GVT-d did.

cp: cannot create regular file '/home/kata/.local/share/x11docker/tini-static-kata': No such file or directory
chmod: cannot access '/home/kata/.local/share/x11docker/tini-static-kata': No such file or directory
x11docker WARNING: Your tini binary is not executeable. Please run
    chmod +x /home/kata/.local/share/x11docker/tini-static-kata

x11docker WARNING: Sharing device file: /dev/vfio

x11docker note: setxkbmap not found. Need setxkbmap and xkbcomp to set keyboard layout.
  You can look for the package name of this command at:
 https://github.com/mviereck/x11docker/wiki/dependencies#table-of-all-packages

x11docker note: Option --wm: Did not find window manager image
      x11docker/openbox
  to provide a containerized window manager. Please run:
      docker pull x11docker/openbox
  If you want to use a host window manager instead and avoid this warning,
  use option                         --wm=host  or  --wm=COMMAND
  or provide a local image with e.g. --wm=x11docker/lxde
  To run without a window manager:   --wm=none  or  --desktop
  Fallback: Will try to run a host window manager: xfwm4

x11docker note: Option --wm: Starting host window manager: xfwm4

I also tried the following
./x11docker --xorg --gpu --share /dev/vfio/15 --runtime=kata-runtime x11docker/check
Here are the errors.

x11docker note: setxkbmap not found. Need setxkbmap and xkbcomp to set keyboard layout.
  You can look for the package name of this command at:
 https://github.com/mviereck/x11docker/wiki/dependencies#table-of-all-packages

x11docker note: Option --wm: Did not find window manager image
      x11docker/openbox
  to provide a containerized window manager. Please run:
      docker pull x11docker/openbox
  If you want to use a host window manager instead and avoid this warning,
  use option                         --wm=host  or  --wm=COMMAND
  or provide a local image with e.g. --wm=x11docker/lxde
  To run without a window manager:   --wm=none  or  --desktop
  Fallback: Will try to run a host window manager: xfwm4

x11docker note: Option --wm: Starting host window manager: xfwm4


x11docker ERROR: Container startup seems to have failed.
  Last lines of log:
Error: No such object: x11docker_X101_0812410_x11docker-check
Error: No such object: x11docker_X101_0812410_x11docker-check
Error: No such object: x11docker_X101_0812410_x11docker-check
Error: No such object: x11docker_X101_0812410_x11docker-check
Error: No such object: x11docker_X101_0812410_x11docker-check
Error: No such object: x11docker_X101_0812410_x11docker-check
Error: No such object: x11docker_X101_0812410_x11docker-check
Error: No such object: x11docker_X101_0812410_x11docker-check
Error: No such object: x11docker_X101_0812410_x11docker-check
docker: Error response from daemon: OCI runtime create failed: QMP command failed: vfio error: 0bb821db-911a-4569-9bb2-414d07f69cfe: failed to open /dev/vfio/15: Device or resource busy: unknown.

  Type 'x11docker --help' for usage information
  Debug options: '--verbose' (full log) or '--debug' (log excerpt).
  Logfile will be: /home/kata/.cache/x11docker/x11docker.log
  Please report issues at https://github.com/mviereck/x11docker

I tried with vfio 16 and got a similar error but this happened QMP command failed: vfio error: 797a3379-4756-470b-b01e-2d6a5e86c77d: error getting device from group 15: No such process: unknown.

Finally I tried the GVT-g setup removing the --gpu option and had similar errors.

Is this the right setup to use? Is my kata-runtime version ok? I didn't see any patches as part of this issue and assumed this relied on the vfio-pci pass through support.

[mviereck]

@eadamsintel Thank you for running the tests!

Let's try to find a working setup without option --gpu but providing docker options directly.

I've previously missed one point in the documentation. It expects to set -v /dev:/dev additionally to --device /dev/vfio/1. Currently x11docker just sets --device /dev/vfio.

Why is -v /dev:/dev needed? I'd prefer to share needed device files only to not expose too much of the host to the container.
Should x11docker share e.g. /dev/dri and/or /dev/vfio with --volume instead of or additionally to --device?

Please try (without --gpu):

 ./x11docker --xorg  --runtime=kata-runtime -- --device /dev/vfio/1 -v /dev:/dev -- x11docker/check

Adjust --device /dev/vfio/1 to your desired setup.
(Options between -- -- are directly submitted to docker run command.)

This setup is basically the same as provided in https://github.com/kata-containers/documentation/blob/master/use-cases/GPU-passthrough-and-Kata.md. x11docker just adds access to Xorg over TCP.

If you add option --debug you can see the generated docker command.

---

cp: cannot create regular file '/home/kata/.local/share/x11docker/tini-static-kata': No such file or directory

Should be fixed now

x11docker ERROR: X server --xorg not ready after 60s.

The error message will show the last lines of xinit log now. If X startup still fails, but the error message is not helpful, please provide me ~/.cache/x1docker/x11docker.log.

---

Is my kata-runtime version ok?
kata-runtime version = 1.6.2

Currently I have installed kata-runtime 1.5.3, I can update it. However, I don't think that makes a difference yet.

I didn't see any patches as part of this issue and assumed this relied on the vfio-pci pass through support.
Is this the right setup to use?

It is supposed to. However, I am not entirely sure if the passthrough in general really works along with the regular Xorg GPU access on host.

I couldn't find nxagent for Clear Linux but when using x11docker with runc it appeared to work so I don't know if this is absolutely necessary for the tests I did below which couldn't boot up a similar display.

It is not needed for the GPU tests. With kata containers only --xorg allows GPU access possible yet.
nxagent, Xephyr or xpra would be needed for nested setups within a running X, but not for --xorg.
With runc none of them is needed because the X unix socket can be shared. For kata containers with X over TCP x11docker must set up an X server with -listen tcp enabled.

I didn't spend a lot of time understanding x11docker but it looks like just a wrapper for docker to add in some GUI components.

Yes, x11docker is basically just a wrapper to set up an X server and to provide it to Docker containers.

---

General X issues/possibilities I cannot fix:

Things would be much easier with X option +iglx / indirect rendering. It would also work for arbitrary GPU's. Unfortunately iGLX is broken in current Xorg versions.

Also more x11docker X setups (others than --xorg) would be possible if Xwayland would support access over TCP without needing a unix socket (X option -listen tcp). That seems to be a build option and is disabled in distributed Xwayland packages.

With the iGLX issue fixed and Xwayland allowing X access over TCP x11docker could provide an easier GPU access and a broader variety of X servers to kata containers. It would work independent from the GPU vendor, too.

[eadamsintel]

I am still having issues with this. It is complaining that tini is missing but it is there in the container. Perhaps it is a permissions issue?

Below is GVT-g

$ ./x11docker --debug --xorg -- --device /dev/vfio/15 --runtime=kata-runtime -v /dev:/dev -- x11docker/check --debug

DEBUGNOTE[20:08:50,808]: New X environment:
  DISPLAY=172.17.0.1:101 XAUTHORITY=/home/kata/.cache/x11docker/x11docker-check-8053042/share/Xclientcookie XSOCKET=  X11DOCKER_CACHE=/home/kata/.cache/x11docker/x11docker-check-8053042
DEBUGNOTE[20:08:50,813]: X server command:
  /usr/bin/Xorg :101 vt10 \
  -dpms -s off -retro \
  +extension RANDR \
  +extension RENDER \
  +extension GLX \
  +extension XVideo \
  +extension DOUBLE-BUFFER \
  +extension SECURITY \
  +extension DAMAGE \
  -extension X-Resource \
  -extension XINERAMA -xinerama \
  -extension MIT-SHM \
  -auth /home/kata/.cache/x11docker/x11docker-check-8053042/Xservercookie \
  -listen tcp \
  +extension Composite +extension COMPOSITE \
  -extension XTEST -tst \
  -verbose
DEBUGNOTE[20:08:50,879]: Users and terminal:
  x11docker was started by:                       kata
  As host user serves (running X, storing cache): kata
  Container user will be:                         kata
  Container user password:                        x11docker
  Getting permission to run docker with:          bash -c
  Running X and other user commands with:         bash -c
  Terminal for password frontend:                 bash -c
  Running on console:                             no
  Running over SSH:                               yes
cp: cannot create regular file '/home/kata/.local/share/x11docker/tini-static-kata': No such file or directory
chmod: cannot access '/home/kata/.local/share/x11docker/tini-static-kata': No such file or directory
x11docker WARNING: Your tini binary is not executeable. Please run
    chmod +x /home/kata/.local/share/x11docker/tini-static-kata

DEBUGNOTE[20:08:50,887]: Found tini binary: /home/kata/.local/share/x11docker/tini-static-kata
DEBUGNOTE[20:08:51,047]: Generated docker command:
  docker run --tty --rm --detach \
  --name x11docker_X101_8053042_x11docker-check \
  --user 1000:1000 \
  --env USER=kata \
  --userns host \
  --group-add 39 \
  --group-add 63 \
  --runtime='kata-runtime' \
  --cap-drop ALL \
  --security-opt no-new-privileges \
  --security-opt label=type:container_runtime_t \
  --tmpfs /run --tmpfs /run/lock \
  --volume '/home/kata/.cache/x11docker/x11docker-check-8053042/share':'/x11docker':rw \
  --workdir '/tmp' \
  --entrypoint env \
  --env 'container=docker' \
  --env 'XAUTHORITY=/x11docker/Xclientcookie' \
  --env 'DISPLAY=172.17.0.1:101' \
   '--device' '/dev/vfio/16' '--runtime=kata-runtime' '-v' '/dev:/dev' \
  -- x11docker/check /bin/sh - /x11docker/container.CMD.sh
DEBUGNOTE[20:08:51,055]: Command at Line 4017 returned with error code 1:
  grep -e '^DOCKER_'
  7399 - ::create_dockerrc::main::main
x11docker note: setxkbmap not found. Need setxkbmap and xkbcomp to set keyboard layout.
  You can look for the package name of this command at:
 https://github.com/mviereck/x11docker/wiki/dependencies#table-of-all-packages

DEBUGNOTE[20:08:51,095]: Stored background pid 5635 of containershell
DEBUGNOTE[20:08:51,097]: Waiting for X server --xorg to be ready.
DEBUGNOTE[20:08:52,131]: --xorg is ready
DEBUGNOTE[20:08:52,142]: Set pid 5650 on watchlist: xinit
DEBUGNOTE[20:08:52,146]: Stored background pid 5650 of xinit
DEBUGNOTE[20:08:52,151]: Stored background pid 5650 of xinit
DEBUGNOTE[20:08:52,170]: Set pid 5652 on watchlist: Xserver
DEBUGNOTE[20:08:52,173]: Stored background pid 5652 of Xserver
DEBUGNOTE[20:08:52,177]: Stored background pid 5652 of Xserver
DEBUGNOTE[20:08:52,781]: Watching pids:
 5650 pts/0    00:00:00 xinit
DEBUGNOTE[20:08:52,783]: Running xinitrc
DEBUGNOTE[20:08:52,802]: Failed to retrieve trusted cookie from X server. Will bake one myself.
DEBUGNOTE[20:08:52,813]: Created cookie: kata3gfx/unix:101  MIT-MAGIC-COOKIE-1  206f3274ea5880a660853ff5b524b26f
#ffff#6b61746133676678#:101  MIT-MAGIC-COOKIE-1  206f3274ea5880a660853ff5b524b26f
DEBUGNOTE[20:08:52,865]: Running dockerrc
DEBUGNOTE[20:08:52,875]: Found default runtime: runc
DEBUGNOTE[20:08:52,884]: All Runtimes: kata-runtime runc
x11docker note: Option --wm: Did not find window manager image
      x11docker/openbox
  to provide a containerized window manager. Please run:
      docker pull x11docker/openbox
  If you want to use a host window manager instead and avoid this warning,
  use option                         --wm=host  or  --wm=COMMAND
  or provide a local image with e.g. --wm=x11docker/lxde
  To run without a window manager:   --wm=none  or  --desktop
  Fallback: Will try to run a host window manager: xfwm4

x11docker note: Option --wm: Starting host window manager: xfwm4

DEBUGNOTE[20:08:52,914]: Stored background pid 5998 of windowmanager
DEBUGNOTE[20:08:54,805]: Watching pids:
 5650 pts/0    00:00:00 xinit
 5652 tty10    00:00:00 Xorg
DEBUGNOTE[20:08:54,926]: Container ID: 02b21d5948eacd10655bd2b5cbff852be155bdbeb786423cfdadc973735d6f31
DEBUGNOTE[20:08:54,946]: Container is up and running.
DEBUGNOTE[20:08:54,956]: 1. check for PID 1: 6304
DEBUGNOTE[20:08:54,966]: Host PID of container PID 1: 6304
DEBUGNOTE[20:08:54,975]: Container IP: 172.17.0.3
DEBUGNOTE[20:08:55,321]: Waiting for file creation of /home/kata/.cache/x11docker/x11docker-check-8053042/xtermready
/x11docker/container.CMD.sh: 95: exec: /x11docker/tini: not found
DEBUGNOTE[20:08:56,337]: Stored background pid 6304 of containerpid1
DEBUGNOTE[20:08:56,342]: Set pid 6304 on watchlist: containerpid1
DEBUGNOTE[20:08:56,348]: Stored background pid 6304 of containerpid1
DEBUGNOTE[20:08:56,833]: Watching pids:
 5650 pts/0    00:00:00 xinit
 5652 tty10    00:00:00 Xorg
DEBUGNOTE[20:08:56,837]: watchpidlist: PID 6304 has terminated
DEBUGNOTE[20:08:56,841]: time to say goodbye (watchpidlist 6304)
DEBUGNOTE[20:08:57,368]: Process tree of container:

DEBUGNOTE[20:08:57,387]: Process tree of x11docker:
  bash(5014)-+-bash(5326)---tail(6719)
           |-bash(5340)---tail(5342)
           |-bash(5365)---sleep(6869)
           |-bash(5635)---bash(6879)---pstree(6880)
           |-tail(5325)
           `-xinit(5650)-+-Xorg(5652)---{Xorg}(5687)
                         `-sh(5688)---sleep(6790)
DEBUGNOTE[20:08:58,846]: time to say goodbye (watchpidlist)
DEBUGNOTE[20:08:58,956]: time to say goodbye (main)
DEBUGNOTE[20:08:58,960]: Terminating x11docker.
DEBUGNOTE[20:08:58,966]: List of stored background processes:
5326 tailstdout
5340 tailstderr
5365 watchpidlist
5373 watchmessagefifo
5635 containershell
5650 xinit
5650 xinit
5652 Xserver
5652 Xserver
5998 windowmanager
6304 containerpid1
6304 containerpid1
DEBUGNOTE[20:08:58,984]: Checking: 6304 (containerpid1):
DEBUGNOTE[20:08:59,000]: Checking: 6304 (containerpid1):
DEBUGNOTE[20:08:59,016]: Checking: 5998 (windowmanager):
DEBUGNOTE[20:08:59,032]: Checking: 5652 (Xserver):
DEBUGNOTE[20:08:59,049]: Checking: 5652 (Xserver):
DEBUGNOTE[20:08:59,065]: Checking: 5650 (xinit):
DEBUGNOTE[20:08:59,081]: Checking: 5650 (xinit):
DEBUGNOTE[20:08:59,098]: Checking: 5635 (containershell):
DEBUGNOTE[20:08:59,114]: Checking: 5373 (watchmessagefifo):
DEBUGNOTE[20:08:59,130]: Checking: 5365 (watchpidlist):
DEBUGNOTE[20:08:59,146]: Checking: 5340 (tailstderr):  5340 pts/0    00:00:00 bash
DEBUGNOTE[20:08:59,162]: Checking: 5326 (tailstdout):  5326 pts/0    00:00:00 bash
DEBUGNOTE[20:08:59,167]: time to say goodbye (finish)
DEBUGNOTE[20:08:59,171]: Exitcode 0

I exec'd into the running runc container with Docker and see that /x11docker/tini is indeed there.

Here is GVT-d. With GVT-d I don't have a display since it is bound to vfio-pci so I don't really expect this scenario to work.

$ /x11docker --xorg -- --device /dev/vfio/1 --runtime=kata-runtime -v /dev:/dev -- x11docker/check --debug

x11docker WARNING: On debian 9, switching often between multiple X servers can
  cause a crash of one X server. This bug may be debian specific and is probably
  some sort of race condition. If you know more about this or it occurs on
  other systems, too, please report at https://github.com/mviereck/x11docker.

  You can avoid this issue with switching to a black tty before switching to X.

cp: cannot create regular file '/home/kata/.local/share/x11docker/tini-static-kata': No such file or directory
chmod: cannot access '/home/kata/.local/share/x11docker/tini-static-kata': No such file or directory
x11docker WARNING: Your tini binary is not executeable. Please run
    chmod +x /home/kata/.local/share/x11docker/tini-static-kata

x11docker note: setxkbmap not found. Need setxkbmap and xkbcomp to set keyboard layout.
  You can look for the package name of this command at:
 https://github.com/mviereck/x11docker/wiki/dependencies#table-of-all-packages



x11docker ERROR: X server --xorg not ready after 60s.

  Type 'x11docker --help' for usage information
  Debug options: '--verbose' (full log) or '--debug' (log excerpt).
  Logfile will be: /home/kata/.cache/x11docker/x11docker.log
  Please report issues at https://github.com/mviereck/x11docker

[mviereck]

I am not sure yet why the tini issue occurs. It causes the entire test to fail.
Did you update to latest master version yet?
However, to avoid the tini issue and to focus on the GPU test, please add option --init=none.

/x11docker --debug --xorg --init=none -- --device /dev/vfio/15 --runtime=kata-runtime -v /dev:/dev -- x11docker/check

[eadamsintel]

Good news @mviereck Setting --init=none worked. The GUI popped up in a kata container and it seems to work. The glxgears test executed showing some gears rotating around at 1100 fps. The glxspheres64 test worked and so did xeyes. I verified this was indeed running inside a Kata container VM.

Edit: I did a docker pull x11docker/check and what I had was already the latest so I don't know why the tini binary has issues.

[mviereck]

Good news @mviereck Setting --init=none worked.

:-)

I did a docker pull x11docker/check and what I had was already the latest so I don't know why the tini binary has issues.

I mean the latest master version of x11docker. It includes a fix for tini.

The glxgears test executed showing some gears rotating around at 1100 fps.

High frame rates rather indicate software rendering. With GPU access glxgears mostly shows about 60 fps matching the monitor frame rate.
Can you check the glxinfo button, too? It should show the Intel GPU, but not llvmpipe.

[eadamsintel]

The glxinfo button shows that GPU not accessible. Software rendering indicated with llvmpipe. I exec'd into the running container and fortunately you had lspci in there. I did see the graphics card in the container.

kata@kata3gfx~ $ docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED                                                              STATUS              PORTS               NAMES
984c9e4c3a5f        x11docker/check     "env /bin/sh - /x11d…"   55 seconds ago                                                       Up 53 seconds                           x11docker_X100_8415532_x11docker-ch                                                 eck
kata@kata3gfx~ $ docker exec -it 984c9e4c3a5f bash
kata@984c9e4c3a5f:/tmp$ lspci
00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02)
00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II]
00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 03)
00:02.0 PCI bridge: Red Hat, Inc. QEMU PCI-PCI bridge
00:03.0 Communication controller: Red Hat, Inc Virtio console
00:04.0 SCSI storage controller: Red Hat, Inc Virtio SCSI
00:05.0 Unclassified device [00ff]: Red Hat, Inc Virtio RNG
00:06.0 Unclassified device [0002]: Red Hat, Inc Virtio filesystem
00:07.0 Ethernet controller: Red Hat, Inc Virtio network device
00:08.0 VGA compatible controller: Intel Corporation Iris Pro Graphics 580 (rev 09)

As an FYI I deleted x11docker, did a git clone https://github.com/mviereck/x11docker.git and ran without the --init=none and it did work, However, I found that executing this docker and then exiting out that eventually the container is hung for docker and it ties up that vfio device number. It still did not recognize the GPU in glxinfo.

Note, even with the new binary it fails with the --gpu option on the x11docker command line.

$ ./x11docker --debug --xorg --gpu  -- --device /dev/vfio/17 --runtime=kata-runtime -v /dev:/dev -- x11docker/check
x11docker note: Found option --runtime in custom DOCKER_RUN_OPTIONS.
  Please use x11docker option --runtime instead.

DEBUGNOTE[21:19:36,572]: ps can watch root processes: yes
x11docker WARNING: User kata is member of group docker.
  That allows unprivileged processes on host to gain root privileges.

DEBUGNOTE[21:19:36,864]:
x11docker version: 6.1.0-beta
docker version:    Docker version 18.06.3, build d7080c17a580919f5340a15a8e5e013133089680
Host system:       Clear Linux OS
Command:           './x11docker' '--debug' '--xorg' '--gpu' '--' '--device' '/dev/vfio/17' '--runtime=kata-runtime' '-v' '/dev:/dev' '--' 'x11docker/check'
Parsed options:     --debug --xorg --gpu -- '--device' '/dev/vfio/17' '--runtime=kata-runtime' '-v' '/dev:/dev' '--' 'x11docker/check'
DEBUGNOTE[21:19:36,869]: Dependency check for --xorg: 0
DEBUGNOTE[21:19:36,873]: Dependency check for --xorg: 0
DEBUGNOTE[21:19:36,878]: Dependency check for --xorg: 0
DEBUGNOTE[21:19:36,882]: Using X server option --xorg
x11docker WARNING: Your configuration seems not to allow to start
  a second core Xorg server from within X. Option --xorg will probably fail.
  (Per default, only root or console users are allowed to run an Xorg server).

  Possible solutions:
  1.) Install one of nested X servers 'Xephyr', 'Xnest' or 'nxagent'.
      For --gpu support: install 'weston' and 'Xwayland'.
  2.) Switch to console tty1...tty6 with <CTRL><ALT><F1>...<F6>
      and start x11docker there.
  3.) Run x11docker as root.
  4.) Edit file '/etc/X11/Xwrapper.config' and replace line:
          allowed_users=console
      with lines
          allowed_users=anybody
          needs_root_rights=yes
      If the file does not exist already, you can create it.
      On Ubuntu 16.04 and debian 9 you need package xserver-xorg-legacy.

x11docker note: Option --xorg: x11docker assumes that you need
  a window manager. If you don't want this, run with option --desktop.
  Enabling option --wm to provide a window manager.

x11docker WARNING: Option --gpu degrades container isolation.
  Container gains access to GPU hardware.
  This allows reading host window content (palinopsia leak)
  and GPU rootkits (compare proof of concept: jellyfish).

x11docker note: Option --runtime=kata-runtime: Be aware not to share
  the same files with runc and kata-runtime containers at the same time.
  Otherwise container startup may fail.

x11docker WARNING: x11docker can run Xorg on another tty (option --xorg),
  but you won't see it in your SSH session.
  Rather install e.g. Xephyr on ssh server and use option --xephyr.

x11docker WARNING: Found custom DOCKER_RUN_OPTIONS.
  x11docker will add them to 'docker run' command without
  a serious check for validity or security. Found options:
   '--device' '/dev/vfio/17' '--runtime=kata-runtime' '-v' '/dev:/dev'

DEBUGNOTE[21:19:36,903]: Stored background pid 1799 of tailstdout
DEBUGNOTE[21:19:36,904]: Waiting for file content in /home/kata/.cache/x11docker/x11docker-check-8477649/container.pid1pid
DEBUGNOTE[21:19:36,908]: Stored background pid 1814 of tailstderr
DEBUGNOTE[21:19:36,927]: Stored background pid 1838 of watchpidlist
DEBUGNOTE[21:19:36,931]: Stored background pid 1846 of watchmessagefifo
x11docker note: Option --gpu: Hardware acceleration with --runtime=kata-runtime
  is possible with Intel hardware only and needs some special setup. Compare:
  https://github.com/kata-containers/documentation/blob/master/use-cases/GPU-passthrough-and-Kata.md

DEBUGNOTE[21:19:36,965]: TTYs currently known to kernel:    1  2  3  4  5  6
x11docker note: New Xorg server  will run on tty 8.
  Access it with [CTRL][ALT][F8].

x11docker WARNING: On debian 9, switching often between multiple X servers can
  cause a crash of one X server. This bug may be debian specific and is probably
  some sort of race condition. If you know more about this or it occurs on
  other systems, too, please report at https://github.com/mviereck/x11docker.

  You can avoid this issue with switching to a black tty before switching to X.

DEBUGNOTE[21:19:36,973]: New X environment:
  DISPLAY=172.17.0.1:100 XAUTHORITY=/home/kata/.cache/x11docker/x11docker-check-8477649/share/Xclientcookie XSOCKET=  X11DOCKER_CACHE=/home/kata/.cache/x11docker/x11docker-check-8477649
DEBUGNOTE[21:19:36,979]: X server command:
  /usr/bin/Xorg :100 vt8 \
  -dpms -s off -retro \
  +extension RANDR \
  +extension RENDER \
  +extension GLX \
  +extension XVideo \
  +extension DOUBLE-BUFFER \
  +extension SECURITY \
  +extension DAMAGE \
  -extension X-Resource \
  -extension XINERAMA -xinerama \
  -extension MIT-SHM \
  -auth /home/kata/.cache/x11docker/x11docker-check-8477649/Xservercookie \
  -listen tcp \
  +extension Composite +extension COMPOSITE \
  -extension XTEST -tst \
  -verbose \
  -iglx
DEBUGNOTE[21:19:37,051]: Users and terminal:
  x11docker was started by:                       kata
  As host user serves (running X, storing cache): kata
  Container user will be:                         kata
  Container user password:                        x11docker
  Getting permission to run docker with:          bash -c
  Running X and other user commands with:         bash -c
  Terminal for password frontend:                 bash -c
  Running on console:                             no
  Running over SSH:                               yes
DEBUGNOTE[21:19:37,060]: Found tini binary: /home/kata/.local/share/x11docker/tini-static-kata
x11docker WARNING: Sharing device file: /dev/vfio

DEBUGNOTE[21:19:37,235]: Generated docker command:
  docker run --tty --rm --detach \
  --name x11docker_X100_8477649_x11docker-check \
  --user 1000:1000 \
  --env USER=kata \
  --userns host \
  --group-add 39 \
  --group-add 63 \
  --runtime='kata-runtime' \
  --cap-drop ALL \
  --security-opt no-new-privileges \
  --security-opt label=type:container_runtime_t \
  --volume '/home/kata/.local/share/x11docker/tini-static-kata':'/x11docker/tini':ro \
  --tmpfs /run --tmpfs /run/lock \
  --volume '/home/kata/.cache/x11docker/x11docker-check-8477649/share':'/x11docker':rw \
  --device '/dev/vfio':'/dev/vfio':rw \
  --workdir '/tmp' \
  --entrypoint env \
  --env 'container=docker' \
  --env 'XAUTHORITY=/x11docker/Xclientcookie' \
  --env 'DISPLAY=172.17.0.1:100' \
   '--device' '/dev/vfio/17' '--runtime=kata-runtime' '-v' '/dev:/dev' \
  -- x11docker/check /bin/sh - /x11docker/container.CMD.sh
DEBUGNOTE[21:19:37,245]: Command at Line 4038 returned with error code 1:
  grep -e '^DOCKER_'
  7431 - ::create_dockerrc::main::main
x11docker note: setxkbmap not found. Need setxkbmap and xkbcomp to set keyboard layout.
  You can look for the package name of this command at:
 https://github.com/mviereck/x11docker/wiki/dependencies#table-of-all-packages

DEBUGNOTE[21:19:37,297]: Stored background pid 2184 of containershell
DEBUGNOTE[21:19:37,299]: Waiting for X server --xorg to be ready.
DEBUGNOTE[21:19:38,334]: --xorg is ready
DEBUGNOTE[21:19:38,346]: Set pid 2198 on watchlist: xinit
DEBUGNOTE[21:19:38,350]: Stored background pid 2198 of xinit
DEBUGNOTE[21:19:38,354]: Stored background pid 2198 of xinit
DEBUGNOTE[21:19:38,373]: Set pid 2202 on watchlist: Xserver
DEBUGNOTE[21:19:38,378]: Stored background pid 2202 of Xserver
DEBUGNOTE[21:19:38,382]: Stored background pid 2202 of Xserver
DEBUGNOTE[21:19:38,937]: Watching pids:
 2198 pts/0    00:00:00 xinit
DEBUGNOTE[21:19:38,940]: Running xinitrc
DEBUGNOTE[21:19:38,967]: Failed to retrieve trusted cookie from X server. Will bake one myself.
DEBUGNOTE[21:19:38,979]: Created cookie: kata3gfx/unix:100  MIT-MAGIC-COOKIE-1  e622c1f42f0a434f0c38f7cdf835332a
#ffff#6b61746133676678#:100  MIT-MAGIC-COOKIE-1  e622c1f42f0a434f0c38f7cdf835332a
DEBUGNOTE[21:19:39,032]: Running dockerrc
DEBUGNOTE[21:19:39,042]: Found default runtime: runc
DEBUGNOTE[21:19:39,053]: All Runtimes: kata-runtime runc
x11docker note: Option --wm: Did not find window manager image
      x11docker/openbox
  to provide a containerized window manager. Please run:
      docker pull x11docker/openbox
  If you want to use a host window manager instead and avoid this warning,
  use option                         --wm=host  or  --wm=COMMAND
  or provide a local image with e.g. --wm=x11docker/lxde
  To run without a window manager:   --wm=none  or  --desktop
  Fallback: Will try to run a host window manager: xfwm4

x11docker note: Option --wm: Starting host window manager: xfwm4

DEBUGNOTE[21:19:39,085]: Stored background pid 2569 of windowmanager
DEBUGNOTE[21:19:40,958]: Watching pids:
 2198 pts/0    00:00:00 xinit
 2202 tty8     00:00:00 Xorg

It just kind of got stuck here with an all grey screen and no dialog boxes.

[mviereck]

I exec'd into the running container and fortunately you had lspci in there. I did see the graphics card in the container.

Good.

The glxinfo button shows that GPU not accessible. Software rendering indicated with llvmpipe.

A thought: Maybe only root can access the Intel GPU shared this way.
Normally x11docker sets up an unprivileged user and adds it to group video.
Try with --user=root to exclude permission issues:

/x11docker --debug --user=root --xorg --runtime=kata-runtime  -- --device /dev/vfio/15 -v /dev:/dev -- x11docker/check

EDIT: What does ls -l /dev/dri in container show?

Note, even with the new binary it fails with the --gpu option on the x11docker command line.
It just kind of got stuck here with an all grey screen and no dialog boxes.

That is odd. Currently --gpu only adds --device /dev/vfio. But somehow x11docker stucks after X is ready and some first container messages should appear.
Anyway, let's try without --gpu for now.

However, I found that executing this docker and then exiting out that eventually the container is hung for docker and it ties up that vfio device number.

Odd. x11docker always runs containers with --rm to delete the container afterwards. I don't know why the vfio device number is not released.

[eadamsintel]

A thought: Maybe only root can access the Intel GPU shared this way.
Normally x11docker sets up an unprivileged user and adds it to group video.
Try with --user=root to exclude permission issues:

/x11docker --debug --user=root --xorg --runtime=kata-runtime  -- --device /dev/vfio/15 -v /dev:/dev -- x11docker/check

It still shows that the GPU is not accessible.

EDIT: What does ls -l /dev/dri in container show?

root@a4c0f017fe61:/tmp# ls -l /dev/dri
total 0
crw------- 1 root root 226,   0 Jul 18 23:01 card0
crw------- 1 root root 226, 128 Jul 18 23:01 renderD128

Odd. x11docker always runs containers with --rm to delete the container afterwards. I don't know why the vfio device number is not released.

What I mean to say is that even though the container has exited docker ps shows that it is still running and the vfio device is tied up. Trying systemctl restart docker hangs. This could possibly be a kata bug with GVT-g when a less than clean exit happens. I just reproduced it by clicking the "exit" button on the GUI. The container appeared to stop from the log messages and it returned me back to my bash session. However, docker ps still shows the container running and it hangs if I try to exec into it.

[mviereck]

Currently I am not sure how to debug further.

• The setup --device /dev/vfio/15 -v /dev:/dev is the same as suggested in the kata documentation.
• The root root owner in /dev/dri should be set by kata to root video instead, but we already bypassed this issue with --user=root.

---

Maybe there is some sort of mismatch in how the X server on host and the container applications in kata access the GPU hardware. It works well for runc containers where both directly access /dev/dri from host.

The documentation does not give a recommendation how to check GPU access in kata except lspci and ls /dev/dri.
I have added package intel-gpu-tools to image x11docker/check yet. The image now provides the following intel* commands:

intel-gen4asm           intel_firmware_decode   intel_l3_parity
intel-gen4disasm        intel_forcewaked        intel_lid
intel-gpu-overlay       intel_framebuffer_dump  intel_opregion_decode
intel_aubdump           intel_gem_info          intel_panel_fitter
intel_audio_dump        intel_gpu_abrt          intel_perf_counters
intel_backlight         intel_gpu_frequency     intel_reg
intel_bios_dumper       intel_gpu_time          intel_reg_checker
intel_display_crc       intel_gpu_top           intel_residency
intel_display_poller    intel_gtt               intel_stepping
intel_dp_compliance     intel_guc_logger        intel_vbt_decode
intel_dump_decode       intel_gvtg_test         intel_watermark

Commands like intel_gpu_top or intel_gem_info might be useful to check GPU access independent from the X server. Please update the image with docker pull x11docker/check for these commands.

Just in case I've added package xserver-xorg-video-all that includes the Xorg driver for Intel GPU. Should not be needed for X clients, but who knows.

---

What I mean to say is that even though the container has exited docker ps shows that it is still running and the vfio device is tied up. Trying systemctl restart docker hangs. This could possibly be a kata bug with GVT-g when a less than clean exit happens.

I'd say it is a kata bug.

[jodh-intel]

It would be helpful if those affected by this could:

1. Enable full debug.
2. Re-run the command.
3. Run sudo kata-collect-data.sh
4. Review the output.
5. Paste it into this issue.

That would help us understand the problem in more detail.

[mviereck]

I found that GPU access fails with runc using X over IP, too, except I set docker option --net=host or x11docker option --hostnet. Sorry, I should have checked that before. x11docker's default for runc is a shared X socket, but default for kata is X over IP.

Please try with --hostnet:

/x11docker --debug --user=root --hostnet --xorg --runtime=kata-runtime  -- --device /dev/vfio/15 -v /dev:/dev -- x11docker/check

[grahamwhaley]

presuming that is the same as --net=host?, we don't support that with kata afaik, as then kata tries to reconfig the host networking, and things go downhill from there...

[eadamsintel]

When I enable --hostnet the container boots up, grey screen shows, and then it crashes with Exitcode 1. I tried --hostnet but as expected it didn't work and crashed consistently.

/x11docker --debug --user=root --hostnet --xorg --runtime=kata-runtime -- --device /dev/vfio/17 -v /dev:/dev -- x11docker/check

I also can now reliably reproduce the crash on exit issue when I use my first vfio device just by launching and clicking the "exit" button. The container appears to properly shutdown but docker ps still shows the container is running and the vfio 15 device is locked up. A systemctl restart docker never seems to complete. It is really hung. This might be a real bug if the first vfio device can never be released. I wonder if there is something else going on with my system because I remember when I tested out transcoding with GVT-g months ago for a demo that the first vfio in a GVT-g setup sometimes failed to work. I would just use the other 3 for showing the demo and never saw an issue. I'll try to get another Intel NUC with a different Iris Pro graphics generation to see if the first vfio shows issue. Perhaps 'x11docker is exposing an issue with GVT-g. I should note that I've never gotten GVT-d to work. When the graphics drivers is bound to vfio there is no display on my monitor. The video card is in the container but I'm not sure how output is supposed to go to the display.

I ran the kata-collect-data.sh script in both cases (--hostnet on and also with it not specified). It always crashes with --hostnet specified. The one with vfio 15 in the file name is from when the container is left dangling in Docker on what looks like a clean exit by clicking the exit button.

x11docker_user_root_hostnet_xorg_vfio17_x11docker_check.txt
x11docker_user_root_xorg_vfio15_x11docker_check.txt

[eadamsintel]

OK I was able to test this out on another Clear Linux based system with Iris Plus Graphics 655. It has the exact same issues as my other system and hangs when cleanly exiting a container only when I use the first vfio. There is clearly an issue with the first vfio devices on two systems when I exit a kata container at least on a Clear Linux system. I am working to update an Ubuntu system and test there.

Edit: I tested in Ubuntu and it doesn't hang on the first vfio device. It shows the same issues though with GPU not accessible. Software rendering indicated with llvmpipe. I had to use sudo to run x11docker to get it to work with both runc and kata-runtime.

[mviereck]

presuming that is the same as --net=host?, we don't support that with kata

I tried --hostnet but as expected it didn't work and crashed consistently.

Ok, I see. Anyway, a setup with --net=host (same as --hostnet) would not be a desirable permanent solution as it severely degrades container isolation.

Just a thought: Maybe it works if the network of the kata VM can be shared with the container (something like --net=kata). If I understand correctly, /dev/dri with this Intel GPU setup is provided by the VM and not by the host, so the kata/qemu network stack might be appropriate.

It is odd that the combination 'X over TCP' + '--net=host' + 'access to /dev/dri' works at all. I just found that it works (at least with runc), but can not say why. I did some research, but did not find a hint.
We can try further to get a working setup this way, but I am not sure if that would be a good way.

I had to use sudo to run x11docker to get it to work with both runc and kata-runtime.

Probably your Ubuntu user is not in group docker. However, that is not needed, running x11docker with sudo is ok, it still runs unprivileged as much as possible.

The container appears to properly shutdown but docker ps still shows the container is running and the vfio 15 device is locked up. A systemctl restart docker never seems to complete.

Does docker stop CONTAINERID work? x11docker does this if the container does not terminate itself, but maybe something has gone wrong here.

---

For now I don't see how to provide X access with GPU acceleration for kata containers.

It would most likely work with iGLX / indirect rendering and would furthermore be independent from the GPU vendor. But iGLX is broken in Xorg since 1.18.4 from 2017:
https://bugs.launchpad.net/ubuntu/+source/xorg-server/+bug/1776447
https://gitlab.freedesktop.org/xorg/xserver/issues/211
Unfortunately there seems to be no active work on this bug.

If you have an idea how to encourage the Xorg developers to fix this bug, we could set up a clean and easy way to access the GPU.

---

Thanks @eadamsintel for your tests and thanks @all for your interest!
At least an x11docker bug with tini was detected and fixed during the tests, and maybe a kata issue detected with the first vfio device.

Currently I have no ideas what to test or develop further in x11docker for kata. If you come up with something, I am glad to hear it.

---

Finally two good news aside from X and GPU:

• Sound: x11docker supports pulseaudio sound in kata-containers with --pulseaudio. For pulseaudio setup without x11docker see https://github.com/mviereck/x11docker/wiki/Container-sound:-ALSA-or-Pulseaudio#pulseaudio-over-tcp
• Printer: x11docker supports access to host CUPS server with --printer. Might need a CUPS configuration change in /etc/cups/cupsd.conf:
  Port 631 <Location /> # Allow remote access... Order allow,deny Allow 172.17.0.* Allow 127.0.0.1 </Location>"
172.17.0.* is the IP mask of docker container IP's.
  With this CUPS configuration kata containers can access host printers with --env CUPS_SERVER=$HOSTIP:631 (or x11docker option --printer). $HOSTIP is the IP of the docker daemon itself, here usually 172.17.0.1. Compare: https://github.com/mviereck/x11docker/wiki/CUPS-printer-in-container#cups-over-tcp