Runtime: failure to start debug console with cloud hypervisor

[egernst]

Description of problem

Unable to start a console connection to the guest when utilizing Cloud Hypervisor. Cloud Hypervisor should be able to provide a /dev/console (unlike FC, which requires setting up a listener on the particular vsock port).

When enabling console debug and connecting to clh.sock (the vsock connection to the virtual machine) with socat, the connection closes after the first character is transmitted from the host to the guest, and an error can be seen on the cloud-hypervisor logs.

ie:

sudo socat "stdin,raw,echo=0,escape=0x11" "unix-connect:clh.sock"

Expected result

A shell is available.

Actual result

socat returns as soon as a character is transmitted.If you observe the hypervisor logs, you'll see:

kata-hypervisor[683757]: cloud-hypervisor: 19.050775789s: INFO:virtio-devices/src/vsock/unix/muxer.rs:440 -- vsock: error adding local-init connection: UnixRead(Os { code: 11, kind: WouldBlock, message: "Resource temporarily unavailable" })

Setup

• install latest kata:

docker run -v /opt/kata:/opt/kata -v /var/run/dbus:/var/run/dbus -v /run/systemd:/run/systemd -v /etc/docker:/etc/docker -it katadocker/kata-deploy kata-deploy-docker install

• build rootfs which contains sh or bash (use the centos rootfs)
• Tested with each of the modified kernel command line parameters below with no success:
  
  
  
  • agent.debug_console_vport=1026
  
  
  • agent.debug_console
  
  

Full hypervisor log available: https://gist.github.com/egernst/eec3723ea6342133ee2528cc334b7d02

[egernst]

/cc @sboeuf @likebreath @jcvenegas @amshinde @devimc -- heads up

[likebreath]

As @sboeuf advised, I did a quick experiment using fc+kata, and I did not get the debug console work.

Here is my setup: I was using everything upstream 1.x (tests/runtime, etc). I built a centOS debug image based on the Deveopler-Guide.md (https://github.com/kata-containers/documentation/pull/704), and also updated the configuration.tomlto use this debug image and kernel command kernel_params="agent.debug_console_vport=1026" . From there, I launched a simple container w/ docker: sudo docker run --runtime=kata-runtime -ti busybox sh, and tried to connect to the debug console from the host.

With inputs from @egernst, I located the kata.hvsock under the path /run/vc/firecracker/$container_id/root. After going to that folder with root, the socat stdin unix-connect:kata.hvsock simply hangs. If I press another enter during the hang, the socat command will simply finish silently after few (1-2) seconds. This is how it looks like:

root@kata-fc-0820:/run/vc/firecracker/208fdb781f66273265c10a5f25ef2e41/root# ls
drive_0  drive_1  drive_2  drive_3  drive_4  drive_5  drive_6  drive_7  fcConfig.json  kata.hvsock  logs.fifo  metrics.fifo  rootfs  run  vmlinux
root@kata-fc-0820:/run/vc/firecracker/208fdb781f66273265c10a5f25ef2e41/root# socat stdin unix-connect:kata.hvsock

root@kata-fc-0820:/run/vc/firecracker/208fdb781f66273265c10a5f25ef2e41/root# 

I shared this observation with @egernst and he thought this is very similar to what he observed w/ clh+kata. If my experiment setup was correct, it looks like an issue of using the debug console w/ vsock in general. Any insights/thoughts?

/cc @devimc @amshinde @jcvenegas @sboeuf @egernst

[amshinde]

@likebreath I tried to get the debug console running with firecracker in connection to my PR https://github.com/kata-containers/documentation/pull/704#pullrequestreview-472071387 and saw exactly the same observations as yours.
Debug console with hybrid vsocks seems to be broken in general.
I'd like to get input from @devimc as he was the one who implemented debug console with hybrid vsocks.

[devimc]

works for me, with its expected limitations, did you read the documentation? https://github.com/kata-containers/agent/#enable-debug-console-for-firecracker

# socat stdin unix-connect:kata.hvsock
CONNECT 1027
OK 1073742109
bash: cannot set terminal process group (-1): Inappropriate ioctl for device
bash: no job control in this shell
bash: grep: command not found
bash: grep: command not found
bash: tty: command not found
bash: expr: command not found
bash: [: : integer expression expected
bash-5.0# 

[devimc]

same result using cloud-hypervisor

# socat stdin unix-connect:clh.sock
CONNECT 1027
OK 1073742979
bash: cannot set terminal process group (-1): Inappropriate ioctl for device
bash: no job control in this shell
bash: grep: command not found
bash: grep: command not found
bash: tty: command not found
bash: expr: command not found
bash: [: : integer expression expected
bash-5.0#

[egernst]

ACK. I see same behavior. I'll work on updating the docs to cover each VMM (probably setup a single debug document). Thx Julio.

[amshinde]

@devimc @egernst I'd really like to see what was causing issues on my side to figure out if this does not work with the configuration I was using.
@devimc I did follow all the instructions you documented. Did you test this with a clearlinux image and a particular version of kata?

[devimc]

@amshinde no, I'm using master, it's worth mentioning that you have to type CONNECT $PORT, where $PORT is the port number specified through agent.debug_console_vport=