Runtime: SPDK vhost failed with message: Could not open '/dev/vda': No such file or directory: unknown
Description of problem
I am trying to run SPDK vhost-user devices with Kata Containers following every step of this tutorial :
https://github.com/kata-containers/documentation/blob/master/use-cases/using-SPDK-vhostuser-and-kata.md
All setups are with default configuration, because I am new to SPDK and kata.
Actual result
Every thing went well, but at last step, it failed with below message
$ sudo docker run --runtime kata-runtime --device=/var/run/kata-containers/vhost-user/block/devices/vhostblk0:/dev/vda -it busybox sh
docker: Error response from daemon: OCI runtime create failed: QMP command failed: Could not open '/dev/vda': No such file or directory: unknown.
ERRO[0001] error waiting for container: context canceled
Below is my setup
- SPDK version v20.01.1
$ uname -a
Linux ip-172-31-3-100 4.15.0-1057-aws #59-Ubuntu SMP Wed Dec 4 10:02:00 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
$ docker --version
Docker version 19.03.8, build afacb8b7f0
$ kata-runtime version
kata-runtime : 1.11.0-alpha0
commit : <<unknown>>
OCI specs: 1.0.1-dev
$ kata-runtime kata-env
[Meta]
Version = "1.0.23"
[Runtime]
Debug = true
Trace = false
DisableGuestSeccomp = true
DisableNewNetNs = false
SandboxCgroupOnly = false
Path = "/usr/bin/kata-runtime"
[Runtime.Version]
OCI = "1.0.1-dev"
[Runtime.Version.Version]
Semver = "1.11.0-alpha0"
Major = 1
Minor = 11
Patch = 0
Commit = ""
[Runtime.Config]
Path = "/etc/kata-containers/configuration.toml"
[Hypervisor]
MachineType = "pc"
Version = "QEMU emulator version 4.1.1\nCopyright (c) 2003-2019 Fabrice Bellard and the QEMU Project developers"
Path = "/usr/bin/qemu-vanilla-system-x86_64"
BlockDeviceDriver = "virtio-scsi"
EntropySource = "/dev/urandom"
SharedFS = "virtio-9p"
Msize9p = 8192
MemorySlots = 10
PCIeRootPort = 0
HotplugVFIOOnRootBus = false
Debug = true
UseVSock = false
[Image]
Path = "/usr/share/kata-containers/kata-containers-image_clearlinux_1.11.0-alpha0_agent_d26a505efd.img"
[Kernel]
Path = "/usr/share/kata-containers/vmlinuz-5.4.15.66-45.container"
Parameters = "systemd.unit=kata-containers.target systemd.mask=systemd-networkd.service systemd.mask=systemd-networkd.socket agent.log=debug agent.log=debug initcall_debug"
[Initrd]
Path = ""
[Proxy]
Type = "kataProxy"
Path = "/usr/libexec/kata-containers/kata-proxy"
Debug = true
[Proxy.Version]
Semver = "1.11.0-alpha0-4ec94c8"
Major = 1
Minor = 11
Patch = 0
Commit = "4ec94c8"
[Shim]
Type = "kataShim"
Path = "/usr/libexec/kata-containers/kata-shim"
Debug = true
[Shim.Version]
Semver = "1.11.0-alpha0-834e0b3"
Major = 1
Minor = 11
Patch = 0
Commit = "834e0b3"
[Agent]
Type = "kata"
Debug = true
Trace = false
TraceMode = ""
TraceType = ""
[Host]
Kernel = "4.15.0-1057-aws"
Architecture = "amd64"
VMContainerCapable = true
SupportVSocks = true
[Host.Distro]
Name = "Ubuntu"
Version = "18.04"
[Host.CPU]
Vendor = "GenuineIntel"
Model = "Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz"
[Netmon]
Path = "/usr/libexec/kata-containers/kata-netmon"
Debug = true
Enable = false
[Netmon.Version]
Semver = "1.11.0-alpha0"
Major = 1
Minor = 11
Patch = 0
Commit = "<<unknown>>"
Full debug log
I tried to enable debug logs, and got below:
Mar 30 05:28:03 ip-172-31-3-100 kernel: [36515.366224] docker0: port 1(veth728c472) entered blocking state
Mar 30 05:28:03 ip-172-31-3-100 kernel: [36515.366227] docker0: port 1(veth728c472) entered disabled state
Mar 30 05:28:03 ip-172-31-3-100 kernel: [36515.366352] device veth728c472 entered promiscuous mode
Mar 30 05:28:03 ip-172-31-3-100 kernel: [36515.366526] IPv6: ADDRCONF(NETDEV_UP): veth728c472: link is not ready
Mar 30 05:28:03 ip-172-31-3-100 kata-runtime[13949]: time="2020-03-30T05:28:03.094494707Z" level=info msg="loaded configuration" arch=amd64 command=create file=/etc/kata-containers/configuration.toml format=TOML name=kata-runtime pid=13949 source=katautils
Mar 30 05:28:03 ip-172-31-3-100 kata-runtime[13949]: time="2020-03-30T05:28:03.094619134Z" level=debug arch=amd64 command=create default-kernel-parameters="systemd.unit=kata-containers.target systemd.mask=systemd-networkd.service systemd.mask=systemd-networkd.socket" name=kata-runtime pid=13949 source=katautils
Mar 30 05:28:03 ip-172-31-3-100 kata-runtime[13949]: time="2020-03-30T05:28:03.094661646Z" level=info arch=amd64 arguments="\"create --bundle /run/containerd/io.containerd.runtime.v1.linux/moby/1f86a80671cedb53b6fc7b9846cf44991743b18d029d9b570233ce32d76a2e9a --pid-file /run/containerd/io.containerd.runtime.v1.linux/moby/1f86a80671cedb53b6fc7b9846cf44991743b18d029d9b570233ce32d76a2e9a/init.pid --console-socket /tmp/pty997844819/pty.sock 1f86a80671cedb53b6fc7b9846cf44991743b18d029d9b570233ce32d76a2e9a\"" command=create commit= name=kata-runtime pid=13949 source=runtime version=1.11.0-alpha0
Mar 30 05:28:03 ip-172-31-3-100 kata-runtime[13949]: time="2020-03-30T05:28:03.094919746Z" level=debug msg="converting /run/containerd/io.containerd.runtime.v1.linux/moby/1f86a80671cedb53b6fc7b9846cf44991743b18d029d9b570233ce32d76a2e9a/config.json" arch=amd64 command=create container=1f86a80671cedb53b6fc7b9846cf44991743b18d029d9b570233ce32d76a2e9a name=kata-runtime pid=13949 source=virtcontainers subsystem=compatoci
Mar 30 05:28:03 ip-172-31-3-100 kata-runtime[13949]: time="2020-03-30T05:28:03.095940326Z" level=debug msg="container rootfs: /var/lib/docker/overlay2/a262356a7737b21b3438e7968cdd0a29d0e65b73e6b85f8d3017f4f83b92ba47/merged" arch=amd64 command=create container=1f86a80671cedb53b6fc7b9846cf44991743b18d029d9b570233ce32d76a2e9a name=kata-runtime pid=13949 source=virtcontainers subsystem=oci
Mar 30 05:28:03 ip-172-31-3-100 kata-runtime[13949]: time="2020-03-30T05:28:03.095977966Z" level=info msg="shm-size detected: 67108864" arch=amd64 command=create container=1f86a80671cedb53b6fc7b9846cf44991743b18d029d9b570233ce32d76a2e9a name=kata-runtime pid=13949 source=virtcontainers subsystem=oci
Mar 30 05:28:03 ip-172-31-3-100 kata-runtime[13949]: time="2020-03-30T05:28:03.097763012Z" level=info msg="create netns" arch=amd64 command=create container=1f86a80671cedb53b6fc7b9846cf44991743b18d029d9b570233ce32d76a2e9a name=kata-runtime netns=/var/run/netns/cnitest-4d8cae33-507b-9fdb-fb4b-ff0b7a4b619b pid=13949 source=katautils
Mar 30 05:28:03 ip-172-31-3-100 kernel: [36515.567525] eth0: renamed from veth1a1fed0
Mar 30 05:28:03 ip-172-31-3-100 kernel: [36515.587487] IPv6: ADDRCONF(NETDEV_CHANGE): veth728c472: link becomes ready
Mar 30 05:28:03 ip-172-31-3-100 kernel: [36515.587568] docker0: port 1(veth728c472) entered blocking state
Mar 30 05:28:03 ip-172-31-3-100 kernel: [36515.587571] docker0: port 1(veth728c472) entered forwarding state
Mar 30 05:28:03 ip-172-31-3-100 kata-runtime[13949]: time="2020-03-30T05:28:03.252749866Z" level=debug msg="restore sandbox failed" arch=amd64 command=create container=1f86a80671cedb53b6fc7b9846cf44991743b18d029d9b570233ce32d76a2e9a error="open /run/vc/sbs/1f86a80671cedb53b6fc7b9846cf44991743b18d029d9b570233ce32d76a2e9a/persist.json: no such file or directory" name=kata-runtime pid=13949 sandbox=1f86a80671cedb53b6fc7b9846cf44991743b18d029d9b570233ce32d76a2e9a source=virtcontainers subsystem=sandbox
Mar 30 05:28:03 ip-172-31-3-100 kata-runtime[13949]: time="2020-03-30T05:28:03.252916869Z" level=debug msg="Creating bridges" arch=amd64 command=create container=1f86a80671cedb53b6fc7b9846cf44991743b18d029d9b570233ce32d76a2e9a name=kata-runtime pid=13949 source=virtcontainers subsystem=qemu
Mar 30 05:28:03 ip-172-31-3-100 kata-runtime[13949]: time="2020-03-30T05:28:03.252985696Z" level=debug msg="Creating UUID" arch=amd64 command=create container=1f86a80671cedb53b6fc7b9846cf44991743b18d029d9b570233ce32d76a2e9a name=kata-runtime pid=13949 source=virtcontainers subsystem=qemu
Mar 30 05:28:03 ip-172-31-3-100 kata-runtime[13949]: time="2020-03-30T05:28:03.274819175Z" level=debug msg="Disable nesting environment checks" arch=amd64 command=create container=1f86a80671cedb53b6fc7b9846cf44991743b18d029d9b570233ce32d76a2e9a inside-vm=false name=kata-runtime pid=13949 source=virtcontainers subsystem=qemu
Mar 30 05:28:03 ip-172-31-3-100 kata-runtime[13949]: time="2020-03-30T05:28:03.275105576Z" level=info msg="adding volume" arch=amd64 command=create container=1f86a80671cedb53b6fc7b9846cf44991743b18d029d9b570233ce32d76a2e9a name=kata-runtime pid=13949 source=virtcontainers subsystem=qemu volume-type=virtio-9p
Mar 30 05:28:03 ip-172-31-3-100 kata-runtime[13949]: time="2020-03-30T05:28:03.275936991Z" level=info msg="Endpoints found after scan" arch=amd64 command=create container=1f86a80671cedb53b6fc7b9846cf44991743b18d029d9b570233ce32d76a2e9a endpoints="[0xc0002bcc80]" name=kata-runtime pid=13949 source=virtcontainers subsystem=network
Mar 30 05:28:03 ip-172-31-3-100 kata-runtime[13949]: time="2020-03-30T05:28:03.276784145Z" level=info msg="Attaching endpoint" arch=amd64 command=create container=1f86a80671cedb53b6fc7b9846cf44991743b18d029d9b570233ce32d76a2e9a endpoint-type=virtual hotplug=false name=kata-runtime pid=13949 source=virtcontainers subsystem=network
Mar 30 05:28:03 ip-172-31-3-100 kernel: [36515.618733] eth0: Caught tx_queue_len zero misconfig
Mar 30 05:28:03 ip-172-31-3-100 kata-runtime[13949]: time="2020-03-30T05:28:03.27849213Z" level=debug msg="Network added" arch=amd64 command=create container=1f86a80671cedb53b6fc7b9846cf44991743b18d029d9b570233ce32d76a2e9a name=kata-runtime pid=13949 source=virtcontainers subsystem=network
Mar 30 05:28:03 ip-172-31-3-100 kata-runtime[13949]: time="2020-03-30T05:28:03.278577655Z" level=info msg="Starting VM" arch=amd64 command=create container=1f86a80671cedb53b6fc7b9846cf44991743b18d029d9b570233ce32d76a2e9a name=kata-runtime pid=13949 sandbox=1f86a80671cedb53b6fc7b9846cf44991743b18d029d9b570233ce32d76a2e9a source=virtcontainers subsystem=sandbox
Mar 30 05:28:03 ip-172-31-3-100 kata-runtime[13949]: time="2020-03-30T05:28:03.278716032Z" level=debug arch=amd64 command=create container=1f86a80671cedb53b6fc7b9846cf44991743b18d029d9b570233ce32d76a2e9a default-kernel-parameters="tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k console=hvc0 console=hvc1 iommu=off cryptomgr.notests net.ifnames=0 pci=lastbus=0 root=/dev/pmem0p1 rootflags=dax,data=ordered,errors=remount-ro ro rootfstype=ext4 debug systemd.show_status=true systemd.log_level=debug" name=kata-runtime pid=13949 source=virtcontainers subsystem=qemu
Mar 30 05:28:03 ip-172-31-3-100 kata-runtime[13949]: time="2020-03-30T05:28:03.278934838Z" level=info msg="Adding extra file [0xc0000ad4a8 0xc0000ad490]" arch=amd64 command=create container=1f86a80671cedb53b6fc7b9846cf44991743b18d029d9b570233ce32d76a2e9a name=kata-runtime pid=13949 source=virtcontainers subsystem=qmp
Mar 30 05:28:03 ip-172-31-3-100 kata-runtime[13949]: time="2020-03-30T05:28:03.2790445Z" level=info msg="launching /usr/bin/qemu-vanilla-system-x86_64 with: [-name sandbox-1f86a80671cedb53b6fc7b9846cf44991743b18d029d9b570233ce32d76a2e9a -uuid 3b748c3e-317f-4f5d-9859-5681f273d919 -machine pc,accel=kvm,kernel_irqchip,nvdimm -cpu host -qmp unix:/run/vc/vm/1f86a80671cedb53b6fc7b9846cf44991743b18d029d9b570233ce32d76a2e9a/qmp.sock,server,nowait -m 2048M,slots=10,maxmem=774724M-device pci-bridge,bus=pci.0,id=pci-bridge-0,chassis_nr=1,shpc=on,addr=2,romfile= -device virtio-serial-pci,disable-modern=false,id=serial0,romfile= -device virtconsole,chardev=charconsole0,id=console0 -chardev socket,id=charconsole0,path=/run/vc/vm/1f86a80671cedb53b6fc7b9846cf44991743b18d029d9b570233ce32d76a2e9a/console.sock,server,nowait -device nvdimm,id=nv0,memdev=mem0 -object memory-backend-file,id=mem0,mem-path=/usr/share/kata-containers/kata-containers-image_clearlinux_1.11.0-alpha0_agent_d26a505efd.img,size=134217728 -device virtio-scsi-pci,id=scsi0,disable-modern=false,romfile= -object rng-random,id=rng0,filename=/dev/urandom -device virtio-rng,rng=rng0,romfile= -device virtserialport,chardev=charch0,id=channel0,name=agent.channel.0 -chardev socket,id=charch0,path=/run/vc/vm/1f86a80671cedb53b6fc7b9846cf44991743b18d029d9b570233ce32d76a2e9a/kata.sock,server,nowait -device virtio-9p-pci,disable-modern=false,fsdev=extra-9p-kataShared,mount_tag=kataShared,romfile= -fsdev local,id=extra-9p-kataShared,path=/run/kata-containers/shared/sandboxes/1f86a80671cedb53b6fc7b9846cf44991743b18d029d9b570233ce32d76a2e9a,security_model=none -netdev tap,id=network-0,vhost=on,vhostfds=3,fds=4 -device driver=virtio-net-pci,netdev=network-0,mac=02:42:ac:11:00:02,disable-modern=false,mq=on,vectors=4,romfile= -global kvm-pit.lost_tick_policy=discard -vga none -no-user-config -nodefaults -nographic -daemonize -object memory-backend-ram,id=dimm1,size=2048M -numa node,memdev=dimm1 -kernel /usr/share/kata-containers/vmlinuz-5.4.15.66-45.container -append tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k console=hvc0 console=hvc1 iommu=off cryptomgr.notests net.ifnames=0 pci=lastbus=0 root=/dev/pmem0p1 rootflags=dax,data=ordered,errors=remount-ro ro rootfstype=ext4 debug systemd.show_status=true systemd.log_level=debug panic=1 nr_cpus=96 agent.use_vsock=false systemd.unit=kata-containers.target systemd.mask=systemd-networkd.service systemd.mask=systemd-networkd.socket agent.log=debug agent.log=debug initcall_debug -pidfile /run/vc/vm/1f86a80671cedb53b6fc7b9846cf44991743b18d029d9b570233ce32d76a2e9a/pid -D /run/vc/vm/1f86a80671cedb53b6fc7b9846cf44991743b18d029d9b570233ce32d76a2e9a/qemu.log -smp 1,cores=1,threads=1,sockets=96,maxcpus=96]" arch=amd64 command=create container=1f86a80671cedb53b6fc7b9846cf44991743b18d029d9b570233ce32d76a2e9a name=kata-runtime pid=13949 source=virtcontainers subsystem=qmp
Mar 30 05:28:03 ip-172-31-3-100 kata-runtime[13949]: time="2020-03-30T05:28:03.376598989Z" level=info msg="{\"QMP\": {\"version\": {\"qemu\": {\"micro\": 1, \"minor\": 1, \"major\": 4}, \"package\": \"\"}, \"capabilities\": [\"oob\"]}}" arch=amd64 command=create container=1f86a80671cedb53b6fc7b9846cf44991743b18d029d9b570233ce32d76a2e9a name=kata-runtime pid=13949 source=virtcontainers subsystem=qmp
Mar 30 05:28:03 ip-172-31-3-100 kata-runtime[13949]: time="2020-03-30T05:28:03.376932605Z" level=info msg="QMP details" arch=amd64 command=create container=1f86a80671cedb53b6fc7b9846cf44991743b18d029d9b570233ce32d76a2e9a name=kata-runtime pid=13949 qmp-capabilities=oob qmp-major-version=4 qmp-micro-version=1 qmp-minor-version=1 source=virtcontainers subsystem=qemu
Mar 30 05:28:03 ip-172-31-3-100 kata-runtime[13949]: time="2020-03-30T05:28:03.377110391Z" level=info msg="{\"execute\":\"qmp_capabilities\"}" arch=amd64 command=create container=1f86a80671cedb53b6fc7b9846cf44991743b18d029d9b570233ce32d76a2e9a name=kata-runtime pid=13949 source=virtcontainers subsystem=qmp
Mar 30 05:28:03 ip-172-31-3-100 kata-runtime[13949]: time="2020-03-30T05:28:03.391541405Z" level=info msg="{\"return\": {}}" arch=amd64 command=create container=1f86a80671cedb53b6fc7b9846cf44991743b18d029d9b570233ce32d76a2e9a name=kata-runtime pid=13949 source=virtcontainers subsystem=qmp
Mar 30 05:28:03 ip-172-31-3-100 kata-runtime[13949]: time="2020-03-30T05:28:03.391836433Z" level=info msg="sanner return error: read unix @->/run/vc/vm/1f86a80671cedb53b6fc7b9846cf44991743b18d029d9b570233ce32d76a2e9a/qmp.sock: use of closed network connection" arch=amd64 command=create container=1f86a80671cedb53b6fc7b9846cf44991743b18d029d9b570233ce32d76a2e9a name=kata-runtime pid=13949 source=virtcontainers subsystem=qmp
Mar 30 05:28:03 ip-172-31-3-100 kata-runtime[13949]: time="2020-03-30T05:28:03.391992468Z" level=info msg="VM started" arch=amd64 command=create container=1f86a80671cedb53b6fc7b9846cf44991743b18d029d9b570233ce32d76a2e9a name=kata-runtimepid=13949 sandbox=1f86a80671cedb53b6fc7b9846cf44991743b18d029d9b570233ce32d76a2e9a source=virtcontainers subsystem=sandbox
Mar 30 05:28:03 ip-172-31-3-100 kata-runtime[13949]: time="2020-03-30T05:28:03.392194331Z" level=debug msg="Starting regular Kata proxy rather than built-in" arch=amd64 command=create container=1f86a80671cedb53b6fc7b9846cf44991743b18d029d9b570233ce32d76a2e9a name=kata-runtime pid=13949 sandbox=1f86a80671cedb53b6fc7b9846cf44991743b18d029d9b570233ce32d76a2e9a source=virtcontainers subsystem=kata_agent
Mar 30 05:28:03 ip-172-31-3-100 kata-runtime[13949]: time="2020-03-30T05:28:03.392733491Z" level=info msg="proxy started" arch=amd64 command=create container=1f86a80671cedb53b6fc7b9846cf44991743b18d029d9b570233ce32d76a2e9a name=kata-runtime pid=13949 proxy-pid=14006 proxy-url="unix:///run/vc/sbs/1f86a80671cedb53b6fc7b9846cf44991743b18d029d9b570233ce32d76a2e9a/proxy.sock" sandbox=1f86a80671cedb53b6fc7b9846cf44991743b18d029d9b570233ce32d76a2e9a source=virtcontainers subsystem=kata_agent
Mar 30 05:28:03 ip-172-31-3-100 kata-runtime[13949]: time="2020-03-30T05:28:03.392895805Z" level=info msg="New client" arch=amd64 command=create container=1f86a80671cedb53b6fc7b9846cf44991743b18d029d9b570233ce32d76a2e9a name=kata-runtimepid=13949 proxy=14006 source=virtcontainers subsystem=kata_agent url="unix:///run/vc/sbs/1f86a80671cedb53b6fc7b9846cf44991743b18d029d9b570233ce32d76a2e9a/proxy.sock"
Mar 30 05:28:03 ip-172-31-3-100 kata-runtime[13949]: time="2020-03-30T05:28:03.3981181Z" level=debug msg="sending request" arch=amd64 command=create container=1f86a80671cedb53b6fc7b9846cf44991743b18d029d9b570233ce32d76a2e9a name=grpc.CheckRequest pid=13949 req= source=virtcontainers subsystem=kata_agent
Mar 30 05:28:03 ip-172-31-3-100 kata-runtime[13949]: time="2020-03-30T05:28:03.969550298Z" level=info msg="New client" arch=amd64 command=create container=1f86a80671cedb53b6fc7b9846cf44991743b18d029d9b570233ce32d76a2e9a name=kata-runtimepid=13949 proxy=14006 source=virtcontainers subsystem=kata_agent url="unix:///run/vc/sbs/1f86a80671cedb53b6fc7b9846cf44991743b18d029d9b570233ce32d76a2e9a/proxy.sock"
Mar 30 05:28:03 ip-172-31-3-100 kata-runtime[13949]: time="2020-03-30T05:28:03.970162336Z" level=debug msg="sending request" arch=amd64 command=create container=1f86a80671cedb53b6fc7b9846cf44991743b18d029d9b570233ce32d76a2e9a name=grpc.UpdateInterfaceRequest pid=13949 req="interface:<device:\"eth0\" name:\"eth0\" IPAddresses:<address:\"172.17.0.2\" mask:\"16\" > mtu:1500 hwAddr:\"02:42:ac:11:00:02\" > " source=virtcontainers subsystem=kata_agent
Mar 30 05:28:03 ip-172-31-3-100 kata-runtime[13949]: time="2020-03-30T05:28:03.972103184Z" level=info msg="New client" arch=amd64 command=create container=1f86a80671cedb53b6fc7b9846cf44991743b18d029d9b570233ce32d76a2e9a name=kata-runtimepid=13949 proxy=14006 source=virtcontainers subsystem=kata_agent url="unix:///run/vc/sbs/1f86a80671cedb53b6fc7b9846cf44991743b18d029d9b570233ce32d76a2e9a/proxy.sock"
Mar 30 05:28:03 ip-172-31-3-100 kata-runtime[13949]: time="2020-03-30T05:28:03.972621231Z" level=debug msg="sending request" arch=amd64 command=create container=1f86a80671cedb53b6fc7b9846cf44991743b18d029d9b570233ce32d76a2e9a name=grpc.UpdateRoutesRequest pid=13949 req="routes:<Routes:<gateway:\"172.17.0.1\" device:\"eth0\" > > " source=virtcontainers subsystem=kata_agent
Mar 30 05:28:03 ip-172-31-3-100 kata-runtime[13949]: time="2020-03-30T05:28:03.973746368Z" level=info msg="New client" arch=amd64 command=create container=1f86a80671cedb53b6fc7b9846cf44991743b18d029d9b570233ce32d76a2e9a name=kata-runtimepid=13949 proxy=14006 source=virtcontainers subsystem=kata_agent url="unix:///run/vc/sbs/1f86a80671cedb53b6fc7b9846cf44991743b18d029d9b570233ce32d76a2e9a/proxy.sock"
Mar 30 05:28:03 ip-172-31-3-100 kata-runtime[13949]: time="2020-03-30T05:28:03.97432828Z" level=debug msg="sending request" arch=amd64 command=create container=1f86a80671cedb53b6fc7b9846cf44991743b18d029d9b570233ce32d76a2e9a name=grpc.CreateSandboxRequest pid=13949 req="hostname:\"1f86a80671ce\" dns:\"# This file is managed by man:systemd-resolved(8). Do not edit.\" dns:\"#\" dns:\"# This is a dynamic resolv.conf file for connecting local clients directly to\" dns:\"# all known uplink DNS servers. This file lists all configured search domains.\" dns:\"#\" dns:\"# Third party programs must not access this file directly, but only through the\" dns:\"# symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a different way,\" dns:\"# replace this symlink by a static file or a different symlink.\" dns:\"#\" dns:\"# See man:systemd-resolved.service(8) for details about the supported modes of\" dns:\"# operation for /etc/resolv.conf.\" dns:\"\" dns:\"nameserver 172.31.0.2\" dns:\"search ec2.internal\" dns:\"\" storages:<driver:\"9p\" source:\"kataShared\" fstype:\"9p\" options:\"trans=virtio,version=9p2000.L,cache=mmap\" options:\"nodev\" options:\"msize=8192\" mount_point:\"/run/kata-containers/shared/containers/\" > storages:<driver:\"ephemeral\" source:\"shm\" fstype:\"tmpfs\" options:\"noexec\" options:\"nosuid\" options:\"nodev\" options:\"mode=1777\" options:\"size=67108864\" mount_point:\"/run/kata-containers/sandbox/shm\" > sandbox_id:\"1f86a80671cedb53b6fc7b9846cf44991743b18d029d9b570233ce32d76a2e9a\" " source=virtcontainers subsystem=kata_agent
Mar 30 05:28:03 ip-172-31-3-100 kata-runtime[13949]: time="2020-03-30T05:28:03.979541832Z" level=info msg="Agent started in the sandbox" arch=amd64 command=create container=1f86a80671cedb53b6fc7b9846cf44991743b18d029d9b570233ce32d76a2e9aname=kata-runtime pid=13949 sandbox=1f86a80671cedb53b6fc7b9846cf44991743b18d029d9b570233ce32d76a2e9a source=virtcontainers subsystem=sandbox
Mar 30 05:28:03 ip-172-31-3-100 kata-runtime[13949]: time="2020-03-30T05:28:03.979674863Z" level=info msg="New client" arch=amd64 command=create container=1f86a80671cedb53b6fc7b9846cf44991743b18d029d9b570233ce32d76a2e9a name=kata-runtimepid=13949 proxy=14006 source=virtcontainers subsystem=kata_agent url="unix:///run/vc/sbs/1f86a80671cedb53b6fc7b9846cf44991743b18d029d9b570233ce32d76a2e9a/proxy.sock"
Mar 30 05:28:03 ip-172-31-3-100 kata-runtime[13949]: time="2020-03-30T05:28:03.980991054Z" level=debug msg="sending request" arch=amd64 command=create container=1f86a80671cedb53b6fc7b9846cf44991743b18d029d9b570233ce32d76a2e9a name=grpc.GuestDetailsRequest pid=13949 req="mem_block_size:true mem_hotplug_probe:true " source=virtcontainers subsystem=kata_agent
Mar 30 05:28:03 ip-172-31-3-100 kata-runtime[13949]: time="2020-03-30T05:28:03.982323892Z" level=info msg="device details" arch=amd64 command=create container=1f86a80671cedb53b6fc7b9846cf44991743b18d029d9b570233ce32d76a2e9a device-major=0 device-minor=51 mount-point=/var/lib/docker/overlay2/a262356a7737b21b3438e7968cdd0a29d0e65b73e6b85f8d3017f4f83b92ba47/merged name=kata-runtime pid=13949 sandbox=1f86a80671cedb53b6fc7b9846cf44991743b18d029d9b570233ce32d76a2e9a source=virtcontainers subsystem=container
Mar 30 05:28:03 ip-172-31-3-100 kata-runtime[13949]: time="2020-03-30T05:28:03.982523979Z" level=info msg="Attaching virtio-scsi device" VirtPath= arch=amd64 command=create container=1f86a80671cedb53b6fc7b9846cf44991743b18d029d9b570233ce32d76a2e9a device=/dev/vda name=kata-runtime pid=13949 source=virtcontainers subsystem=device
Mar 30 05:28:03 ip-172-31-3-100 kata-runtime[13949]: time="2020-03-30T05:28:03.982991793Z" level=info msg="{\"QMP\": {\"version\": {\"qemu\": {\"micro\": 1, \"minor\": 1, \"major\": 4}, \"package\": \"\"}, \"capabilities\": [\"oob\"]}}" arch=amd64 command=create container=1f86a80671cedb53b6fc7b9846cf44991743b18d029d9b570233ce32d76a2e9a name=kata-runtime pid=13949 source=virtcontainers subsystem=qmp
Mar 30 05:28:03 ip-172-31-3-100 kata-runtime[13949]: time="2020-03-30T05:28:03.983247977Z" level=info msg="{\"execute\":\"qmp_capabilities\"}" arch=amd64 command=create container=1f86a80671cedb53b6fc7b9846cf44991743b18d029d9b570233ce32d76a2e9a name=kata-runtime pid=13949 source=virtcontainers subsystem=qmp
Mar 30 05:28:03 ip-172-31-3-100 kata-runtime[13949]: time="2020-03-30T05:28:03.983774996Z" level=info msg="{\"return\": {}}" arch=amd64 command=create container=1f86a80671cedb53b6fc7b9846cf44991743b18d029d9b570233ce32d76a2e9a name=kata-runtime pid=13949 source=virtcontainers subsystem=qmp
Mar 30 05:28:03 ip-172-31-3-100 kata-runtime[13949]: time="2020-03-30T05:28:03.983965106Z" level=info msg="{\"arguments\":{\"driver\":\"raw\",\"file\":{\"driver\":\"file\",\"filename\":\"/dev/vda\"},\"node-name\":\"drive-dfa9be07c5f87fcc\"},\"execute\":\"blockdev-add\"}" arch=amd64 command=create container=1f86a80671cedb53b6fc7b9846cf44991743b18d029d9b570233ce32d76a2e9a name=kata-runtime pid=13949 source=virtcontainers subsystem=qmp
Mar 30 05:28:03 ip-172-31-3-100 kata-runtime[13949]: time="2020-03-30T05:28:03.986097057Z" level=info msg="{\"error\": {\"class\": \"GenericError\", \"desc\": \"Could not open '/dev/vda': No such file or directory\"}}" arch=amd64 command=create container=1f86a80671cedb53b6fc7b9846cf44991743b18d029d9b570233ce32d76a2e9a name=kata-runtime pid=13949 source=virtcontainers subsystem=qmp
Mar 30 05:28:03 ip-172-31-3-100 kata-runtime[13949]: time="2020-03-30T05:28:03.986281526Z" level=warning msg="no such file or directory: /run/kata-containers/shared/sandboxes/1f86a80671cedb53b6fc7b9846cf44991743b18d029d9b570233ce32d76a2e9a/1f86a80671cedb53b6fc7b9846cf44991743b18d029d9b570233ce32d76a2e9a/rootfs"
Mar 30 05:28:03 ip-172-31-3-100 kata-runtime[13949]: time="2020-03-30T05:28:03.986364368Z" level=info msg="Stopping sandbox in the VM" arch=amd64 command=create container=1f86a80671cedb53b6fc7b9846cf44991743b18d029d9b570233ce32d76a2e9a name=kata-runtime pid=13949 sandbox=1f86a80671cedb53b6fc7b9846cf44991743b18d029d9b570233ce32d76a2e9a source=virtcontainers subsystem=sandbox
Mar 30 05:28:03 ip-172-31-3-100 kata-runtime[13949]: time="2020-03-30T05:28:03.986468404Z" level=info msg="New client" arch=amd64 command=create container=1f86a80671cedb53b6fc7b9846cf44991743b18d029d9b570233ce32d76a2e9a name=kata-runtimepid=13949 proxy=14006 source=virtcontainers subsystem=kata_agent url="unix:///run/vc/sbs/1f86a80671cedb53b6fc7b9846cf44991743b18d029d9b570233ce32d76a2e9a/proxy.sock"
Mar 30 05:28:03 ip-172-31-3-100 kata-runtime[13949]: time="2020-03-30T05:28:03.987001772Z" level=debug msg="sending request" arch=amd64 command=create container=1f86a80671cedb53b6fc7b9846cf44991743b18d029d9b570233ce32d76a2e9a name=grpc.DestroySandboxRequest pid=13949 req= source=virtcontainers subsystem=kata_agent
Mar 30 05:28:04 ip-172-31-3-100 kata-runtime[13949]: time="2020-03-30T05:28:04.000172783Z" level=info msg="Stopping VM" arch=amd64 command=create container=1f86a80671cedb53b6fc7b9846cf44991743b18d029d9b570233ce32d76a2e9a name=kata-runtime pid=13949 sandbox=1f86a80671cedb53b6fc7b9846cf44991743b18d029d9b570233ce32d76a2e9a source=virtcontainers subsystem=sandbox
Mar 30 05:28:04 ip-172-31-3-100 kata-runtime[13949]: time="2020-03-30T05:28:04.000241752Z" level=info msg="Stopping Sandbox" arch=amd64 command=create container=1f86a80671cedb53b6fc7b9846cf44991743b18d029d9b570233ce32d76a2e9a name=kata-runtime pid=13949 source=virtcontainers subsystem=qemu
Mar 30 05:28:04 ip-172-31-3-100 kata-runtime[13949]: time="2020-03-30T05:28:04.000340536Z" level=info msg="{\"execute\":\"quit\"}" arch=amd64 command=create container=1f86a80671cedb53b6fc7b9846cf44991743b18d029d9b570233ce32d76a2e9a name=kata-runtime pid=13949 source=virtcontainers subsystem=qmp
Mar 30 05:28:04 ip-172-31-3-100 kata-runtime[13949]: time="2020-03-30T05:28:04.000738688Z" level=info msg="{\"return\": {}}" arch=amd64 command=create container=1f86a80671cedb53b6fc7b9846cf44991743b18d029d9b570233ce32d76a2e9a name=kata-runtime pid=13949 source=virtcontainers subsystem=qmp
Mar 30 05:28:04 ip-172-31-3-100 kata-runtime[13949]: time="2020-03-30T05:28:04.00082401Z" level=info msg="{\"timestamp\": {\"seconds\": 1585546084, \"microseconds\": 731}, \"event\": \"SHUTDOWN\", \"data\": {\"guest\": false, \"reason\":\"host-qmp-quit\"}}" arch=amd64 command=create container=1f86a80671cedb53b6fc7b9846cf44991743b18d029d9b570233ce32d76a2e9a name=kata-runtime pid=13949 source=virtcontainers subsystem=qmp
Mar 30 05:28:04 ip-172-31-3-100 kata-runtime[13949]: time="2020-03-30T05:28:04.000965468Z" level=info msg="cleanup vm path" arch=amd64 command=create container=1f86a80671cedb53b6fc7b9846cf44991743b18d029d9b570233ce32d76a2e9a dir=/run/vc/vm/1f86a80671cedb53b6fc7b9846cf44991743b18d029d9b570233ce32d76a2e9a link=/run/vc/vm/1f86a80671cedb53b6fc7b9846cf44991743b18d029d9b570233ce32d76a2e9a name=kata-runtime pid=13949 source=virtcontainers subsystem=qemu
Mar 30 05:28:04 ip-172-31-3-100 kata-runtime[13949]: time="2020-03-30T05:28:04.001148516Z" level=info msg="Detaching endpoint" arch=amd64 command=create container=1f86a80671cedb53b6fc7b9846cf44991743b18d029d9b570233ce32d76a2e9a endpoint-type=virtual name=kata-runtime pid=13949 source=virtcontainers subsystem=network
Mar 30 05:28:04 ip-172-31-3-100 kata-runtime[13949]: time="2020-03-30T05:28:04.03866273Z" level=info msg="sanner return error: <nil>" arch=amd64 command=create container=1f86a80671cedb53b6fc7b9846cf44991743b18d029d9b570233ce32d76a2e9a name=kata-runtime pid=13949 source=virtcontainers subsystem=qmp
Mar 30 05:28:04 ip-172-31-3-100 kata-runtime[13949]: time="2020-03-30T05:28:04.039154814Z" level=debug msg="Network removed" arch=amd64 command=create container=1f86a80671cedb53b6fc7b9846cf44991743b18d029d9b570233ce32d76a2e9a name=kata-runtime pid=13949 source=virtcontainers subsystem=network
Mar 30 05:28:04 ip-172-31-3-100 kata-runtime[13949]: time="2020-03-30T05:28:04.039256091Z" level=info msg="Network namespace \"/var/run/netns/cnitest-4d8cae33-507b-9fdb-fb4b-ff0b7a4b619b\" deleted" arch=amd64 command=create container=1f86a80671cedb53b6fc7b9846cf44991743b18d029d9b570233ce32d76a2e9a name=kata-runtime pid=13949 source=virtcontainers subsystem=network
Mar 30 05:28:04 ip-172-31-3-100 kernel: [36516.379893] docker0: port 1(veth728c472) entered disabled state
Mar 30 05:28:04 ip-172-31-3-100 kata-runtime[13949]: time="2020-03-30T05:28:04.062628415Z" level=debug msg="Deleting sandbox cgroup" arch=amd64 command=create container=1f86a80671cedb53b6fc7b9846cf44991743b18d029d9b570233ce32d76a2e9a name=kata-runtime pid=13949 sandbox=1f86a80671cedb53b6fc7b9846cf44991743b18d029d9b570233ce32d76a2e9a source=virtcontainers subsystem=sandbox
Mar 30 05:28:04 ip-172-31-3-100 kata-runtime[13949]: time="2020-03-30T05:28:04.062729648Z" level=warning msg="sandbox cgroups path is empty" arch=amd64 command=create container=1f86a80671cedb53b6fc7b9846cf44991743b18d029d9b570233ce32d76a2e9a name=kata-runtime pid=13949 sandbox=1f86a80671cedb53b6fc7b9846cf44991743b18d029d9b570233ce32d76a2e9a source=virtcontainers subsystem=sandbox
Mar 30 05:28:04 ip-172-31-3-100 kata-runtime[13949]: time="2020-03-30T05:28:04.062803052Z" level=info msg="cleanup agent" arch=amd64 command=create container=1f86a80671cedb53b6fc7b9846cf44991743b18d029d9b570233ce32d76a2e9a name=kata-runtime path=/run/kata-containers/shared/sandboxes/1f86a80671cedb53b6fc7b9846cf44991743b18d029d9b570233ce32d76a2e9a pid=13949 source=virtcontainers subsystem=kata_agent
Mar 30 05:28:04 ip-172-31-3-100 kata-runtime[13949]: time="2020-03-30T05:28:04.062917539Z" level=warning msg="failed to cleanup netns" arch=amd64 command=create container=1f86a80671cedb53b6fc7b9846cf44991743b18d029d9b570233ce32d76a2e9a error="failed to get netns /var/run/netns/cnitest-4d8cae33-507b-9fdb-fb4b-ff0b7a4b619b: failed to Statfs \"/var/run/netns/cnitest-4d8cae33-507b-9fdb-fb4b-ff0b7a4b619b\": no such file or directory" name=kata-runtime path=/var/run/netns/cnitest-4d8cae33-507b-9fdb-fb4b-ff0b7a4b619b pid=13949 source=katautils
Mar 30 05:28:04 ip-172-31-3-100 kata-runtime[13949]: time="2020-03-30T05:28:04.06299249Z" level=error msg="QMP command failed: Could not open '/dev/vda': No such file or directory" arch=amd64 command=create container=1f86a80671cedb53b6fc7b9846cf44991743b18d029d9b570233ce32d76a2e9a name=kata-runtime pid=13949 source=runtime
Mar 30 05:28:04 ip-172-31-3-100 kernel: [36516.430592] veth1a1fed0: renamed from eth0
Mar 30 05:28:04 ip-172-31-3-100 kernel: [36516.490506] docker0: port 1(veth728c472) entered disabled state
Mar 30 05:28:04 ip-172-31-3-100 kernel: [36516.493546] device veth728c472 left promiscuous mode
Mar 30 05:28:04 ip-172-31-3-100 kernel: [36516.493553] docker0: port 1(veth728c472) entered disabled state
coderplay
All 24 comments
@coderplay thanks for raising this, @dong-liuliu you're the expert on SPDK, any thoughts?
devimc
on 30 Mar 2020
thanks for the notice, @devimc
let me try to reproduce the steps coderplay had.
dong-liuliu
on 30 Mar 2020
hi, @coderplay
I just tried on newly release kata-container, no error reported on it.
So I checked your log, seems you are using kata with version "1.11.0-alpha0", not "1.11.0-alpha1".
Patchset for Vhost-user-blk device enablement starts released in 1.11.0-alpha1, so you need to update your kata containers.
dong-liuliu
on 30 Mar 2020
Another key point I have missed to update in host-setup-for-vhost-user-devices is that:
In kata config file configuration.toml， together with parameters vhost_user_store_path = /var/run/kata-containers/vhost-user, you need to add an explicit delaration enable_vhost_user_store = true. Also currently, hugepage are required by vhost-user target, so it should be enabled by adding enable_hugepages = true.
let me adding this revision to page Setup to run SPDK vhost-user devices with Kata Containers and Docker now.
dong-liuliu
on 30 Mar 2020
@dong-liuliu thanks for the tips. I installed the alpha0 version following this guide https://github.com/kata-containers/documentation/blob/master/install/ubuntu-installation-guide.md
I didn't see alpha1 version on the repository. Could you please update? http://download.opensuse.org/repositories/home:/katacontainers:/releases:/x86_64:/master/xUbuntu_18.04/
coderplay
on 30 Mar 2020
hi @coderplay,
From kata release, alpha1 was released 12 days ago.
I'm not very clear about how to get it from package repo, too....
Since previously I already set a Kata dev env, so I pulled the new version from github source repo, and compile them manualy.
But I don't think you need to compile them manualy for evaluation.
dong-liuliu
on 31 Mar 2020
Hi @dong-liuliu , thanks for your help. I've upgraded to 1.11.0-alpha1 through this repository: http://download.opensuse.org/repositories/home:/katacontainers:/releases:/x86_64:/alpha/xUbuntu_18.04/
but the failure was still there. I've added all configuration items you suggested.
$ kata-runtime kata-env
[Meta]
Version = "1.0.24"
[Runtime]
Debug = true
Trace = false
DisableGuestSeccomp = true
DisableNewNetNs = false
SandboxCgroupOnly = false
Path = "/usr/bin/kata-runtime"
[Runtime.Version]
OCI = "1.0.1-dev"
[Runtime.Version.Version]
Semver = "1.11.0-alpha1"
Major = 1
Minor = 11
Patch = 0
Commit = ""
[Runtime.Config]
Path = "/etc/kata-containers/configuration.toml"
[Hypervisor]
MachineType = "pc"
Version = "QEMU emulator version 4.1.1\nCopyright (c) 2003-2019 Fabrice Bellard and the QEMU Project developers"
Path = "/usr/bin/qemu-vanilla-system-x86_64"
BlockDeviceDriver = "virtio-scsi"
EntropySource = "/dev/urandom"
SharedFS = "virtio-9p"
VirtioFSDaemon = "/usr/bin/virtiofsd"
Msize9p = 8192
MemorySlots = 10
PCIeRootPort = 0
HotplugVFIOOnRootBus = false
Debug = true
UseVSock = false
[Image]
Path = "/usr/share/kata-containers/kata-containers-image_clearlinux_1.11.0-alpha1_agent_1f432c6896.img"
[Kernel]
Path = "/usr/share/kata-containers/vmlinuz-5.4.15.69-42.container"
Parameters = "systemd.unit=kata-containers.target systemd.mask=systemd-networkd.service systemd.mask=systemd-networkd.socket agent.log=debugagent.log=debug initcall_debug"
[Initrd]
Path = ""
[Proxy]
Type = "kataProxy"
Path = "/usr/libexec/kata-containers/kata-proxy"
Debug = true
[Proxy.Version]
Semver = "1.11.0-alpha1-e7d2214"
Major = 1
Minor = 11
Patch = 0
Commit = "e7d2214"
[Shim]
Type = "kataShim"
Path = "/usr/libexec/kata-containers/kata-shim"
Debug = true
[Shim.Version]
Semver = "1.11.0-alpha1-6a828a4"
Major = 1
Minor = 11
Patch = 0
Commit = "6a828a4"
[Agent]
Type = "kata"
Debug = true
Trace = false
TraceMode = ""
TraceType = ""
[Host]
Kernel = "4.15.0-1057-aws"
Architecture = "amd64"
VMContainerCapable = true
SupportVSocks = true
[Host.Distro]
Name = "Ubuntu"
Version = "18.04"
[Host.CPU]
Vendor = "GenuineIntel"
Model = "Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz"
[Netmon]
Path = "/usr/libexec/kata-containers/kata-netmon"
Debug = true
Enable = false
[Netmon.Version]
Semver = "1.11.0-alpha1"
Major = 1
Minor = 11
Patch = 0
Commit = "<<unknown>>"
I'm using a fedora30, let me verify it on a Ubuntu 1604.
dong-liuliu
on 31 Mar 2020
@coderplay can you also attach your kata configure file?
As I know, without enable_vhost_user_store = false, the type of error you met will occur.
dong-liuliu
on 31 Mar 2020
content of /etc/kata-containers/configuration.toml
# Copyright (c) 2017-2019 Intel Corporation
#
# SPDX-License-Identifier: Apache-2.0
#
# XXX: WARNING: this file is auto-generated.
# XXX:
# XXX: Source file: "cli/config/configuration-qemu.toml.in"
# XXX: Project:
# XXX: Name: Kata Containers
# XXX: Type: kata
[hypervisor.qemu]
path = "/usr/bin/qemu-vanilla-system-x86_64"
kernel = "/usr/share/kata-containers/vmlinuz.container"
image = "/usr/share/kata-containers/kata-containers.img"
machine_type = "pc"
# Optional space-separated list of options to pass to the guest kernel.
# For example, use `kernel_params = "vsyscall=emulate"` if you are having
# trouble running pre-2.15 glibc.
#
# WARNING: - any parameter specified here will take priority over the default
# parameter value of the same name used to start the virtual machine.
# Do not set values here unless you understand the impact of doing so as you
# may stop the virtual machine from booting.
# To see the list of default parameters, enable hypervisor debug, create a
# container and look for 'default-kernel-parameters' log entries.
kernel_params = " agent.log=debug initcall_debug"
# Path to the firmware.
# If you want that qemu uses the default firmware leave this option empty
firmware = ""
# Machine accelerators
# comma-separated list of machine accelerators to pass to the hypervisor.
# For example, `machine_accelerators = "nosmm,nosmbus,nosata,nopit,static-prt,nofw"`
machine_accelerators=""
# Default number of vCPUs per SB/VM:
# unspecified or 0 --> will be set to 1
# < 0 --> will be set to the actual number of physical cores
# > 0 <= number of physical cores --> will be set to the specified number
# > number of physical cores --> will be set to the actual number of physical cores
default_vcpus = 1
# Default maximum number of vCPUs per SB/VM:
# unspecified or == 0 --> will be set to the actual number of physical cores or to the maximum number
# of vCPUs supported by KVM if that number is exceeded
# > 0 <= number of physical cores --> will be set to the specified number
# > number of physical cores --> will be set to the actual number of physical cores or to the maximum number
# of vCPUs supported by KVM if that number is exceeded
# WARNING: Depending of the architecture, the maximum number of vCPUs supported by KVM is used when
# the actual number of physical cores is greater than it.
# WARNING: Be aware that this value impacts the virtual machine's memory footprint and CPU
# the hotplug functionality. For example, `default_maxvcpus = 240` specifies that until 240 vCPUs
# can be added to a SB/VM, but the memory footprint will be big. Another example, with
# `default_maxvcpus = 8` the memory footprint will be small, but 8 will be the maximum number of
# vCPUs supported by the SB/VM. In general, we recommend that you do not edit this variable,
# unless you know what are you doing.
default_maxvcpus = 0
# Bridges can be used to hot plug devices.
# Limitations:
# * Currently only pci bridges are supported
# * Until 30 devices per bridge can be hot plugged.
# * Until 5 PCI bridges can be cold plugged per VM.
# This limitation could be a bug in qemu or in the kernel
# Default number of bridges per SB/VM:
# unspecified or 0 --> will be set to 1
# > 1 <= 5 --> will be set to the specified number
# > 5 --> will be set to 5
default_bridges = 1
# Default memory size in MiB for SB/VM.
# If unspecified then it will be set 2048 MiB.
default_memory = 2048
#
# Default memory slots per SB/VM.
# If unspecified then it will be set 10.
# This is will determine the times that memory will be hotadded to sandbox/VM.
#memory_slots = 10
# The size in MiB will be plused to max memory of hypervisor.
# It is the memory address space for the NVDIMM devie.
# If set block storage driver (block_device_driver) to "nvdimm",
# should set memory_offset to the size of block device.
# Default 0
#memory_offset = 0
# Specifies virtio-mem will be enabled or not.
# Please note that this option should be used with the command
# "echo 1 > /proc/sys/vm/overcommit_memory".
# Default false
#enable_virtio_mem = true
# Disable block device from being used for a container's rootfs.
# In case of a storage driver like devicemapper where a container's
# root file system is backed by a block device, the block device is passed
# directly to the hypervisor for performance reasons.
# This flag prevents the block device from being passed to the hypervisor,
# 9pfs is used instead to pass the rootfs.
disable_block_device_use = false
# Shared file system type:
# - virtio-9p (default)
# - virtio-fs
shared_fs = "virtio-9p"
# Path to vhost-user-fs daemon.
virtio_fs_daemon = "/usr/bin/virtiofsd"
# Default size of DAX cache in MiB
virtio_fs_cache_size = 1024
# Extra args for virtiofsd daemon
#
# Format example:
# ["-o", "arg1=xxx,arg2", "-o", "hello world", "--arg3=yyy"]
#
# see `virtiofsd -h` for possible options.
virtio_fs_extra_args = []
# Cache mode:
#
# - none
# Metadata, data, and pathname lookup are not cached in guest. They are
# always fetched from host and any changes are immediately pushed to host.
#
# - auto
# Metadata and pathname lookup cache expires after a configured amount of
# time (default is 1 second). Data is cached while the file is open (close
# to open consistency).
#
# - always
# Metadata, data, and pathname lookup are cached in guest and never expire.
virtio_fs_cache = "always"
# Block storage driver to be used for the hypervisor in case the container
# rootfs is backed by a block device. This is virtio-scsi, virtio-blk
# or nvdimm.
block_device_driver = "virtio-scsi"
# Specifies cache-related options will be set to block devices or not.
# Default false
#block_device_cache_set = true
# Specifies cache-related options for block devices.
# Denotes whether use of O_DIRECT (bypass the host page cache) is enabled.
# Default false
#block_device_cache_direct = true
# Specifies cache-related options for block devices.
# Denotes whether flush requests for the device are ignored.
# Default false
#block_device_cache_noflush = true
# Enable iothreads (data-plane) to be used. This causes IO to be
# handled in a separate IO thread. This is currently only implemented
# for SCSI.
#
enable_iothreads = false
# Enable pre allocation of VM RAM, default false
# Enabling this will result in lower container density
# as all of the memory will be allocated and locked
# This is useful when you want to reserve all the memory
# upfront or in the cases where you want memory latencies
# to be very predictable
# Default false
#enable_mem_prealloc = true
# Enable huge pages for VM RAM, default false
# Enabling this will result in the VM memory
# being allocated using huge pages.
# This is useful when you want to use vhost-user network
# stacks within the container. This will automatically
# result in memory pre allocation
#enable_hugepages = true
# Enable file based guest memory support. The default is an empty string which
# will disable this feature. In the case of virtio-fs, this is enabled
# automatically and '/dev/shm' is used as the backing folder.
# This option will be ignored if VM templating is enabled.
#file_mem_backend = ""
# Enable swap of vm memory. Default false.
# The behaviour is undefined if mem_prealloc is also set to true
#enable_swap = true
# This option changes the default hypervisor and kernel parameters
# to enable debug output where available. This extra output is added
# to the proxy logs, but only when proxy debug is also enabled.
#
# Default false
enable_debug = true
# Disable the customizations done in the runtime when it detects
# that it is running on top a VMM. This will result in the runtime
# behaving as it would when running on bare metal.
#
#disable_nesting_checks = true
# This is the msize used for 9p shares. It is the number of bytes
# used for 9p packet payload.
#msize_9p = 8192
# If true and vsocks are supported, use vsocks to communicate directly
# with the agent and no proxy is started, otherwise use unix
# sockets and start a proxy to communicate with the agent.
# Default false
#use_vsock = true
# If false and nvdimm is supported, use nvdimm device to plug guest image.
# Otherwise virtio-block device is used.
# Default is false
#disable_image_nvdimm = true
# VFIO devices are hotplugged on a bridge by default.
# Enable hotplugging on root bus. This may be required for devices with
# a large PCI bar, as this is a current limitation with hotplugging on
# a bridge. This value is valid for "pc" machine type.
# Default false
#hotplug_vfio_on_root_bus = true
# Before hot plugging a PCIe device, you need to add a pcie_root_port device.
# Use this parameter when using some large PCI bar devices, such as Nvidia GPU
# The value means the number of pcie_root_port
# This value is valid when hotplug_vfio_on_root_bus is true and machine_type is "q35"
# Default 0
#pcie_root_port = 2
# If vhost-net backend for virtio-net is not desired, set to true. Default is false, which trades off
# security (vhost-net runs ring0) for network I/O performance.
#disable_vhost_net = true
#
# Default entropy source.
# The path to a host source of entropy (including a real hardware RNG)
# /dev/urandom and /dev/random are two main options.
# Be aware that /dev/random is a blocking source of entropy. If the host
# runs out of entropy, the VMs boot time will increase leading to get startup
# timeouts.
# The source of entropy /dev/urandom is non-blocking and provides a
# generally acceptable source of entropy. It should work well for pretty much
# all practical purposes.
#entropy_source= "/dev/urandom"
# Path to OCI hook binaries in the *guest rootfs*.
# This does not affect host-side hooks which must instead be added to
# the OCI spec passed to the runtime.
#
# You can create a rootfs with hooks by customizing the osbuilder scripts:
# https://github.com/kata-containers/osbuilder
#
# Hooks must be stored in a subdirectory of guest_hook_path according to their
# hook type, i.e. "guest_hook_path/{prestart,postart,poststop}".
# The agent will scan these directories for executable files and add them, in
# lexicographical order, to the lifecycle of the guest container.
# Hooks are executed in the runtime namespace of the guest. See the official documentation:
# https://github.com/opencontainers/runtime-spec/blob/v1.0.1/config.md#posix-platform-hooks
# Warnings will be logged if any error is encountered will scanning for hooks,
# but it will not abort container execution.
#guest_hook_path = "/usr/share/oci/hooks"
[factory]
# VM templating support. Once enabled, new VMs are created from template
# using vm cloning. They will share the same initial kernel, initramfs and
# agent memory by mapping it readonly. It helps speeding up new container
# creation and saves a lot of memory if there are many kata containers running
# on the same host.
#
# When disabled, new VMs are created from scratch.
#
# Note: Requires "initrd=" to be set ("image=" is not supported).
#
# Default false
#enable_template = true
# Specifies the path of template.
#
# Default "/run/vc/vm/template"
#template_path = "/run/vc/vm/template"
# The number of caches of VMCache:
# unspecified or == 0 --> VMCache is disabled
# > 0 --> will be set to the specified number
#
# VMCache is a function that creates VMs as caches before using it.
# It helps speed up new container creation.
# The function consists of a server and some clients communicating
# through Unix socket. The protocol is gRPC in protocols/cache/cache.proto.
# The VMCache server will create some VMs and cache them by factory cache.
# It will convert the VM to gRPC format and transport it when gets
# requestion from clients.
# Factory grpccache is the VMCache client. It will request gRPC format
# VM and convert it back to a VM. If VMCache function is enabled,
# kata-runtime will request VM from factory grpccache when it creates
# a new sandbox.
#
# Default 0
#vm_cache_number = 0
# Specify the address of the Unix socket that is used by VMCache.
#
# Default /var/run/kata-containers/cache.sock
#vm_cache_endpoint = "/var/run/kata-containers/cache.sock"
[proxy.kata]
path = "/usr/libexec/kata-containers/kata-proxy"
# If enabled, proxy messages will be sent to the system log
# (default: disabled)
enable_debug = true
[shim.kata]
path = "/usr/libexec/kata-containers/kata-shim"
# If enabled, shim messages will be sent to the system log
# (default: disabled)
enable_debug = true
# If enabled, the shim will create opentracing.io traces and spans.
# (See https://www.jaegertracing.io/docs/getting-started).
#
# Note: By default, the shim runs in a separate network namespace. Therefore,
# to allow it to send trace details to the Jaeger agent running on the host,
# it is necessary to set 'disable_new_netns=true' so that it runs in the host
# network namespace.
#
# (default: disabled)
#enable_tracing = true
[agent.kata]
# If enabled, make the agent display debug-level messages.
# (default: disabled)
enable_debug = true
# Enable agent tracing.
#
# If enabled, the default trace mode is "dynamic" and the
# default trace type is "isolated". The trace mode and type are set
# explicity with the `trace_type=` and `trace_mode=` options.
#
# Notes:
#
# - Tracing is ONLY enabled when `enable_tracing` is set: explicitly
# setting `trace_mode=` and/or `trace_type=` without setting `enable_tracing`
# will NOT activate agent tracing.
#
# - See https://github.com/kata-containers/agent/blob/master/TRACING.md for
# full details.
#
# (default: disabled)
#enable_tracing = true
#
#trace_mode = "dynamic"
#trace_type = "isolated"
# Comma separated list of kernel modules and their parameters.
# These modules will be loaded in the guest kernel using modprobe(8).
# The following example can be used to load two kernel modules with parameters
# - kernel_modules=["e1000e InterruptThrottleRate=3000,3000,3000 EEE=1", "i915 enable_ppgtt=0"]
# The first word is considered as the module name and the rest as its parameters.
# Container will not be started when:
# * A kernel module is specified and the modprobe command is not installed in the guest
# or it fails loading the module.
# * The module is not available in the guest or it doesn't met the guest kernel
# requirements, like architecture and version.
#
kernel_modules=[]
[netmon]
# If enabled, the network monitoring process gets started when the
# sandbox is created. This allows for the detection of some additional
# network being added to the existing network namespace, after the
# sandbox has been created.
# (default: disabled)
#enable_netmon = true
# Specify the path to the netmon binary.
path = "/usr/libexec/kata-containers/kata-netmon"
# If enabled, netmon messages will be sent to the system log
# (default: disabled)
enable_debug = true
[runtime]
# If enabled, the runtime will log additional debug messages to the
# system log
# (default: disabled)
enable_debug = true
#
# Internetworking model
# Determines how the VM should be connected to the
# the container network interface
# Options:
#
# - macvtap
# Used when the Container network interface can be bridged using
# macvtap.
#
# - none
# Used when customize network. Only creates a tap device. No veth pair.
#
# - tcfilter
# Uses tc filter rules to redirect traffic from the network interface
# provided by plugin to a tap interface connected to the VM.
#
internetworking_model="tcfilter"
# disable guest seccomp
# Determines whether container seccomp profiles are passed to the virtual
# machine and applied by the kata agent. If set to true, seccomp is not applied
# within the guest
# (default: true)
disable_guest_seccomp=true
# If enabled, the runtime will create opentracing.io traces and spans.
# (See https://www.jaegertracing.io/docs/getting-started).
# (default: disabled)
#enable_tracing = true
# If enabled, the runtime will not create a network namespace for shim and hypervisor processes.
# This option may have some potential impacts to your host. It should only be used when you know what you're doing.
# `disable_new_netns` conflicts with `enable_netmon`
# `disable_new_netns` conflicts with `internetworking_model=tcfilter` and `internetworking_model=macvtap`. It works only
# with `internetworking_model=none`. The tap device will be in the host network namespace and can connect to a bridge
# (like OVS) directly.
# If you are using docker, `disable_new_netns` only works with `docker run --net=none`
# (default: false)
#disable_new_netns = true
# if enabled, the runtime will add all the kata processes inside one dedicated cgroup.
# The container cgroups in the host are not created, just one single cgroup per sandbox.
# The runtime caller is free to restrict or collect cgroup stats of the overall Kata sandbox.
# The sandbox cgroup path is the parent cgroup of a container with the PodSandbox annotation.
# The sandbox cgroup is constrained if there is no container type annotation.
# See: https://godoc.org/github.com/kata-containers/runtime/virtcontainers#ContainerType
sandbox_cgroup_only=false
# Enabled experimental feature list, format: ["a", "b"].
# Experimental features are features not stable enough for production,
# they may break compatibility, and are prepared for a big version bump.
# Supported experimental features:
# (default: [])
experimental=[]
vhost_user_store_path="/var/run/kata-containers/vhost-user"
enable_vhost_user_store=true
enable_hugepages=true
I've tested with enable_vhost_user_store=false as well, still with the same failure. BTW, should I restart anything after a configuration change?
I did
sudo systemctl daemon-reload
sudo systemctl restart docker
In addition, content of /usr/share/defaults/kata-containers/configuration.toml
# Copyright (c) 2017-2019 Intel Corporation
#
# SPDX-License-Identifier: Apache-2.0
#
# XXX: WARNING: this file is auto-generated.
# XXX:
# XXX: Source file: "cli/config/configuration-qemu.toml.in"
# XXX: Project:
# XXX: Name: Kata Containers
# XXX: Type: kata
[hypervisor.qemu]
path = "/usr/bin/qemu-vanilla-system-x86_64"
kernel = "/usr/share/kata-containers/vmlinuz.container"
image = "/usr/share/kata-containers/kata-containers.img"
machine_type = "pc"
# Optional space-separated list of options to pass to the guest kernel.
# For example, use `kernel_params = "vsyscall=emulate"` if you are having
# trouble running pre-2.15 glibc.
#
# WARNING: - any parameter specified here will take priority over the default
# parameter value of the same name used to start the virtual machine.
# Do not set values here unless you understand the impact of doing so as you
# may stop the virtual machine from booting.
# To see the list of default parameters, enable hypervisor debug, create a
# container and look for 'default-kernel-parameters' log entries.
kernel_params = ""
# Path to the firmware.
# If you want that qemu uses the default firmware leave this option empty
firmware = ""
# Machine accelerators
# comma-separated list of machine accelerators to pass to the hypervisor.
# For example, `machine_accelerators = "nosmm,nosmbus,nosata,nopit,static-prt,nofw"`
machine_accelerators=""
# Default number of vCPUs per SB/VM:
# unspecified or 0 --> will be set to 1
# < 0 --> will be set to the actual number of physical cores
# > 0 <= number of physical cores --> will be set to the specified number
# > number of physical cores --> will be set to the actual number of physical cores
default_vcpus = 1
# Default maximum number of vCPUs per SB/VM:
# unspecified or == 0 --> will be set to the actual number of physical cores or to the maximum number
# of vCPUs supported by KVM if that number is exceeded
# > 0 <= number of physical cores --> will be set to the specified number
# > number of physical cores --> will be set to the actual number of physical cores or to the maximum number
# of vCPUs supported by KVM if that number is exceeded
# WARNING: Depending of the architecture, the maximum number of vCPUs supported by KVM is used when
# the actual number of physical cores is greater than it.
# WARNING: Be aware that this value impacts the virtual machine's memory footprint and CPU
# the hotplug functionality. For example, `default_maxvcpus = 240` specifies that until 240 vCPUs
# can be added to a SB/VM, but the memory footprint will be big. Another example, with
# `default_maxvcpus = 8` the memory footprint will be small, but 8 will be the maximum number of
# vCPUs supported by the SB/VM. In general, we recommend that you do not edit this variable,
# unless you know what are you doing.
default_maxvcpus = 0
# Bridges can be used to hot plug devices.
# Limitations:
# * Currently only pci bridges are supported
# * Until 30 devices per bridge can be hot plugged.
# * Until 5 PCI bridges can be cold plugged per VM.
# This limitation could be a bug in qemu or in the kernel
# Default number of bridges per SB/VM:
# unspecified or 0 --> will be set to 1
# > 1 <= 5 --> will be set to the specified number
# > 5 --> will be set to 5
default_bridges = 1
# Default memory size in MiB for SB/VM.
# If unspecified then it will be set 2048 MiB.
default_memory = 2048
#
# Default memory slots per SB/VM.
# If unspecified then it will be set 10.
# This is will determine the times that memory will be hotadded to sandbox/VM.
#memory_slots = 10
# The size in MiB will be plused to max memory of hypervisor.
# It is the memory address space for the NVDIMM devie.
# If set block storage driver (block_device_driver) to "nvdimm",
# should set memory_offset to the size of block device.
# Default 0
#memory_offset = 0
# Specifies virtio-mem will be enabled or not.
# Please note that this option should be used with the command
# "echo 1 > /proc/sys/vm/overcommit_memory".
# Default false
#enable_virtio_mem = true
# Disable block device from being used for a container's rootfs.
# In case of a storage driver like devicemapper where a container's
# root file system is backed by a block device, the block device is passed
# directly to the hypervisor for performance reasons.
# This flag prevents the block device from being passed to the hypervisor,
# 9pfs is used instead to pass the rootfs.
disable_block_device_use = false
# Shared file system type:
# - virtio-9p (default)
# - virtio-fs
shared_fs = "virtio-9p"
# Path to vhost-user-fs daemon.
virtio_fs_daemon = "/usr/bin/virtiofsd"
# Default size of DAX cache in MiB
virtio_fs_cache_size = 1024
# Extra args for virtiofsd daemon
#
# Format example:
# ["-o", "arg1=xxx,arg2", "-o", "hello world", "--arg3=yyy"]
#
# see `virtiofsd -h` for possible options.
virtio_fs_extra_args = []
# Cache mode:
#
# - none
# Metadata, data, and pathname lookup are not cached in guest. They are
# always fetched from host and any changes are immediately pushed to host.
#
# - auto
# Metadata and pathname lookup cache expires after a configured amount of
# time (default is 1 second). Data is cached while the file is open (close
# to open consistency).
#
# - always
# Metadata, data, and pathname lookup are cached in guest and never expire.
virtio_fs_cache = "always"
# Block storage driver to be used for the hypervisor in case the container
# rootfs is backed by a block device. This is virtio-scsi, virtio-blk
# or nvdimm.
block_device_driver = "virtio-scsi"
# Specifies cache-related options will be set to block devices or not.
# Default false
#block_device_cache_set = true
# Specifies cache-related options for block devices.
# Denotes whether use of O_DIRECT (bypass the host page cache) is enabled.
# Default false
#block_device_cache_direct = true
# Specifies cache-related options for block devices.
# Denotes whether flush requests for the device are ignored.
# Default false
#block_device_cache_noflush = true
# Enable iothreads (data-plane) to be used. This causes IO to be
# handled in a separate IO thread. This is currently only implemented
# for SCSI.
#
enable_iothreads = false
# Enable pre allocation of VM RAM, default false
# Enabling this will result in lower container density
# as all of the memory will be allocated and locked
# This is useful when you want to reserve all the memory
# upfront or in the cases where you want memory latencies
# to be very predictable
# Default false
#enable_mem_prealloc = true
# Enable huge pages for VM RAM, default false
# Enabling this will result in the VM memory
# being allocated using huge pages.
# This is useful when you want to use vhost-user network
# stacks within the container. This will automatically
# result in memory pre allocation
#enable_hugepages = true
# Enable vhost-user storage device, default false
# Enabling this will result in some Linux reserved block type
# major range 240-254 being chosen to represent vhost-user devices.
enable_vhost_user_store = false
# The base directory specifically used for vhost-user devices.
# Its sub-path "block" is used for block devices; "block/sockets" is
# where we expect vhost-user sockets to live; "block/devices" is where
# simulated block device nodes for vhost-user devices to live.
vhost_user_store_path = "/var/run/kata-containers/vhost-user"
# Enable file based guest memory support. The default is an empty string which
# will disable this feature. In the case of virtio-fs, this is enabled
# automatically and '/dev/shm' is used as the backing folder.
# This option will be ignored if VM templating is enabled.
#file_mem_backend = ""
# Enable swap of vm memory. Default false.
# The behaviour is undefined if mem_prealloc is also set to true
#enable_swap = true
# This option changes the default hypervisor and kernel parameters
# to enable debug output where available. This extra output is added
# to the proxy logs, but only when proxy debug is also enabled.
#
# Default false
#enable_debug = true
# Disable the customizations done in the runtime when it detects
# that it is running on top a VMM. This will result in the runtime
# behaving as it would when running on bare metal.
#
#disable_nesting_checks = true
# This is the msize used for 9p shares. It is the number of bytes
# used for 9p packet payload.
#msize_9p = 8192
# If true and vsocks are supported, use vsocks to communicate directly
# with the agent and no proxy is started, otherwise use unix
# sockets and start a proxy to communicate with the agent.
# Default false
#use_vsock = true
# If false and nvdimm is supported, use nvdimm device to plug guest image.
# Otherwise virtio-block device is used.
# Default is false
#disable_image_nvdimm = true
# VFIO devices are hotplugged on a bridge by default.
# Enable hotplugging on root bus. This may be required for devices with
# a large PCI bar, as this is a current limitation with hotplugging on
# a bridge. This value is valid for "pc" machine type.
# Default false
#hotplug_vfio_on_root_bus = true
# Before hot plugging a PCIe device, you need to add a pcie_root_port device.
# Use this parameter when using some large PCI bar devices, such as Nvidia GPU
# The value means the number of pcie_root_port
# This value is valid when hotplug_vfio_on_root_bus is true and machine_type is "q35"
# Default 0
#pcie_root_port = 2
# If vhost-net backend for virtio-net is not desired, set to true. Default is false, which trades off
# security (vhost-net runs ring0) for network I/O performance.
#disable_vhost_net = true
#
# Default entropy source.
# The path to a host source of entropy (including a real hardware RNG)
# /dev/urandom and /dev/random are two main options.
# Be aware that /dev/random is a blocking source of entropy. If the host
# runs out of entropy, the VMs boot time will increase leading to get startup
# timeouts.
# The source of entropy /dev/urandom is non-blocking and provides a
# generally acceptable source of entropy. It should work well for pretty much
# all practical purposes.
#entropy_source= "/dev/urandom"
# Path to OCI hook binaries in the *guest rootfs*.
# This does not affect host-side hooks which must instead be added to
# the OCI spec passed to the runtime.
#
# You can create a rootfs with hooks by customizing the osbuilder scripts:
# https://github.com/kata-containers/osbuilder
#
# Hooks must be stored in a subdirectory of guest_hook_path according to their
# hook type, i.e. "guest_hook_path/{prestart,postart,poststop}".
# The agent will scan these directories for executable files and add them, in
# lexicographical order, to the lifecycle of the guest container.
# Hooks are executed in the runtime namespace of the guest. See the official documentation:
# https://github.com/opencontainers/runtime-spec/blob/v1.0.1/config.md#posix-platform-hooks
# Warnings will be logged if any error is encountered will scanning for hooks,
# but it will not abort container execution.
#guest_hook_path = "/usr/share/oci/hooks"
[factory]
# VM templating support. Once enabled, new VMs are created from template
# using vm cloning. They will share the same initial kernel, initramfs and
# agent memory by mapping it readonly. It helps speeding up new container
# creation and saves a lot of memory if there are many kata containers running
# on the same host.
#
# When disabled, new VMs are created from scratch.
#
# Note: Requires "initrd=" to be set ("image=" is not supported).
#
# Default false
#enable_template = true
# Specifies the path of template.
#
# Default "/run/vc/vm/template"
#template_path = "/run/vc/vm/template"
# The number of caches of VMCache:
# unspecified or == 0 --> VMCache is disabled
# > 0 --> will be set to the specified number
#
# VMCache is a function that creates VMs as caches before using it.
# It helps speed up new container creation.
# The function consists of a server and some clients communicating
# through Unix socket. The protocol is gRPC in protocols/cache/cache.proto.
# The VMCache server will create some VMs and cache them by factory cache.
# It will convert the VM to gRPC format and transport it when gets
# requestion from clients.
# Factory grpccache is the VMCache client. It will request gRPC format
# VM and convert it back to a VM. If VMCache function is enabled,
# kata-runtime will request VM from factory grpccache when it creates
# a new sandbox.
#
# Default 0
#vm_cache_number = 0
# Specify the address of the Unix socket that is used by VMCache.
#
# Default /var/run/kata-containers/cache.sock
#vm_cache_endpoint = "/var/run/kata-containers/cache.sock"
[proxy.kata]
path = "/usr/libexec/kata-containers/kata-proxy"
# If enabled, proxy messages will be sent to the system log
# (default: disabled)
#enable_debug = true
[shim.kata]
path = "/usr/libexec/kata-containers/kata-shim"
# If enabled, shim messages will be sent to the system log
# (default: disabled)
#enable_debug = true
# If enabled, the shim will create opentracing.io traces and spans.
# (See https://www.jaegertracing.io/docs/getting-started).
#
# Note: By default, the shim runs in a separate network namespace. Therefore,
# to allow it to send trace details to the Jaeger agent running on the host,
# it is necessary to set 'disable_new_netns=true' so that it runs in the host
# network namespace.
#
# (default: disabled)
#enable_tracing = true
[agent.kata]
# If enabled, make the agent display debug-level messages.
# (default: disabled)
#enable_debug = true
# Enable agent tracing.
#
# If enabled, the default trace mode is "dynamic" and the
# default trace type is "isolated". The trace mode and type are set
# explicity with the `trace_type=` and `trace_mode=` options.
#
# Notes:
#
# - Tracing is ONLY enabled when `enable_tracing` is set: explicitly
# setting `trace_mode=` and/or `trace_type=` without setting `enable_tracing`
# will NOT activate agent tracing.
#
# - See https://github.com/kata-containers/agent/blob/master/TRACING.md for
# full details.
#
# (default: disabled)
#enable_tracing = true
#
#trace_mode = "dynamic"
#trace_type = "isolated"
# Comma separated list of kernel modules and their parameters.
# These modules will be loaded in the guest kernel using modprobe(8).
# The following example can be used to load two kernel modules with parameters
# - kernel_modules=["e1000e InterruptThrottleRate=3000,3000,3000 EEE=1", "i915 enable_ppgtt=0"]
# The first word is considered as the module name and the rest as its parameters.
# Container will not be started when:
# * A kernel module is specified and the modprobe command is not installed in the guest
# or it fails loading the module.
# * The module is not available in the guest or it doesn't met the guest kernel
# requirements, like architecture and version.
#
kernel_modules=[]
[netmon]
# If enabled, the network monitoring process gets started when the
# sandbox is created. This allows for the detection of some additional
# network being added to the existing network namespace, after the
# sandbox has been created.
# (default: disabled)
#enable_netmon = true
# Specify the path to the netmon binary.
path = "/usr/libexec/kata-containers/kata-netmon"
# If enabled, netmon messages will be sent to the system log
# (default: disabled)
#enable_debug = true
[runtime]
# If enabled, the runtime will log additional debug messages to the
# system log
# (default: disabled)
#enable_debug = true
#
# Internetworking model
# Determines how the VM should be connected to the
# the container network interface
# Options:
#
# - macvtap
# Used when the Container network interface can be bridged using
# macvtap.
#
# - none
# Used when customize network. Only creates a tap device. No veth pair.
#
# - tcfilter
# Uses tc filter rules to redirect traffic from the network interface
# provided by plugin to a tap interface connected to the VM.
#
internetworking_model="tcfilter"
# disable guest seccomp
# Determines whether container seccomp profiles are passed to the virtual
# machine and applied by the kata agent. If set to true, seccomp is not applied
# within the guest
# (default: true)
disable_guest_seccomp=true
# If enabled, the runtime will create opentracing.io traces and spans.
# (See https://www.jaegertracing.io/docs/getting-started).
# (default: disabled)
#enable_tracing = true
# If enabled, the runtime will not create a network namespace for shim and hypervisor processes.
# This option may have some potential impacts to your host. It should only be used when you know what you're doing.
# `disable_new_netns` conflicts with `enable_netmon`
# `disable_new_netns` conflicts with `internetworking_model=tcfilter` and `internetworking_model=macvtap`. It works only
# with `internetworking_model=none`. The tap device will be in the host network namespace and can connect to a bridge
# (like OVS) directly.
# If you are using docker, `disable_new_netns` only works with `docker run --net=none`
# (default: false)
#disable_new_netns = true
# if enabled, the runtime will add all the kata processes inside one dedicated cgroup.
# The container cgroups in the host are not created, just one single cgroup per sandbox.
# The runtime caller is free to restrict or collect cgroup stats of the overall Kata sandbox.
# The sandbox cgroup path is the parent cgroup of a container with the PodSandbox annotation.
# The sandbox cgroup is constrained if there is no container type annotation.
# See: https://godoc.org/github.com/kata-containers/runtime/virtcontainers#ContainerType
sandbox_cgroup_only=false
# Enabled experimental feature list, format: ["a", "b"].
# Experimental features are features not stable enough for production,
# they may break compatibility, and are prepared for a big version bump.
# Supported experimental features:
# (default: [])
experimental=[]
Ha, I know where the problem is, @coderplay
These related parameters should be under section [hypervisor.qemu], not the end of the config file.
Thanks for your feedback~, let me also emphasize this point in the document revision.
dong-liuliu
on 31 Mar 2020
Unfortunately, It's the same after I moved the 3 items under [hypervisor.qemu]
coderplay
on 31 Mar 2020
content of /usr/share/defaults/kata-containers/configuration.toml
# Copyright (c) 2017-2019 Intel Corporation
#
# SPDX-License-Identifier: Apache-2.0
#
# XXX: WARNING: this file is auto-generated.
# XXX:
# XXX: Source file: "cli/config/configuration-qemu.toml.in"
# XXX: Project:
# XXX: Name: Kata Containers
# XXX: Type: kata
[hypervisor.qemu]
path = "/usr/bin/qemu-vanilla-system-x86_64"
kernel = "/usr/share/kata-containers/vmlinuz.container"
image = "/usr/share/kata-containers/kata-containers.img"
machine_type = "pc"
# Optional space-separated list of options to pass to the guest kernel.
# For example, use `kernel_params = "vsyscall=emulate"` if you are having
# trouble running pre-2.15 glibc.
#
# WARNING: - any parameter specified here will take priority over the default
# parameter value of the same name used to start the virtual machine.
# Do not set values here unless you understand the impact of doing so as you
# may stop the virtual machine from booting.
# To see the list of default parameters, enable hypervisor debug, create a
# container and look for 'default-kernel-parameters' log entries.
kernel_params = ""
# Path to the firmware.
# If you want that qemu uses the default firmware leave this option empty
firmware = ""
# Machine accelerators
# comma-separated list of machine accelerators to pass to the hypervisor.
# For example, `machine_accelerators = "nosmm,nosmbus,nosata,nopit,static-prt,nofw"`
machine_accelerators=""
# Default number of vCPUs per SB/VM:
# unspecified or 0 --> will be set to 1
# < 0 --> will be set to the actual number of physical cores
# > 0 <= number of physical cores --> will be set to the specified number
# > number of physical cores --> will be set to the actual number of physical cores
default_vcpus = 1
# Default maximum number of vCPUs per SB/VM:
# unspecified or == 0 --> will be set to the actual number of physical cores or to the maximum number
# of vCPUs supported by KVM if that number is exceeded
# > 0 <= number of physical cores --> will be set to the specified number
# > number of physical cores --> will be set to the actual number of physical cores or to the maximum number
# of vCPUs supported by KVM if that number is exceeded
# WARNING: Depending of the architecture, the maximum number of vCPUs supported by KVM is used when
# the actual number of physical cores is greater than it.
# WARNING: Be aware that this value impacts the virtual machine's memory footprint and CPU
# the hotplug functionality. For example, `default_maxvcpus = 240` specifies that until 240 vCPUs
# can be added to a SB/VM, but the memory footprint will be big. Another example, with
# `default_maxvcpus = 8` the memory footprint will be small, but 8 will be the maximum number of
# vCPUs supported by the SB/VM. In general, we recommend that you do not edit this variable,
# unless you know what are you doing.
default_maxvcpus = 0
# Bridges can be used to hot plug devices.
# Limitations:
# * Currently only pci bridges are supported
# * Until 30 devices per bridge can be hot plugged.
# * Until 5 PCI bridges can be cold plugged per VM.
# This limitation could be a bug in qemu or in the kernel
# Default number of bridges per SB/VM:
# unspecified or 0 --> will be set to 1
# > 1 <= 5 --> will be set to the specified number
# > 5 --> will be set to 5
default_bridges = 1
# Default memory size in MiB for SB/VM.
# If unspecified then it will be set 2048 MiB.
default_memory = 2048
#
# Default memory slots per SB/VM.
# If unspecified then it will be set 10.
# This is will determine the times that memory will be hotadded to sandbox/VM.
#memory_slots = 10
# The size in MiB will be plused to max memory of hypervisor.
# It is the memory address space for the NVDIMM devie.
# If set block storage driver (block_device_driver) to "nvdimm",
# should set memory_offset to the size of block device.
# Default 0
#memory_offset = 0
# Specifies virtio-mem will be enabled or not.
# Please note that this option should be used with the command
# "echo 1 > /proc/sys/vm/overcommit_memory".
# Default false
#enable_virtio_mem = true
# Disable block device from being used for a container's rootfs.
# In case of a storage driver like devicemapper where a container's
# root file system is backed by a block device, the block device is passed
# directly to the hypervisor for performance reasons.
# This flag prevents the block device from being passed to the hypervisor,
# 9pfs is used instead to pass the rootfs.
disable_block_device_use = false
# Shared file system type:
# - virtio-9p (default)
# - virtio-fs
shared_fs = "virtio-9p"
# Path to vhost-user-fs daemon.
virtio_fs_daemon = "/usr/bin/virtiofsd"
# Default size of DAX cache in MiB
virtio_fs_cache_size = 1024
# Extra args for virtiofsd daemon
#
# Format example:
# ["-o", "arg1=xxx,arg2", "-o", "hello world", "--arg3=yyy"]
#
# see `virtiofsd -h` for possible options.
virtio_fs_extra_args = []
# Cache mode:
#
# - none
# Metadata, data, and pathname lookup are not cached in guest. They are
# always fetched from host and any changes are immediately pushed to host.
#
# - auto
# Metadata and pathname lookup cache expires after a configured amount of
# time (default is 1 second). Data is cached while the file is open (close
# to open consistency).
#
# - always
# Metadata, data, and pathname lookup are cached in guest and never expire.
virtio_fs_cache = "always"
# Block storage driver to be used for the hypervisor in case the container
# rootfs is backed by a block device. This is virtio-scsi, virtio-blk
# or nvdimm.
block_device_driver = "virtio-scsi"
# Specifies cache-related options will be set to block devices or not.
# Default false
#block_device_cache_set = true
# Specifies cache-related options for block devices.
# Denotes whether use of O_DIRECT (bypass the host page cache) is enabled.
# Default false
#block_device_cache_direct = true
# Specifies cache-related options for block devices.
# Denotes whether flush requests for the device are ignored.
# Default false
#block_device_cache_noflush = true
# Enable iothreads (data-plane) to be used. This causes IO to be
# handled in a separate IO thread. This is currently only implemented
# for SCSI.
#
enable_iothreads = false
# Enable pre allocation of VM RAM, default false
# Enabling this will result in lower container density
# as all of the memory will be allocated and locked
# This is useful when you want to reserve all the memory
# upfront or in the cases where you want memory latencies
# to be very predictable
# Default false
#enable_mem_prealloc = true
# Enable huge pages for VM RAM, default false
# Enabling this will result in the VM memory
# being allocated using huge pages.
# This is useful when you want to use vhost-user network
# stacks within the container. This will automatically
# result in memory pre allocation
enable_hugepages = true
# Enable vhost-user storage device, default false
# Enabling this will result in some Linux reserved block type
# major range 240-254 being chosen to represent vhost-user devices.
enable_vhost_user_store = false
# The base directory specifically used for vhost-user devices.
# Its sub-path "block" is used for block devices; "block/sockets" is
# where we expect vhost-user sockets to live; "block/devices" is where
# simulated block device nodes for vhost-user devices to live.
vhost_user_store_path = "/var/run/kata-containers/vhost-user"
# Enable file based guest memory support. The default is an empty string which
# will disable this feature. In the case of virtio-fs, this is enabled
# automatically and '/dev/shm' is used as the backing folder.
# This option will be ignored if VM templating is enabled.
#file_mem_backend = ""
# Enable swap of vm memory. Default false.
# The behaviour is undefined if mem_prealloc is also set to true
#enable_swap = true
# This option changes the default hypervisor and kernel parameters
# to enable debug output where available. This extra output is added
# to the proxy logs, but only when proxy debug is also enabled.
#
# Default false
#enable_debug = true
# Disable the customizations done in the runtime when it detects
# that it is running on top a VMM. This will result in the runtime
# behaving as it would when running on bare metal.
#
#disable_nesting_checks = true
# This is the msize used for 9p shares. It is the number of bytes
# used for 9p packet payload.
#msize_9p = 8192
# If true and vsocks are supported, use vsocks to communicate directly
# with the agent and no proxy is started, otherwise use unix
# sockets and start a proxy to communicate with the agent.
# Default false
#use_vsock = true
# If false and nvdimm is supported, use nvdimm device to plug guest image.
# Otherwise virtio-block device is used.
# Default is false
#disable_image_nvdimm = true
# VFIO devices are hotplugged on a bridge by default.
# Enable hotplugging on root bus. This may be required for devices with
# a large PCI bar, as this is a current limitation with hotplugging on
# a bridge. This value is valid for "pc" machine type.
# Default false
#hotplug_vfio_on_root_bus = true
# Before hot plugging a PCIe device, you need to add a pcie_root_port device.
# Use this parameter when using some large PCI bar devices, such as Nvidia GPU
# The value means the number of pcie_root_port
# This value is valid when hotplug_vfio_on_root_bus is true and machine_type is "q35"
# Default 0
#pcie_root_port = 2
# If vhost-net backend for virtio-net is not desired, set to true. Default is false, which trades off
# security (vhost-net runs ring0) for network I/O performance.
#disable_vhost_net = true
#
# Default entropy source.
# The path to a host source of entropy (including a real hardware RNG)
# /dev/urandom and /dev/random are two main options.
# Be aware that /dev/random is a blocking source of entropy. If the host
# runs out of entropy, the VMs boot time will increase leading to get startup
# timeouts.
# The source of entropy /dev/urandom is non-blocking and provides a
# generally acceptable source of entropy. It should work well for pretty much
# all practical purposes.
#entropy_source= "/dev/urandom"
# Path to OCI hook binaries in the *guest rootfs*.
# This does not affect host-side hooks which must instead be added to
# the OCI spec passed to the runtime.
#
# You can create a rootfs with hooks by customizing the osbuilder scripts:
# https://github.com/kata-containers/osbuilder
#
# Hooks must be stored in a subdirectory of guest_hook_path according to their
# hook type, i.e. "guest_hook_path/{prestart,postart,poststop}".
# The agent will scan these directories for executable files and add them, in
# lexicographical order, to the lifecycle of the guest container.
# Hooks are executed in the runtime namespace of the guest. See the official documentation:
# https://github.com/opencontainers/runtime-spec/blob/v1.0.1/config.md#posix-platform-hooks
# Warnings will be logged if any error is encountered will scanning for hooks,
# but it will not abort container execution.
#guest_hook_path = "/usr/share/oci/hooks"
[factory]
# VM templating support. Once enabled, new VMs are created from template
# using vm cloning. They will share the same initial kernel, initramfs and
# agent memory by mapping it readonly. It helps speeding up new container
# creation and saves a lot of memory if there are many kata containers running
# on the same host.
#
# When disabled, new VMs are created from scratch.
#
# Note: Requires "initrd=" to be set ("image=" is not supported).
#
# Default false
#enable_template = true
# Specifies the path of template.
#
# Default "/run/vc/vm/template"
#template_path = "/run/vc/vm/template"
# The number of caches of VMCache:
# unspecified or == 0 --> VMCache is disabled
# > 0 --> will be set to the specified number
#
# VMCache is a function that creates VMs as caches before using it.
# It helps speed up new container creation.
# The function consists of a server and some clients communicating
# through Unix socket. The protocol is gRPC in protocols/cache/cache.proto.
# The VMCache server will create some VMs and cache them by factory cache.
# It will convert the VM to gRPC format and transport it when gets
# requestion from clients.
# Factory grpccache is the VMCache client. It will request gRPC format
# VM and convert it back to a VM. If VMCache function is enabled,
# kata-runtime will request VM from factory grpccache when it creates
# a new sandbox.
#
# Default 0
#vm_cache_number = 0
# Specify the address of the Unix socket that is used by VMCache.
#
# Default /var/run/kata-containers/cache.sock
#vm_cache_endpoint = "/var/run/kata-containers/cache.sock"
[proxy.kata]
path = "/usr/libexec/kata-containers/kata-proxy"
# If enabled, proxy messages will be sent to the system log
# (default: disabled)
#enable_debug = true
[shim.kata]
path = "/usr/libexec/kata-containers/kata-shim"
# If enabled, shim messages will be sent to the system log
# (default: disabled)
#enable_debug = true
# If enabled, the shim will create opentracing.io traces and spans.
# (See https://www.jaegertracing.io/docs/getting-started).
#
# Note: By default, the shim runs in a separate network namespace. Therefore,
# to allow it to send trace details to the Jaeger agent running on the host,
# it is necessary to set 'disable_new_netns=true' so that it runs in the host
# network namespace.
#
# (default: disabled)
#enable_tracing = true
[agent.kata]
# If enabled, make the agent display debug-level messages.
# (default: disabled)
#enable_debug = true
# Enable agent tracing.
#
# If enabled, the default trace mode is "dynamic" and the
# default trace type is "isolated". The trace mode and type are set
# explicity with the `trace_type=` and `trace_mode=` options.
#
# Notes:
#
# - Tracing is ONLY enabled when `enable_tracing` is set: explicitly
# setting `trace_mode=` and/or `trace_type=` without setting `enable_tracing`
# will NOT activate agent tracing.
#
# - See https://github.com/kata-containers/agent/blob/master/TRACING.md for
# full details.
#
# (default: disabled)
#enable_tracing = true
#
#trace_mode = "dynamic"
#trace_type = "isolated"
# Comma separated list of kernel modules and their parameters.
# These modules will be loaded in the guest kernel using modprobe(8).
# The following example can be used to load two kernel modules with parameters
# - kernel_modules=["e1000e InterruptThrottleRate=3000,3000,3000 EEE=1", "i915 enable_ppgtt=0"]
# The first word is considered as the module name and the rest as its parameters.
# Container will not be started when:
# * A kernel module is specified and the modprobe command is not installed in the guest
# or it fails loading the module.
# * The module is not available in the guest or it doesn't met the guest kernel
# requirements, like architecture and version.
#
kernel_modules=[]
[netmon]
# If enabled, the network monitoring process gets started when the
# sandbox is created. This allows for the detection of some additional
# network being added to the existing network namespace, after the
# sandbox has been created.
# (default: disabled)
#enable_netmon = true
# Specify the path to the netmon binary.
path = "/usr/libexec/kata-containers/kata-netmon"
# If enabled, netmon messages will be sent to the system log
# (default: disabled)
#enable_debug = true
[runtime]
# If enabled, the runtime will log additional debug messages to the
# system log
# (default: disabled)
#enable_debug = true
#
# Internetworking model
# Determines how the VM should be connected to the
# the container network interface
# Options:
#
# - macvtap
# Used when the Container network interface can be bridged using
# macvtap.
#
# - none
# Used when customize network. Only creates a tap device. No veth pair.
#
# - tcfilter
# Uses tc filter rules to redirect traffic from the network interface
# provided by plugin to a tap interface connected to the VM.
#
internetworking_model="tcfilter"
# disable guest seccomp
# Determines whether container seccomp profiles are passed to the virtual
# machine and applied by the kata agent. If set to true, seccomp is not applied
# within the guest
# (default: true)
disable_guest_seccomp=true
# If enabled, the runtime will create opentracing.io traces and spans.
# (See https://www.jaegertracing.io/docs/getting-started).
# (default: disabled)
#enable_tracing = true
# If enabled, the runtime will not create a network namespace for shim and hypervisor processes.
# This option may have some potential impacts to your host. It should only be used when you know what you're doing.
# `disable_new_netns` conflicts with `enable_netmon`
# `disable_new_netns` conflicts with `internetworking_model=tcfilter` and `internetworking_model=macvtap`. It works only
# with `internetworking_model=none`. The tap device will be in the host network namespace and can connect to a bridge
# (like OVS) directly.
# If you are using docker, `disable_new_netns` only works with `docker run --net=none`
# (default: false)
#disable_new_netns = true
# if enabled, the runtime will add all the kata processes inside one dedicated cgroup.
# The container cgroups in the host are not created, just one single cgroup per sandbox.
# The runtime caller is free to restrict or collect cgroup stats of the overall Kata sandbox.
# The sandbox cgroup path is the parent cgroup of a container with the PodSandbox annotation.
# The sandbox cgroup is constrained if there is no container type annotation.
# See: https://godoc.org/github.com/kata-containers/runtime/virtcontainers#ContainerType
sandbox_cgroup_only=false
# Enabled experimental feature list, format: ["a", "b"].
# Experimental features are features not stable enough for production,
# they may break compatibility, and are prepared for a big version bump.
# Supported experimental features:
# (default: [])
experimental=[]
content of /etc/kata-containers/configuration.toml
# Copyright (c) 2017-2019 Intel Corporation
#
# SPDX-License-Identifier: Apache-2.0
#
# XXX: WARNING: this file is auto-generated.
# XXX:
# XXX: Source file: "cli/config/configuration-qemu.toml.in"
# XXX: Project:
# XXX: Name: Kata Containers
# XXX: Type: kata
[hypervisor.qemu]
path = "/usr/bin/qemu-vanilla-system-x86_64"
kernel = "/usr/share/kata-containers/vmlinuz.container"
image = "/usr/share/kata-containers/kata-containers.img"
machine_type = "pc"
enable_vhost_user_store=true
# Optional space-separated list of options to pass to the guest kernel.
# For example, use `kernel_params = "vsyscall=emulate"` if you are having
# trouble running pre-2.15 glibc.
#
# WARNING: - any parameter specified here will take priority over the default
# parameter value of the same name used to start the virtual machine.
# Do not set values here unless you understand the impact of doing so as you
# may stop the virtual machine from booting.
# To see the list of default parameters, enable hypervisor debug, create a
# container and look for 'default-kernel-parameters' log entries.
kernel_params = " agent.log=debug initcall_debug"
# Path to the firmware.
# If you want that qemu uses the default firmware leave this option empty
firmware = ""
# Machine accelerators
# comma-separated list of machine accelerators to pass to the hypervisor.
# For example, `machine_accelerators = "nosmm,nosmbus,nosata,nopit,static-prt,nofw"`
machine_accelerators=""
# Default number of vCPUs per SB/VM:
# unspecified or 0 --> will be set to 1
# < 0 --> will be set to the actual number of physical cores
# > 0 <= number of physical cores --> will be set to the specified number
# > number of physical cores --> will be set to the actual number of physical cores
default_vcpus = 1
# Default maximum number of vCPUs per SB/VM:
# unspecified or == 0 --> will be set to the actual number of physical cores or to the maximum number
# of vCPUs supported by KVM if that number is exceeded
# > 0 <= number of physical cores --> will be set to the specified number
# > number of physical cores --> will be set to the actual number of physical cores or to the maximum number
# of vCPUs supported by KVM if that number is exceeded
# WARNING: Depending of the architecture, the maximum number of vCPUs supported by KVM is used when
# the actual number of physical cores is greater than it.
# WARNING: Be aware that this value impacts the virtual machine's memory footprint and CPU
# the hotplug functionality. For example, `default_maxvcpus = 240` specifies that until 240 vCPUs
# can be added to a SB/VM, but the memory footprint will be big. Another example, with
# `default_maxvcpus = 8` the memory footprint will be small, but 8 will be the maximum number of
# vCPUs supported by the SB/VM. In general, we recommend that you do not edit this variable,
# unless you know what are you doing.
default_maxvcpus = 0
# Bridges can be used to hot plug devices.
# Limitations:
# * Currently only pci bridges are supported
# * Until 30 devices per bridge can be hot plugged.
# * Until 5 PCI bridges can be cold plugged per VM.
# This limitation could be a bug in qemu or in the kernel
# Default number of bridges per SB/VM:
# unspecified or 0 --> will be set to 1
# > 1 <= 5 --> will be set to the specified number
# > 5 --> will be set to 5
default_bridges = 1
# Default memory size in MiB for SB/VM.
# If unspecified then it will be set 2048 MiB.
default_memory = 2048
#
# Default memory slots per SB/VM.
# If unspecified then it will be set 10.
# This is will determine the times that memory will be hotadded to sandbox/VM.
#memory_slots = 10
# The size in MiB will be plused to max memory of hypervisor.
# It is the memory address space for the NVDIMM devie.
# If set block storage driver (block_device_driver) to "nvdimm",
# should set memory_offset to the size of block device.
# Default 0
#memory_offset = 0
# Specifies virtio-mem will be enabled or not.
# Please note that this option should be used with the command
# "echo 1 > /proc/sys/vm/overcommit_memory".
# Default false
#enable_virtio_mem = true
# Disable block device from being used for a container's rootfs.
# In case of a storage driver like devicemapper where a container's
# root file system is backed by a block device, the block device is passed
# directly to the hypervisor for performance reasons.
# This flag prevents the block device from being passed to the hypervisor,
# 9pfs is used instead to pass the rootfs.
disable_block_device_use = false
# Shared file system type:
# - virtio-9p (default)
# - virtio-fs
shared_fs = "virtio-9p"
# Path to vhost-user-fs daemon.
virtio_fs_daemon = "/usr/bin/virtiofsd"
# Default size of DAX cache in MiB
virtio_fs_cache_size = 1024
# Extra args for virtiofsd daemon
#
# Format example:
# ["-o", "arg1=xxx,arg2", "-o", "hello world", "--arg3=yyy"]
#
# see `virtiofsd -h` for possible options.
virtio_fs_extra_args = []
# Cache mode:
#
# - none
# Metadata, data, and pathname lookup are not cached in guest. They are
# always fetched from host and any changes are immediately pushed to host.
#
# - auto
# Metadata and pathname lookup cache expires after a configured amount of
# time (default is 1 second). Data is cached while the file is open (close
# to open consistency).
#
# - always
# Metadata, data, and pathname lookup are cached in guest and never expire.
virtio_fs_cache = "always"
# Block storage driver to be used for the hypervisor in case the container
# rootfs is backed by a block device. This is virtio-scsi, virtio-blk
# or nvdimm.
block_device_driver = "virtio-scsi"
# Specifies cache-related options will be set to block devices or not.
# Default false
#block_device_cache_set = true
# Specifies cache-related options for block devices.
# Denotes whether use of O_DIRECT (bypass the host page cache) is enabled.
# Default false
#block_device_cache_direct = true
# Specifies cache-related options for block devices.
# Denotes whether flush requests for the device are ignored.
# Default false
#block_device_cache_noflush = true
# Enable iothreads (data-plane) to be used. This causes IO to be
# handled in a separate IO thread. This is currently only implemented
# for SCSI.
#
enable_iothreads = false
# Enable pre allocation of VM RAM, default false
# Enabling this will result in lower container density
# as all of the memory will be allocated and locked
# This is useful when you want to reserve all the memory
# upfront or in the cases where you want memory latencies
# to be very predictable
# Default false
#enable_mem_prealloc = true
# Enable huge pages for VM RAM, default false
# Enabling this will result in the VM memory
# being allocated using huge pages.
# This is useful when you want to use vhost-user network
# stacks within the container. This will automatically
# result in memory pre allocation
#enable_hugepages = true
# Enable file based guest memory support. The default is an empty string which
# will disable this feature. In the case of virtio-fs, this is enabled
# automatically and '/dev/shm' is used as the backing folder.
# This option will be ignored if VM templating is enabled.
#file_mem_backend = ""
# Enable swap of vm memory. Default false.
# The behaviour is undefined if mem_prealloc is also set to true
#enable_swap = true
# This option changes the default hypervisor and kernel parameters
# to enable debug output where available. This extra output is added
# to the proxy logs, but only when proxy debug is also enabled.
#
# Default false
enable_debug = true
# Disable the customizations done in the runtime when it detects
# that it is running on top a VMM. This will result in the runtime
# behaving as it would when running on bare metal.
#
#disable_nesting_checks = true
# This is the msize used for 9p shares. It is the number of bytes
# used for 9p packet payload.
#msize_9p = 8192
# If true and vsocks are supported, use vsocks to communicate directly
# with the agent and no proxy is started, otherwise use unix
# sockets and start a proxy to communicate with the agent.
# Default false
#use_vsock = true
# If false and nvdimm is supported, use nvdimm device to plug guest image.
# Otherwise virtio-block device is used.
# Default is false
#disable_image_nvdimm = true
# VFIO devices are hotplugged on a bridge by default.
# Enable hotplugging on root bus. This may be required for devices with
# a large PCI bar, as this is a current limitation with hotplugging on
# a bridge. This value is valid for "pc" machine type.
# Default false
#hotplug_vfio_on_root_bus = true
# Before hot plugging a PCIe device, you need to add a pcie_root_port device.
# Use this parameter when using some large PCI bar devices, such as Nvidia GPU
# The value means the number of pcie_root_port
# This value is valid when hotplug_vfio_on_root_bus is true and machine_type is "q35"
# Default 0
#pcie_root_port = 2
# If vhost-net backend for virtio-net is not desired, set to true. Default is false, which trades off
# security (vhost-net runs ring0) for network I/O performance.
#disable_vhost_net = true
#
# Default entropy source.
# The path to a host source of entropy (including a real hardware RNG)
# /dev/urandom and /dev/random are two main options.
# Be aware that /dev/random is a blocking source of entropy. If the host
# runs out of entropy, the VMs boot time will increase leading to get startup
# timeouts.
# The source of entropy /dev/urandom is non-blocking and provides a
# generally acceptable source of entropy. It should work well for pretty much
# all practical purposes.
#entropy_source= "/dev/urandom"
# Path to OCI hook binaries in the *guest rootfs*.
# This does not affect host-side hooks which must instead be added to
# the OCI spec passed to the runtime.
#
# You can create a rootfs with hooks by customizing the osbuilder scripts:
# https://github.com/kata-containers/osbuilder
#
# Hooks must be stored in a subdirectory of guest_hook_path according to their
# hook type, i.e. "guest_hook_path/{prestart,postart,poststop}".
# The agent will scan these directories for executable files and add them, in
# lexicographical order, to the lifecycle of the guest container.
# Hooks are executed in the runtime namespace of the guest. See the official documentation:
# https://github.com/opencontainers/runtime-spec/blob/v1.0.1/config.md#posix-platform-hooks
# Warnings will be logged if any error is encountered will scanning for hooks,
# but it will not abort container execution.
#guest_hook_path = "/usr/share/oci/hooks"
[factory]
# VM templating support. Once enabled, new VMs are created from template
# using vm cloning. They will share the same initial kernel, initramfs and
# agent memory by mapping it readonly. It helps speeding up new container
# creation and saves a lot of memory if there are many kata containers running
# on the same host.
#
# When disabled, new VMs are created from scratch.
#
# Note: Requires "initrd=" to be set ("image=" is not supported).
#
# Default false
#enable_template = true
# Specifies the path of template.
#
# Default "/run/vc/vm/template"
#template_path = "/run/vc/vm/template"
# The number of caches of VMCache:
# unspecified or == 0 --> VMCache is disabled
# > 0 --> will be set to the specified number
#
# VMCache is a function that creates VMs as caches before using it.
# It helps speed up new container creation.
# The function consists of a server and some clients communicating
# through Unix socket. The protocol is gRPC in protocols/cache/cache.proto.
# The VMCache server will create some VMs and cache them by factory cache.
# It will convert the VM to gRPC format and transport it when gets
# requestion from clients.
# Factory grpccache is the VMCache client. It will request gRPC format
# VM and convert it back to a VM. If VMCache function is enabled,
# kata-runtime will request VM from factory grpccache when it creates
# a new sandbox.
#
# Default 0
#vm_cache_number = 0
# Specify the address of the Unix socket that is used by VMCache.
#
# Default /var/run/kata-containers/cache.sock
#vm_cache_endpoint = "/var/run/kata-containers/cache.sock"
[proxy.kata]
path = "/usr/libexec/kata-containers/kata-proxy"
# If enabled, proxy messages will be sent to the system log
# (default: disabled)
enable_debug = true
[shim.kata]
path = "/usr/libexec/kata-containers/kata-shim"
# If enabled, shim messages will be sent to the system log
# (default: disabled)
enable_debug = true
# If enabled, the shim will create opentracing.io traces and spans.
# (See https://www.jaegertracing.io/docs/getting-started).
#
# Note: By default, the shim runs in a separate network namespace. Therefore,
# to allow it to send trace details to the Jaeger agent running on the host,
# it is necessary to set 'disable_new_netns=true' so that it runs in the host
# network namespace.
#
# (default: disabled)
#enable_tracing = true
[agent.kata]
# If enabled, make the agent display debug-level messages.
# (default: disabled)
enable_debug = true
# Enable agent tracing.
#
# If enabled, the default trace mode is "dynamic" and the
# default trace type is "isolated". The trace mode and type are set
# explicity with the `trace_type=` and `trace_mode=` options.
#
# Notes:
#
# - Tracing is ONLY enabled when `enable_tracing` is set: explicitly
# setting `trace_mode=` and/or `trace_type=` without setting `enable_tracing`
# will NOT activate agent tracing.
#
# - See https://github.com/kata-containers/agent/blob/master/TRACING.md for
# full details.
#
# (default: disabled)
#enable_tracing = true
#
#trace_mode = "dynamic"
#trace_type = "isolated"
# Comma separated list of kernel modules and their parameters.
# These modules will be loaded in the guest kernel using modprobe(8).
# The following example can be used to load two kernel modules with parameters
# - kernel_modules=["e1000e InterruptThrottleRate=3000,3000,3000 EEE=1", "i915 enable_ppgtt=0"]
# The first word is considered as the module name and the rest as its parameters.
# Container will not be started when:
# * A kernel module is specified and the modprobe command is not installed in the guest
# or it fails loading the module.
# * The module is not available in the guest or it doesn't met the guest kernel
# requirements, like architecture and version.
#
kernel_modules=[]
[netmon]
# If enabled, the network monitoring process gets started when the
# sandbox is created. This allows for the detection of some additional
# network being added to the existing network namespace, after the
# sandbox has been created.
# (default: disabled)
#enable_netmon = true
# Specify the path to the netmon binary.
path = "/usr/libexec/kata-containers/kata-netmon"
# If enabled, netmon messages will be sent to the system log
# (default: disabled)
enable_debug = true
[runtime]
# If enabled, the runtime will log additional debug messages to the
# system log
# (default: disabled)
enable_debug = true
#
# Internetworking model
# Determines how the VM should be connected to the
# the container network interface
# Options:
#
# - macvtap
# Used when the Container network interface can be bridged using
# macvtap.
#
# - none
# Used when customize network. Only creates a tap device. No veth pair.
#
# - tcfilter
# Uses tc filter rules to redirect traffic from the network interface
# provided by plugin to a tap interface connected to the VM.
#
internetworking_model="tcfilter"
# disable guest seccomp
# Determines whether container seccomp profiles are passed to the virtual
# machine and applied by the kata agent. If set to true, seccomp is not applied
# within the guest
# (default: true)
disable_guest_seccomp=true
# If enabled, the runtime will create opentracing.io traces and spans.
# (See https://www.jaegertracing.io/docs/getting-started).
# (default: disabled)
#enable_tracing = true
# If enabled, the runtime will not create a network namespace for shim and hypervisor processes.
# This option may have some potential impacts to your host. It should only be used when you know what you're doing.
# `disable_new_netns` conflicts with `enable_netmon`
# `disable_new_netns` conflicts with `internetworking_model=tcfilter` and `internetworking_model=macvtap`. It works only
# with `internetworking_model=none`. The tap device will be in the host network namespace and can connect to a bridge
# (like OVS) directly.
# If you are using docker, `disable_new_netns` only works with `docker run --net=none`
# (default: false)
#disable_new_netns = true
# if enabled, the runtime will add all the kata processes inside one dedicated cgroup.
# The container cgroups in the host are not created, just one single cgroup per sandbox.
# The runtime caller is free to restrict or collect cgroup stats of the overall Kata sandbox.
# The sandbox cgroup path is the parent cgroup of a container with the PodSandbox annotation.
# The sandbox cgroup is constrained if there is no container type annotation.
# See: https://godoc.org/github.com/kata-containers/runtime/virtcontainers#ContainerType
sandbox_cgroup_only=false
# Enabled experimental feature list, format: ["a", "b"].
# Experimental features are features not stable enough for production,
# they may break compatibility, and are prepared for a big version bump.
# Supported experimental features:
# (default: [])
experimental=[]
From your previous feedback, seems your kata prefer /etc/kata-containers/configuration.toml. So you should add them to that file, not /usr/share/defaults/kata-containers/configuration.toml:
vhost_user_store_path = "/var/run/kata-containers/vhost-user"
enable_vhost_user_store = true
enable_hugepages = true
Another note is that if there is duplicated parameters like
enable_hugepages=false
enable_vhost_user_store=false
you also need to comment them out by #, directly modify them:
#enable_hugepages=false
#enable_vhost_user_store=false
Are you meaning duplication between /etc/kata-containers/configuration.toml and /usr/share/defaults/kata-containers/configuration.toml? I thought /etc/kata-containers/configuration.toml should always take higher priority, it should override the setting in /usr/share/defaults/kata-containers/configuration.toml, right?
Otherwise, I didn't see any duplication within the same file.
Also, I've add the settings you mentioned to /etc/kata-containers/configuration.toml rather than /usr/share/defaults/kata-containers/configuration.toml. The failure was still there.
coderplay
on 31 Mar 2020
Quick question, will /usr/share/defaults/kata-containers/configuration.toml also take effect with lower priority? Or kata-runtime won't read it from my setting?
coderplay
on 31 Mar 2020
yes, /etc/kata-containers/configuration.toml has a higher priority, so you need to add
enable_vhost_user_store = true
enable_hugepages = true
into it. I saw in your /etc/kata-containers/configuration.toml , hugepage enablement is not true.
I just verified this feature both on Ubuntu and Fedora with a clean installation. they all worked well without error.
Your case is really strange, if hugepage is not enabled, you will get this error:
docker: Error response from daemon: OCI runtime create failed: Vhost-user-blk/scsi is enabled without HugePages.
It is different from your error...
dong-liuliu
on 31 Mar 2020
Quick question, will
/usr/share/defaults/kata-containers/configuration.tomlalso take effect with lower priority? Or kata-runtime won't read it from my setting?
I think it would only chose one of them, not all of them. So I think the lower priority one will not take any effect.
dong-liuliu
on 31 Mar 2020
yeah, let me reinstall those packages and start everything over.
coderplay
on 31 Mar 2020
can you first try
yes,
/etc/kata-containers/configuration.tomlhas a higher priority, so you need to addenable_vhost_user_store = true enable_hugepages = trueinto it. I saw in your
/etc/kata-containers/configuration.toml, hugepage enablement is not true.
dong-liuliu
on 31 Mar 2020
Sure, I've already reinstalled everything. and added the 3 items you mentioned to /etc/kata-containers/configuration.toml. It worked! Thanks a lot!
coderplay
on 31 Mar 2020
😄1
👍1
Good Work! @coderplay
dong-liuliu
on 31 Mar 2020
thanks @dong-liuliu @coderplay , closing issue...
devimc
on 31 Mar 2020
Was this page helpful?
0 / 5 - 0 ratings
Related issues
mcastelino
·
8Comments
maartenvdezz
·
4Comments
AkihiroSuda
·
4Comments
eadamsintel
·
11Comments
gnawux
·
11Comments
Most helpful comment
Sure, I've already reinstalled everything. and added the 3 items you mentioned to
/etc/kata-containers/configuration.toml. It worked! Thanks a lot!