Runtime: The number of percpu_usages is incorrect

[ZeroMagic]

Description of problem

When I used func (s *Sandbox) StatsContainer(containerID string) (ContainerStats, error) to get stats of the container, I found the number of percpu_usages was not consistent with the cpuinfo inside the container.

Then I searched the related code in kata-runtime and kata-agent. I could not find any mistakes. The function just use libcontainer's stats() interface to get stats.

So I think there are some mistakes in qemu. But I am not familiar with the qemu. Could anyone help me with the issue ?

My host info:

$cat /proc/cpuinfo| grep "processor"| wc -l
24

The kata configuration.toml:

default_vcpus = 1
default_maxvcpus = 0

The kata container info which I get from inside the container:

~ # cat /proc/cpuinfo
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 45
model name      : Intel(R) Xeon(R) CPU E5-2630 0 @ 2.30GHz
stepping        : 7
microcode       : 0x1
cpu MHz         : 2299.998
cache size      : 16384 KB
physical id     : 0
siblings        : 1
core id         : 0
cpu cores       : 1
apicid          : 0
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl xtopology pni pclmulqdq ssse3 cx16 pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx hypervisor lahf_lm tsc_adjust xsaveopt arat
bugs            : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf
bogomips        : 4599.99
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:

The guest log:

time="2019-05-09T11:17:16.305294421+08:00" level=debug msg="reading guest console" 
ID=test8 sandbox=test8 source=virtcontainers subsystem=kata_agent 
vmconsole="time=\"2019-05-09T03:17:16.254315739Z\" level=debug 
msg=\"request end\" 
duration=\"955.621µs\" name=kata-agent pid=103 
request=/grpc.AgentService/StatsContainer 
resp=\"cgroup_stats:<cpu_stats:<cpu_usage:<total_usage:39279514 percpu_usage:39279514 percpu_usage:0 percpu_usage:0 percpu_usage:0 percpu_usage:0 percpu_usage:0 percpu_usage:0 percpu_usage:0 percpu_usage:0 percpu_usage:0 percpu_usage:0 percpu_usage:0 percpu_usage:0 percpu_usage:0 percpu_usage:0 percpu_usage:0 percpu_usage:0 percpu_usage:0 percpu_usage:0 percpu_usage:0 percpu_usage:0 percpu_usage:0 percpu_usage:0 percpu_usage:0 usage_in_kernelmode:20000000 > throttling_data:<> > 

Expected result

In the above environment, it should return only one percpu_usage.

Actual result

It returns 24 percpu_usages. The number is not consistent with what I config. But the number is equal to that in host.

---

[ZeroMagic]

ping @xuliker

[grahamwhaley]

@devimc @jcvenegas for any input on the kata side, and maybe @dagrh might have some thoughts from the qemu side.

[dagrh]

I'm guessing (without checking) that kata is starting qemu with max_cpus=host but then only actually starting one; and something somewhere (runtime?) is listing the state of all of them.

[jodh-intel]

I think that's correct @dagrh:

$ grep -c ^processor /proc/cpuinfo 
4
$ docker run --runtime kata-runtime -ti busybox grep -c ^processor /proc/cpuinfo
1

When running on my test system, qemu is using -smp 1,cores=1,threads=1,sockets=4,maxcpus=4.

The agent asks libcontainer for stats (implicitly for all cpus) here:

• https://github.com/kata-containers/agent/blob/master/grpc.go#L1180

Hence one of the following is going to have to happen I think:

• libcontainer augments these stats with a bool showing if the cpu is online/offline.
• libcontainer is changed to only return online cpu stat details.
• we change the agent so that agentGRPC.StatsContainer() only returns stats for onlined cpus.
• we change the agent so that agentGRPC.StatsContainer() adds in cpu online/offline details so the runtime can display whatever is appropriate.

[ZeroMagic]

Thanks for your analysis. @jodh-intel @dagrh @grahamwhaley

I don't know what scenario the offline CPU information is for.
If none, the third option will be a better one.