Runtime: Containers are not started when pod is configured with hugepages

[liangxianlong]

Description of problem

The kuebeletandkata-runtimeversion:

[root@host-69 yaml]# kubelet --version
Kubernetes v1.16.0
[root@host-69 yaml]# kata-runtime --version
kata-runtime  : 1.9.0
   commit   : c772b3e15b403745be4ec51e32d9a58f0d65e499
   OCI specs: 1.0.1-dev
[root@host-69 yaml]#

Set the sandbox_cgroup_onlytotrue:

# if enabled, the runtime will add all the kata processes inside one dedicated cgroup.
# The container cgroups in the host are not created, just one single cgroup per sandbox.
# The sandbox cgroup is not constrained by the runtime
# The runtime caller is free to restrict or collect cgroup stats of the overall Kata sandbox.
# The sandbox cgroup path is the parent cgroup of a container with the PodSandbox annotation.
# See: https://godoc.org/github.com/kata-containers/runtime/virtcontainers#ContainerType
sandbox_cgroup_only=true

The pod config,yaml:

apiVersion: v1
kind: Pod
metadata:
  name: centos-vlan
spec:
  runtimeClassName: kata-shimv1
  containers:
  - name: vlan-1
    image: centos:latest
    stdin: true
    tty: true
    imagePullPolicy: IfNotPresent
    volumeMounts:
    - mountPath: /dev/hugepages
      name: hugepage
    resources:
      limits:
        hugepages-2Mi: 512Mi
        memory: 512Mi
      requests:
        memory: 512Mi
  volumes:
  - name: hugepage
    emptyDir:
      medium: HugePages

The host hugepages configure:

[root@host-69 yaml]# sysctl vm.nr_hugepages=2048
vm.nr_hugepages = 2048
[root@host-69 yaml]# cat /proc/meminfo | grep -i huge                                         
AnonHugePages:    319488 kB
HugePages_Total:    2048
HugePages_Free:     2048
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
[root@host-69 yaml]#

Importantly, the kata also enable the hugepage:

# Enable huge pages for VM RAM, default false
# Enabling this will result in the VM memory
# being allocated using huge pages.
# This is useful when you want to use vhost-user network
# stacks within the container. This will automatically 
# result in memory pre allocation
enable_hugepages = true

Then run the pod:

[root@host-69 yaml]# kubectl apply -f centos-vlan.yaml 
pod/centos-vlan created
[root@host-69 yaml]# kubectl get pods                  
NAME          READY   STATUS              RESTARTS   AGE
centos-vlan   0/1     ContainerCreating   0          13s
[root@host-69 yaml]# kubectl describe pod centos-vlan
...
Events:
  Type     Reason                  Age        From               Message
  ----     ------                  ----       ----               -------
  Normal   Scheduled               <unknown>  default-scheduler  Successfully assigned default/centos-vlan to host-69
  Warning  FailedCreatePodSandBox  3s         kubelet, host-69   Failed create pod sandbox: rpc error: code = Unknown desc = failed to create containerd task: OCI runtime create failed: Failed to check if grpc server is working: rpc error: code = Unavailable desc = transport is closing: unknown

This is because kata-agent did not start successfully.

defer func() {
        if err != nil {
            s.hypervisor.stopSandbox()
        }
    }()

    // In case of vm factory, network interfaces are hotplugged
    // after vm is started.
    if s.factory != nil {
        endpoints, err := s.network.Add(s.ctx, &s.config.NetworkConfig, s.hypervisor, true)
        if err != nil {
            return err
        }

        s.networkNS.Endpoints = endpoints

        if s.config.NetworkConfig.NetmonConfig.Enable {
            if err := s.startNetworkMonitor(); err != nil {
                return err
            }
        }

        if !s.supportNewStore() {
            if err := s.store.Store(store.Network, s.networkNS); err != nil {
                return err
            }
        }
    }

    s.Logger().Info("VM started")

    // Once the hypervisor is done starting the sandbox,
    // we want to guarantee that it is manageable.
    // For that we need to ask the agent to start the
    // sandbox inside the VM.
    if err := s.agent.startSandbox(s); err != nil {
        return err
    }

Expected result

The pod should run successfully.

[root@host-69 yaml]# kubectl get pods
NAME          READY   STATUS              RESTARTS   AGE
centos-vlan   1/1     running   0          10m
[root@host-69 yaml]#

Actual result

[root@host-69 yaml]# kubectl get pods
NAME          READY   STATUS              RESTARTS   AGE
centos-vlan   0/1     ContainerCreating   0          10m
[root@host-69 yaml]#

I think there should be some modifications for both kata-runtime and kata-agent.

@liwei @bryteise @jbryce @gnawux @bergwolf @

[gnawux]

@liangxianlong Thanks for raising this issue, and I will keep follow this issue as there was practice on hugepages in our productions. It's really appreciated that you could raise issues when you had gotten into trouble.

However, I don't think you should involve so many unrelated developers (or not developers at alll, such as Jonathon) here, which may be annoying them and harm for them to receive related notifications. Just my two cents.

[liangxianlong]

@gnawux I'm sorry,I just don't know who to inform.

[gnawux]

@liangxianlong No worries, normally no issue will be ignored, the maintainers who read the issue first will point to related guys.

Hugepages is frequently used in production environments, and your issue is important for the project.

[ptptptptptpt]

Hi @liangxianlong, as far as I know, there is no need to add any hugepage-relative specs to pod yaml.

After enabling hugepage on host and in kata config, the new-created kata pod will use hugepage memory automatically.

[liangxianlong]

@ptptptptptpt
[Hugepages are not allocated when pod is deployed with k8s using kata-runtime

2109](https://github.com/kata-containers/runtime/issues/2109)

[
Add support for huge-pages with k8s empty-dir

1548](https://github.com/kata-containers/runtime/issues/1548)

[gnawux]

Looks like it's a missing feature for a long time.

@amshinde Do you know the current status?

[liangxianlong]

Correct,currently, it is the error caused by enable_hugepages = true and sandbox_cgroup_only=true.
my busybox-two.yam:

apiVersion: v1
kind: Pod
metadata:
  name: busybox-two
spec:
  runtimeClassName: kata-shimv1
  containers:
  - name: busybox-1
    image: busybox
    stdin: true
    tty: true
    imagePullPolicy: IfNotPresent
  - name: busybox-2
    image: busybox
    stdin: true
    tty: true
    imagePullPolicy: IfNotPresent

1. set enable_hugepages to trueonly

enable_hugepages = true
sandbox_cgroup_only = false

Run the busybox-two.yaml:

[root@host-69 yaml]# kubectl apply -f busybox-two.yaml
pod/busybox-two created
[root@host-69 yaml]# kubectl get pods
NAME          READY   STATUS    RESTARTS   AGE
busybox-two   2/2     Running   0          3s
[root@host-69 yaml]#

1. set sandbox_cgroup_only to trueonly

enable_hugepages = false
sandbox_cgroup_only = true

Run the busybox-two.yaml:

[root@host-69 yaml]# kubectl apply -f busybox-two.yaml
pod/busybox-two created
[root@host-69 yaml]# kubectl get pods
NAME          READY   STATUS              RESTARTS   AGE
busybox-two   0/2     ContainerCreating   0          3s
[root@host-69 yaml]# kubectl get pods
NAME          READY   STATUS    RESTARTS   AGE
busybox-two   2/2     Running   0          7s
[root@host-69 yaml]#

1. The two options are set to trueat the same time:

enable_hugepages = true
sandbox_cgroup_only = true

Run the busybox-two.yaml:

[root@host-69 yaml]# kubectl apply -f busybox-two.yaml
[root@host-69 yaml]# kubectl get pods
NAME          READY   STATUS              RESTARTS   AGE
busybox-two   0/2     ContainerCreating   0          35s
[root@host-69 yaml]# kubectl describe pod busybox-two
...
Events:
  Type     Reason                  Age                From               Message
  ----     ------                  ----               ----               -------
  Normal   Scheduled               <unknown>          default-scheduler  Successfully assigned default/busybox-two to host-69
  Warning  FailedCreatePodSandBox  23s (x2 over 54s)  kubelet, host-69   Failed create pod sandbox: rpc error: code = Unknown desc = failed to create containerd task: OCI runtime create failed: Failed to check if grpc server is working: rpc error: code = Unavailable desc = transport is closing: unknown

enable_hugepagesand sandbox_cgroup_only can not set to trueat the same time,shall we make some changes to modify this bug?
@gnawux

[gnawux]

@liangxianlong Could you submit the fix?

[grahamwhaley]

/cc @chavafg - is this something we could add to the CI test matrix?

[liangxianlong]

@gnawux，i need some time to figure out what cause this.

[chavafg]

@grahamwhaley yeah, definitely we need a test for hugepages and another with hugepages + sandbox_cgroups_only. Will open an issue on tests repo to add them.

[GabyCT]

@liangxianlong , I tried to run a k8s test with hugepages=true and sandbox_cgroups_only=false but the pods stay in created mode, here it is the reference for that test https://github.com/kata-containers/tests/pull/2071

[liangxianlong]

@GabyCT I am trying to solve this problem, but for the time being there is no progress.Today I will continue this work until I find the final reason.

[liangxianlong]

@gnawux Hi, now I am confused. I did not find out the specific reason for this failure.I tried to log in to the virtual machine and restart kata-agent, but I still can't log in if I use the debug image.Later, I started the pause container using kata-runtime alone, and I also configured enable_hugepages=true and sandbox_cgroup_only=true,it can start successfully.This means the bug only appers in k8s.Maybe i need some help or suggestions.Can you help me?

[gnawux]

@amshinde Any comments here?

[liangxianlong]

@gnawux @amshinde There are some advances and attempts.
1.turn on all debug options,we can see the kvm run failed

[root@host-69 yaml]# cat /run/vc/vm/17659326091092d9b10190d3edecde74bece68e4529d71138713c972f8f34a13/qemu.log
device_mem_size align up to 0x1a00000000
vplat x86_cpu_machine_reset_cb 4716: vcpu =0 thread_id=22580
error: kvm run failed Bad address
EAX=00000000 EBX=00000000 ECX=00000010 EDX=000f1bcd
ESI=00000000 EDI=00000000 EBP=00000000 ESP=00007000
EIP=000f1bcd EFL=00010006 [-----P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =0010 00000000 ffffffff 00c09300 DPL=0 DS   [-WA]
CS =0008 00000000 ffffffff 00c09b00 DPL=0 CS32 [-RA]
SS =0010 00000000 ffffffff 00c09300 DPL=0 DS   [-WA]
DS =0010 00000000 ffffffff 00c09300 DPL=0 DS   [-WA]
FS =0010 00000000 ffffffff 00c09300 DPL=0 DS   [-WA]
GS =0010 00000000 ffffffff 00c09300 DPL=0 DS   [-WA]
LDT=0000 00000000 0000ffff 00008200 DPL=0 LDT
TR =0000 00000000 0000ffff 00008b00 DPL=0 TSS32-busy
GDT=     000f6140 00000037
IDT=     000f617e 00000000
CR0=00000011 CR2=00000000 CR3=00000000 CR4=00000000
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000 
DR6=00000000ffff0ff0 DR7=0000000000000400
EFER=0000000000000000
Code=c0 e2 0f 00 0f b7 d2 8d 44 24 02 e8 81 c6 ff ff 83 c4 28 c3 <53> 83 ec 2c e8 51 c9 ff ff bb 00 00 00 40 8d 44 24 18 89 44 24 04 8d 44 24 14 89 04 24 8d
[root@host-69 yaml]# cat /proc/meminfo | grep -i huge
AnonHugePages:    391168 kB
HugePages_Total:    1024
HugePages_Free:     1024
HugePages_Rsvd:     1024
HugePages_Surp:        0
Hugepagesize:       2048 kB
[root@host-69 yaml]#

2.modify kata-runtime code

@@ -1045,6 +1046,18 @@ func (s *Sandbox) startVM() (err error) {

        s.Logger().Info("VM started")

+       pid := s.hypervisor.getPids()
+       s.Logger().Infof("hypervisor pid is %d", pid[0])
+       time.Sleep(time.Duration(1)*time.Second)
+       cgroup, err := cgroupsNewFunc(cgroups.V1, cgroups.StaticPath(s.state.CgroupPath), &specs.LinuxResources{})
+       if err != nil {
+               return fmt.Errorf("Could not create sandbox cgroup in %v: %v", s.state.CgroupPath, err)
+
+       }
+       if err := cgroup.Add(cgroups.Process{Pid: pid[0]}); err != nil {
+               return fmt.Errorf("Could not add runtime PID %d to sandbox cgroup:  %v", pid, err)
+       }

@@ -2113,11 +2126,12 @@ func (s *Sandbox) setupSandboxCgroup() error {
                s.Logger().WithField("sandboxid", s.id).Warning("no cgroup path provided for pod sandbox, not creating sandbox cgroup")
                return nil
        }
+       //time.Sleep(time.Duration(120)*time.Second)
        validContainerCgroup := utils.ValidCgroupPath(spec.Linux.CgroupsPath)

        // Create a Kata sandbox cgroup with the cgroup of the sandbox container as the parent
        s.state.CgroupPath = filepath.Join(filepath.Dir(validContainerCgroup), cgroupKataPrefix+"_"+s.id)
-       cgroup, err := cgroupsNewFunc(cgroups.V1, cgroups.StaticPath(s.state.CgroupPath), &specs.LinuxResources{})
+       /*cgroup, err := cgroupsNewFunc(cgroups.V1, cgroups.StaticPath(s.state.CgroupPath), &specs.LinuxResources{})
        if err != nil {
                return fmt.Errorf("Could not create sandbox cgroup in %v: %v", s.state.CgroupPath, err)

@@ -2127,7 +2141,7 @@ func (s *Sandbox) setupSandboxCgroup() error {
        runtimePid := os.Getpid()
        if err := cgroup.Add(cgroups.Process{Pid: runtimePid}); err != nil {
                return fmt.Errorf("Could not add runtime PID %d to sandbox cgroup:  %v", runtimePid, err)
-       }
+       }*/

And the container can run successfully with k8s.

[amshinde]

@gnawux Missed this, will try to take a look at this in a couple of days.

[liangxianlong]

@gnawux @amshinde I have found the reason. This is because k8s does not inherit the Hugetlb when creating the pod, but clears the hugetlb-control group hugetlb.2MB.limit_in_bytes to zero.hugetlb.2MB.limit_in_bytes is very important for huge page usage. When putting the qemu process into the hugetlb-control group, you must ensure that the value of hubetlb.2MB.limit_in_bytes is greater than or equal to the huge page that qemu needs to use.We can see the kernel code,Please pay attention to functionhugetlb_cgroup_charge_cgroupand the functionpage_counter_try_charge will make the problem clearer.

struct page *alloc_huge_page(struct vm_area_struct *vma,
                    unsigned long addr, int avoid_reserve)
{
    struct hugepage_subpool *spool = subpool_vma(vma);
    struct hstate *h = hstate_vma(vma);
    struct page *page;
    long map_chg, map_commit;
    long gbl_chg;
    int ret, idx;
    struct hugetlb_cgroup *h_cg;
        ...
    ret = hugetlb_cgroup_charge_cgroup(idx, pages_per_huge_page(h), &h_cg);
    ...
out_subpool_put:
    if (map_chg || avoid_reserve)
        hugepage_subpool_put_pages(spool, 1);
    vma_end_reservation(h, vma, addr);
    return ERR_PTR(-ENOSPC);
}

/**
 * page_counter_try_charge - try to hierarchically charge pages
 * @counter: counter
 * @nr_pages: number of pages to charge
 * @fail: points first counter to hit its limit, if any
 *
 * Returns 0 on success, or -ENOMEM and @fail if the counter or one of
 * its ancestors has hit its configured limit.
 */
int page_counter_try_charge(struct page_counter *counter,
                unsigned long nr_pages,
                struct page_counter **fail)
{
    struct page_counter *c;

    for (c = counter; c; c = c->parent) {
        long new;
        /*
         * Charge speculatively to avoid an expensive CAS.  If
         * a bigger charge fails, it might falsely lock out a
         * racing smaller charge and send it into reclaim
         * early, but the error is limited to the difference
         * between the two sizes, which is less than 2M/4M in
         * case of a THP locking out a regular page charge.
         *
         * The atomic_long_add_return() implies a full memory
         * barrier between incrementing the count and reading
         * the limit.  When racing with page_counter_limit(),
         * we either see the new limit or the setter sees the
         * counter has changed and retries.
         */
        new = atomic_long_add_return(nr_pages, &c->count);
        if (new > c->limit) {
            atomic_long_sub(nr_pages, &c->count);
            /*
             * This is racy, but we can live with some
             * inaccuracy in the failcnt.
             */
            c->failcnt++;
            *fail = c;
            goto failed;
        }
        /*
         * Just like with failcnt, we can live with some
         * inaccuracy in the watermark.
         */
        if (new > c->watermark)
            c->watermark = new;
    }
    return 0;

failed:
    for (c = counter; c != *fail; c = c->parent)
        page_counter_cancel(c, nr_pages);

    return -ENOMEM;
}

I think,we should drive the k8s community to solve the problem.

[liangxianlong]

@gnawux @amshinde Hi,This problem has been going on for a long time, and we should find a way to promote the k8s community to solve it.

[liangxianlong]

I make a sum in https://github.com/kata-containers/runtime/issues/2353