Runc: Kubernetes failures with latest runc

[dims]

By mistake we ended up using containerd 1.3.x with latest runc and ended up with some intermittent failures

@fuweid looked at it and here is the diagnosis:
https://github.com/kubernetes/kubernetes/issues/91579#issuecomment-639296103

@AkihiroSuda mentioned that there may be a fix in progress:
https://github.com/kubernetes/kubernetes/issues/91579#issuecomment-639297499

With the following PR:
https://github.com/opencontainers/runc/pull/2338

Opening this issue so we can follow up later to see if this is indeed fixed when we cut a new release of runc.

cc @liggitt @bentheelder

[AkihiroSuda]

Thanks for the report. Would it be possible to git bisect the issue?

[mrunalp]

Can you clarify what version of runc was used in the failing test? @dims said latest runc, the comments in the linked issue say rc10.

[BenTheElder]

rc10+dev .. we don't have the exact commits in these, since make release wipes out the commit and we were building "releases" ... (and the only reason we even had an untagged version is broken lookup of the commit from containerd vendor.conf)

EDIT: some of the (fine) runs were with rc10 actual, which was what we expected to have.

[BenTheElder]

bisecting may be possible but will be ... a bit expensive, It wasn't 100% reliable failure, we saw some flakiness with container deletion, more in the KIND jobs (presumably due to some resource starvation issues in the CI platform?) but also in kube-up GCE.

[mrunalp]

Have we ruled out any changes to containerd oom detection entirely?
Reason I ask is because we use runc master in crio master for running k8s e2e (using k8s master) and haven't run into this.
One big difference is that CRI-O defaults to systemd cgroup driver while containerd uses cgroupfs.

[BenTheElder]

containerd with rc10 is working fine across many (containerd, kubernetes) versions, I don't think anyone has ruled out OOM detection though.

One big difference is that CRI-O defaults to systemd cgroup driver while containerd uses cgroupfs.

I'm pretty sure this is the relevant difference here.

[AkihiroSuda]

@dims Could you try the latest ?

[AkihiroSuda]

This is almost the last blocker for rc91.

If this issue is still reproducible with the current master, anyone please post a reproducer for a local laptop?

[BenTheElder]

I'll try to rig up some tests in Kubernetes CI, I intended to do it earlier this week but we had unrelated broad issues (looking a bit better now).

I don't supposed there are existing binaries / nightlies anywhere?

[AkihiroSuda]

I don't supposed there are existing binaries / nightlies anywhere?

https://github.com/AkihiroSuda/moby-snapshot

The 2020-06-19 release contains runc binary built from 9748b48742fad065666bcded00f2db5ded7e44c1

[BenTheElder]

I think this is picking up the nightly against containerd 1.3.4 + kubernetes at head now here:
https://github.com/kubernetes/kubernetes/pull/92333#issuecomment-647930308
will run it more times and see if flakes crop up.
EDIT: pull-kubernetes-e2e-gce-ubuntu-containerd is the relevant job.

[AkihiroSuda]

What's current status?

[dims]

@AkihiroSuda we'll run it a few more times in the weekend, let's circle back on this next monday. I've cloned Ben's PR into https://github.com/kubernetes/kubernetes/pull/92540 so we can run it more times.

[dims]

@AkihiroSuda @BenTheElder based on the runs i have in https://github.com/kubernetes/kubernetes/pull/92540, feels like we are out of the woods as i do not see the specific problem we saw before with the e2e test mentioned in https://github.com/kubernetes/kubernetes/pull/92333#issuecomment-649520905

[AkihiroSuda]

Thanks, can we close this issue and cut a release for rc91?

cc @cyphar @mrunalp

[dims]

@AkihiroSuda yes please!