Kube-router: Is restarting containerd after configuring container networking required?

Created on 15 Jan 2018 · 10Comments · Source: cloudnativelabs/kube-router

First, thank you for kube-router. I've been using it a lot lately and it works wonderfully!

I'm experiencing an issue related to bootstrapping a cluster that I am hoping you all can help with.

If I don't restart containerd after deploying my kube-router pods, any pod not connected to the host network fails to start.

I have a full example process, including the containerd restart documented here:
https://github.com/tkellen/kwm#first-time-user-guide

If you skip the containerd restart step you'll see that kube-dns (or any other pod not connected to the host network) fails to start with the error "Failed create pod sandbox". with. As soon as you restart containerd, the issue resolves itself. Is this expected? If not, can you help me understand what the correct process for this would be?

Thanks so much!

bug

Source

tkellen

All 10 comments

I've posted this over on the cri-containerd repo because I'm unclear if this is an issue with kube-router or cri-containerd (or neither, maybe it's just me!). Here's the link:
https://github.com/containerd/cri-containerd/issues/545

tkellen on 15 Jan 2018

Not sure of exact reason. But many users seems faced "Failed create pod sandbox" with various CNI plugins

https://github.com/kubernetes/kubernetes/issues/56902
https://github.com/kubernetes/kubeadm/issues/578
https://github.com/kubernetes/kubernetes/issues/55094

murali-reddy on 16 Jan 2018

Thanks @murali-reddy! It looks like we've gotten some traction over on containerd/cri-containerd#545 so I am going to close this.

tkellen on 16 Jan 2018

👍1

tl;dr Just coming to say that I just ran into this, and restarting cri-containerd/containerd is required.

I wasn't getting "Failed to create pod sandbox" errors, but the network was just silently failing. kube-router logs looked good and it wrote configuration like you'd expect, automatically detecting a cidr to use for my cluster (10.244.1.0/24), but I noticed that all the nodes actually had IPs like 10.244.0.41. I thought maybe this wasn't a problem (I also was using canal before switching to kube-router), but it was.

After much head scratching, checking configuration, creating tutum/curl, alpine, and nginx pods trying to access one from the other, I had the thought to systemctl restart cri-containerd containerd and that's what did it. I would have saved myself some time if I'd seen/read https://github.com/containerd/cri-containerd/issues/545 sooner

t3hmrman on 20 Feb 2018

Thanks @t3hmrman for sharing.

murali-reddy on 20 Feb 2018

👍1

@tkellen looks like you are still running into issues as noted in https://github.com/containerd/cri/issues/545 I am reopening here to track and work for resolution.

murali-reddy on 30 Apr 2018

@murali-reddy @tkellen We think that we should not support dynamically loading and changing cni config, because it will only break the system.

I'm not quite familiar with kube-router, but based on what @tkellen told me, it is doing multi-stage cni config initialization. Given that, I think it is safer to initialize the cni config in a different directory, and move the config to actual cni directly at the end. Or else, node may become ready during the initialization, and any pod created during initialization will be in a bad state.

Random-Liu on 2 May 2018

👍1

Isn't this fixed by containerd/cri/#825?

wilmardo on 8 Jul 2019

My testing showed that it was not but I have since moved on from bare-metal installations to using EKS. I appreciate all the attention and effort deeply but I'm not presently able to validate :/.

tkellen on 8 Jul 2019

Closing as stale given the status of the containerd ticket. Feel free to re-open with current specifics.