Cilium: Pod can be stuck in ContainerCreating state for ~15m with identity-allocation-mode=kvstore

[christarazi]

Bug report

A Pod remains stuck in ContainerCreating state for up to 15m with identity-allocation-mode=kvstore. After ~15m, the pod transitions to Running state, and everything proceeds normally. This delay (presumably in identity allocation) occurs when at least one etcd node is down. If the etcd cluster has quorum (e.g. 2/3), then this delay should not happen.

General Information

• 1.7.7 & 1.8.2
• K8s with etcd backends
• Sysdumps:
  
  
  
  • cilium-sysdump-20200820-185312-1.8.2.zip
  
  
  • cilium-sysdump-20200820-181830-1.7.7-heartbeats-after-pod-running.zip
  
  
  • cilium-sysdump-20200820-181535-1.7.7-heartbeats-during-containercreating.zip
  
  

How to reproduce the issue

1. Deploy K8s cluster with etcd nodes
2. Deploy Cilium with the following config:
   
   
   
   • kvstore: etcd
   
   
   • identity-allocation-mode: kvstore
   
   
3. Block one etcd node via https://gist.github.com/christarazi/11aadf01d353112eb10ed82373569155. Note this must be done before deploying the pods in the next step.
4. Deploy nginx (ensure that at least one replica lands on the Cilium instance that you've applied the above etcd iptables rules to):

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
    cool: stuff
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 80

[aanm]

This can easily be replicated it the dev environment:

docker run -d --network host \
 --name etcd0 quay.io/coreos/etcd:v3.3.20 \
  etcd \
 -name etcd0 \
 -advertise-client-urls http://192.168.33.11:23790 \
 -listen-client-urls http://0.0.0.0:23790 \
 -initial-advertise-peer-urls http://192.168.33.11:23800 \
 -listen-peer-urls http://0.0.0.0:23800 \
 -initial-cluster-token etcd-cluster-1 \
 -initial-cluster etcd0=http://192.168.33.11:23800,etcd1=http://192.168.33.11:23801,etcd2=http://192.168.33.11:23802 \
 -initial-cluster-state new
docker run -d --network host \
 --name etcd1 quay.io/coreos/etcd:v3.3.20 \
  etcd \
 -name etcd1 \
 -advertise-client-urls http://192.168.33.11:23791 \
 -listen-client-urls http://0.0.0.0:23791 \
 -initial-advertise-peer-urls http://192.168.33.11:23801 \
 -listen-peer-urls http://0.0.0.0:23801 \
 -initial-cluster-token etcd-cluster-1 \
 -initial-cluster etcd0=http://192.168.33.11:23800,etcd1=http://192.168.33.11:23801,etcd2=http://192.168.33.11:23802 \
 -initial-cluster-state new
docker run -d --network host \
 --name etcd2 quay.io/coreos/etcd:v3.3.20 \
  etcd \
 -name etcd2 \
 -advertise-client-urls http://192.168.33.11:23792 \
 -listen-client-urls http://0.0.0.0:23792 \
 -initial-advertise-peer-urls http://192.168.33.11:23802 \
 -listen-peer-urls http://0.0.0.0:23802 \
 -initial-cluster-token etcd-cluster-1 \
 -initial-cluster etcd0=http://192.168.33.11:23800,etcd1=http://192.168.33.11:23801,etcd2=http://192.168.33.11:23802 \
 -initial-cluster-state new