Amazon-vpc-cni-k8s: Error message: failed to setup network: setup NS network: failed to add host route: file exists

[max-rocket-internet]

In pod events:

Events:
  Type     Reason                  Age   From                                 Message
  ----     ------                  ----  ----                                 -------
  Normal   Scheduled               4m    default-scheduler                    Successfully assigned us01-stg-spinnaker-spinn-halyard-0 to ip-10-0-22-21.ec2.internal
  Normal   SuccessfulMountVolume   4m    kubelet, ip-10-0-22-21.ec2.internal  MountVolume.SetUp succeeded for volume "halyard-config"
  Normal   SuccessfulMountVolume   4m    kubelet, ip-10-0-22-21.ec2.internal  MountVolume.SetUp succeeded for volume "us01-stg-spinnaker-spinn-halyard-token-zcg7q"
  Normal   SuccessfulMountVolume   4m    kubelet, ip-10-0-22-21.ec2.internal  MountVolume.SetUp succeeded for volume "reg-secrets"
  Normal   SuccessfulAttachVolume  4m    attachdetach-controller              AttachVolume.Attach succeeded for volume "pvc-d17d527b-aad3-11e8-81e7-xxxxxxxxx"
  Normal   SuccessfulMountVolume   4m    kubelet, ip-10-0-22-21.ec2.internal  MountVolume.SetUp succeeded for volume "pvc-d17d527b-aad3-11e8-81e7-xxxxxxxxx"
  Warning  FailedCreatePodSandBox  4m    kubelet, ip-10-0-22-21.ec2.internal  Failed create pod sandbox: rpc error: code = Unknown desc = NetworkPlugin cni failed to set up pod "us01-stg-spinnaker-spinn-halyard-0_default" network: add command: failed to setup network: setup NS network: failed to add host route: file exists
  Normal   SandboxChanged          4m    kubelet, ip-10-0-22-21.ec2.internal  Pod sandbox changed, it will be killed and re-created.
  Normal   Pulled                  4m    kubelet, ip-10-0-22-21.ec2.internal  Container image "gcr.io/spinnaker-marketplace/halyard:stable" already present on machine
  Normal   Created                 4m    kubelet, ip-10-0-22-21.ec2.internal  Created container

Pod is created and started OK but these errors are still in the events and look bad.

[max-rocket-internet]

I've emailed the aws-cni-support.tar.gz file to [email protected].

[max-rocket-internet]

I've had this issue again today and emailed you the details.

[max-rocket-internet]

Still seeing this:

  Warning  FailedCreatePodSandBox  2s (x4 over 5s)  kubelet, ip-10-0-27-182.eu-west-1.compute.internal  Failed create pod sandbox: rpc error: code = Unknown desc = NetworkPlugin cni failed to set up pod "elder-horse-postgres-controller-68756dc9b8-jt9l9_default" network: add command: failed to setup network: setup NS network: failed to add host route: file exists

[Deepthidharwar]

I am seeing these issues too on AWS while trying to deploy pod using the amazon-vpc-cni

[ERROR] Failed to setup NS network setup NS network: failed to add default route: file exists
Can some one please look into this.

[aleerizw-zz]

@liwenwu-amazon
We are getting the same issue in our production environment with amazon-vpc-cni-k8s v1.2.1 on kops (kubernetes 1.10.6).

[liwenwu-amazon]

Here are error msgs from plugin.log.xxx file

2018-09-24T09:28:22Z [INFO] Received add network response for pod <xxxx>  namespace default container e54bb96550d63c07d5a8a80c48793729211f3fea5b355a3ebfcc30c555f0452e: 10.0.23.133, table 0
2018-09-24T09:28:22Z [ERROR] Failed SetupPodNetwork for pod <xxxx>  namespace default container e54bb96550d63c07d5a8a80c48793729211f3fea5b355a3ebfcc30c555f0452e: setup NS network: failed to add host route: file exists

[liwenwu-amazon]

here is snips of route.output

ip route show table all
default via 10.0.22.1 dev eth0
10.0.22.0/23 dev eth0 proto kernel scope link src 10.0.23.172
10.0.22.56 dev enic0a80e334d5 scope link
10.0.22.94 dev enid336d91eb1a scope link
10.0.22.105 dev enie7834888824 scope link
10.0.22.137 dev enieda03e67426 scope link
10.0.22.168 dev eni995da829552 scope link
10.0.22.173 dev enicd72a6e3741 scope link
10.0.22.183 dev enic42d8b48626 scope link
10.0.23.8 dev enic382c044fa7 scope link
10.0.23.44 dev eni5fdca8f8634 scope link
10.0.23.93 dev eni7ce0ea50d68 scope link
10.0.23.133 dev enia7153239861 scope link <---- existing
10.0.23.140 dev enie5b1ee2eef9 scope link
10.0.23.145 dev eni9092c5a5932 scope link
10.0.23.174 dev enifd23e38f68c scope link

[liwenwu-amazon]

here is the snips of ifconfig.out

enia7153239861: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::bc70:abff:fe37:ccfe  prefixlen 64  scopeid 0x20<link>
        ether be:70:ab:37:cc:fe  txqueuelen 0  (Ethernet)
        RX packets 2743  bytes 329200 (321.4 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 1832  bytes 229940 (224.5 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

[nitin302]

Since this fix is now merged, how can I get this deployed into my EKS cluster?

[max-rocket-internet]

@nitin302

kubectl apply -f https://raw.githubusercontent.com/aws/amazon-vpc-cni-k8s/master/config/v1.3/aws-k8s-cni.yaml

[tsuna]

I'm seeing this (or something that looks exactly like the symptoms/logs described above) in a new kops cluster with k8s 1.11.8 and the plugin version 1.3.0 (I know it's not the latest but I thought the fix was in 1.3.0).

[robottaway]

@tsuna please create another ticket and link to this one. I too ran into this issue and I am using 1.3.2. Issue for sure still exists, only one route to failure was fixed there must be others.

[tsuna]

Alright I filed a new issue.