Cluster-api-provider-aws: [Investigate] Failure creating new cluster

[ashish-amarnath]

/kind bug

What steps did you take and what happened:
[A clear and concise description of what the bug is.]

1. make clean manifests-dev
2. Edit cluster name and ssh key
3. clusterctl create with all params

New cluster failed to get created because the cluster network status wasn't getting updated.

E1126 05:57:54.840455       1 scope.go:122] Failed to update status for cluster "ashisha-2": the server could not find the requested resource (put clusters.cluster.k8s.io ashisha-2)

What did you expect to happen:
New cluster to be created.

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]
To debug the problem, I am running a private build of the repo.
Creating this issue for tracking purposes only.

Environment:

k8s_cluster=minikube$ kc version
Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.3", GitCommit:"a4529464e4629c21224b3d52edfe0ea91b072862", GitTreeState:"clean", BuildDate:"2018-09-10T11:44:36Z", GoVersion:"go1.11", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.0", GitCommit:"fc32d2f3698e36b93322a3465f63a14e9f0eaead", GitTreeState:"clean", BuildDate:"2018-03-26T16:44:10Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

k8s_cluster=minikube$ minikube version
minikube version: v0.27.0

• Cluster-api-provider-aws version:
• Kubernetes version: (use kubectl version):
• OS (e.g. from /etc/os-release):

[ashish-amarnath]

/assign

[ashish-amarnath]

aamarnath@aamarnath-ltm0:~/go/src/sigs.k8s.io/cluster-api-provider-aws on my-dev [!]
k8s_cluster=minikube$ kc get clusters
NAME        CREATED AT
ashisha-2   6m

aamarnath@aamarnath-ltm0:~/go/src/sigs.k8s.io/cluster-api-provider-aws on my-dev [!]
k8s_cluster=minikube$ kc get machines
NAME                     CREATED AT
aws-controlplane-xb9w5   6m

from aws-provider-controller-manager-0 logs in the private build with enhanced debugging logs

I1126 05:52:17.395892       1 controller.go:124] reconciling cluster object ashisha-2 triggers idempotent reconcile.
E1126 05:52:17.396374       1 actuator.go:67] cluster.Status.ProviderStatus is nil
E1126 05:52:20.715101       1 scope.go:110] Updated status for cluster "ashisha-2"(encoded): "{\"metadata\":{\"creationTimestamp\":null},\"region\":\"us-east-1\",\"network\":{\"vpc\":{\"id\":\"vpc-01ca5c1431f84d048\",\"cidrBlock\":\"10.0.0.0/16\"},\"internetGatewayId\":\"igw-04d56297fd77678a6\",\"securityGroups\":{\"bastion\":{\"id\":\"sg-07cd2133770e0c9c9\",\"name\":\"ashisha-2-bastion\",\"ingressRule\":[{\"description\":\"SSH\",\"protocol\":\"tcp\",\"fromPort\":22,\"toPort\":22,\"cidrBlocks\":[\"0.0.0.0/0\"],\"sourceSecurityGroupId\":null}]},\"controlplane\":{\"id\":\"sg-0703a9c77957e6df2\",\"name\":\"ashisha-2-controlplane\",\"ingressRule\":[{\"description\":\"Kubernetes API\",\"protocol\":\"tcp\",\"fromPort\":6443,\"toPort\":6443,\"cidrBlocks\":[\"0.0.0.0/0\"],\"sourceSecurityGroupId\":null},{\"description\":\"SSH\",\"protocol\":\"tcp\",\"fromPort\":22,\"toPort\":22,\"cidrBlocks\":null,\"sourceSecurityGroupId\":\"sg-07cd2133770e0c9c9\"},{\"description\":\"etcd\",\"protocol\":\"tcp\",\"fromPort\":2379,\"toPort\":2379,\"cidrBlocks\":null,\"sourceSecurityGroupId\":\"sg-0703a9c77957e6df2\"},{\"description\":\"etcd peer\",\"protocol\":\"tcp\",\"fromPort\":2380,\"toPort\":2380,\"cidrBlocks\":null,\"sourceSecurityGroupId\":\"sg-0703a9c77957e6df2\"},{\"description\":\"bgp (calico)\",\"protocol\":\"tcp\",\"fromPort\":179,\"toPort\":179,\"cidrBlocks\":null,\"sourceSecurityGroupId\":\"sg-092c4b76104353f53\"}]},\"node\":{\"id\":\"sg-092c4b76104353f53\",\"name\":\"ashisha-2-node\",\"ingressRule\":[{\"description\":\"Node Port Services\",\"protocol\":\"tcp\",\"fromPort\":30000,\"toPort\":32767,\"cidrBlocks\":[\"0.0.0.0/0\"],\"sourceSecurityGroupId\":null},{\"description\":\"SSH\",\"protocol\":\"tcp\",\"fromPort\":22,\"toPort\":22,\"cidrBlocks\":null,\"sourceSecurityGroupId\":\"sg-07cd2133770e0c9c9\"},{\"description\":\"Kubelet API\",\"protocol\":\"tcp\",\"fromPort\":10250,\"toPort\":10250,\"cidrBlocks\":null,\"sourceSecurityGroupId\":\"sg-0703a9c77957e6df2\"},{\"description\":\"bgp (calico)\",\"protocol\":\"tcp\",\"fromPort\":179,\"toPort\":179,\"cidrBlocks\":null,\"sourceSecurityGroupId\":\"sg-092c4b76104353f53\"}]}},\"subnets\":[{\"id\":\"subnet-07eb7d3b862aad726\",\"vpcId\":\"vpc-01ca5c1431f84d048\",\"availabilityZone\":\"us-east-1a\",\"cidrBlock\":\"10.0.0.0/24\",\"public\":false,\"routeTableId\":null,\"natGatewayId\":null},{\"id\":\"subnet-0b276033221f1bc27\",\"vpcId\":\"vpc-01ca5c1431f84d048\",\"availabilityZone\":\"us-east-1a\",\"cidrBlock\":\"10.0.1.0/24\",\"public\":true,\"routeTableId\":null,\"natGatewayId\":null}],\"apiServerElb\":{}},\"bastion\":{\"id\":\"\"}}"
E1126 05:52:20.715187       1 scope.go:111] Network status for cluster "ashisha-2":
E1126 05:52:20.715193       1 scope.go:112] Subnets (count = 2)
E1126 05:52:20.715198       1 scope.go:114] 0: "id=subnet-07eb7d3b862aad726/az=us-east-1a/public=false"
E1126 05:52:20.715202       1 scope.go:114] 1: "id=subnet-0b276033221f1bc27/az=us-east-1a/public=true"
E1126 05:52:20.715204       1 scope.go:116] API server ELB=""
E1126 05:52:20.717560       1 scope.go:122] Failed to update status for cluster "ashisha-2": the server could not find the requested resource (put clusters.cluster.k8s.io ashisha-2)
E1126 05:52:20.718041       1 scope.go:136] [scope] failed to store provider status for cluster "ashisha-2" in namespace "default": the server could not find the requested resource (put clusters.cluster.k8s.io ashisha-2)
E1126 05:52:20.718497       1 controller.go:131] Error reconciling cluster object ashisha-2; unable to reconcile network: failed to authorize security group "sg-0703a9c77957e6df2" ingress rules: [protocol=tcp/range=[179-179]/description=bgp (calico)]: InvalidPermission.Duplicate: the specified rule "peer: sg-0703a9c77957e6df2, TCP, from port: 179, to port: 179, ALLOW" already exists
    status code: 400, request id: d53e9184-aa62-4c7a-8a3c-d1304cf790f0

The last error message is kinda misleading.

[ashish-amarnath]

https://github.com/kubernetes-sigs/cluster-api-provider-aws/pull/424 seems to have addressed this issue. I built the controller image from master in my private image repo and was able to use that to spin up a cluster.

$ kc get clusters
NAME       AGE
ashisha1   13m

$ kc get machines
NAME                     AGE
aws-controlplane-hjpz6   13m
aws-node-9lmbs           13m
$

[ashish-amarnath]

/close

[k8s-ci-robot]

@ashish-amarnath: Closing this issue.

In response to this:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.