Cluster-api: Bootstrap cluster cleaned up despite failed pivot

[dlipovetsky]

/kind bug

When the clusterdeployer.Create method fails _after_ deploying the control plane machine, but _before_ pivoting (i.e. copying over the CAPI and CAPA control planes, and Cluster, Machine, and ancillary objects to the workload cluster), the bootstrap cluster is cleaned up.

Without the Cluster, Machine, and ancillary objects, it is not possible to use clusterctl to delete the AWS resources. The AWS resources must be deleted manually.

What steps did you take and what happened:
I deployed the cluster:

clusterctl  create cluster -v 5   --bootstrap-type kind   --provider aws   -m ./aws-examples/out/machines.yaml   -c ./aws-examples/out/cluster.yaml   -p ./aws-examples/out/provider-components.yaml 

But, importantly, I forgot to set the AWS_PROFILE environment variable. So when it came time for clusterdeployer.Create to update the cluster endpoint by making a direct AWS API call, it was unable to do so:

I0715 15:59:02.922821   32034 clusterdeployer.go:125] Saving provider components to the target cluster                                                                       
I0715 15:59:03.296593   32034 clusterdeployer.go:133] Updating target cluster object with control plane endpoint running on machine                                          
I0715 15:59:23.687017   32034 createbootstrapcluster.go:36] Cleaning up bootstrap cluster.                                                                                   
I0715 15:59:23.687079   32034 kind.go:69] Running: kind [delete cluster --name=clusterapi]
I0715 15:59:25.390908   32034 kind.go:72] Ran: kind [delete cluster --name=clusterapi] Output: Deleting cluster "clusterapi" ...                                             
$KUBECONFIG is still set to use /home/daniel/.kube/kind-config-clusterapi even though that file has been deleted, remember to unset it                                       
F0715 15:59:25.390952   32034 create_cluster.go:61] unable to update target cluster endpoint: unable to get cluster endpoint: unexpected aws error: NoCredentialProviders: n$
 valid providers in chain. Deprecated.                                                                                                                                       
        For verbose messaging see aws.Config.CredentialsChainVerboseErrors

What did you expect to happen:
The bootstrap cluster should not have been cleaned up.

Anything else you would like to add:
The bootstrap cluster is cleaned up in a defer call. If the pivot fails, the bootstrap cluster must not be cleaned up.

Environment:

• Cluster-api-provider-aws version: v0.3.4
• Kubernetes version: (use kubectl version): v1.14.1
• OS (e.g. from /etc/os-release): Fedora 29

[dlipovetsky]

If the pivot fails, the bootstrap cluster must not be cleaned up.

If that's uncontroversial, I'm happy to take up the fix.

[chuckha]

There is a flag for this!

clusterctl create cluster --bootstrap-cluster-cleanup=false <other args>...

However this issue might be about the concept that the cluster always tries to clean up but should be smarter about how it does so. In which case, I think this should be an issue logged in cluster-api and cc'd @timothysc

[dlipovetsky]

Thanks @chuckha! I filed this on the wrong repo.

Does an admin want move this issue to cluster-api, or should I just close and file a new one?

[vincepri]

I can try, not sure what's gonna happen though :D

[vincepri]

Seems it worked 🎉