Cluster-api: clusterctl panics on init

[jichenjc]

What steps did you take and what happened:
[A clear and concise description on how to REPRODUCE the bug.]

What did you expect to happen:

I am using openstack provider and doing
clusterctl init --infrastructure openstack --kubeconfig ./capi-quickstart.kubeconfig

first time it last for long time (a few minutes and I ctrl + c)

# clusterctl init --infrastructure openstack --kubeconfig ./capi-quickstart.kubeconfig
Fetching providers
Installing cert-manager
Waiting for cert-manager to be available...
^C

# kubectl --kubeconfig=./capi-quickstart.kubeconfig get nodes
NAME                                 STATUS   ROLES    AGE   VERSION
capi-openstack-control-plane-h6l47   Ready    master   83m   v1.17.3
capi-openstack-md-0-jzb69            Ready    <none>   81m   v1.17.3

run the command again, it panic ...

# clusterctl init --infrastructure openstack --kubeconfig ./capi-quickstart.kubeconfig -v 10
No default config file available
Fetching providers
Fetching File="core-components.yaml" Provider="cluster-api" Version="v0.3.7"
Fetching File="bootstrap-components.yaml" Provider="bootstrap-kubeadm" Version="v0.3.7"
Fetching File="control-plane-components.yaml" Provider="control-plane-kubeadm" Version="v0.3.7"
Fetching File="infrastructure-components.yaml" Provider="infrastructure-openstack" Version="v0.3.1"
Fetching File="metadata.yaml" Provider="cluster-api" Version="v0.3.7"
Fetching File="metadata.yaml" Provider="bootstrap-kubeadm" Version="v0.3.7"
Fetching File="metadata.yaml" Provider="control-plane-kubeadm" Version="v0.3.7"
Fetching File="metadata.yaml" Provider="infrastructure-openstack" Version="v0.3.1"
Installing Provider="cluster-api" Version="v0.3.7" TargetNamespace="capi-system"
Creating shared objects Provider="cluster-api" Version="v0.3.7"
Creating Namespace="capi-webhook-system"
Creating CustomResourceDefinition="clusterresourcesetbindings.addons.cluster.x-k8s.io"
Creating CustomResourceDefinition="clusterresourcesets.addons.cluster.x-k8s.io"
Creating CustomResourceDefinition="clusters.cluster.x-k8s.io"
Creating CustomResourceDefinition="machinedeployments.cluster.x-k8s.io"
Creating CustomResourceDefinition="machinehealthchecks.cluster.x-k8s.io"
Creating CustomResourceDefinition="machinepools.exp.cluster.x-k8s.io"
Creating CustomResourceDefinition="machines.cluster.x-k8s.io"
Creating CustomResourceDefinition="machinesets.cluster.x-k8s.io"
Creating MutatingWebhookConfiguration="capi-mutating-webhook-configuration"
Creating Service="capi-webhook-service" Namespace="capi-webhook-system"
Creating Deployment="capi-controller-manager" Namespace="capi-webhook-system"
Creating Certificate="capi-serving-cert" Namespace="capi-webhook-system"
Operation failed, retrying with backoff Cause="failed to create provider object cert-manager.io/v1alpha2, Kind=Certificate, capi-webhook-system/capi-serving-cert: Internal error occurred: failed calling webhook \"webhook.cert-manager.io\": the server is currently unable to handle the request"
Creating Certificate="capi-serving-cert" Namespace="capi-webhook-system"
Operation failed, retrying with backoff Cause="failed to create provider object cert-manager.io/v1alpha2, Kind=Certificate, capi-webhook-system/capi-serving-cert: Internal error occurred: failed calling webhook \"webhook.cert-manager.io\": the server is currently unable to handle the request"
Creating Certificate="capi-serving-cert" Namespace="capi-webhook-system"
Operation failed, retrying with backoff Cause="failed to create provider object cert-manager.io/v1alpha2, Kind=Certificate, capi-webhook-system/capi-serving-cert: Internal error occurred: failed calling webhook \"webhook.cert-manager.io\": the server is currently unable to handle the request"
Creating Certificate="capi-serving-cert" Namespace="capi-webhook-system"
Operation failed, retrying with backoff Cause="failed to create provider object cert-manager.io/v1alpha2, Kind=Certificate, capi-webhook-system/capi-serving-cert: Internal error occurred: failed calling webhook \"webhook.cert-manager.io\": the server is currently unable to handle the request"
Creating Certificate="capi-serving-cert" Namespace="capi-webhook-system"
Operation failed, retrying with backoff Cause="failed to create provider object cert-manager.io/v1alpha2, Kind=Certificate, capi-webhook-system/capi-serving-cert: Internal error occurred: failed calling webhook \"webhook.cert-manager.io\": the server is currently unable to handle the request"
Creating Certificate="capi-serving-cert" Namespace="capi-webhook-system"
Operation failed, retrying with backoff Cause="failed to create provider object cert-manager.io/v1alpha2, Kind=Certificate, capi-webhook-system/capi-serving-cert: Internal error occurred: failed calling webhook \"webhook.cert-manager.io\": the server is currently unable to handle the request"
Creating Certificate="capi-serving-cert" Namespace="capi-webhook-system"
Operation failed, retrying with backoff Cause="failed to create provider object cert-manager.io/v1alpha2, Kind=Certificate, capi-webhook-system/capi-serving-cert: Internal error occurred: failed calling webhook \"webhook.cert-manager.io\": the server is currently unable to handle the request"
Creating Certificate="capi-serving-cert" Namespace="capi-webhook-system"
Operation failed, retrying with backoff Cause="failed to create provider object cert-manager.io/v1alpha2, Kind=Certificate, capi-webhook-system/capi-serving-cert: Internal error occurred: failed calling webhook \"webhook.cert-manager.io\": the server is currently unable to handle the request"
Creating Certificate="capi-serving-cert" Namespace="capi-webhook-system"
Operation failed, retrying with backoff Cause="failed to create provider object cert-manager.io/v1alpha2, Kind=Certificate, capi-webhook-system/capi-serving-cert: Internal error occurred: failed calling webhook \"webhook.cert-manager.io\": the server is currently unable to handle the request"
Creating Certificate="capi-serving-cert" Namespace="capi-webhook-system"
Error: action failed after 10 attempts: failed to create provider object cert-manager.io/v1alpha2, Kind=Certificate, capi-webhook-system/capi-serving-cert: Internal error occurred: failed calling webhook "webhook.cert-manager.io": the server is currently unable to handle the request
sigs.k8s.io/cluster-api/cmd/clusterctl/client/cluster.retryWithExponentialBackoff
        /workspace/cmd/clusterctl/client/cluster/client.go:248
sigs.k8s.io/cluster-api/cmd/clusterctl/client/cluster.(*providerComponents).Create
        /workspace/cmd/clusterctl/client/cluster/components.go:66
sigs.k8s.io/cluster-api/cmd/clusterctl/client/cluster.installComponentsAndUpdateInventory
        /workspace/cmd/clusterctl/client/cluster/installer.go:104
sigs.k8s.io/cluster-api/cmd/clusterctl/client/cluster.(*providerInstaller).Install
        /workspace/cmd/clusterctl/client/cluster/installer.go:71
sigs.k8s.io/cluster-api/cmd/clusterctl/client.(*clusterctlClient).Init
        /workspace/cmd/clusterctl/client/init.go:112
sigs.k8s.io/cluster-api/cmd/clusterctl/cmd.runInit
        /workspace/cmd/clusterctl/cmd/init.go:146
sigs.k8s.io/cluster-api/cmd/clusterctl/cmd.glob..func5
        /workspace/cmd/clusterctl/cmd/init.go:88
github.com/spf13/cobra.(*Command).execute
        /go/pkg/mod/github.com/spf13/[email protected]/command.go:840
github.com/spf13/cobra.(*Command).ExecuteC
        /go/pkg/mod/github.com/spf13/[email protected]/command.go:945
github.com/spf13/cobra.(*Command).Execute
        /go/pkg/mod/github.com/spf13/[email protected]/command.go:885
sigs.k8s.io/cluster-api/cmd/clusterctl/cmd.Execute
        /workspace/cmd/clusterctl/cmd/root.go:52
main.main
        /workspace/cmd/clusterctl/main.go:25
runtime.main
        /usr/local/go/src/runtime/proc.go:203
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1357

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

Environment:

• Cluster-api version:
• Minikube/KIND version:
• Kubernetes version: (use kubectl version):
• OS (e.g. from /etc/os-release):

/kind bug
[One or more /area label. See https://github.com/kubernetes-sigs/cluster-api/labels?q=area for the list of labels]

[fabriziopandini]

I agree that clusterctl should not panic and this should be fixed.

With regard to the specific problem, it seems that for some reason the first cert-manager installation did not end up correctly (I saw this happen if you don't have a CNI installed). When you run the second init, clusterctl detected cert manager already installed, but then get stuck when trying to create a Certificate.

FYI There is already a PR (https://github.com/kubernetes-sigs/cluster-api/pull/3313) that is improving the cert-manager check in order to validate if cert-manager is properly functioning (instead of simply checking if it is installed); in case cert-manager will be detected as not working, a new installation will be re-tried also during the second init.

[jichenjc]

@fabriziopandini thanks for the info ,yes, I don't know what happened at 1st time to try cert-manager
the CNI is not a problem as I am using kind ...

[wfernandes]

/retitle clusterctl panics on init
/area clusterctl

@jichenjc Thanks for reporting this bug!
If this is reproducible, can you provide some additional details such as the CAPI version, kind version, clusterctl version and any other info that might be causing this issue?
if clusterctl hangs, could we see the state of the cert-manager deployment and services.

[benmoss]

It's not a panic, it's just a stack trace from this code. It only happens if you have verbosity set to 5 or higher.

[wfernandes]

@benmoss Yeah you are actually correct. I should've noticed that actually.
So may be the actual problem is the initial clusterctl init and it hanging on cert-manager installation but I think that can be opened as a separate issue and we can close this one.

[jichenjc]

ok, works for me ,thanks for the comments~