Cluster-api: clusterctl init -i vsphere fails

Created on 29 Jul 2020  路  20Comments  路  Source: kubernetes-sigs/cluster-api

What steps did you take and what happened:

$ export CLUSTERCTL_LOG_LEVEL=5
$ clusterctl init -i vsphere
Using configuration File="/Users/wfernandes/.cluster-api/clusterctl.yaml"
Installing the clusterctl inventory CRD
Creating CustomResourceDefinition="providers.clusterctl.cluster.x-k8s.io"
Fetching providers
Fetching File="core-components.yaml" Provider="cluster-api" Version="v0.3.7"
Fetching File="bootstrap-components.yaml" Provider="bootstrap-kubeadm" Version="v0.3.7"
Fetching File="control-plane-components.yaml" Provider="control-plane-kubeadm" Version="v0.3.7"
Fetching File="infrastructure-components.yaml" Provider="infrastructure-vsphere" Version="v0.7.0-alpha.0"
Fetching File="metadata.yaml" Provider="cluster-api" Version="v0.3.7"
Fetching File="metadata.yaml" Provider="bootstrap-kubeadm" Version="v0.3.7"
Fetching File="metadata.yaml" Provider="control-plane-kubeadm" Version="v0.3.7"
Fetching File="metadata.yaml" Provider="infrastructure-vsphere" Version="v0.7.0-alpha.0"
Error: invalid provider metadata: version v0.7.0-alpha.0 for the provider capv-system/infrastructure-vsphere does not match any release series
sigs.k8s.io/cluster-api/cmd/clusterctl/client/cluster.(*providerInstaller).getProviderContract
        /Users/wfernandes/workspace/cluster-api/cmd/clusterctl/client/cluster/installer.go:244
sigs.k8s.io/cluster-api/cmd/clusterctl/client/cluster.(*providerInstaller).Validate
        /Users/wfernandes/workspace/cluster-api/cmd/clusterctl/client/cluster/installer.go:200
sigs.k8s.io/cluster-api/cmd/clusterctl/client.(*clusterctlClient).Init
        /Users/wfernandes/workspace/cluster-api/cmd/clusterctl/client/init.go:103
sigs.k8s.io/cluster-api/cmd/clusterctl/cmd.runInit
        /Users/wfernandes/workspace/cluster-api/cmd/clusterctl/cmd/init.go:146
sigs.k8s.io/cluster-api/cmd/clusterctl/cmd.glob..func6
        /Users/wfernandes/workspace/cluster-api/cmd/clusterctl/cmd/init.go:88
github.com/spf13/cobra.(*Command).execute
        /Users/wfernandes/go/pkg/mod/github.com/spf13/[email protected]/command.go:840
github.com/spf13/cobra.(*Command).ExecuteC
        /Users/wfernandes/go/pkg/mod/github.com/spf13/[email protected]/command.go:945
github.com/spf13/cobra.(*Command).Execute
        /Users/wfernandes/go/pkg/mod/github.com/spf13/[email protected]/command.go:885
sigs.k8s.io/cluster-api/cmd/clusterctl/cmd.Execute
        /Users/wfernandes/workspace/cluster-api/cmd/clusterctl/cmd/root.go:52
main.main
        /Users/wfernandes/workspace/cluster-api/cmd/clusterctl/main.go:25
runtime.main
        /usr/local/go/src/runtime/proc.go:203
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1357

What did you expect to happen:
Expected it to work.

Anything else you would like to add:
https://github.com/kubernetes-sigs/cluster-api/pull/3412 was merged in recently.

Environment:

  • clusterctl version: clusterctl version: &version.Info{Major:"0", Minor:"3", GitVersion:"v0.3.7", GitCommit:"846ca08db5939e5bf88104b2f34427e7316ee7b9", GitTreeState:"clean", BuildDate:"2020-07-14T14:34:36Z", GoVersion:"go1.13.12", Compiler:"gc", Platform:"linux/amd64"}
  • OS (e.g. from /etc/os-release): macos

/kind bug
/area clusterctl

areclusterctl kinbug lifecyclactive
Source
picture wfernandes

All 20 comments

@wfernandes: The label(s) area/ cannot be applied, because the repository doesn't have them

In response to this:

What steps did you take and what happened:

$ export CLUSTERCTL_LOG_LEVEL=5
$ clusterctl init -i vsphere
Using configuration File="/Users/wfernandes/.cluster-api/clusterctl.yaml"
Installing the clusterctl inventory CRD
Creating CustomResourceDefinition="providers.clusterctl.cluster.x-k8s.io"
Fetching providers
Fetching File="core-components.yaml" Provider="cluster-api" Version="v0.3.7"
Fetching File="bootstrap-components.yaml" Provider="bootstrap-kubeadm" Version="v0.3.7"
Fetching File="control-plane-components.yaml" Provider="control-plane-kubeadm" Version="v0.3.7"
Fetching File="infrastructure-components.yaml" Provider="infrastructure-vsphere" Version="v0.7.0-alpha.0"
Fetching File="metadata.yaml" Provider="cluster-api" Version="v0.3.7"
Fetching File="metadata.yaml" Provider="bootstrap-kubeadm" Version="v0.3.7"
Fetching File="metadata.yaml" Provider="control-plane-kubeadm" Version="v0.3.7"
Fetching File="metadata.yaml" Provider="infrastructure-vsphere" Version="v0.7.0-alpha.0"
Error: invalid provider metadata: version v0.7.0-alpha.0 for the provider capv-system/infrastructure-vsphere does not match any release series
sigs.k8s.io/cluster-api/cmd/clusterctl/client/cluster.(*providerInstaller).getProviderContract
       /Users/wfernandes/workspace/cluster-api/cmd/clusterctl/client/cluster/installer.go:244
sigs.k8s.io/cluster-api/cmd/clusterctl/client/cluster.(*providerInstaller).Validate
       /Users/wfernandes/workspace/cluster-api/cmd/clusterctl/client/cluster/installer.go:200
sigs.k8s.io/cluster-api/cmd/clusterctl/client.(*clusterctlClient).Init
       /Users/wfernandes/workspace/cluster-api/cmd/clusterctl/client/init.go:103
sigs.k8s.io/cluster-api/cmd/clusterctl/cmd.runInit
       /Users/wfernandes/workspace/cluster-api/cmd/clusterctl/cmd/init.go:146
sigs.k8s.io/cluster-api/cmd/clusterctl/cmd.glob..func6
       /Users/wfernandes/workspace/cluster-api/cmd/clusterctl/cmd/init.go:88
github.com/spf13/cobra.(*Command).execute
       /Users/wfernandes/go/pkg/mod/github.com/spf13/[email protected]/command.go:840
github.com/spf13/cobra.(*Command).ExecuteC
       /Users/wfernandes/go/pkg/mod/github.com/spf13/[email protected]/command.go:945
github.com/spf13/cobra.(*Command).Execute
       /Users/wfernandes/go/pkg/mod/github.com/spf13/[email protected]/command.go:885
sigs.k8s.io/cluster-api/cmd/clusterctl/cmd.Execute
       /Users/wfernandes/workspace/cluster-api/cmd/clusterctl/cmd/root.go:52
main.main
       /Users/wfernandes/workspace/cluster-api/cmd/clusterctl/main.go:25
runtime.main
       /usr/local/go/src/runtime/proc.go:203
runtime.goexit
       /usr/local/go/src/runtime/asm_amd64.s:1357

What did you expect to happen:
Expected it to work.

Anything else you would like to add:
https://github.com/kubernetes-sigs/cluster-api/pull/3412 was merged in recently.

Environment:

  • clusterctl version: clusterctl version: &version.Info{Major:"0", Minor:"3", GitVersion:"v0.3.7", GitCommit:"846ca08db5939e5bf88104b2f34427e7316ee7b9", GitTreeState:"clean", BuildDate:"2020-07-14T14:34:36Z", GoVersion:"go1.13.12", Compiler:"gc", Platform:"linux/amd64"}
  • OS (e.g. from /etc/os-release): macos

/kind bug
/area clusterctl

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

k8s-ci-robot picture k8s-ci-robot on 29 Jul 2020

/milestone v0.3.x

vincepri picture vincepri on 29 Jul 2020

@wfernandes I had the same issue this morning. Are you intentionally pulling v0.7.0-alpha.0 Pre-release? I was not trying to pull v0.7.0-alpha.0 but it pulled anyway during "clusterctl init"

rushyrush picture rushyrush on 29 Jul 2020

@rushyrush you need to specify the version when running init. For some reason it is defaulting to the pre-release. Try this:
clusterctl init --infrastructure=vsphere:v0.6.6

BrandenCobb picture BrandenCobb on 29 Jul 2020
👍1

@rushyrush you need to specify the version when running init. For some reason it is defaulting to the pre-release. Try this:
clusterctl init --infrastructure=vsphere:v0.6.6

@BrandenCobb, thanks! specifying the version worked.

rushyrush picture rushyrush on 29 Jul 2020

@rushyrush So by default when you specify clusterctl init -i vsphere it pulls the latest release. I believe that clusterctl should be ignoring pre-release versions but that doesn't seem to be happening.
I'll take a look into this soon.
@BrandenCobb thanks for responding with the fix. 馃檪

wfernandes picture wfernandes on 29 Jul 2020

/assign
/lifecycle active

wfernandes picture wfernandes on 29 Jul 2020

I think I know what's going on.
Users need to update clusterctl.
Since we are relying on embedded metadata and not pulling them from the release, if you have an older clusterctl it only knows of the following versions/contracts.

v0.6 --> v1alpha3
v0.5 --> v1alpha2

So when it pulls the "latest" release of CAPV provider, which is of version v0.7.0-alpha.0, it doesn't know which contract to adhere to for the v0.7.x releases.

After building the latest clusterctl binary, I was not able to reproduce this error.

wfernandes picture wfernandes on 30 Jul 2020

This issue arose because providers are relying on the "out-of-the-box" behavior of clusterctl using its embedded metadata instead of fetching it from the provider release.
In the documentation we say that providers MUST provide a metadata and components file.

Maybe we should document this negative side-effect to user experience until we find a way to warn the users that they are using an out-of-date clusterctl.

Can we advocate for providers to ship a metadata.yaml as part of their releases?
Any thoughts on this? @vincepri @ncdc

wfernandes picture wfernandes on 30 Jul 2020

@wfernandes thanks for digging into this. I think your suggestion makes sense.

ncdc picture ncdc on 30 Jul 2020

I'm fine shipping a metadata.yaml for each provider, I'd like to hear from Fabrizio when he comes back from vacation next week, I remember there were some downsides although this was long ago

vincepri picture vincepri on 30 Jul 2020

It's an extra GitHub API request, which counts against the low anonymous rate limit unless you've configured your token. But presumably it's a one-time cost, as I would expect us to cache the contents for each release.

ncdc picture ncdc on 30 Jul 2020
👍1

Makes sense, if we require each infrastructure provider to use the main branch (or any other fixed branch) to publish their metadata.yaml, we could get around this using raw.githubusercontent.com instead.

vincepri picture vincepri on 30 Jul 2020

I'd like to avoid pulling from branches. I think it's most appropriate to continue to pull from releases.

ncdc picture ncdc on 30 Jul 2020

Agree for published components, which are versioned. Although metadata isn't exactly version, but rather an object that covers the whole repository across any version published, the suggestion to use a branch above either main or something else would make that clear.

That said, we could make sure to document that the behavior would be that metadata.yaml is always considered from the _latest_ release, regardless of which one is selected during init/install/upgrade

vincepri picture vincepri on 30 Jul 2020

I prefer we have the metadata as part of the release vs. raw.githubusercontent.com because of how the repository client is currently setup. We would have to include special logic to handle the raw.githubusercontent.com.
Currently we have the following checks to validate the github url which needs to be in a specific format.
https://github.com/kubernetes-sigs/cluster-api/blob/95fe9e2c2c48cb7c765e40fe97861f22765441ff/cmd/clusterctl/client/repository/repository_github.go#L110-L133

This could also interfere with the overrides logic.

wfernandes picture wfernandes on 30 Jul 2020

How do we overcome the issue we just had if the metadata is tied to a specific release?

Let's work on an example, provider foo is on version v0.6.4 which matches v1alpha3 API types, its metadata on v0.6.4 has only the release series v0.6.x. Provider has a breaking API change, and it has to bump the version to v0.7. Provider cuts an alpha pre-release v0.7.0-beta.0, which contains the new metadata.yaml with the newly included v0.7.x -> v1alpha3 mapping.

If I use clusterctl init --provider=foo, does clusterctl know to look at the pre-release version metadata.yaml but offer the latest v0.6.4 to install? This use case was brought up because we said we'd exclude pre-release versions when installing without a specific version.

vincepri picture vincepri on 30 Jul 2020

We are not currently excluding GH prereleases.

ncdc picture ncdc on 30 Jul 2020

We are not currently excluding GH prereleases.

Yeah, although we talked about excluding them unless a user specifies the version.

vincepri picture vincepri on 30 Jul 2020

This use case was brought up because we said we'd exclude pre-release versions when installing without a specific version.

Should we file an issue/feature request for this? (if one hasn't already been created)

How do we overcome the issue we just had if the metadata is tied to a specific release?

I think you explained the intended workflow. So if the v0.7.x-beta.0 was cut, it would have a metadata file with this contract rule included v0.7.x -> v1alpha3. And if we implement the feature of ignoring pre-releases, then clusterctl init -i foo would move on to v0.6.4 release which would have it's own metadata file BUT if the v0.6.4 release didn't have a metadata file then it would revert to using the embedded metadata.

The idea is that the embedded metadata would just be a fallback but it shouldn't be used as the sole source of truth.

Also, as you mentioned, this metadata.yaml file could be in the repo and the automation to cut a release includes it in the github release artifacts.

This issue is essentially trying to balance the UX for the infra provider authors/developers vs. the users of clusterctl.

wfernandes picture wfernandes on 30 Jul 2020
👍1
Was this page helpful?
0 / 5 - 0 ratings

Related issues

dlipovetsky picture dlipovetsky  路  5Comments
mboersma picture mboersma  路  5Comments
fabriziopandini picture fabriziopandini  路  5Comments
oneilcin picture oneilcin  路  6Comments
rsmitty picture rsmitty  路  5Comments