Cluster-api: clusterctl config repositories point to invalid URLs

[scottd018]

What steps did you take and what happened:
Run the following:

clusterctl config repositories
NAME          TYPE                     URL
cluster-api   CoreProvider             https://github.com/kubernetes-sigs/cluster-api/releases/latest/core-components.yaml
kubeadm       BootstrapProvider        https://github.com/kubernetes-sigs/cluster-api/releases/latest/bootstrap-components.yaml
kubeadm       ControlPlaneProvider     https://github.com/kubernetes-sigs/cluster-api/releases/latest/control-plane-components.yaml
aws           InfrastructureProvider   https://github.com/kubernetes-sigs/cluster-api-provider-aws/releases/latest/infrastructure-components.yaml
azure         InfrastructureProvider   https://github.com/kubernetes-sigs/cluster-api-provider-azure/releases/latest/infrastructure-components.yaml
metal3        InfrastructureProvider   https://github.com/metal3-io/cluster-api-provider-metal3/releases/latest/infrastructure-components.yaml
openstack     InfrastructureProvider   https://github.com/kubernetes-sigs/cluster-api-provider-openstack/releases/latest/infrastructure-components.yaml
vsphere       InfrastructureProvider   https://github.com/kubernetes-sigs/cluster-api-provider-vsphere/releases/latest/infrastructure-components.yaml

Pick any of the repositories and you will get a 'Not Found' message:

curl https://github.com/kubernetes-sigs/cluster-api-provider-vsphere/releases/latest/infrastructure-components.yaml
Not Found

What did you expect to happen:

The repositories will reference valid YAML repositories AND the specific version of clusterctl should be used in the repository URL (not latest).

Anything else you would like to add:

N/A

Environment:

• Cluster-api version: clusterctl version: &version.Info{Major:"0", Minor:"3", GitVersion:"v0.3.3", GitCommit:"ecff70af1b839c4086335234d88b1b8c00c3383c", GitTreeState:"clean", BuildDate:"2020-03-27T22:25:55Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"darwin/amd64"}
• Minikube/KIND version: N/A
• Kubernetes version: (use kubectl version): N/A
• OS (e.g. from /etc/os-release): MacOS

/kind bug

[vincepri]

Seems the correct URL should be https://github.com/kubernetes-sigs/cluster-api-provider-vsphere/releases/latest/download/infrastructure-components.yaml

missing the download/ subpath

/assign @fabriziopandini @wfernandes
/milestone v0.3.x

[scottd018]

@vincepri yes, sorry, I forgot to notate what the correct URL should be. Was thinking it also should reference the correct version that is associated with the currently installed clusterctl binary (as opposed to latest), but perhaps that is taking it a bit too far.

[vincepri]

Sounds like a valid ask to me

[fabriziopandini]

@vincepri @scottd018

missing the download/ subpath

WRT to the suggestion of making the url corresponding to the real url I agree this can be less confusing for the users. However, we should make sure that the download/ subpath is ignored otherwise the api calls will fail.

it also should reference the correct version that is associated with the currently installed clusterctl binary (as opposed to latest)

The idea behind latest is to always use the most recent versions of a provider when creating a management cluster; this behavior was copied by kubeadm, and it was explicitly required during the initial phases of the clusterctl implementation.

Nevertheless, the idea of pinning versions IMO is not applicable to the list of out the box providers, because as of today this will require a coordination/synchronization between 6 different projects to define the correct version to be associated with the currently installed clusterctl binary, and a considerable churn as each of the 6 projects move forward with different timelines. (TL;DR; the correct version is hard to define)

Please note that, if necessary, each user can pin the provider definition to a specific version by replacing "latest" with a version tag using the clusterctl config file as described here https://cluster-api.sigs.k8s.io/clusterctl/configuration.html#provider-repositories, but TBH i think that in this case, the CLI syntax for pinning version is much more simple and explicit (eg. -infrastructure aws:v0.5.1)

[vincepri]

Maybe we should remove the full URL from the output and just point to the base repository instead?

[vincepri]

we can also output the version separately

[fabriziopandini]

How the URL should be interpreted to determine what is the base repository vs what is latest/version depends on the type of repository (Github vs FileSystem).

Instantiating those repository objects might be overkilling for the current command output, but if instead, we decide to redesign the output and show available versions this could make sense (there is also an issue for this https://github.com/kubernetes-sigs/cluster-api/issues/2171).
On the backside, we should also consider that a scan of all the repositories might consume the full free quota of the github APIs in a single shot...

[fabriziopandini]

/area clusterctl

[wfernandes]

Maybe we can present the URLs in the output of clusterctl config repositories as just
https://github.com/kubernetes-sigs/cluster-api-provider-aws/releases/latest for example.
That way when a user happens to go to that url, it redirects them to the specific version/tag of the provider.

I believe our documentation already refers to infrastructure-components.yaml for InfrastructureProviders and likewise for the others to guide users what components they might be looking for.

Again, the internal defaults would be the original URL and wouldn't change, but the CLI output would just trim out the yaml filename in the "presentation layer". WDYT?

[wfernandes]

I can work on this issue regarding the solution I proposed above. What do y'all think?

[scottd018]

Hey @wfernandes apologies for the delayed response. I think your solution is acceptable. It would certainly fix the confusion surrounding broken links and inaccessible URLs. Probably the best approach considering what @fabriziopandini has pointed out.

Other thoughts???

[wfernandes]

Cool...I can get started on this and I'm open to feedback and suggestions @fabriziopandini if you have any input on this.

[wfernandes]

/assign

[fabriziopandini]

@wfernandes +1 to the idea
What about moving the file name to a separated column? This will make clear the difference between the three provider that are reading from the cluster API repository.
Also please make sure the spot mechanism works for local repository as well.

[wfernandes]

/lifecycle active