Cluster-api: Testing of clusterctl exhausts Github API limits

[randomvariable]

What steps did you take and what happened:
[A clear and concise description on how to REPRODUCE the bug.]
Am testing invocations of clusterctl init and ran into

Fetching providers
Error: failed to get repository client for the BootstrapProvider with name kubeadm: error creating the GitHub repository client: failed to get GitHub latest version: failed to get the list of versions: rate limit for github api has been reached. Please wait one hour or get a personal API tokens a assign it to the GITHUB_TOKEN environment variable

What did you expect to happen:

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]
Can set the token, but not ideal.

Environment:

• Cluster-api version: 0.3.0
• Minikube/KIND version:
• Kubernetes version: (use kubectl version):
• OS (e.g. from /etc/os-release):

/kind bug

[vincepri]

/milestone Next
/help
/priority important-longterm

This will need a spike to understand possible solutions

[k8s-ci-robot]

@vincepri:
This request has been marked as needing help from a contributor.

Please ensure the request meets the requirements listed here.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-help command.

In response to this:

/milestone Next
/help
/priority important-longterm

This will need a spike to understand possible solutions

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[kacole2]

@randomvariable I just received this same error. using v0.3.0-rc.2

Fetching providers
Error: failed to get provider components for the "cluster-api:v0.3.0-rc.2" provider: failed to read "core-components.yaml" from provider's repository "cluster-api": failed to download files from GitHub release v0.3.0-rc.2: rate limit for github api has been reached. Please wait one hour or get a personal API tokens a assign it to the GITHUB_TOKEN environment variable

[CecileRobertMichon]

+1 I just got the same error

Error: failed to get provider components for the "cluster-api:v0.3.0" provider: failed to get repository client for the CoreProvider with name cluster-api: error creating the GitHub repository client: failed to get GitHub latest version: failed to get the list of versions: rate limit for github api has been reached. Please wait one hour or get a personal API tokens a assign it to the GITHUB_TOKEN environment variable

This works as a workaround https://github.com/settings/tokens/new but it's strange that we hit the limits so quickly...

[eshanks16]

+1 I got the same error after trying to deploy it a few times <10.

Error: failed to get provider components for the "cluster-api" provider: failed to get repository client for the CoreProvider with name cluster-api: error creating the GitHub repository client: failed to get GitHub latest version: failed to get the list of versions: rate limit for github api has been reached. Please wait one hour or get a personal API tokens a assign it to the GITHUB_TOKEN environment variable

[davidmontoyago]

+1 Consistently seeing this issue when doing:

clusterctl --v=99 init --infrastructure=aws

version: Major:"0", Minor:"3", GitVersion:"v0.3.0-dirty", GitCommit:"b6aad253dfcdafc263e8b399d3a97732006b3069"

[ncdc]

@vincepri to fill in some details for how to potential solve this problem

[bboreham]

Would this be any help? https://github.com/kubernetes/test-infra/tree/master/ghproxy

[vincepri]

In https://github.com/kubernetes-sigs/cluster-api/blob/master/hack/tools/mdbook/releaselink/releaselink.go#L56-L65

We use go proxy to get the latest version, and the go raw.github.com which doesn't have the same API limits that other endpoints have.

If we go down this path, we should make sure we let users use their preferred proxy from GOPROXY environment variable, or default to proxy.golang.org

[fabriziopandini]

/area clusterctl

[fabriziopandini]

/lifecycle active

From my tests, clusterctl init --infrastructure aws currently executes 20 GitHub calls

ListReleases   owner="kubernetes-sigs" repository="cluster-api"
GetReleaseByTag   owner="kubernetes-sigs" repository="cluster-api"   tag="v0.3.3"
DownloadReleaseAsset   owner="kubernetes-sigs" repository="cluster-api"   release="v0.3.3" fileName="core-components.yaml"
ListReleases   owner="kubernetes-sigs" repository="cluster-api"
GetReleaseByTag   owner="kubernetes-sigs" repository="cluster-api"   tag="v0.3.3"
DownloadReleaseAsset   owner="kubernetes-sigs" repository="cluster-api"   release="v0.3.3" fileName="bootstrap-components.yaml"
ListReleases   owner="kubernetes-sigs" repository="cluster-api"
GetReleaseByTag   owner="kubernetes-sigs" repository="cluster-api"   tag="v0.3.3"
DownloadReleaseAsset   owner="kubernetes-sigs" repository="cluster-api"   release="v0.3.3" fileName="control-plane-components.yaml"
ListReleases   owner="kubernetes-sigs"   repository="cluster-api-provider-aws"
GetReleaseByTag   owner="kubernetes-sigs"   repository="cluster-api-provider-aws" tag="v0.5.2"
DownloadReleaseAsset   owner="kubernetes-sigs"   repository="cluster-api-provider-aws" release="v0.5.2"   fileName="infrastructure-components.yaml"
ListReleases   owner="kubernetes-sigs" repository="cluster-api"
GetReleaseByTag   owner="kubernetes-sigs" repository="cluster-api"   tag="v0.3.3"
ListReleases   owner="kubernetes-sigs" repository="cluster-api"
GetReleaseByTag   owner="kubernetes-sigs" repository="cluster-api"   tag="v0.3.3"
ListReleases   owner="kubernetes-sigs" repository="cluster-api"
GetReleaseByTag   owner="kubernetes-sigs" repository="cluster-api"   tag="v0.3.3"
ListReleases   owner="kubernetes-sigs"   repository="cluster-api-provider-aws"
GetReleaseByTag   owner="kubernetes-sigs"   repository="cluster-api-provider-aws" tag="v0.5.2"

With some caching, I can cut the number down to 8

ListReleases   owner="kubernetes-sigs" repository="cluster-api"
GetReleaseByTag   owner="kubernetes-sigs" repository="cluster-api"   tag="v0.3.3"
DownloadReleaseAsset   owner="kubernetes-sigs" repository="cluster-api"   release="v0.3.3" fileName="core-components.yaml"
DownloadReleaseAsset   owner="kubernetes-sigs" repository="cluster-api"   release="v0.3.3" fileName="bootstrap-components.yaml"
DownloadReleaseAsset   owner="kubernetes-sigs" repository="cluster-api"   release="v0.3.3" fileName="control-plane-components.yaml"
ListReleases   owner="kubernetes-sigs"   repository="cluster-api-provider-aws"
GetReleaseByTag   owner="kubernetes-sigs"   repository="cluster-api-provider-aws" tag="v0.5.2"
DownloadReleaseAsset   owner="kubernetes-sigs"   repository="cluster-api-provider-aws" release="v0.5.2"   fileName="infrastructure-components.yaml"

This number is not affected by the -v flag.

Considering that clusterctl config cluster test --infrastructure aws executes 3 GitHub calls, this gives room for executing 5 Cluster API quick-start per hour without a token (for unauthenticated requests, the rate limit allows for up to 60 requests per hour).

With a token instead, you can make up to 5000 requests per hour, and this means ~450 Cluster API quick-start per hour.

With the change suggested by @vincepri we can go down from 8/3 calls to 6/2, but TBH, I don't think this is required anymore after implementing caching

@ncdc @vincepri opinions?