Cluster-api-provider-aws: Audit & review our e2e & conformance Prow jobs

[ncdc]

I created a spreadsheet that attempts to describe our various e2e and conformance Prow jobs: https://docs.google.com/spreadsheets/d/1uu7bSkOvbV2qa6ul1rvfgqkXatoviJf1MIFkPHMNC8I/edit?folder=0AP4mO3u-5wuhUk9PVA#gid=0.

We currently have some issues with some jobs because they build the e2e.test from the Kubernetes master branch and try to run the tests against older clusters (master is now 1.17+ but the clusters we test against are 1.15 and 1.16). There are some tests that require 1.17.

We also have two ways that tests deploy Cluster API and create clusters, and two kinds of tests:

1. Using go code - basic functional testing
2. Using make generate-examples and make create-cluster - conformance tests

The latter includes a lot of hacks on top of the in-repo kustomize examples to support overriding an AMI's baked-in Kubernetes artifacts with a specific Kubernetes version, either an official release or a CI build. I'm not positive, but I think we're interested in testing that the examples continue to function. I think we perhaps can split this into multiple categories:

1. Test that the examples function (no hacks)
2. Basic functional tests
3. Run conformance tests against various official Kubernetes releases
4. Run conformance tests against Kubernetes master/CI latest

I'm thinking we could potentially adjust our jobs to be the following (I'll address periodics but the same applies to post and presubmits):

1. Test that the examples function
   
   
   
   1. CAPA variations: release-0.4, master
   
   
   2. Kubernetes variations: 1.15, 1.16, 1.17
      
      
      
      
      
      1. Rotate jobs in/out as Kubernetes versions change (drop 1.15, add 1.18, etc)
      
      
      
   
   
2. Basic functional test (Go)
   
   
   
   1. CAPA variations: release-0.4, master
   
   
   2. Kubernetes variations: 1.15, 1.16, 1.17
      
      
      
      
      
      1. Rotate jobs in/out as Kubernetes versions change (drop 1.15, add 1.18, etc)
      
      
      
   
   
3. Run conformance tests (Go)
   
   
   
   1. CAPA variations: release-0.4, master
   
   
   2. Kubernetes variations: 1.15, 1.16, 1.17, CI/master
      
      
      
      
      
      1. Rotate jobs in/out as Kubernetes versions change (drop 1.15, add 1.18, etc)
      
      
      
   
   

Now there's the question of which AMIs to use. Assuming we continue to stay up to date building AMIs as new patch versions come out, we could manually update the branches in CAPA to change the defaults for 1.15/1.16/1.17, etc. For testing Kubernetes CI/master, ideally we would build a new temporary AMI for that, instead of using a preKubeadmCommand to override the baked-in artifacts. It sounds like maybe there is/was a problem building images in CI, so hopefully we can get that resolved if it isn't already.

Thoughts?

@detiber @vincepri @dims @rudoi

[ncdc]

In the future, "test examples" probably turns into "test clusterctl"

[ncdc]

Current list of failing CI jobs as of 12/19/2019:

Periodics

• ci-cluster-api-provider-aws-make-conformance-master: running conformance tests from k/k master against a 1.16.x cluster, and the tests require a 1.17 cluster
• ci-cluster-api-provider-aws-make-conformance-stable: unable to correctly override using released Kubernetes bits (#1436 might fix)

Postsubmits

• ci-cluster-api-provider-aws-e2e-conformance-release-0-4 (seems to either be timing out running the tests, and/or having issues with sonobuoy)

PRs

• pull-cluster-api-provider-aws-e2e-conformance: timing out - needs more than 2h?

[detiber]

@ncdc This mostly lines up with my thoughts on where we want to get to for testing as well... One caveat is that I don't think that having a test that compares both capa/master and k8s/master for conformance testing will be too valuable and we should focus on keeping at least one of the two projects at "stable" version for testing.

[ncdc]

Thanks. I agree that too much instability between capa & k8s will probably yield a lot of spurious failures. Maybe we limit conformance to capa release branches, and create a release-0.5 when we're getting close to releasing v1alpha3?

[detiber]

@ncdc I do like the idea of testing conformance of the clusters we create with the head of our development branch, but think we should limit it to just released versions of k8s, so that we know it is likely some change we made to the way we are deploying the clusters rather than a new bug in k8s :)

[ncdc]

Ok, that sounds reasonable!

[ncdc]

So capa release branches can test against k/k master, but capa master only tests against released k/k?

[ncdc]

Ok, so now on to AMIs.

1. How do you want to handle building them - ideally we could automate?
2. How should we get the jobs updated to test against the latest patch releases of each minor version?

[ncdc]

I would love to be able to edit a single kubernetes version setting in a single file and have a postsubmit job build an ami for that version if it doesn't already exist...

[vincepri]

How do you want to handle building them - ideally we could automate?

This should be very doable with AWS and correct permissions. Either prow or github actions can push the images from image-builder repository.

How should we get the jobs updated to test against the latest patch releases of each minor version?

We can probably query EC2 APIs for the latest patch image, instead of getting a fixed one like we're doing today?

[detiber]

Ok, so now on to AMIs.

1. How do you want to handle building them - ideally we could automate?

This probably ties into work being done on the image-builder side. I know @moshloop has started digging into this, but we basically need to work with wg-k8s-infra to get some dedicated cloud accounts for this, likely a dedicated prow cluster, and ideally some automation in place to create/publish images as needed.

1. How should we get the jobs updated to test against the latest patch releases of each minor version?

+1 to @vincepri's suggestion of querying EC2 to determine the latest published image for each minor version and using that.

[ncdc]

How should we get the jobs updated to test against the latest patch releases of each minor version?

Rather than updating the jobs, if we do ^^^ for image building, but have something a bit more expanded like

{
  "versions": ["1.15.3", "1.16.4", "1.17.0"]
}

And then in the prow jobs, we have one each for 1.15.x, 1.16.x, 1.17.x, and each job would have an env var indicating which minor version to test, and then we'd find a way to connect the dots from the env var to the json file to a kubernetesVersion that matches the right minor version?

[vincepri]

Can we just say we want to test against 1.15.x, 1.16.x, etc? Then the controller picks whatever latest image we have published upon creation

[ncdc]

We don't have any code that does that? Don't we have to specify major.minor.patch to get the AMI lookup to work?

[vincepri]

We do, but it includes the patch version. I'm thinking about what Azure and GCP provider do today, which is to allow images to be picked by major.minor, and exclude the patch from the lookup

[detiber]

Since the images are public, we could query the list of images with the proper organization id, and determine which versions have published images that way as well.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[ncdc]

/reopen
/remove-lifecycle rotten

[k8s-ci-robot]

@ncdc: Reopened this issue.

In response to this:

/reopen
/remove-lifecycle rotten

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[ncdc]

This should be done when the e2e cleanup PRs are merged.

[randomvariable]

/lifecycle active