Cluster-api-provider-aws: Periodic e2e testing

[detiber]

/kind feature

wire up periodic full e2e tests

/assign @liztio

[detiber]

/priority critical-urgent

[timothysc]

This should be the conformance suite on a full cluster deployment.

[liztio]

Current Status

E2E Tests

Background

We spent a lot of time discussing what exactly the E2E tests should entail.
The current "e2e" tests just build Docker images and test that they can boot.
This is a decent smoke test, but to validate our assumptions more detailed tests.

The proposed minimum viable test stack is as follows:

1. The tests get an AWS account
2. A kind kubernetes cluster is booted
3. The ClusterAPI and ClusterAPIProviderAWS controllers are created in the cluster
4. A cluster object is created using the Cluster API
5. A machine object is created in said cluster
6. We wait for the cluster api aws provider controller to mark the created machine as "ready"

Current Status

_E2E Pull Request_

I've wrapped up a pull request that implements the above steps.
It makes several assumptions about the environment it's running in.

First, AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.
The checkin_account.py scripts will set these.
It's intended to be used as eval $(./hack/checkin_account.py)
Those scripts expect a BOSKOS_HOST environment variable, in the form of localhost:8080.
checkout_account.py also needs the name of the account from boskos.
The eval() command will set this.

Future

Deployment

The existing integration tests are run as E2E in scripts/ci-bazel-e2e.sh. If we choose to make
the new tests E2E, we will only need to modify that script.

If we wish to instead use periodic E2E (as this issue requests), we will need to [create a new prow job][job].

More E2E tests

All we currently exercise is that AWS, and therefore Cluster API, thinks our booted control plane node is healthy. Some future ideas include:

1. Connecting to to the control plane and running health checks
2. Pivoting away from the Kind cluster
3. Tearing down a cluster
4. Worker nodes joining the cluster
5. Probably a bunch of things I'm not thinking of!

Boskos Test Infrastructure

_Main tracking issue here_

Background

To fully test the AWS cluster provider, we need access to AWS accounts.
This presents a problem for CI, as these accounts can't be simply committed to the repository.
A more complex management solution was needed.

After considering many alternatives, we decided that a central place to "check in" and "check out" AWS accounts was the best solution.
[Boskos][bosksos] is a service for managing shared resources between tests.
It is owned by sig-testing and therefore already deployed and maintained.

Boskos works on CRDs that move through arbitrary "states."
In our case, two applications will be checking in and out the AWS accounts: the tests themselves and the janitor.
Janitors are responible for taking resources and making them ready for use again.
In our case, that means emptying the AWS account to the extent possible so tests do not intefere with each other over multiple runs.

Unfortunately, the only way to clean an AWS account is to simply enumerate all resources and delete them.
Some resources depend on others, so deletion order is very important.
We make use of the existing aws-janitor codebase to delete in dependency order.

Current Status

The new aws-boskos-janitor has been tested and merged.

This comment sums up the work that still remains to be done.
In summary, the Janitor itself still needs be deployed in the Boskos Cluster.

Boskos also needs to populated with new resources for the aws accounts.
This is unfortunately blocked on upstream CNCF issues with our AWS accounts.

Once this is resolved, the new accounts should be addeded to resource.yaml.
After the resources have been added, AWS credentials will need to be added to account resource.
An IAM account with Administrator Privileges should be sufficient.
The creds can either be added to boskos using either the update API or kubectl edit directly on the Boskos CRDs.

Potential Future Issues

The AWS Janitor codebase is not comprehensive.
It doesn't presently contain Cloudformation and SSH Keypair objects for example, both of which are created by the Cluster API bootstrap. Added in https://github.com/kubernetes/test-infra/pull/11323
If AWS account isn't completely cleaned, repeated test runs could bump up against account limits that will be difficult to debug.

AWS Janitor may need retries as well.
I suspect in practice the dependency order is brittle and subject to race conditions.
For example, if NAT Gateways haven't completed deletion (which can take several minutes) by the time VPC deletion is called, deletion of the VPC will fail.

[randomvariable]

/assign

Can sort out some of the janitor tasks

[akutz]

Hi @liztio,

Thank you for this. I look forward to our chat tomorrow. I also sent you some questions at https://github.com/kubernetes/test-infra/issues/10693#issuecomment-463840731.

I’m sorry you are being asked to hand this off to at least one newbie. When it comes to Boskos and this test workflow, I’m still trying to wrap my head around the larger picture, and the brain dump assumes pre-existing experience with some concepts and tooling. There is a brief, high-level overview of Boskos -- but I’m afraid at least I will have to spend a lot more time understanding Boskos as a service, how it’s deployed, where it’s deployed, and how it works before I’m able to provide any value to the e2e discussion or process.

Is there possibly a good link or starting point for Boskos in action?

Boskos for dummies?

I think after the brain dump above it’s more clear to me that Boskos is simply a way to manage access to shared AWS accounts, whereas I thought it was for tracking all the resources created as part of a CAPA CI run (VPCs, instances, subnets, volumes, etc). This does not appear to be the case. It seems Boskos is test-infra’s version of Hashicorp Vault and its dynamic secrets engine with support for AWS? At the risk of being hit in the head with a digital can of soup, was Vault considered as part of the exploratory process?

Thank you!

--
-a

[liztio]

@akutz

I replied to your comment on test-infra. Hopefully that helps at least a little? Boskos isn't something we run or manage, it's something we can assume is available. I don't really have any information that isn't what I've linked to, unfortunately, but I think you will find the sig-testing folks very friendly and helpful.

We didn't consider Vault mostly because a service in the hand is worth two in the bush :)

[akutz]

Hi @liztio,

Your answers were spectacularly succinct solutions to my inquiries, thank you!

[akutz]

Hi all,

I have tried to condense some of the call (recording) we had last week regarding the next steps for periodic e2e testing as well as restate some of what Liz said above as a way to teach myself about the subject matter. Please let me know if the following is as clear as mud :)

Local e2e test workflow

The following steps illustrate a basic e2e test workflow:

1. Acquire AWS credentials
2. Bootstraps a local Kubernetes cluster using Kind
3. Applies the Cluster API (CAPI) and CAPA controllers to the Kind cluster
4. Uses the CAPI and CAPA controllers to boot an AWS cluster, and therefore machine, in AWS
5. Verify the CAPA-deployed cluster is healthy and online

A golang-based test, aws_test.go, already exists that implements most of the above workflow. The existing test does not implement all of step five, instead the verification process is currently limited to querying AWS to see if the deployed instance is in the _Ready_ state.

Periodic e2e test workflow

This section assumes periodic tests, but the same principals apply to PR-blocking tests as well. Either type of test is configured as a Prow job. The following steps illustrate a periodic e2e test workflow:

1. Wait on clean AWS creds from Boskos, a service running on the same cluster as Prow
2. With the AWS credentials, execute the local e2e test workflow (see above)
3. Return AWS credentials as dirty back to Boskos

AWS account cleanup

CAPA should work in a basically pristine environment, since that is what customers will have when they use CAPA for the first time. Because the AWS accounts used for periodic e2e tests are reused, any resources created by a test must be cleaned up before any other test can use the same shared account.

The final step of a test job is to return the AWS credentials to Boskos in the dirty state. Because of the requirement that an account should be in a pristine state prior to a test run, the accounts in a dirty state must be cleaned. That is achieved by configuring another service in the same cluster where Prow and Boskos are running. The new service is a Boskos janitor -- a process that queries Boskos for resources of a certain type and state and takes actions on said resources.

Thanks to Liz there is now an AWS janitor for Boskos that may be used to clean the shared AWS accounts and return them to the clean state.

Next steps

• [x] The AWS accounts need to be added to Boskos. Liz pinged the on-call contact for test-infra last week, and I've followed up with Erick Fejta to find out if the credentials have been added. According to Cole Wagner the accounts have been added.
• [ ] The AWS janitor for Boskos has been merged, but it is not yet configured as a service on the Prow cluster to connect to Boskos. This piece of glue needs to be written. I have pinged @randomvariable as he expressed interest in helping with this above.
• [ ] Finally, the Prow job(s) for periodic CAPA e2e tests will need to be written.

(cc @liztio @randomvariable @vincepri @rdodev)

[randomvariable]

Seems about right. I have touched the Prow jobs test-infra a small amount, but Slack will have more expertise anyway.

Also can probably rattle off some of the AWS API specific janitors and sort out dependency management where we have gaps.

[randomvariable]

One more task:

To get a e2e MVP in, the controller is started with the root credentials coming from boskos. Actually want to test it with the IAM policies in our CloudFormation template, which will mean creating an AWS IAM keypair for the bootstrap user and feeding that into the e2e test.

[randomvariable]

and

• [ ] Rename pull-cluster-api-provider-aws-bazel-e2e to pull-cluster-api-provider-aws-bazel-integration and make the latter blocking.

Optionally:

• [ ] Enable Bazel caching

[detiber]

@randomvariable Do we actually want to make the e2e jobs blocking? It seems like it would be more prudent to have them live as periodics to avoid having long running blocking jobs.

[randomvariable]

I'm agnostic. The current e2e is actually an integration test bazel run //test/integration that makes sure the controller runs and that's it. It could complete in two minutes if we turn on bazel caching, but also not sure how useful it is in and of itself?

bazel run //test/e2e actually spins up a cluster.

[detiber]

In that case :+1: to running the integration test as PR blocking

[akutz]

Hi @randomvariable,

While I agree that the integration test doesn't validate the e2e workflow, doesn't it at least validate the CRDs and any activity involved in deploying the CAPI/CAPA controllers themselves? That alone seems like it would be nice to block if some error is introduced.

[akutz]

Hi All,

Here's a summary of the completed and outstanding work towards periodic e2e testing for CAPA. This is not an exhaustive list -- only the work for which I'm responsible in case there are any issues that necessitate a revert of a merged PR. Hopefully that won't happen, but in case it does, the following PRs are a good place to start looking:

Test-infra PRs

• [x] boskos-client: HTTP retry on timeout (https://github.com/kubernetes/test-infra/pull/11377)
• [x] boskos-client: AcquireWait/AcquireByStateWait (https://github.com/kubernetes/test-infra/pull/11395)
• [x] boskos: Create a Boskos client Dialer (https://github.com/kubernetes/test-infra/pull/11409)
• [x] boskos: AWS janitor region via env var (https://github.com/kubernetes/test-infra/pull/11425)
• [x] capa: Update CAPA job/grid config e2e->integration (https://github.com/kubernetes/test-infra/pull/11459)
• [x] boskos: Remove glog from AWS janitor (https://github.com/kubernetes/test-infra/pull/11462)
• [x] AWS janitor clean -all option (https://github.com/kubernetes/test-infra/pull/11482)
• [x] aws-janitor: Support for CloudFormation Stacks (https://github.com/kubernetes/test-infra/pull/11504)
• [ ] boskos: Deploying a new Boskos janitor for AWS (https://github.com/kubernetes/test-infra/pull/11511)
• [ ] Add CAPA e2e periodic and post-submit jobs (https://github.com/kubernetes/test-infra/pull/11524)

CAPA PRs

• [x] Configurable region (https://github.com/kubernetes-sigs/cluster-api-provider-aws/pull/598)
• [x] Separate e2e and integration scripts (https://github.com/kubernetes-sigs/cluster-api-provider-aws/pull/603)
• [x] e2e tests (https://github.com/kubernetes-sigs/cluster-api-provider-aws/pull/606)

I will be OOTO Wed-Fri of this week, so I wanted to at least enqueue the PRs needed to close Phase 1 of CAPA e2e testing.

cc @detiber @clintkitson @timh

[akutz]

Hi All,

I'm going offline a bit to get ready for my trip. Two things:

1. I've addressed all of the outstanding comments/suggestions in the un-merged PRs above. Please take a moment to give the three PRs another once-over so they can hopefully be merged to provide e2e testing.
2. Before I leave I will amend this issue with a list of new/existing issues that should be addressed for Phase 2 of e2e testing. Some of them Liz identified early on, such as the resource ordering and how its dependency graph complicates the janitor cleanup. I just want to make sure all of the priority concerns with respect to e2e are captured in GH issues and listed here to provide the same level of convenience and help Liz provided me with her thorough brain-dump above.

cc @detiber

[timothysc]

@ncdc - Can we help offload Andrew here in the next couple of days?

[akutz]

Hi @timothysc,

I engaged with Andy on Slack, and he pointed me in the direction of resources. Thank you!

[chuckha]

/close

this is done