Cluster-api-provider-azure: Implement better testing workflow

Created on 15 Oct 2018 · 11Comments · Source: kubernetes-sigs/cluster-api-provider-azure

The integration tests are currently triggered manually after a preliminary code review.

It would be nice to have this somewhat automated. Previously, Travis was set to trigger builds on PRs (before the repo was made public). However, there are a few issues with this. Most importantly, a PR could contain specially crafted code to dump secrets/POST them to some server somewhere.

In addition, we could probably split the integration tests and the unit tests.

help wanted kinfeature lifecyclstale prioritimportant-soon

Source

vannrt

Most helpful comment

@Pothulapati -- thanks for offering to work on this, we can definitely use the help! :)

To answer your questions:

I see that Azure Pipelines is being used. It should be replaced with K8s Test-infra, right?

Yes, eventually it will be. Please work with SIG Testing to gather a list of requirements to make the switch. @nikhita, @BenTheElder, and @fejta are all good resources to reach out to help you get started. Document this in a Google Doc or here on this issue and tag me for visibility.

For Now, I have thought only about Pre-checks for PR's. Any more ideas?

Pre-Checks for PR, what are the things that you expect apart from build, test and e2e?

For now, the pre-checks will suffice. SIG Testing folks will be able to give you more granular details.

For builds and tests, I see that Bazel is being used in aws and others. I think it helps faster the builds by caching in test-infra. Should we also use it?

Yes, we will be using Bazel in future, but it's a long-term goal at the current moment. If you have time to get to it, then it would be appreciated!

e2e Tests would need Azure Cred's to spin up the machines. Boskos can be used to manage the creds. Will CNCF be helping us with creds?

I'm currently working with @lachie83 at Microsoft to secure credentials. Once we're further along in the process, I'll provide everyone an update.

justaugustus on 14 Jan 2019

👍2

All 11 comments

Hello,
I am willing to take this Issue, and have researched how Cluster-Api-Provider-AWS was doing it. I have some questions :)

I see that Azure Pipelines is being used. It should be replaced with K8s Test-infra, right?
For Now, I have thought only about Pre-checks for PR's. Any more ideas?
Pre-Checks for PR, what are the things that you expect apart from build, test and e2e?
For builds and tests, I see that Bazel is being used in aws and others. I think it helps faster the builds by caching in test-infra. Should we also use it?
e2e Tests would need Azure Cred's to spin up the machines. Boskos can be used to manage the creds. Will CNCF be helping us with creds?

Feel free to add any more suggestions/ideas,
Thanks.

@justaugustus @tariq1890

Pothulapati on 13 Jan 2019

Oops! Looks like Bazel is already being added in #42 . Should I wait till the PR is merged, to get a better view of what is needed?

Pothulapati on 13 Jan 2019

@Pothulapati -- thanks for offering to work on this, we can definitely use the help! :)

To answer your questions:

I see that Azure Pipelines is being used. It should be replaced with K8s Test-infra, right?

For Now, I have thought only about Pre-checks for PR's. Any more ideas?

Pre-Checks for PR, what are the things that you expect apart from build, test and e2e?

For now, the pre-checks will suffice. SIG Testing folks will be able to give you more granular details.

For builds and tests, I see that Bazel is being used in aws and others. I think it helps faster the builds by caching in test-infra. Should we also use it?

Yes, we will be using Bazel in future, but it's a long-term goal at the current moment. If you have time to get to it, then it would be appreciated!

e2e Tests would need Azure Cred's to spin up the machines. Boskos can be used to manage the creds. Will CNCF be helping us with creds?

I'm currently working with @lachie83 at Microsoft to secure credentials. Once we're further along in the process, I'll provide everyone an update.

justaugustus on 14 Jan 2019

👍2

/milestone mvp

soggiest on 7 Feb 2019

/milestone baseline

justaugustus on 13 Feb 2019

/kind feature
/priority important-soon

justaugustus on 6 Mar 2019

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

fejta-bot on 4 Jun 2019

/remove-lifecycle stale

justaugustus on 6 Jun 2019

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

fejta-bot on 4 Sep 2019

This is good to close as we've been wired up to Prow for some time now.
/assign
/close

justaugustus on 2 Oct 2019

@justaugustus: Closing this issue.

In response to this:

This is good to close as we've been wired up to Prow for some time now.
/assign
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.