Katib: Long term solution for AutoML CI/CD test infrastructure

[andreyvelich]

/kind feature

We need to figure out a long term plan to run CI test infrastructure under AutoML projects.
Right now, we use Travis to run our unit and integration tests, kubeflow-ci and automl-ci project to run our e2e tests, using Prow.
During Prow test we follow these steps:

• We submit Argo workflow from ksonnet app to kubeflow-testing cluster in kubeflow-ci project. Do we want to refactor this workflow using Tekton, for example?

• For pre-submit we use gcr.io/automl-ci registry and automl-ci project to build and push images.
  For post-submit we use gcr.io/kubeflow-images-public registry and kubeflow-ci project to build and push images.
  As mentioned here: https://github.com/kubeflow/katib/pull/1326#issuecomment-689512234 we can switch to apps-cd to release our images instead of using post-submit. @gaocegege @johnugeorge Do we want to use our own registry under automl-ci project or continue use gcr.io/kubeflow-images-public ?

• For e2e test we create Kubernetes cluster under automl-ci project, deploy manifests and submit required Experiments.

Since kubeflow-ci is maintaining only for 20% (https://github.com/kubeflow/testing/issues/737) we need to figure out the way to deploy our test infra. Some possible solutions can be:

• Use Kind with GitHub actions to build images and run our e2e tests.
• Switch to AWS common test infra on which @Jeffwan is currently working.
• Continue maintain automl-ci project, create cluster there to trigger Prow jobs.

/cc @gaocegege @johnugeorge @jlewi

[issue-label-bot[bot]]

Issue Label Bot is not confident enough to auto-label this issue.
See dashboard for more details.

[andreyvelich]

/priority p1

[andreyvelich]

As discussed on the meeting, we should try to investigate GitHub actions in Katib e2e workflow.
@gaocegege Can you show repo example with GitHub action and k8s server that you mentioned before?

[jlewi]

@kubeflow/wg-automl-leads What is an ETA for migrating off shared infra? Is 4 weeks enough time? If not please suggest an alternative.

[andreyvelich]

@jlewi We are waiting for the @Jeffwan implementation with AWS restructuring.
@PatrickXYS has already created the initial PR: https://github.com/kubeflow/testing/pull/755.
Once the AWS infra will be ready, we will give ETA for the Katib project.

At the meantime, we were thinking about switch to GitHub actions. I debugged it here: https://github.com/andreyvelich/katib/pull/3 and that can work for Katib's Experiments.
We are investigating, if the GitHub actions can be our solution for the all CI testing later.

[jlewi]

@andreyvelich that is an insufficient committement from the @kubeflow/wg-automl-leads. @Jeffwan and @PatrickXYS are not AutoML leads; they are not responsible for @kubeflow/wg-automl-leads test and CI release.

Absent another agreed upon deadline, the default will be 4 weeks from now, October 17. That will be 3 months from the original kubeflow/testing#737

I have configured the IAM permissions for kubeflow-testing@ to write to kubeflow-images-public to expire on 2020/10/17. So at a minimum you will need to find an alternative solution for building and publishing your images by then.

[andreyvelich]

@jlewi October 17 as a deadline looks good for us.
We verified that GitHub actions can work.

If we don't switch to AWS test infra until the deadline, we will use GitHub actions.
/cc @johnugeorge @gaocegege

[jlewi]

Fantastic thank you!

[andreyvelich]

For the AutoML we are waiting for the resolving this comment: https://github.com/kubeflow/internal-acls/issues/348#issuecomment-704614217.
After set the webhook we can start to use AWS test infra.

@kubeflow/google-admins Since you have admin access to all repositories, can you help us with the webhook set-up?

/cc @johnugeorge

[issue-label-bot[bot]]

Issue-Label Bot is automatically applying the labels:

| Label | Probability |
| ------------- | ------------- |
| area/engprod | 1.00 |

Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback!
Links: app homepage, dashboard and code for this bot.

[jlewi]

@andreyvelich per my earlier comment permissions for publishing to kubeflow-images-public have now expired.

[jlewi]

@andreyvelich it looks like the cloudbuild service account [email protected] for project kubeflow-ci should also have had its permissions removed as part of the lockdown of kubeflow-images-public.
I have updated it to expire on 11/02

[andreyvelich]

We are using the AWS test infra currently: https://github.com/kubeflow/katib/pull/1356.
Let's continue discussion for the release infra and removing ksonnet in: https://github.com/kubeflow/katib/issues/1367, https://github.com/kubeflow/katib/issues/1376.