Tf-operator: [Test] Add support for public ECR image

[PatrickXYS]

TL;DR
tf-operator can switch to use public ECR image to support better testing user experience, which support CI + CD simultaneously.

Details
We've started work in Kaniko and ECR Credential Helper (https://github.com/awslabs/amazon-ecr-credential-helper/issues/248 + https://github.com/GoogleContainerTools/kaniko/issues/1515), after those work done, we can start engineering work in tf-operator for Public ECR Image support.

[PatrickXYS]

/cc @ChanYiLin @gaocegege @Jeffwan

[gaocegege]

Hi

Is there any progress?

[PatrickXYS]

@gaocegege There are some engineering work ongoing, and we look forward to making this work around March

[PatrickXYS]

FYI, we have added support for public ECR. From now on, we can use latest kaniko to build and push images to public ECR registry.

I think only a few changes need to be made:

1. replace image url from private ecr to public ecr
2. create public ecr registry
3. Update kaniko to latest version

/cc @gaocegege @ChanYiLin @Jeffwan

[gaocegege]

cc @andreyvelich @johnugeorge

[PatrickXYS]

Hi all, personally I plan to invest more in training part, so I could help finish this part to begin my training project contributions.

[andreyvelich]

Thanks a lot @PatrickXYS!
That would be great.

So do we want to have 2 public registries for each project ?
One for the CI and one for the release ?

[PatrickXYS]

@andreyvelich There are two options we can move forward:

1. One public ECR registry, in presubmit, we enable kaniko --no-push flag to not push to public ECR.
2. Two public ECR registries, one for presubmit, one for postsubmit.

The no.1 is development-easy but may mess up carelessly, e.g, someone mistakely remove --no-push in their PR, thus, pushing into public registry directly.
The no.2 is more robust, which need to explicitely specificy two registries in presubmit and postsubmit.

WDYT?

[andreyvelich]

If we have enough resources I prefer 2nd option since some of the project might not re-build all of the images in the CI and just use images from the release.
Also it's more safety as you mentioned.

What do others think @kubeflow/wg-training-leads ?

[gaocegege]

Hi all, personally I plan to invest more in training part, so I could help finish this part to begin my training project contributions.

Cool!

@andreyvelich There are two options we can move forward:

1. One public ECR registry, in presubmit, we enable kaniko --no-push flag to not push to public ECR.
2. Two public ECR registries, one for presubmit, one for postsubmit.

The no.1 is development-easy but may mess up carelessly, e.g, someone mistakely remove --no-push in their PR, thus, pushing into public registry directly.
The no.2 is more robust, which need to explicitely specificy two registries in presubmit and postsubmit.

WDYT?

Personally, prefer the latter.

[PatrickXYS]

Created a public ECR registry: public.ecr.aws/j1r0q0g6/training/tf-operator

Public View: https://gallery.ecr.aws/j1r0q0g6/training/tf-operator

[PatrickXYS]

Will send a follow up PR to fix test part