Pipelines: [Hosted Pipelines] Deployment fails with `BackoffLimitExceeded`

[numerology]

What steps did you take:

• Create a new GKE cluster and deploy KFP 0.2.5 through GKE MKP.

What happened:

Deployment failed in assembly phase:

unable to recognize "/data/resources.yaml": no matches for kind "Deployment" in version "apps/v1beta2

What did you expect to happen:

Successfully deploy KFP.

Environment:

How did you deploy Kubeflow Pipelines (KFP)?
MKP deployment, version = 0.2.5

Anything else you would like to add:

I believe it's a MKP side regression.
/kind bug
/priority p0

[numerology]

/assign @numerology
/assign @rmgogogo

[numerology]

Workaround: manually deploy a GKE cluster with older k8s version and deploy KFP to it.

1. Deploy a GKE cluster with k8s 1.14 by running (either in your local machine or cloud shell):

gcloud container clusters create "my-kfp-cluster" --region=<GCE_REGION>, --cluster-version="1.14" --num-nodes=3 --machine-type="n1-standard-2" --scopes=storage-rw,cloud-platform --no-enable-autoupgrade

1. Deploy KFP to "my-kfp-cluster" according to this guide, from step 8. You need to skip the cluster creation.

[rmgogogo]

Workaround: manually deploy a GKE cluster with older k8s version and deploy KFP to it.

1. Deploy a GKE cluster with k8s 1.14 by running (either in your local machine or cloud shell):

gcloud container clusters create "my-kfp-cluster" --region=<GCE_REGION>, --cluster-version="1.14" --num-nodes=3 --machine-type="n1-standard-2" --scopes=storage-rw,cloud-platform --no-enable-autoupgrade

1. Deploy KFP to "my-kfp-cluster" according to this guide, from step 8. You need to skip the cluster creation.

a patch to existing 0.2.5/0.5.1 is on the way. The 0.5.2 (not released) already has the fix.

[rmgogogo]

Workaround: manually deploy a GKE cluster with older k8s version and deploy KFP to it.

1. Deploy a GKE cluster with k8s 1.14 by running (either in your local machine or cloud shell):

gcloud container clusters create "my-kfp-cluster" --region=<GCE_REGION>, --cluster-version="1.14" --num-nodes=3 --machine-type="n1-standard-2" --scopes=storage-rw,cloud-platform --no-enable-autoupgrade

1. Deploy KFP to "my-kfp-cluster" according to this guide, from step 8. You need to skip the cluster creation.

a quick patching is WIP, ETA: as soon as possible.

[rmgogogo]

All done. 0.5.1 is in Hosted. No issue. Here close.

[rmgogogo]

BTW: Hosted/GCP 0.5.1 has few big features to be documented soon.
(1) managed storage: use CloudSQL & GCS-minio for storage. So reinstallation is pretty safe.
(2) KFP SDK Cache: if no changes in certain pipeline/step, rerun will use cached data
(3) For new installations, default bucket name is changed to be "projectId-kubeflowpipelines-default". It's detailed in gcp-default-config configmap.