Pipelines: controller-manager stuck in CrashLoopBackOff

[rcleere]

What happened:
When attempting to install v0.2.0+ via kustomize I get the following error:
error: unable to recognize "manifests/kustomize/env/dev": no matches for kind "Application" in version "app.k8s.io/v1beta1"

I think more relevant is the controller-manager pod gets stuck in CrashLoopBackOff with the following log lines:

{"level":"info","ts":1580920889.5703506,"logger":"entrypoint","caller":"manager/main.go:42","msg":"setting up client for manager"}
{"level":"info","ts":1580920889.5705976,"logger":"entrypoint","caller":"manager/main.go:55","msg":"setting up manager"}
{"level":"info","ts":1580920890.067755,"logger":"entrypoint","caller":"manager/main.go:67","msg":"Registering Components."}
{"level":"info","ts":1580920890.0678134,"logger":"entrypoint","caller":"manager/main.go:70","msg":"setting up scheme"}
{"level":"info","ts":1580920890.067924,"logger":"entrypoint","caller":"manager/main.go:77","msg":"Setting up controller"}
{"level":"info","ts":1580920890.0680432,"logger":"kubebuilder.controller","caller":"controller/controller.go:120","msg":"Starting EventSource","Controller":"application-ctrl","Source":"kind source: /, Kind="}
{"level":"dpanic","ts":1580920890.0681956,"logger":"kubebuilder.source","caller":"zapr/zapr.go:129","msg":"odd number of arguments passed as key-value pairs for logging","ignored key":{"Group":"app.k8s.io","Kind":"Application"},"stacktrace":"github.com/kubernetes-sigs/application/vendor/github.com/go-logr/zapr.handleFields\n\t/go/src/github.com/kubernetes-sigs/application/vendor/github.com/go-logr/zapr/zapr.go:106\ngithub.com/kubernetes-sigs/application/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/kubernetes-sigs/application/vendor/github.com/go-logr/zapr/zapr.go:129\ngithub.com/kubernetes-sigs/application/vendor/sigs.k8s.io/controller-runtime/pkg/source.(*Kind).Start\n\t/go/src/github.com/kubernetes-sigs/application/vendor/sigs.k8s.io/controller-runtime/pkg/source/source.go:89\ngithub.com/kubernetes-sigs/application/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Watch\n\t/go/src/github.com/kubernetes-sigs/application/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:121\ngithub.com/kubernetes-sigs/application/pkg/kbcontroller.CreateController\n\t/go/src/github.com/kubernetes-sigs/application/pkg/kbcontroller/kbcontroller.go:35\ngithub.com/kubernetes-sigs/application/pkg/controller/application.Add\n\t/go/src/github.com/kubernetes-sigs/application/pkg/controller/application/application_controller.go:40\ngithub.com/kubernetes-sigs/application/pkg/controller.AddToManager\n\t/go/src/github.com/kubernetes-sigs/application/pkg/controller/controller.go:29\nmain.main\n\t/go/src/github.com/kubernetes-sigs/application/cmd/manager/main.go:78\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:198"}
{"level":"error","ts":1580920890.0681434,"logger":"kubebuilder.source","caller":"source/source.go:89","msg":"if %s is a CRD, should install it before calling Start","error":"no matches for kind \"Application\" in version \"app.k8s.io/v1beta1\"","stacktrace":"github.com/kubernetes-sigs/application/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/kubernetes-sigs/application/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/kubernetes-sigs/application/vendor/sigs.k8s.io/controller-runtime/pkg/source.(*Kind).Start\n\t/go/src/github.com/kubernetes-sigs/application/vendor/sigs.k8s.io/controller-runtime/pkg/source/source.go:89\ngithub.com/kubernetes-sigs/application/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Watch\n\t/go/src/github.com/kubernetes-sigs/application/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:121\ngithub.com/kubernetes-sigs/application/pkg/kbcontroller.CreateController\n\t/go/src/github.com/kubernetes-sigs/application/pkg/kbcontroller/kbcontroller.go:35\ngithub.com/kubernetes-sigs/application/pkg/controller/application.Add\n\t/go/src/github.com/kubernetes-sigs/application/pkg/controller/application/application_controller.go:40\ngithub.com/kubernetes-sigs/application/pkg/controller.AddToManager\n\t/go/src/github.com/kubernetes-sigs/application/pkg/controller/controller.go:29\nmain.main\n\t/go/src/github.com/kubernetes-sigs/application/cmd/manager/main.go:78\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:198"}
{"level":"error","ts":1580920890.068303,"logger":"entrypoint","caller":"manager/main.go:79","msg":"unable to register controllers to the manager","error":"no matches for kind \"Application\" in version \"app.k8s.io/v1beta1\"","stacktrace":"github.com/kubernetes-sigs/application/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/kubernetes-sigs/application/vendor/github.com/go-logr/zapr/zapr.go:128\nmain.main\n\t/go/src/github.com/kubernetes-sigs/application/cmd/manager/main.go:79\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:198"}

What did you expect to happen:
For there to not be any error messages on deploy and the controller-manager pod to run.

What steps did you take:
I locally cloned the source and used the Standalone Deployment guide to make 2 modifications:

1. Changed target namespace in file manifests/kustomize/env/dev/kustomization.yaml
2. Removed the inverse-proxy-agentlines from manifests/kustomize/base/kustomization.yaml

I then proceeded to use kubectl apply -k manifests/kustomize/env/dev to deploy to my GKE Private cluster.

Anything else you would like to add:
I have been able to run Kubeflow Pipelines v0.1.X deployments in GKE Private clusters without issue.

[parthmishra]

In KFP version 0.2.0+, you need to apply the base and wait for the application to come up:

kubectl apply -k github.com/kubeflow/pipelines//manifests/kustomize/base/crds?ref=$PIPELINE_VERSION
kubectl wait --for condition=established --timeout=60s crd/applications.app.k8s.io
kubectl apply -k github.com/kubeflow/pipelines//manifests/kustomize/env/dev?ref=$PIPELINE_VERSION

Did you do that and it's still giving the above error?

[rmgogogo]

It introduced App CRD. It's required to install App CRD first.
@IronPan

[rcleere]

Thank you, that worked. Sorry I missed that.