Website: Test and update the GCP end to end Kubeflow tutorial for Kubeflow v0.6

[sarahmaddox]

Test and update the GCP end to end Kubeflow tutorial. Many of the docs have been updated since we last tested it, including the OAuth docs. We need to make sure the tutorial works end to end.

https://www.kubeflow.org/docs/gke/gcp-e2e/

[issue-label-bot[bot]]

Issue-Label Bot is automatically applying the label improvement/enhancement to this issue, with a confidence of 0.66. Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback!

Links: app homepage, dashboard and code for this bot.

[sarahmaddox]

I've marked this one for tackling during the Kubeflow Doc Sprint. This is a good task for a tech writer to tackle during the doc spritn.

[bd12]

hi there,
I've just recently tried to deploy kf on GCP and want to share my experiences. I tried one time per GUI and one time per cli. Both happened over the course of the last two weeks and unfortunately both shots were not successful.
Here's what I have faced following the docs:

1. shot: gui approach included usage of IAP --> Seemed like gke creation and kf deployment were successful, but the IAP integration did not work for me (having followed the kf docs).
2. shot: cli approach: with basic auth: script and packaging problems (at least on macOS). Creation of gke cluster seemed to be working, but couldn't even bring up the namespaces inside with ksonnet.

I will try out stuff over the next days and can loop back my experiences. I'm very familiar with k8s and openshift, but less firm with the GCP services and terms. Gonna try to describe the tested scenarios as concisely as possible, though.

edit1: After a third try I can say that a clean gui deployment with basic auth works and apparently still uses ksonnett according to the logs.

[gogasca]

I will take a look at this one

[sarahmaddox]

/assign @gogasca

[dwightjl]

I went through this tutorial and encountered an issue with the "Train the model on GKE" section. I get the following error when I run kustomize build . |kubectl apply -f -:

$ kustomize build . |kubectl apply -f -
Error: no matches for OriginalId kubeflow.org_v1beta2_TFJob|~X|$(trainingName); no matches for CurrentId
 kubeflow.org_v1beta2_TFJob|~X|$(trainingName); failed to find unique target for patch kubeflow.org_v1be
ta2_TFJob|$(trainingName)

I'm not familiar with kustomize, so I suspect it might an issue with one of the .yaml files that kustomize uses. I'm pretty sure I followed the earlier steps correctly.

[numerology]

@dwightjl Same issue.

[sarahmaddox]

Hallo @gogasca How are you doing with testing and updating this tutorial? I'd like to have it working in v0.6 before we archive off the v0.6 doc branch.

[ricoms]

I went through this tutorial and encountered an issue with the "Train the model on GKE" section. I get the following error when I run kustomize build . |kubectl apply -f -:

$ kustomize build . |kubectl apply -f -
Error: no matches for OriginalId kubeflow.org_v1beta2_TFJob|~X|$(trainingName); no matches for CurrentId
 kubeflow.org_v1beta2_TFJob|~X|$(trainingName); failed to find unique target for patch kubeflow.org_v1be
ta2_TFJob|$(trainingName)

I'm not familiar with kustomize, so I suspect it might an issue with one of the .yaml files that kustomize uses. I'm pretty sure I followed the earlier steps correctly.

I'm following this tutorial. I'm pretty sure I followed the instructions correctly and I reached the same problem at step kustomize build .

$ kustomize build .
Error: no matches for OriginalId kubeflow.org_v1beta2_TFJob|~X|$(trainingName); no matches for CurrentId kubeflow.org_v1beta2_TFJob|~X|$(trainingName); failed to find unique target for patch kubeflow.org_v1beta2_TFJob|$(trainingName)

I installed kustomize from "Quickly curl the latest" here

my version is as below:
Version: {KustomizeVersion:3.1.0 GitCommit:95f3303493fdea243ae83b767978092396169baf BuildDate:2019-07-26T18:11:16Z GoOs:linux GoArch:amd64}

[ricoms]

this thread might help

[ricoms]

At the end of "Deploying the UI" it says: "Now in your Cloud Shell interface, press the web preview button and select "Preview on port 8080" to open the web interface in your browser:"... but I'm running everything on my local shell, and I was not warned that I should use Cloud Shell the entire time.

I accessed http://localhost:8080/ and I could see the UI, although I'm not sure: Am I serving it locally or it's all available at the kubernetes cluster?

Would the idea of this to run kubectl port-forward svc/web-ui 8080:80 on a kubernetes pod so anyone can play with the model?

[gogasca]

@sarahmaddox Zane @zanedurante will be taking over this PR.

[sarahmaddox]

Great, thanks @gogasca

/assign @zanedurante

[k8s-ci-robot]

@sarahmaddox: GitHub didn't allow me to assign the following users: zanedurante.

Note that only kubeflow members, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time.
For more information please see the contributor guide

In response to this:

Great, thanks @gogasca

/assign @zanedurante

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[sarahmaddox]

@zanedurante Please would you comment on this issue so that I can assign it to you?

[zanedurante]

Sure thing--I'll work on this for the remainder of my internship.

[sarahmaddox]

/assign @zanedurante

That's great news! Thanks Zane.

[zanedurante]

Since this is my last week, I don't think I'll be able to finish fixing this tutorial. However, I did run into the same problems that many people mentioned. IAP integration gave me an error as was mentioned by @bd12 (however I could still connect to the endpoint despite receiving the IAP error in the GUI deployer). I ran into the same issues as @ricoms when using the newest Kustomize version as well. @sarahmaddox who do you think would be a good person to reassign this issue to?

[sarahmaddox]

See also the comments on issue https://github.com/kubeflow/website/issues/1012

[sarahmaddox]

I'll work through this tutorial and liaise with @texasmichelle if the sample needs fixing.

/assign

[sarahmaddox]

I've run through the tutorial at https://www.kubeflow.org/docs/gke/gcp-e2e/ using a Kubeflow v0.6.1 installation (installed via the CLI). I verify that I encountered the problems reported by others:

1. When attempting to run model.py in a notebook (this _optional_ step), I get the same error as reported in issue https://github.com/kubeflow/website/issues/1012.

2. When running kustomize build . |kubectl apply -f - (this step) I get the error reported above.

@texasmichelle Please would you take a look at the sample? Perhaps @DanSanche can help to shed some light on the problems.

My PR https://github.com/kubeflow/website/pull/1079 shows the steps I took - I've updated the tutorial as far as I got. Here's a preview of the updated doc from the PR).

[SinaChavoshi]

@ricoms regarding:

I accessed http://localhost:8080/ and I could see the UI, although I'm not sure: Am I serving it locally or it's all available at the kubernetes cluster?

Could shell is just like your local environment, it is basically a VM running on the cloud. So the instructions should work similarity regardless on local dev as long as you have setup gcloud.

The UI itself is hosted on the k8 cluster, when you run kubectl port-forward svc/web-ui 8080:80 it creates a tunnel so you can access the UI via localhost:8080 from your local machine. This is similar to ssh port forwarding.

[SinaChavoshi]

@ricoms & @dwightjl regarding the issue with kustomize build . I'm going through the sample I should be able to share an update shortly.

In the mean time just FYI, kustomize simply allows to mix and match yamls based on the name space. though the core logic of what is happening is kept in the YAML files, you can modify and run them separately.

[Daniel-Sanche]

@sarahmaddox

1. When attempting to run model.py in a notebook (this optional step), I get the same error as reported in issue #1012.

I opened a PR to partially address this. I think we should also change the docs to instruct readers to use a specific image for the notebook server (maybe 1.13?), so we can ensure that everyone is using a compatible version. I don't know of any incompatible ones, but it might be good to be safe

[jinchihe]

I went through this tutorial and encountered an issue with the "Train the model on GKE" section. I get the following error when I run kustomize build . |kubectl apply -f -:

$ kustomize build . |kubectl apply -f -
Error: no matches for OriginalId kubeflow.org_v1beta2_TFJob|~X|$(trainingName); no matches for CurrentId
 kubeflow.org_v1beta2_TFJob|~X|$(trainingName); failed to find unique target for patch kubeflow.org_v1be
ta2_TFJob|$(trainingName)

I'm not familiar with kustomize, so I suspect it might an issue with one of the .yaml files that kustomize uses. I'm pretty sure I followed the earlier steps correctly.

This is a bug for kustomize, pls use kustomize v2.0.3, thanks.

[sarahmaddox]

Thanks @jinchihe! I saw the chain of issues that led to the conclusion we should use kustomize v2.0.3, but that doesn't seem like a sustainable solution. kustomize is already at v3.1.0. Do you know when the kustomize issue will be fixed so that kustomize behavior suits what we want? From reading the issues, it seems this may never happen?

[jinchihe]

@sarahmaddox The bug of kustomize exists on Kustonize V2.1 and V3.0, I'm not sure if that's fixed in the lastest version, but will confirm with kustomize dev or check code once get chance, thanks.

[sarahmaddox]

@DanSanche

I opened a PR to partially address this.

Thanks so much! This fixed the problem. I've updated the tutorial in my PR to reflect the new behaviour.