Website: End-to-end example needs to be changed to deploy tfjob into user profile namespace

[amygdala]

https://www.kubeflow.org/docs/gke/gcp-e2e/
(I'll file a kubeflow/examples bug too).

Looks like this example is broken right now; looks like the tfjob needs to be deployed into a user profile namespace now. The quick workaround is to just direct the output of 'kustomize build' to a yaml file rather than piping directly to kubectl; then editing that file to change the namespace, then applying it. The more principled fix would be to change the set of kustomize commands.

(for googlers, see this discussion for background)

[issue-label-bot[bot]]

Issue-Label Bot is automatically applying the labels:

| Label | Probability |
| ------------- | ------------- |
| kind/bug | 0.82 |

Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback!
Links: app homepage, dashboard and code for this bot.

[amygdala]

Also, I assume that we need to update the tfjob docs as well (https://www.kubeflow.org/docs/components/training/tftraining/). Looks like either they need to deploy into a kubeflow-<user> namespace, or give the correct permissions to a service account in the namespace they want to use.
(Perhaps this should be a separate bug).

[jtfogarty]

/area docs
/priority p2

[amygdala]

I'd put this as a 'p0' since it's the "getting started" tutorial.

[jal06]

When I run the examples for Katib (doc https://www.kubeflow.org/docs/components/hyperparameter-tuning/hyperparameter/ ), they run in the namespace kubeflow. Looks like they also must also run in a user namespace. Do you agree ? Do we need to open another ticket ? (sorry for asking those questions, I am new to kubeflow)

[jlewi]

@jal06 please open a separate issue for katib examples.

[jlewi]

Bump to P0 because we should fix for 1.0.

[sarahmaddox]

@amygdala Are you working on this tutorial and if so, can I assign this issue to you?

[jal06]

@jal06 please open a separate issue for katib examples.

Thanks for your answer
I open a new issue for Katib : https://github.com/kubeflow/website/issues/1599

[sarahmaddox]

/cc @DanSanche

@DanSanche I'm cc-ing you on this bug about the end-to-end tutorial (not the afore-mentioned Katib issue). @amygdala is planning to chat to you about it.

[sarahmaddox]

Related issue in the kubeflow/examples repo: https://github.com/kubeflow/examples/issues/713

[jlewi]

I might suggest that we create a notebook to run this mnist example. Similar to what we do for xgboost_synthetic

The guide would then be

1. Deploy Kubeflow
2. Start a notebook
3. Clone kubeflow/examples
4. Open the notebook
5. Run the notebook

This has a couple advantages

1. It should make it easier to add E2E testing for the example
   
   
   
   • We can just run the notebook like we do for xgboost_synthetic and make sure it runs to completion
   
   
2. It should be much easier for users they can just run the notebook
3. We can potentially simplify the example by using RAW YAML rather than introducing kustomize
   
   
   
   • We can write simple python snippets to do any substitution necessary
   
   

[amygdala]

A notebook SGTM; leaving this issue open. We'd want to change the instructions both in the docs e2e page (kf/website), and the mnist example README (kf/examples).
This would address https://github.com/kubeflow/website/issues/1553 as well.

See also https://github.com/kubeflow/examples/issues/713