Pipelines: Error: "Sorry, /notebook/kubeflow/kfp-notebook is not a valid page"

[Ark-kun]

I've deployed KFP 0.1.20 with IAP using web deployer.
I've created a notebook using the Kubeflow UI.
WHen I click "Connect" I'm redirected to the https://foo.endpoints.bar.cloud.goog/notebook/kubeflow/kfp-notebook page which shows "Sorry, /notebook/kubeflow/kfp-notebook is not a valid page".

[rileyjbauer]

Has anyone else seen this? When I started a new KFP 0.1.20 deployment on a new cluster, I did not encounter this issue

[Edilmo]

I have the same issue. I have try with different names and also deleting the PVC generated before to retry.
The output of kubectl -n kubeflow get notebooks in my case is showing the list of running notebooks:

git:(master) kubectl -n kubeflow get notebooks
NAME      AGE
daniela   23d
edi       5h
hillary   25d
isabel    3d
lino      25d
miguel    22d

The kubernetes cluster is ok from a resource perspective and there is no pods with error or pending state.
The cluster have almost a month working without problems. I'm experiencing the issue since yesterday.
Kubernetes version 1.12 deployed in GCP.
Kubeflow build version 0.5.0

Any advice?

[fabito]

Having the same issue here. I'm using port forward in the meantime.

These are the entries of the Ambassador Router Table:

URL | Service | Weight
------------ | ------------- | -------------
http://localhost:8877/tensorboard/ tensorboard/ | tensorboard.kubeflow:9000 | 100.0%
http://localhost:8877/ambassador/v0/check_ready | 127.0.0.1:8877 | 100.0%
http://localhost:8877/ambassador/v0/check_alive | 127.0.0.1:8877 | 100.0%
http://localhost:8877/ambassador/v0/ | 127.0.0.1:8877 | 100.0%
http://localhost:8877/pipeline | ml-pipeline-ui.kubeflow | 100.0%
http://localhost:8877/jupyter/ | jupyter-web-app.kubeflow | 100.0%
http://localhost:8877/tfjobs/ | tf-job-dashboard.kubeflow | 100.0%
http://localhost:8877/katib/ | katib-ui.kubeflow | 100.0%
http://localhost:8877/argo/ | argo-ui.kubeflow | 100.0%
http://localhost:8877/data | ml-pipeline-ui.kubeflow | 100.0%
http://localhost:8877/ | centraldashboard.kubeflow | 100.0%

Shouldn't we see a entry for /notebook/kubeflow ?

The Ambassador annotation for the notebook service is there:

# kubectl -n kubeflow describe service/chameleon-client

Name:              chameleon-client
Namespace:         kubeflow
Labels:            <none>
Annotations:       getambassador.io/config:
                     ---
                     apiVersion: ambassador/v0
                     kind:  Mapping
                     name: notebook_kubeflow_chameleon-client_mapping
                     prefix: /notebook/kubeflow/chameleon-client
                     rewrite: /notebook/kubeflow/chameleon-client
                     timeout_ms: 300000
                     service: chameleon-client.kubeflow
                     use_websocket: true
Selector:          statefulset=chameleon-client
Type:              ClusterIP
IP:                10.27.252.89
Port:              <unset>  80/TCP
TargetPort:        8888/TCP
Endpoints:         10.24.3.2:8888
Session Affinity:  None
Events:            <none>

But seems like Ambassador haven't detected it.

[fabito]

Forcing Ambassador to restart solved the issue.
See: https://github.com/kubeflow/kubeflow/issues/3414#issuecomment-499964564

[Ark-kun]

I have an issue:
My first notebook works fine, but notebooks spawned later are not accessible.

[aarthipa]

Hi - I see the same error on latest version v1.0.0. Unlike the previous versions, i cannot restart the Ambassador pods either :(

[ankitsingi]

Stuck with same issue. Any help ?

[ConverJens]

Same here. Using KF 1.0.2 which also doesn't use Ambassador anymore.

[ConverJens]

Bump this

[ConverJens]

@Ark-kun Any news on this?

[ConverJens]

@Ark-kun @aarthipa @ankitsingi In the issue linked in my previous comment @jlewi helped me debug this and as he pointed out, the issue was a missing virtualservice. This should have been created by the notebook-controller but it wasn't. Still not sure why that is. So workaround if you only have a very limited number of notebook servers is to manually deploy a virtualservice for each notebook server.

[ConverJens]

@Ark-kun @jlewi Found the issue, for some reason the USE_ISTIO var is hard-coded to false in notebook-controller/base/deployment.yaml. So even though USE_ISTIO is set to true in notebook-controller/overlays/istio/param.env the Kustomize merge will fail since base/deployment.yaml doesn't use the variable USE_ISTIO.

[jlewi]

Closing this issue. Original issue is no longer relevant since it predates ISTIO.