Kubeflow: Jupyter notebook pod scheduling fails with 'serviceaccount "default-editor" not found'

[solovyevt]

I'm experiencing problems during creating of Jupyter notebook pod on my on-premise k8s deployment. StatefulSet fails with the nest error message:

Warning FailedCreate 4m46s (x26 over 10m) statefulset-controller create Pod test-0 in StatefulSet test failed error: pods "test-0" is forbidden: error looking up service account kubeflow-workspace/default-editor: serviceaccount "default-editor" not found

notebook-controller log:

{"level":"info","ts":1557412479.1712413,"logger":"entrypoint","msg":"setting up client for manager"} {"level":"info","ts":1557412479.1717148,"logger":"entrypoint","msg":"setting up manager"} {"level":"info","ts":1557412479.2517176,"logger":"entrypoint","msg":"Registering Components."} {"level":"info","ts":1557412479.2517457,"logger":"entrypoint","msg":"setting up scheme"} {"level":"info","ts":1557412479.2518659,"logger":"entrypoint","msg":"Setting up controller"} {"level":"info","ts":1557412479.2519147,"logger":"kubebuilder.controller","msg":"Starting EventSource","controller":"notebook-controller","source":"kind source: /, Kind="} {"level":"info","ts":1557412479.2520208,"logger":"kubebuilder.controller","msg":"Starting EventSource","controller":"notebook-controller","source":"kind source: /, Kind="} {"level":"info","ts":1557412479.2520826,"logger":"kubebuilder.controller","msg":"Starting EventSource","controller":"notebook-controller","source":"kind source: /, Kind="} {"level":"info","ts":1557412479.2521398,"logger":"kubebuilder.controller","msg":"Starting EventSource","controller":"notebook-controller","source":"kind source: /, Kind="} {"level":"info","ts":1557412479.2521877,"logger":"entrypoint","msg":"setting up webhooks"} {"level":"info","ts":1557412479.252193,"logger":"entrypoint","msg":"Starting the Cmd."} {"level":"info","ts":1557412479.352339,"logger":"kubebuilder.controller","msg":"Starting Controller","controller":"notebook-controller"} {"level":"info","ts":1557412479.452473,"logger":"kubebuilder.controller","msg":"Starting workers","controller":"notebook-controller","worker count":1} {"level":"info","ts":1557412479.452622,"logger":"controller","msg":"Updating StatefulSet","namespace":"kubeflow-workspace","name":"test2"} {"level":"info","ts":1557412479.4628272,"logger":"controller","msg":"Pod not found..."} {"level":"info","ts":1557412953.8582733,"logger":"controller","msg":"Creating StatefulSet","namespace":"kubeflow-workspace","name":"test"} {"level":"info","ts":1557412953.872616,"logger":"controller","msg":"Creating Service","namespace":"kubeflow-workspace","name":"test"} {"level":"info","ts":1557412953.8899689,"logger":"controller","msg":"Pod not found..."} {"level":"info","ts":1557412953.8900712,"logger":"controller","msg":"Updating StatefulSet","namespace":"kubeflow-workspace","name":"test"} {"level":"info","ts":1557412953.8974965,"logger":"controller","msg":"Pod not found..."}

I've checked the code there: https://github.com/kubeflow/kubeflow/blob/869077d015d2e7cdb0d6af9af3d0a718aa786c3c/components/profile-controller/pkg/controller/profile/profile_controller.go
and I also couldn't find default-viewer service account.

What might be the problem?

[issue-label-bot[bot]]

Issue-Label Bot is automatically applying the label kind/bug to this issue, with a confidence of 0.76. Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback!

Links: app homepage, dashboard and code for this bot.

[solovyevt]

Update: my bad, I've used kfctl init $KFAPP without --version=v0.5.0, everything works now. Issue may be closed.

[doschkinow]

I was able to work around this (I am using kfctl v0.6.1-rc.2-1-g3a37cbc6) by creating the service account:

kubectl -n kubeflow create serviceaccount default-editor

(I had created the jupyter server in the kubeflow namespace)

[gar1t]

I'm seeing this in 0.6.2 and command offered by @doschkinow does work-around the problem.

I think this issue should be reopened.