Kubeflow: [v0.5.1] Best practice setting notebook tolerations / node selector

[travishen]

kubeflow version: v0.5.1

Hi, I want to create notebook instances by notebook-controller via jupyter-web-ui, which I prefer to schedule them on special node type, from a custom node-pool for some reasons:

• High CPU/Memory machine needed
• I like it to be preemptible

In my use case, I have a special node-pool n32-highmem-preemptible-autoscale-0-1, then I create a notebook server that allocate 30Gi CPU / 200Gi Memory, assume the node pool will scale up to 1 node, then I remove the notebook server from the web UI, assume the node pool will scale down to 0. So far this use case works, but I think I still need to set the constrains, because I don't want any other pod to schedule on this node.

I'm aware that it's practicable because I've tried using preemptible vm on pipeline, base on the documentation: https://www.kubeflow.org/docs/pipelines/preemptible/

Any recommendation or best practices will be awesome, thanks for your time!

[issue-label-bot[bot]]

Issue-Label Bot is automatically applying the label community/question to this issue, with a confidence of 0.86. Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback!

Links: app homepage, dashboard and code for this bot.

[jlewi]

One option would be to create them via kubectl and not the UI by creating notebook resources. Then you can specify the full podtemplate spec.

[travishen]

Thanks for the reply. For now I'll try what your suggested as a workaround.