Katib: [katib]Is it possible to run a trial on multiple nodes at the same time instead of just one node?

[ffeelers]

Is it possible to run a trial on multiple nodes at the same time instead of just one node?:
I am trying to run katib HP experiment in a cluster gpu environment.
When I run experiment, it runs only on one node. So, trials die due to OOM error.
Should I do distributed learning using TFJob or are there other ways?

random-experiment-katib.yaml

apiVersion: "kubeflow.org/v1alpha3"
kind: Experiment
metadata:
  namespace: admin
  labels:
    controller-tools.k8s.io: "1.0"
  name: katib-test
spec:
  parallelTrialCount: 5 
  maxFailedTrialCount: 3   
  objective:  
    type: maximize  
    goal: 0.99
    objectiveMetricName: F1-score  
    additionalMetricNames:  
      - Exact-matching
  algorithm:  
    algorithmName: random
  trialTemplate: 
    goTemplate:
        rawTemplate: |-
          apiVersion: batch/v1
          kind: Job
          metadata:
            name: {{.Trial}}
            namespace: {{.NameSpace}}
          spec:
            template:
              spec:
                containers:
                - name: {{.Trial}}
                  image: kubeflow-registry.default.svc.cluster.local:30000/katib-job:CD8EB734
                  command:
                  - "python"
                  - "/app/korquad-katib.py"
                  - "--init_checkpoint=./model.ckpt-31250"
                  - "--tfrecord_file=./train.tf_record"
                  {{- with .HyperParameters}} 
                  {{- range .}}
                  - "{{.Name}}={{.Value}}"
                  {{- end}}
                  {{- end}}
                  resources:
                    request:
                      nvidia.com/gpu: 2
                  volumeMounts:
                    - mountPath: "/resources"
                      name: korquad-volume
                restartPolicy: Never
                volumes:
                  - name: korquad-volume
                    persistentVolumeCaim:
                      claimName: korquad-volume
  parameters:
    - name: --learning_rate
      parameterType: double
      feasibleSpace:
        min: "0.00003"
        max: "0.00005"
    - name: --warmup_proportion
      parameterType: double
      feasibleSpace:
        min: "0.1"
        max: "0.2"
    - name: --num_train_epochs
      parameterType: categorical
      feasibleSpace:
        list:
          - "2.0"
          - "3.0"

trials

Environment:

• Kubeflow version (kfctl version): v1.0 istio-dex version
• Kubernetes version: (use kubectl version): v1.15.5
• OS (e.g. from /etc/os-release): CentOS v7.x
• Tensorflow version: v1.15.0 (No keras, using Estimator API)

[issue-label-bot[bot]]

Issue-Label Bot is automatically applying the labels:

| Label | Probability |
| ------------- | ------------- |
| kind/question | 0.89 |
| area/katib | 0.83 |

Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback!
Links: app homepage, dashboard and code for this bot.

[andreyvelich]

@rmard90 You can assign particular Node for the Trial's pod execution: https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/.
Also, you can define GPU that you want to use for your Trial's pod: https://kubernetes.io/docs/tasks/manage-gpus/scheduling-gpus/#node-labeller.

Katib doesn't support BatchJob execution on more than one Pod, but you can use TFJob or PyTorch Job.

[kylepad]

@rmard90 Kubernetes is scheduling all of your Trials on the same node because you are not setting resource constraints for the Trial. (read more here: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/)

Without a resource block present, Kubernetes assumes that each Trial Pod is using 0 cpu and 0 memory, so an infinite amount of these pods should fit on one Node.

You can ensure each Trial is scheduled on a different node by adding resource requests & limits that mirror the shape of the Node. For example: if your node has 4 cores, 12 gb of ram and 1 gpu you could use the following resource block in the trialTemplate:

...
{{- end}}
resources:
  request:
    cpu: 3000m
    memory: 10Gi
    nvidia.com/gpu: 1
  limit:
    cpu: 3000m
    memory: 10Gi
    nvidia.com/gpu: 1
volumeMounts:
...

What this tells Kubernetes is that this pod needs exclusive use of these resources. Since each Node only has enough space to fit a single pod with these limits, Kubernetes has no choice but to schedule the other Trials on separate Nodes!

One important note here: We are not requesting 100% of the cpu or ram that exists on the Node. There is a difference between what the Node hardware has available and how much of that Kubernetes allows to be "allocatable". It is a good idea to leave 20-15% of the Node resources available for overhead and sidecars (such as the metrics-collector).

Hope that helps :)

[ffeelers]

Thanks to @andreyvelich @kylepad
I wasn't familiar with k8s.
I was able to solve this problem with your help.

This is my setting.

apiVersion: "kubeflow.org/v1alpha3"
kind: Experiment
metadata:
  namespace: admin
  labels:
    controller-tools.k8s.io: "1.0"
  name: katib-experiment-30trial-for99
spec:
  nodeSelector:
    system: gpu
  parallelTrialCount: 5
  maxTrialCount: 30
  maxFailedTrialCount: 3
  objective:
    type: maximize
    goal: 0.99
    objectiveMetricName: F1-score
    additionalMetricNames:
      - Exact-matching
  algorithm:
    algorithmName: random
  trialTemplate:
    goTemplate:
        rawTemplate: |-
          apiVersion: batch/v1
          kind: Job
          metadata:
            name: {{.Trial}}
            namespace: {{.NameSpace}}
          spec:
            template:
              metadata:
                labels:
                  app: korquad-katib
              spec:
                affinity:
                  podAntiAffinity:
                    requiredDuringSchedulingIgnoredDuringExecution:
                    - labelSelector:
                        matchExpressions:
                        - key: app
                          operator: In
                          values:
                          - korquad-katib
                      topologyKey: "kubernetes.io/hostname"
                containers:
                - name: {{.Trial}}
                  image: kubeflow-registry.default.svc.cluster.local:30000/katib:20200811
                  command:
                  - "python"
                  - "/app/run_korquad.py"
                  - "--bert_config_file=./albert_base_config.json"
                  - "--vocab_file=./vocab.txt"
                  - "--do_train=True"
                  - "--do_predict=True"
                  - "--do_lower_case=False"
                  - "--train_file=./KorQuAD_v1.0_train.json"
                  - "--predict_file=./KorQuAD_v1.0_dev.json"
                  - "--train_batch_size=12"
                  - "--iterations_per_loop=100"
                  - "--save_checkpoints_steps=1000"
                  - "--init_checkpoint=./model.ckpt-31250"
                  - "--tfrecord_file=./train.tf_record"
                  - "--output_dir=./{{.Trial}}"
                  {{- with .HyperParameters}}
                  {{- range .}}
                  - "{{.Name}}={{.Value}}"
                  {{- end}}
                  {{- end}}
                  resources:
                    request:
                      nvidia.com/gpu: 2
                  volumeMounts:
                    - mountPath: "/resources"
                      name: kubeflow-pvc
                restartPolicy: Never
                volumes:
                  - name: kubeflow-pvc
                    persistentVolumeCaim:
                      claimName: kubeflow-vol
  parameters:
    - name: --learning_rate
      parameterType: double
      feasibleSpace:
        min: "0.00003"
        max: "0.00005"
    - name: --warmup_proportion
      parameterType: double
      feasibleSpace:
        min: "0.1"
        max: "0.2"
    - name: --num_train_epochs
      parameterType: categorical
      feasibleSpace:
        list:
          - "2.0"
          - "3.0"