Katib: SuccessiveHalving and early-stopping not working for Katib Hyperband algorithm

[asispatra]

SuccessiveHalving and early-stopping not working for Katib Hyperband algorithm.
Any idea how to enable SuccessiveHalving and early-stopping features in Katib Hyperband algorithm?

Thanks.

[johnugeorge]

which katib version are you using?

[johnugeorge]

/cc @hougangliu

[hougangliu]

@asispatra can you detail more about what you mean? saying what's your experiment yaml look like and so on.

[asispatra]

@hougangliu,

Hyperband related config in the yaml is as below:

  suggestionSpec:
    suggestionAlgorithm: "hyperband"
    suggestionParameters:
      - name: "ResourceName"
        value: "--max-epoch"
      - name: "eta"
        value: "3"
      - name: "r_l"
        value: "9"

This launches 9 jobs together at the first time. Once these all are completed, the experiment status is completed. Currently I am not seeing anything related to SuccessiveHalving and early-stopping.

Lets say my each Deep-learning job will run for 10 epochs which requires 1 GPU and I have 8 GPUs in total. For hyper-parameter optimization, First time I want to launch 8 parallel jobs and later based on the feedback let the Hyperband algo decide how may job it need to run next and let the algo decide it should do early stopping or not.

For these scenario what should be the Hyparband configuration in the yaml file?

Thanks.

[asispatra]

Kubeflow version: 0.5.1
Kubernetes version: (use kubectl version): 1.14.3
OS (e.g. from /etc/os-release): Ubuntu 18.04.2 LTS

@johnugeorge, How to get the katib version?

[hougangliu]

@asispatra hyperband in kubeflow 0.5.1 cannot work well.
The fix has been merged in Katib master or 0.6 branch.
You can try it as https://github.com/kubeflow/katib/tree/master/examples/v1alpha2

[asispatra]

@hougangliu, in v1alpha2, the pytorch example is not even working. #702

[gaocegege]

/area katib-v1alpha2
/priority p1
/kind bug

[terrykong]

Is there any update here? This appears to still be broken in v1beta1

[issue-label-bot[bot]]

Issue-Label Bot is automatically applying the labels:

| Label | Probability |
| ------------- | ------------- |
| area/katib | 1.00 |

Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback!
Links: app homepage, dashboard and code for this bot.

[andreyvelich]

Hi @terrykong.
I think Hyperband needs to be refactored once we implement Early Stopping: https://github.com/kubeflow/katib/pull/1344.

I will work on this after Katib 0.10 is released.
/cc @gaocegege @johnugeorge

[gaocegege]

SGTM Thanks

[andreyvelich]

Ref issue: #1389.