Katib: Bayesian optimization experiment suggests same hyperparameters

[martham93]

/kind bug

What steps did you take and what happened:
[A clear and concise description of what the bug is.]

I've experienced the same hyperparameters being suggested by a Bayesian optimization experiment several times.

• For example: with 1 parallel trial, and max trials of 4 the exact same hyperparameters are suggested for each trial

part of experiment yml file:

apiVersion: "kubeflow.org/v1alpha3"
kind: Experiment
metadata:
  namespace: kubeflow
  name: poi-experiment35
  labels:
    controller-tools.k8s.io: "1.0"
spec:
  objective:
    type: maximize
    goal: 0.99
    objectiveMetricName: fbeta_m
  metricsCollectorSpec:
    source:
      fileSystemPath:
        path: "/ml/models/abc/eval/"
        kind: Directory
    collector:
      kind: TensorFlowEvent
  algorithm:
    algorithmName: bayesianoptimization
    algorithmSettings:
      - name: "random_state"
        value: "5"
  parallelTrialCount: 1
  maxTrialCount: 4
  maxFailedTrialCount: 3
  parameters:
    - name: --tf_dense_size
      parameterType: int
      feasibleSpace:
        min: "128"
        max: "512"
    - name: --tf_dense_dropout_rate
      parameterType: double
      feasibleSpace:
        min: "0.01"
        max: "0.5"
    - name: --tf_learning_rate
      parameterType: double
      feasibleSpace:
        min: "0.00001"
        max: "0.001"
    - name: --tf_optimizer
      parameterType: categorical
      feasibleSpace:
        list:
        - sgd
        - rmsprop
        - adam

• Another example: with 3 parallel trials, and 9 max trials, different hyperparameters are suggested for the first three trials, but for the following 6 trials the hyper-parameter combination from the best performing trial is suggested again for all six of the remaining trials.
  

part of experiment yml file:

apiVersion: "kubeflow.org/v1alpha3"
kind: Experiment
metadata:
  namespace: kubeflow
  name: poi-experiment36
  labels:
    controller-tools.k8s.io: "1.0"
spec:
  objective:
    type: maximize
    goal: 0.99
    objectiveMetricName: fbeta_m
  metricsCollectorSpec:
    source:
      fileSystemPath:
        path: "/ml/models/abc/eval/"
        kind: Directory
    collector:
      kind: TensorFlowEvent
  algorithm:
    algorithmName: bayesianoptimization
    algorithmSettings:
      - name: "random_state"
        value: "15"
  parallelTrialCount: 3
  maxTrialCount: 9
  maxFailedTrialCount: 3
  parameters:
    - name: --tf_dense_size
      parameterType: int
      feasibleSpace:
        min: "128"
        max: "512"
    - name: --tf_dense_dropout_rate
      parameterType: double
      feasibleSpace:
        min: "0.01"
        max: "0.5"
    - name: --tf_learning_rate
      parameterType: double
      feasibleSpace:
        min: "0.00001"
        max: "0.001"
    - name: --tf_optimizer
      parameterType: categorical
      feasibleSpace:
        list:
        - sgd
        - rmsprop
        - adam

What did you expect to happen:
Different hyperparameters combinations to be suggested.

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]
I was just wondering if other people have experienced something similar.

Environment:
Minikube version: N/A. Deployed on GCP.

• Kubeflow version: kfctl v1.0-rc.1-0-g963c787
• Kubernetes version: (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.8", GitCommit:"211047e9a1922595eaa3a1127ed365e9299a6c23", GitTreeState:"clean", BuildDate:"2019-10-15T12:11:03Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"14+", GitVersion:"v1.14.8-gke.33", GitCommit:"2c6d0ee462cee7609113bf9e175c107599d5213f", GitTreeState:"clean", BuildDate:"2020-01-15T17:47:46Z", GoVersion:"go1.12.11b4", Compiler:"gc", Platform:"linux/amd64"}

OS: linux

[issue-label-bot[bot]]

Issue-Label Bot is automatically applying the labels:

| Label | Probability |
| ------------- | ------------- |
| bug | 0.86 |

Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback!
Links: app homepage, dashboard and code for this bot.

[martham93]

Also, with the example Bayesian experiment from the documentation (3 parallel trials, and 12 max trials) different hyperparameters are suggested for the first three trials, but for the following 9 trials the hyperparameter combination from the best performing trial is suggested again for all 9 of the remaining trials.

This behavior seems a little strange. Are there any algorithm settings that can be modified for bayesian search?

[andreyvelich]

@martham93 Thank you for your issue!

This problem maybe happens because we create new Optimizer class every time we call GetSuggestion (https://github.com/kubeflow/katib/blob/master/pkg/suggestion/v1alpha3/skopt/base_skopt_service.py#L51).

So we call tell and ask methods for different Optimizer instance and Suggestion doesn't have previous steps. If you take a look here: https://github.com/scikit-optimize/scikit-optimize#getting-started, Optimizer instance is created just once.
I changed creating Optimizer instance only for the first GetSuggestion call and got 4 different Trial assignments for Experiment with 1 parallel trial, and max trials of 4.

I think we should create new Optimizer class in 2 cases:

1. First run of GetSuggestion
2. Experiment Search Space was changed.

What do you think @gaocegege @johnugeorge ?

[johnugeorge]

Search space Modification should not be allowed in the same experiment. it should be prevented now. See https://github.com/kubeflow/katib/issues/768

In that case, can we do this when we instantiate the algorithm

[andreyvelich]

In that case, can we do this when we instantiate the algorithm

Yes, we can create Optimizer instance only for the first GetSuggestion call.

[gaocegege]

SGTM

[ever-cheng]

Do you think this issue also happen on random optimization?