Katib: TPE suggestion service failing to provide proposals for new trials

[jimmy-hawkfish]

/kind bug

I am running an experiment with algorithm tpe and the following search space:

[ { "feasibleSpace": { "max": "1", "min": "0" }, "name": "r1", "parameterType": "double" }, { "feasibleSpace": { "max": "1", "min": "0" }, "name": "r2", "parameterType": "double" }, { "feasibleSpace": { "max": "1", "min": "0" }, "name": "r3", "parameterType": "double" } ]

and a maxTrialCount of 10,000. The experiment successfully runs 2,057 trials but fails to propose new ones. Looking at the suggestion pod logs I see:

ERROR:grpc._server:Exception calling application: list index out of range Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/grpc/_server.py", line 434, in _call_behavior response_or_iterator = behavior(argument, context) File "/opt/katib/pkg/suggestion/v1alpha3/hyperopt/service.py", line 38, in GetSuggestions new_assignments = self.base_service.getSuggestions(trials, request.request_number) File "/opt/katib/pkg/suggestion/v1alpha3/hyperopt/base_service.py", line 220, in getSuggestions vals = new_trials[i]['misc']['vals'] IndexError: list index out of range

What did you expect to happen:
I expected suggestions for the next 1000 trials to be proposed. It seems unlikely that the search space should be exhausted in ~2k trials.

Environment:
Running in Google Cloud Kubernetes engine

• Kubeflow version (kfctl version): 1.0.0
• Kubernetes version: (use kubectl version):
  Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.2", GitCommit:"52c56ce7a8272c798dbc29846288d7cd9fbae032", GitTreeState:"clean", BuildDate:"2020-04-16T11:56:40Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"14+", GitVersion:"v1.14.10-gke.42", GitCommit:"42bef28c2031a74fc68840fce56834ff7ea08518", GitTreeState:"clean", BuildDate:"2020-06-02T16:07:00Z", GoVersion:"go1.12.12b4", Compiler:"gc", Platform:"linux/amd64"}

[issue-label-bot[bot]]

Issue-Label Bot is automatically applying the labels:

| Label | Probability |
| ------------- | ------------- |
| area/katib | 0.90 |

Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback!
Links: app homepage, dashboard and code for this bot.

[issue-label-bot[bot]]

Issue Label Bot is not confident enough to auto-label this issue.
See dashboard for more details.

[andreyvelich]

@jimmy-hawkfish Thank you for reporting this!
I was able to run example with this search space for 3180 Trials at least.

Does suggestion pod have any other errors?
Can you show other settings for your experiment, please?

If you take a look at the code, it is not possible that len(new_trials) array will be less than request_number because we iterate over new_trials request_number times, if it is not first call.
https://github.com/kubeflow/katib/blob/2ceed7d5b74ec26128fe882decb1624f08ef9614/pkg/suggestion/v1beta1/hyperopt/base_service.py#L196-L222

Does your suggestion restart during the experiment run?

[andreyvelich]

/cc @gaocegege

[andreyvelich]

I faced with your issue, when my Suggestion pod was OOMKilling after 3815 succeeded trials.

I believe the issue is that.
After restart Suggestion is_first_run=True and self.hyperopt_algorithm() returns only one trial, instead of request_number trials.
Only for n_startup_jobs times self.hyperopt_algorithm() can return more than one trial in the same time.

You can increase your suggestion pod resources using Katib config, explaining here.

If you want to run your suggestion pod without disk ephemeral-storage param, we have pending PR https://github.com/kubeflow/katib/pull/1312 to be able omit this parameter.

For Hyperopt, I think we can refactor code to not CrashLoopBackOff in that case.

[jimmy-hawkfish]

Thanks for the reply @andreyvelich! Yes, I see multiple restarts for the experiment and the explanation sounds reasonable. I'll try your fixes. Let me know if there are any settings you would still be interested in seeing.

[gaocegege]

Thanks for the issue.

One question here, why do we have OOM in hyperopt? /cc @andreyvelich

[andreyvelich]

@gaocegege I think because we specify default memory, cpu and disk storage for the hyperopt suggestion's pod.
When Experiment is large, hyperopt might need more memory to execute.

[andreyvelich]

Feel free to re-open the issue if you face the problem again.