Ray: [tune] mode='min' doesn't work for Bayesian search

[michaelitvin]

Bayesian search maximizes the target function even if using mode='min'.

ray 0.8.6
bayesian-optimization 1.2.0
Ubuntu 18.04

Reproduction (REQUIRED)

import time

from ray import tune
from ray.tune.suggest import ConcurrencyLimiter
from ray.tune.suggest.bayesopt import BayesOptSearch


def evaluation_fn(width, height):
    return abs(width - 17) * abs(height - 6)


def easy_objective(config):
    # Hyperparameters
    width, height = config["width"], config["height"]

    # Iterative training function - can be any arbitrary training procedure
    intermediate_score = evaluation_fn(width, height)
    # Feed the score back back to Tune.
    tune.report(iterations=1, mean_loss=intermediate_score)
    time.sleep(0.1)


if __name__ == "__main__":
    space = {"width": (0, 20), "height": (-100, 100)}

    config = {
        "num_samples": 15,
    }
    algo = BayesOptSearch(
        space,
        metric="mean_loss",
        mode="min",
        random_search_steps=2,
        utility_kwargs={
            "kind": "ucb",
            "kappa": 2.5,
            "xi": 0.0
        })
    algo = ConcurrencyLimiter(algo, max_concurrent=1)

    analysis = tune.run(
        easy_objective,
        name="my_exp",
        search_alg=algo,
        **config)

    print("Best config: ", analysis.get_best_config(metric="mean_loss"))

This outputs:

== Status ==
Memory usage on this node: 4.9/125.5 GiB
Using FIFO scheduling algorithm.
Resources requested: 0/20 CPUs, 0/3 GPUs, 0.0/76.27 GiB heap, 0.0/25.29 GiB objects
Result logdir: /home/razor/ray_results/my_exp
Number of trials: 15 (15 TERMINATED)
+-------------------------+------------+-------+-----------+----------+-----------+--------+------------------+
| Trial name              | status     | loc   |    height |    width |      loss |   iter |   total time (s) |
|-------------------------+------------+-------+-----------+----------+-----------+--------+------------------|
| easy_objective_69ce363c | TERMINATED |       | -25.092   | 19.0143  |  62.6281  |      1 |      0.00034833  |
| easy_objective_69ce3640 | TERMINATED |       |  46.3988  | 11.9732  | 203.078   |      1 |      0.000291109 |
| easy_objective_69ce3644 | TERMINATED |       | -23.9199  | 19.536   |  75.877   |      1 |      0.000339985 |
| easy_objective_69ce3648 | TERMINATED |       | -34.9571  | 14.6231  |  97.35    |      1 |      0.000264406 |
| easy_objective_6ac09169 | TERMINATED |       | -24.7124  | 10.0279  | 214.129   |      1 |      0.000271797 |
| easy_objective_6ac0916d | TERMINATED |       | -30.5173  | 20       | 109.552   |      1 |      0.000296116 |
| easy_objective_6ac09171 | TERMINATED |       | -41.9788  | 13.5983  | 163.211   |      1 |      0.000264406 |
| easy_objective_6ac09175 | TERMINATED |       | -15.314   | 19.9151  |  62.1327  |      1 |      0.000336885 |
| easy_objective_6c04e36b | TERMINATED |       | -10.6867  | 20       |  50.06    |      1 |      0.000285387 |
| easy_objective_6c04e36f | TERMINATED |       | -11.6825  | 15.0632  |  34.2473  |      1 |      0.000282288 |
| easy_objective_6c04e373 | TERMINATED |       |  -6.1038  | 14.8981  |  25.4413  |      1 |      0.000318289 |
| easy_objective_6c04e377 | TERMINATED |       |  -7.46917 |  8.97698 | 108.063   |      1 |      0.00028038  |
| easy_objective_6d30ff77 | TERMINATED |       |  -2.96115 | 19.4633  |  22.0743  |      1 |      0.000296116 |
| easy_objective_6d30ff7b | TERMINATED |       |   1.42222 | 15.2888  |   7.83356 |      1 |      0.000290155 |
| easy_objective_6d30ff7f | TERMINATED |       |   5.42205 | 20       |   1.73384 |      1 |      0.000284195 |
+-------------------------+------------+-------+-----------+----------+-----------+--------+------------------+


Best config:  {'height': -24.712352103501683, 'width': 10.027923731678012}

Whereas the expected result for these trials would be (5.42205, 20) with loss 1.73384.

• [x] I have verified my script runs in a clean environment and reproduces the issue.
• [x] I have verified the issue also occurs with the latest wheels.

[richardliaw]

Hi @michaelitvin, I think you'll want to do get_best_config(metric="..", mode="min").

[michaelitvin]

Yes that did the trick!

[michaelitvin]

mode='min' doesn't appear in get_best_config in any example I saw im the docs. Even not in the first getting started example.
What am I missing?

[richardliaw]

Hmm... i guess it's just not in the examples but there's the api reference: https://docs.ray.io/en/latest/tune/api_docs/analysis.html#ray.tune.ExperimentAnalysis.get_best_config

let's keep this issue open until we patch the documentation :)

Thanks for bringing up that point!

[richardliaw]

Should be fixed with #10663