Ray: [tune] NotImplementedError with BOHB and function-based API

[sumanthratna]

What is the problem?

When running ray.tune with BOHB, I get the following error:

2020-04-19 21:51:49,379 ERROR worker.py:1012 -- Possible unhandled error from worker: ray::WrappedFunc (pid=6643, ip=192.168.1.179)
  File "python/ray/_raylet.pyx", line 417, in ray._raylet.execute_task
  File "python/ray/_raylet.pyx", line 439, in ray._raylet.execute_task
ray.exceptions.RayTaskError: ray::WrappedFunc.save_to_object() (pid=6645, ip=192.168.1.179)
  File "python/ray/_raylet.pyx", line 452, in ray._raylet.execute_task
  File "python/ray/_raylet.pyx", line 407, in ray._raylet.execute_task.function_executor
  File ".../site-packages/ray/tune/trainable.py", line 385, in save_to_object
    checkpoint_path = self.save(tmpdir)
  File ".../site-packages/ray/tune/trainable.py", line 342, in save
    checkpoint = self._save(checkpoint_dir)
  File ".../site-packages/ray/tune/trainable.py", line 612, in _save
    raise NotImplementedError
NotImplementedError

This happens with none of the other search algorithms that Tune provides.

Note: could be related to https://github.com/ray-project/ray/issues/4833.

Ray version and other system information (Python version, TensorFlow version, OS):

• python 3.6.8
• ray 0.8.4
• tensorflow 2.1.0
• macOS 10.15.5 Beta

Reproduction (REQUIRED)

#!/usr/bin/env python

import numpy as np

import ray
from ray.tune import run
from ray.tune.schedulers.hb_bohb import HyperBandForBOHB
from ray.tune.suggest.bohb import TuneBOHB


def trainable(config, reporter):
    global timestep
    timestep += 1
    v = np.tanh(float(timestep) / config['width'])
    v *= config['height']
    reporter(episode_reward_mean=v)


if __name__ == "__main__":
    import ConfigSpace as CS
    ray.init()

    # BOHB uses ConfigSpace for their hyperparameter search space
    config_space = CS.ConfigurationSpace()
    config_space.add_hyperparameter(
        CS.UniformFloatHyperparameter("height", lower=10, upper=100))
    config_space.add_hyperparameter(
        CS.UniformFloatHyperparameter("width", lower=0, upper=100))

    experiment_metrics = dict(metric="episode_reward_mean", mode="max")
    bohb_hyperband = HyperBandForBOHB(
        time_attr="training_iteration",
        max_t=100,
        reduction_factor=4,
        **experiment_metrics)
    bohb_search = TuneBOHB(
        config_space, max_concurrent=4, **experiment_metrics)

    timestep = 0

    run(trainable,
        name="bohb_test",
        scheduler=bohb_hyperband,
        search_alg=bohb_search,
        num_samples=10,
        stop={"training_iteration": 10})

I'm aware that BOHB is still experimental, but I think that the function-based API should work with BOHB.

• [x] I have verified my script runs in a clean environment and reproduces the issue.
• [x] I have verified the issue also occurs with the latest wheels.

[richardliaw]

@sumanthratna sorry for the late reply; we'll be fixing this in the next two weeks.

[AlanHassen]

Are there any updates on this or a timeframe? BOHB has a nice ecosystem (CAVE...) that I would like to use over a manual analysis and sadly in my setting, only the functional api is viable.

[richardliaw]

ETA 3-4 days. Try out #8471 - it's working now, just need to write docs.

[richardliaw]

This should be merged on master. An example of using the FunctionAPI with PBT can be found here. You should be able to swap it out for BOHB.