Ray: [ray] Better UX for finding local modules

[SumNeuron]

System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Ubuntu 16.04
• Ray installed from (source or binary): pip
• Ray version: latest
• Python version: 3.5
• Exact command to reproduce: see linked repo

1. clone this [repo]
2. cd to project folder
3. jupyter-notebook <cloned-project>/jupyter/
4. just run the cells

Describe the problem

In the linked [repo] I provide a M.W.E. of how I envision sacred and tune would work together. Inside there is a local python module named project which contains four submodules model, utils, sacred and tune.

The model submodule has code for a simple twin embedding model in keras (simply to provide some "realism" to the MWE. This code functions and can be tested in the directory jupyter.

The sacred submodule, attempts to decouple sacred specific code as much as possible (e.g. ingredients, observers, etc). If you import the ex from this module it still runs on its own. (again, see the jupyter directory)

The tune submodule is where I envision how tune would wrap the sacred ex for hyperparameter optimization.

Most likely, as I get more familiar with tune (or for users who are) the amount of code here would expand. Nonetheless there are two simple functions, one which wraps sacred, and another which registers the wrapper and expects a hyper-parameter configuration to be passed.

When calling this function ray / tune fails as:

(pid=40079) ModuleNotFoundError: No module named 'project'

which seems to be because of this python module structure.

Notably, if all this code is put at the top level of a module, this works.

I would like to keep this structure though, and I do not see why tune should or would be opinionated about that.

Source code / logs

repo at https://gitlab.com/SumNeuron/extune

[richardliaw]

Hi @SumNeuron, can you try printing sys.path, both in the run_fn and in tune_run_ex?

[robertnishihara]

See also https://stackoverflow.com/questions/55358554/ray-tune-and-sacred-tune-cant-find-local-module.

[SumNeuron]

@richardliaw

# <repo>/project/tune/experiment.py
import ray, os, sys
from ray import tune

from ..sacred.experiment import  ex

# wrapper function over sacred experiment
def tune_run_ex(config, reporter):
    print('TUNE_RUN_EX\n\n', sys.path, '\n\n')
    ex.run(config_updates=config)
    result = ex.current_run.result
    reporter(result=result, done=True)


ray.init(ignore_reinit_error=True)
def run_fn(config):
    print('RUN_FN\n\n', sys.path, '\n\n')
    tune.register_trainable('tune_run_ex', tune_run_ex)
    tune.run_experiments({
        'demo': {
            'run': 'tune_run_ex',
            'stop': {'result': 0.1},
            'config': config,
            'resources_per_trial': {
                'cpu': 1,
                'gpu': 1
            },
            'num_samples': 1,
        }
    })

In the above code block you can see I added two print statements.

# <repo>/jupyter/Test.ipynb
import os, sys
sys.path.insert(0, '../')

import ray
from ray import tune
from project.tune.experiment import run_fn


run_fn(
    config={
        'latent_factors':  tune.grid_search([1, 2, 3])
    }
)

which results in:

# NOTE: this just runs forever

RUN_FN

 ['/home/sumner/anaconda3/lib/python3.6/site-packages/ray/pyarrow_files', '/home/sumner/Projects/extune/jupyter', '/home/sumner/anaconda3/lib/python3.6/site-packages', '/home/sumner/Projects/extune/jupyter', '/home/sumner/anaconda3/lib/python3.6/site-packages', '../', '', '/home/sumner/anaconda3/lib/python36.zip', '/home/sumner/anaconda3/lib/python3.6', '/home/sumner/anaconda3/lib/python3.6/lib-dynload', '/home/sumner/anaconda3/lib/python3.6/site-packages', '/home/sumner/anaconda3/lib/python3.6/site-packages/IPython/extensions', '/home/sumner/.ipython', '/home/sumner/anaconda3/lib/python3.6/site-packages/ray/modin'] 


== Status ==
Using FIFO scheduling algorithm.
Resources requested: 0/40 CPUs, 0/1 GPUs
Memory usage on this node: 8.4/100.0 GB

== Status ==
Using FIFO scheduling algorithm.
Resources requested: 1/40 CPUs, 1/1 GPUs
Memory usage on this node: 8.4/100.0 GB
Result logdir: /home/sumner/ray_results/demo
Number of trials: 3 ({'RUNNING': 1, 'PENDING': 2})
PENDING trials:
 - tune_run_ex_1_latent_factors=2:  PENDING
 - tune_run_ex_2_latent_factors=3:  PENDING
RUNNING trials:
 - tune_run_ex_0_latent_factors=1:  RUNNING

2019-03-27 09:33:43,349 ERROR worker.py:1780 -- Failed to unpickle actor class 'WrappedFunc' for actor ID fb1ec0b30866962cad2b5db15179befe8466c019. Traceback:
Traceback (most recent call last):
  File "/home/sumner/anaconda3/lib/python3.6/site-packages/ray/function_manager.py", line 720, in _load_actor_class_from_gcs
    actor_class = pickle.loads(pickled_class)
ModuleNotFoundError: No module named 'project'

2019-03-27 09:33:43,362 ERROR trial_runner.py:460 -- Error processing event.
Traceback (most recent call last):
  File "/home/sumner/anaconda3/lib/python3.6/site-packages/ray/tune/trial_runner.py", line 409, in _process_trial
    result = self.trial_executor.fetch_result(trial)
  File "/home/sumner/anaconda3/lib/python3.6/site-packages/ray/tune/ray_trial_executor.py", line 314, in fetch_result
    result = ray.get(trial_future[0])
  File "/home/sumner/anaconda3/lib/python3.6/site-packages/ray/worker.py", line 2316, in get
    raise value
ray.exceptions.RayTaskError: ray_worker (pid=9892, host=deltabonn-3)
Exception: The actor with name WrappedFunc failed to be imported, and so cannot execute this method.

2019-03-27 09:33:43,381 INFO ray_trial_executor.py:178 -- Destroying actor for trial tune_run_ex_0_latent_factors=1. If your trainable is slow to initialize, consider setting reuse_actors=True to reduce actor creation overheads.

(pid=9892) 2019-03-27 09:33:43,324  ERROR function_manager.py:723 -- Failed to load actor class %s.
(pid=9892) Traceback (most recent call last):
(pid=9892)   File "/home/sumner/anaconda3/lib/python3.6/site-packages/ray/function_manager.py", line 720, in _load_actor_class_from_gcs
(pid=9892)     actor_class = pickle.loads(pickled_class)
(pid=9892) ModuleNotFoundError: No module named 'project'

[richardliaw]

Sorry for the slow reply. Can you try moving the import into the function, like:

import ray, os, sys
from ray import tune


# wrapper function over sacred experiment
def tune_run_ex(config, reporter):
    print('TUNE_RUN_EX\n\n', sys.path, '\n\n')
    from ..sacred.experiment import  ex
    ex.run(config_updates=config)
    result = ex.current_run.result
    reporter(result=result, done=True)


ray.init(ignore_reinit_error=True)
def run_fn(config):
    print('RUN_FN\n\n', sys.path, '\n\n')
    tune.register_trainable('tune_run_ex', tune_run_ex)
    tune.run_experiments({
        'demo': {
            'run': 'tune_run_ex',
            'stop': {'result': 0.1},
            'config': config,
            'resources_per_trial': {
                'cpu': 1,
                'gpu': 1
            },
            'num_samples': 1,
        }
    })

[SumNeuron]

@richardliaw @robertnishihara 's suggestion to use a setup.py file and pip install -e is a sufficient work around...

However I personally dislike both approaches. In general python style guides encourage all import statements at the top of the file.

Having to setup and manage the module shouldn't be necessary either.

Please clone the repo and try the jupyter notebook as well as the "script.py" file.
I ask this not just because it may help in figuring out exactly why these workarounds are necessary, but because I added a fair amount of documentation throughout the functions. So it might serve as as good starting point should ray / tune want to add an example or official docs on sacred integration

I think the true solution requires the ray module to become relative / local module aware.

[kschuerholt]

Had a similar issue yesterday and resolved it with setuptools. Also, I'd like to add that the issue does not exist in local_mode. Is there any documentation of what environment the actors live in? If so, I couldn't find it.

As state before, I'd very much appreciate if ray actors became aware of local modules.

[crypdick]

@robertnishihara looks like that SO question was "removed from Stack Overflow for reasons of moderation"

[meechos]

@SumNeuron could you please post your setuptools solution here? I am still having issues with this and the relevant stackexhange link posted by @robertnishihara has been deleted.

[kschuerholt]

Sure. Basically, you make your local module a package. Instead of importing it locally from python path, you install it with pip. that way, it becomes available, even if the package isn't in your path.
To that end, create a file called 'setup.py' in the package you want to install. There's a lot of info on that online, you'll find all the details you need. It needs to contain the import of setup,

from setuptools import setup

setup(
    name="name_of_your_package",
    version="0.1.2.3",
    author="Author Name",
    author_email="[email protected]",
    packages=["module_1", "module_2"],
    description="An awesome package that does something",
)

AFAIK that the modules in packages need to be python modules, so they have to be in separate folders with corresponding names and have to have __init__.py files.
If you have that, just navigate to the directory with the setup.py and run pip install -e ., which will install your new package. When you import your package, it imports the actual code that is in the directory, so you don't need to reinstall when you change your code.
Read up on packages e.g. here: https://betterscientificsoftware.github.io/python-for-hpc/tutorials/python-pypi-packaging/