Hyperopt: mongo-workers, ModuleNotFoundError: No module named 'core'

[midasSSS]

Good evening, everyone!

Recently I was trying to use parallelism with MongoDB and was able to successfully run the sin example from Wiki. Then I tried to apply the same logic using, to my objective function, which was working perfectly with Trials() class.
When I launch hyperopt-mongo-worker like this:
hyperopt-mongo-worker --mongo=localhost:1234/foo_db --poll-interval=0.1

Then I get the following issue:

INFO:hyperopt.mongoexp:PROTOCOL mongo
INFO:hyperopt.mongoexp:USERNAME None
INFO:hyperopt.mongoexp:HOSTNAME localhost
INFO:hyperopt.mongoexp:PORT 1234
INFO:hyperopt.mongoexp:PATH /foo_db/jobs
INFO:hyperopt.mongoexp:AUTH DB None
INFO:hyperopt.mongoexp:DB foo_db
INFO:hyperopt.mongoexp:COLLECTION jobs
in moddd hyperopt.base
in moddd dill._dill
in moddd dill._dill
in moddd core.hyperopt_model
INFO:hyperopt.mongoexp:Error while unpickling.
INFO:hyperopt.mongoexp:job exception: No module named 'core'
Traceback (most recent call last):
  File "/Users/midas/venv/bin/hyperopt-mongo-worker", line 6, in <module>
    sys.exit(hyperopt.mongoexp.main_worker())
  File "/Users/midas/venv/lib/python3.6/site-packages/hyperopt/mongoexp.py", line 1313, in main_worker
    return main_worker_helper(options, args)
  File "/Users/midas/venv/lib/python3.6/site-packages/hyperopt/mongoexp.py", line 1260, in main_worker_helper
    mworker.run_one(reserve_timeout=float(options.reserve_timeout))
  File "/Users/midas/venv/lib/python3.6/site-packages/hyperopt/mongoexp.py", line 1075, in run_one
    domain = pickler.loads(blob)
  File "/Users/midas/venv/lib/python3.6/site-packages/dill/_dill.py", line 317, in loads
    return load(file, ignore)
  File "/Users/midas/venv/lib/python3.6/site-packages/dill/_dill.py", line 305, in load
    obj = pik.load()
  File "/Users/midas/venv/lib/python3.6/site-packages/dill/_dill.py", line 475, in find_class
    return StockUnpickler.find_class(self, module, name)
ModuleNotFoundError: No module named 'core'

This issue repeats 4 consecutive times and then throws exception:
INFO:hyperopt.mongoexp:exiting with N=9223372036854775803 after 4 consecutive exceptions

Thanks for helping in advance!

[JonnoFTW]

Are you running the worker on a different machine to the one running MongoTrials? Make sure you have all the correct packages installed on the machine running the worker.

Or you could try setting the environment variable PYTHONPATH=. on the worker machine as well

[midasSSS]

Are you running the worker on a different machine to the one running MongoTrials? Make sure you have all the correct packages installed on the machine running the worker.

Or you could try setting the environment variable PYTHONPATH=. on the worker machine as well

No, I run it on the same machine.
So, on one machine I should launch MongoTrials and set the host address of another machine?
At the same time on the second machine I should have worker running?
Is there a way to do this simultaneously on same machine?

[shoeffner]

Yes, @midasSSS, you can do this on the same machine, and your setup should work.
Just change your call as suggested by @JonnoFTW to

PYTHONPATH=. hyperopt-mongo-worker --mongo=localhost:27017/sacred --poll-interval=0.1

when running hyperopt-mongo-worker from the directory where your source code is.
For accessing non-Python-modules, e.g. datasets, I also have to set the working directory for the worker:

PYTHONPATH=. hyperopt-mongo-worker --mongo=localhost:27017/sacred --poll-interval=0.1 --workdir=$(pwd)

[renan-ms]

Good morning everyone!

I am struggling with a similar problem when trying to paralleilize trials with Mongo but I didn't manage to solve it. It works ok with Trials class, but when I call

hyperopt-mongo-worker --mongo=localhost:27017/foo_db --poll-interval=0.1

I get the following error:

INFO:hyperopt.mongoexp:PROTOCOL mongo
INFO:hyperopt.mongoexp:USERNAME None
INFO:hyperopt.mongoexp:HOSTNAME localhost
INFO:hyperopt.mongoexp:PORT 27017
INFO:hyperopt.mongoexp:PATH /foo_db/jobs
INFO:hyperopt.mongoexp:AUTH DB None
INFO:hyperopt.mongoexp:DB foo_db
INFO:hyperopt.mongoexp:COLLECTION jobs
INFO:hyperopt.mongoexp:Error while unpickling.
INFO:hyperopt.mongoexp:job exception: Can't get attribute 'objective' on
Traceback (most recent call last):
File "/home/renanms/anaconda3/envs/tf/bin/hyperopt-mongo-worker", line 6, in
sys.exit(hyperopt.mongoexp.main_worker())
File "/home/renanms/anaconda3/envs/tf/lib/python3.6/site-packages/hyperopt/mongoexp.py", line 1313, in main_worker
return main_worker_helper(options, args)
File "/home/renanms/anaconda3/envs/tf/lib/python3.6/site-packages/hyperopt/mongoexp.py", line 1260, in main_worker_helper
mworker.run_one(reserve_timeout=float(options.reserve_timeout))
File "/home/renanms/anaconda3/envs/tf/lib/python3.6/site-packages/hyperopt/mongoexp.py", line 1075, in run_one
domain = pickler.loads(blob)
AttributeError: Can't get attribute 'objective' on

As with @midasSSS, this issue repeats 4 consecutive times and then throws exception:
INFO:hyperopt.mongoexp:exiting with N=9223372036854775803 after 4 consecutive exceptions

I tried the PYTHONPATH solution but it didn't work for me. Thanks for helping in advance!

[JonnoFTW]

@renan-ms did you make sure to call that from the directory containing your module main?

[renan-ms]

Hi @JonnoFTW

My python script is in the /home/renanms/Documents/Mestrado/Dissertacao/Codigo/LSTM-Py/

Then I run "python " from the above dir in a terminal

Then, when I run "hyperopt-mongo-worker --mongo=localhost:27017/foo_db --poll-interval=0.1" from the same dir in another terminal I get the error