Ray: RayWorkerError: The worker died unexpectedly while executing this task.

Created on 30 Jul 2019  ·  35Comments  ·  Source: ray-project/ray

Describe the problem


A Modin user has reported a RayWorkerError with running df.head(), which involves some ray.get calls.

Source code / logs

See https://github.com/modin-project/modin/issues/746

cc @masoodimran

picture devin-petersohn

Most helpful comment

I have the same problem while reading csv. Just simple pd.read_csv()

picture Espad0 on 1 Aug 2019
👍9

All 35 comments

Note: I cannot reproduce the error myself, so it is difficult to discover the source of the error.

devin-petersohn picture devin-petersohn on 30 Jul 2019

I have the same problem while reading csv. Just simple pd.read_csv()

Espad0 picture Espad0 on 1 Aug 2019
👍9

Same here.

uli-weltersbach picture uli-weltersbach on 2 Aug 2019

Same issue with modin.pandas.read_csv(), using Python3.7 in Jupyter Notebook.
Modin version is '0.5.4'.
Ray version is '0.7.1'.

lvoursl picture lvoursl on 2 Aug 2019

Does the problem persist if you run pip install -U ray?

devin-petersohn picture devin-petersohn on 5 Aug 2019

Does the problem persist if you run pip install -U ray?

I also met the Ray worker issue when using 'read_csv', or just show a 'df'.
Until now, I didn't successfully run any operations using Modin.

I tried pip install -u ray. The issue still exists.

Jacobzhangjw picture Jacobzhangjw on 6 Aug 2019

By the way, I'm using it in WSL ubuntu18. The WSL environment might be the possible reason for the issue? I have tried it on a real ubuntu18 system.

Jacobzhangjw picture Jacobzhangjw on 6 Aug 2019

Python 3.7.3
Pandas 0.24.2
Modin 0.5.4
Ray 0.7.1

Issue seems to happen with all dataframe exploratory commands

AcWd picture AcWd on 6 Aug 2019

Python 3.7.3
Pandas 0.24.2
Modin 0.5.4
Ray 0.7.1

所有数据框探索命令似乎都会出现问题

I have the same problem as you, buddy!

xjnotxj picture xjnotxj on 6 Aug 2019

Is there any interesting information in the log files under /tmp/ray/session_.../logs/worker...? Any hint about why the worker died?

robertnishihara picture robertnishihara on 6 Aug 2019

@robertnishihara

Sorry, I didn't have any experience of using Ray or Modin. I've checked the path you mentioned and attached the following log files for your reference.

file name: plasma_store.err

external/plasma/cpp/src/plasma/store.cc:1114: Allowing the Plasma store to use up to 41GB of memory.
external/plasma/cpp/src/plasma/store.cc:1141: Starting object store with directory /dev/shm and huge page support disabled
external/plasma/cpp/src/plasma/store.cc:714: Disconnecting client on fd 21
external/plasma/cpp/src/plasma/store.cc:714: Disconnecting client on fd 12
...

file name: log_monitor.err

Traceback (most recent call last):
File "/home/jacobwsl/anaconda3/lib/python3.7/site-packages/ray/log_monitor.py", line 278, in
raise e
File "/home/jacobwsl/anaconda3/lib/python3.7/site-packages/ray/log_monitor.py", line 268, in
log_monitor.run()
File "/home/jacobwsl/anaconda3/lib/python3.7/site-packages/ray/log_monitor.py", line 219, in run
anything_published = self.check_log_files_and_publish_updates()
File "/home/jacobwsl/anaconda3/lib/python3.7/site-packages/ray/log_monitor.py", line 175, in check_log_files_and_publish_updates
next_line = file_info.file_handle.readline()
File "/home/jacobwsl/anaconda3/lib/python3.7/codecs.py", line 322, in decode
(result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x96 in position 16: invalid start byte

Jacobzhangjw picture Jacobzhangjw on 7 Aug 2019

@Jacobzhangjw no problem, thanks for the help!

Does the problem persist if you run the following:

import ray
ray.init()
import modin.pandas as pd
df = pd.read_csv("file.csv")
devin-petersohn picture devin-petersohn on 7 Aug 2019

Tried the same but no luck

AcWd picture AcWd on 7 Aug 2019

Alright, let's see if we can narrow down the issue. Can you run this and see if it causes the issue as well?

import ray
ray.init()

_ = ray.get([ray.put(str(i)) for i in range(1000)])
devin-petersohn picture devin-petersohn on 7 Aug 2019

@devin-petersohn

Seems due to a missing package named 'setproctitle' ?

import ray
ray.init()

_ = ray.get([ray.put(str(i)) for i in range(1000)])

2019-08-08 11:44:20,259 WARNING worker.py:1331 -- WARNING: Not updating worker name since setproctitle is not installed. Install this with pip install setproctitle (or ray[debug]) to enable monitoring of worker processes.
2019-08-08 11:44:20,262 INFO node.py:498 -- Process STDOUT and STDERR is being redirected to /tmp/ray/session_2019-08-08_11-44-20_260809_55/logs.
2019-08-08 11:44:20,404 INFO services.py:409 -- Waiting for redis server at 127.0.0.1:37836 to respond...
2019-08-08 11:44:20,537 INFO services.py:409 -- Waiting for redis server at 127.0.0.1:59815 to respond...
2019-08-08 11:44:20,541 INFO services.py:806 -- Starting Redis shard with 10.0 GB max memory.
2019-08-08 11:44:20,606 INFO node.py:512 -- Process STDOUT and STDERR is being redirected to /tmp/ray/session_2019-08-08_11-44-20_260809_55/logs.
2019-08-08 11:44:20,609 WARNING services.py:1294 -- Warning: Capping object memory store to 20.0GB. To increase this further, specify object_store_memory when calling ray.init() or ray start.
2019-08-08 11:44:20,610 INFO services.py:1442 -- Starting the Plasma object store with 20.0 GB memory using /dev/shm.
​
I already tried 'pip install 'setproctitle', it raised installation error:

Running setup.py clean for setproctitle
Failed to build setproctitle
Installing collected packages: setproctitle
Running setup.py install for setproctitle ... error
....
ERROR: Command "/home/jacobwsl/anaconda3/bin/python -u -c 'import setuptools, tokenize;__file__='"'"'/tmp/pip-install-hzc7xkfg/setproctitle/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'rn'"'"', '"'"'n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-1ky78tlu/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-install-hzc7xkfg/setproctitle/

Jacobzhangjw picture Jacobzhangjw on 8 Aug 2019

I had the same problem. I uninstalled the typing package after reading this issue https://github.com/ray-project/ray/issues/4929 and it worked for me.

With the typing package installed, when I ran the tests on ray using this command python -m pytest -v ray/tests/test_mini.py they all failed.

ricdeblasis picture ricdeblasis on 8 Aug 2019
🎉1

@ricdeblasis @devin-petersohn @robertnishihara

Magic! uninstalled the typing make it (Ray & Modin) work again.
At least modin.pandas.read_csv() works perfectly now. I will report any other issues if found when using modin

By the way, typing is a standard package in python 3 now. Even I haven't used it directly before, I'm concerning some other packages depending on it.

Jacobzhangjw picture Jacobzhangjw on 8 Aug 2019

Alright, let's see if we can narrow down the issue. Can you run this and see if it causes the issue as well?

import ray
ray.init()

_ = ray.get([ray.put(str(i)) for i in range(1000)])

Execute with issue no issue

Also uninstalling "typing" package is not an option for us since packages like RASA have a dependency on it

AcWd picture AcWd on 8 Aug 2019

From https://github.com/python/typing/issues/573, it does seem like typing module is only required for python version less than 3.5.

@AcWd are you using RASA with python2? If you are using python3, then uninstalling typing is safe. Otherwise, I would recommend starting a virtualenv with python3. Thanks!

simon-mo picture simon-mo on 12 Aug 2019

@simon-mo We are on Python3 uninstalling typingwill not be an option for us for various reasons.

AcWd picture AcWd on 12 Aug 2019

@AcWd can I ask what version is the typing module? I'm trying to reproduce it.

You can check the version by pip install typing. Since you already have the package, pip will skip the installation.

This is what mine looks like

(py3.7) > pip install typing
Requirement already satisfied: typing in ./anaconda3/envs/py3.7/lib/python3.7/site-packages (3.7.4)
simon-mo picture simon-mo on 12 Aug 2019

Just checked it is typing 3.7.4

On Mon, 12 Aug 2019, 20:34 Simon Mo, notifications@github.com wrote:

@AcWd https://github.com/AcWd can I ask what version is the typing
module? I'm trying to reproduce it.

You can check the version by pip install typing. Since you already have
the package, pip will skip the installation.

This is what mine looks like

(py3.7) > pip install typing
Requirement already satisfied: typing in ./anaconda3/envs/py3.7/lib/python3.7/site-packages (3.7.4)

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ray-project/ray/issues/5316?email_source=notifications&email_token=ABRQ43XP2VSSHZJ6EMS5NTDQEG3NLA5CNFSM4IH6XQF2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4DS4RI#issuecomment-520564293,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABRQ43RT6OJIUNKCVSASKPLQEG3NLANCNFSM4IH6XQFQ
.

AcWd picture AcWd on 13 Aug 2019
👍1

Trying Modin today for the first time on ubuntu

when running jupyter notebook example,
encountered the same issue reported here

after pip uninstall typing, issue went away.

As a side note:
No issue if I ran read_csv() inside ipython shell or python script on command line (without pip uninstall typing), It only occurred with jupyter notebook

wgong picture wgong on 17 Aug 2019

This is not a reproducible issue, removing the blocker label.

ericl picture ericl on 26 Aug 2019
👎1

We have someone who can reproduce it.

devin-petersohn picture devin-petersohn on 28 Aug 2019

Can you attach a repro script?

ericl picture ericl on 28 Aug 2019

It is environment dependent. We are working on a fix.

devin-petersohn picture devin-petersohn on 28 Aug 2019

Great, let me know

ericl picture ericl on 28 Aug 2019

@AcWd, @wgong

To confirm a hypothesis, can you run the following snippets and attach the output?

import ray
ray.init(object_store_memory=int(1e8))

import sys
print("Driver sys.path")
print(sys.path)

@ray.remote
def print_sys_path():
    return sys.path

print("Worker sys.path")
print(ray.get(print_sys_path.remote()))

I believe the issue is that for some users, site-packages was ahead of standard libs. This cause an issue with python3.7 and typing package.

@ericl I can confirm this is not a release blocker because it's reproducible without ray. Ray play no part in this issue. But please leave it open until hypothesis confirmed.

simon-mo picture simon-mo on 29 Aug 2019

Thanks for investigating. I do want to keep the labels up to date though, it's fine to keep open issues indefinitely (we certainly don't clean the up often), but the labels are more flexible.

Feel free to add it back should it actually be a blocker.

ericl picture ericl on 29 Aug 2019

@simon-mo even if it's not related to Ray, perhaps we can have a simple fix. Would putting the following in ray/__init__.py work?

if sys.version_info >= (3, 7):
    try:
        import typing
        if hasattr(typing, "__version__"):
            # I think this means that we are using typing from PyPI, but we
            # should use the builtin module.
        raise RuntimeError("You should do 'pip uninstall typing'")
    except ImportError:
        pass

@wgong or @AcWd or someone who is able to reproduce the issue, any chance you could see if this helps?

robertnishihara picture robertnishihara on 29 Aug 2019

Coincidentally, this issue came up for me today and I was able to recreate it using an ec2 instance with Ubuntu 18.06 LTS, anaconda3 install of python (https://repo.anaconda.com/archive/Anaconda3-2019.07-Linux-x86_64.sh), and installing modin. Updating to ray 0.7.3 does not fix the issue but uninstalling typing does. This is the case because ray 0.7.1 has a dependency on typing 3.7.4.1

@simon-mo's script output was 

Driver sys.path
['/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/pyarrow_files', '/home/ubuntu', '/home/ubuntu/anaconda3/lib/python3.7/site-packages', '/home/ubuntu', '/home/ubuntu/anaconda3/lib/python37.zip', '/home/ubuntu/anaconda3/lib/python3.7', '/home/ubuntu/anaconda3/lib/python3.7/lib-dynload', '', '/home/ubuntu/anaconda3/lib/python3.7/site-packages', '/home/ubuntu/anaconda3/lib/python3.7/site-packages/IPython/extensions', '/home/ubuntu/.ipython', '/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/modin']
Worker sys.path
['/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/pyarrow_files', '/home/ubuntu', '/home/ubuntu/anaconda3/lib/python3.7/site-packages', '/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/workers', '/home/ubuntu/anaconda3/lib/python37.zip', '/home/ubuntu/anaconda3/lib/python3.7', '/home/ubuntu/anaconda3/lib/python3.7/lib-dynload', '/home/ubuntu/anaconda3/lib/python3.7/site-packages', '/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/modin']

The error is thrown when trying to store the output in the object store. Part of the long traceback that I get is

2019-08-29 06:28:24,189 ERROR worker.py:1714 -- Traceback (most recent call last):
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 967, in _process_task
    self._store_outputs_in_object_store(return_object_ids, outputs)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 880, in _store_outputs_in_object_store
    self.put_object(object_ids[i], outputs[i])
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 398, in put_object
    self._try_store_and_register(object_id, value)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 418, in _try_store_and_register
    self.store_and_register(object_id, value)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 327, in store_and_register
    self.current_job_id))
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 256, in get_serialization_context
    _initialize_serialization(job_id)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 1187, in _initialize_serialization
    serialization_context = pyarrow.default_serialization_context()
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/pyarrow_files/pyarrow/serialization.py", line 361, in default_serialization_context
    register_default_serialization_handlers(context)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/pyarrow_files/pyarrow/serialization.py", line 356, in register_default_serialization_handlers
    _register_custom_pandas_handlers(serialization_context)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/pyarrow_files/pyarrow/serialization.py", line 150, in _register_custom_pandas_handlers
    import pandas as pd
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/pandas/__init__.py", line 23, in <module>
    from pandas.compat.numpy import *
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/pandas/compat/__init__.py", line 446, in <module>
    import typing
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/typing.py", line 1357, in <module>
    class Callable(extra=collections_abc.Callable, metaclass=CallableMeta):
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/typing.py", line 1005, in __new__
    self._abc_registry = extra._abc_registry
AttributeError: type object 'Callable' has no attribute '_abc_registry'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/workers/default_worker.py", line 98, in <module>
    ray.worker.global_worker.main_loop()
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 1088, in main_loop
    self._wait_for_and_process_task(task)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 1039, in _wait_for_and_process_task
    self._process_task(task, execution_info)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 971, in _process_task
    ray.utils.format_error_message(traceback.format_exc()))
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 980, in _handle_process_task_failure
    self._store_outputs_in_object_store(return_object_ids, failure_objects)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 880, in _store_outputs_in_object_store
    self.put_object(object_ids[i], outputs[i])
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 398, in put_object
    self._try_store_and_register(object_id, value)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 418, in _try_store_and_register
    self.store_and_register(object_id, value)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 327, in store_and_register
    self.current_job_id))
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 256, in get_serialization_context
    _initialize_serialization(job_id)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 1187, in _initialize_serialization
    serialization_context = pyarrow.default_serialization_context()
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/pyarrow_files/pyarrow/serialization.py", line 361, in default_serialization_context
    register_default_serialization_handlers(context)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/pyarrow_files/pyarrow/serialization.py", line 356, in register_default_serialization_handlers
    _register_custom_pandas_handlers(serialization_context)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/pyarrow_files/pyarrow/serialization.py", line 150, in _register_custom_pandas_handlers
    import pandas as pd
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/pandas/__init__.py", line 23, in <module>
    from pandas.compat.numpy import *
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/pandas/compat/__init__.py", line 446, in <module>
    import typing
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/typing.py", line 1357, in <module>
    class Callable(extra=collections_abc.Callable, metaclass=CallableMeta):
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/typing.py", line 1005, in __new__
    self._abc_registry = extra._abc_registry
AttributeError: type object 'Callable' has no attribute '_abc_registry'

  This error is unexpected and should not have happened. Somehow a worker
  crashed in an unanticipated way causing the main_loop to throw an exception,
  which is being caught in "python/ray/workers/default_worker.py".

2019-08-29 06:28:24,194 ERROR worker.py:1714 -- A worker died or was killed while executing task cb9c0f35cd7fed78ddcce8f4701afc6f.
2019-08-29 06:28:24,196 ERROR worker.py:1714 -- Traceback (most recent call last):
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 967, in _process_task
    self._store_outputs_in_object_store(return_object_ids, outputs)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 880, in _store_outputs_in_object_store
    self.put_object(object_ids[i], outputs[i])
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 398, in put_object
    self._try_store_and_register(object_id, value)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 418, in _try_store_and_register
    self.store_and_register(object_id, value)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 327, in store_and_register
    self.current_job_id))
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 256, in get_serialization_context
    _initialize_serialization(job_id)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 1187, in _initialize_serialization
    serialization_context = pyarrow.default_serialization_context()
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/pyarrow_files/pyarrow/serialization.py", line 361, in default_serialization_context
    register_default_serialization_handlers(context)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/pyarrow_files/pyarrow/serialization.py", line 356, in register_default_serialization_handlers
    _register_custom_pandas_handlers(serialization_context)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/pyarrow_files/pyarrow/serialization.py", line 150, in _register_custom_pandas_handlers
    import pandas as pd
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/pandas/__init__.py", line 23, in <module>
    from pandas.compat.numpy import *
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/pandas/compat/__init__.py", line 446, in <module>
    import typing
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/typing.py", line 1357, in <module>
    class Callable(extra=collections_abc.Callable, metaclass=CallableMeta):
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/typing.py", line 1005, in __new__
    self._abc_registry = extra._abc_registry
AttributeError: type object 'Callable' has no attribute '_abc_registry'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/workers/default_worker.py", line 98, in <module>
    ray.worker.global_worker.main_loop()
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 1088, in main_loop
    self._wait_for_and_process_task(task)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 1039, in _wait_for_and_process_task
    self._process_task(task, execution_info)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 971, in _process_task
    ray.utils.format_error_message(traceback.format_exc()))
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 980, in _handle_process_task_failure
    self._store_outputs_in_object_store(return_object_ids, failure_objects)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 880, in _store_outputs_in_object_store
    self.put_object(object_ids[i], outputs[i])
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 398, in put_object
    self._try_store_and_register(object_id, value)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 418, in _try_store_and_register
    self.store_and_register(object_id, value)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 327, in store_and_register
    self.current_job_id))
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 256, in get_serialization_context
    _initialize_serialization(job_id)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 1187, in _initialize_serialization
    serialization_context = pyarrow.default_serialization_context()
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/pyarrow_files/pyarrow/serialization.py", line 361, in default_serialization_context
    register_default_serialization_handlers(context)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/pyarrow_files/pyarrow/serialization.py", line 356, in register_default_serialization_handlers
    _register_custom_pandas_handlers(serialization_context)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/pyarrow_files/pyarrow/serialization.py", line 150, in _register_custom_pandas_handlers
    import pandas as pd
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/pandas/__init__.py", line 23, in <module>
    from pandas.compat.numpy import *
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/pandas/compat/__init__.py", line 446, in <module>
    import typing
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/typing.py", line 1357, in <module>
    class Callable(extra=collections_abc.Callable, metaclass=CallableMeta):
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/typing.py", line 1005, in __new__
    self._abc_registry = extra._abc_registry
AttributeError: type object 'Callable' has no attribute '_abc_registry'

  This error is unexpected and should not have happened. Somehow a worker
  crashed in an unanticipated way causing the main_loop to throw an exception,
  which is being caught in "python/ray/workers/default_worker.py".

2019-08-29 06:28:24,197 ERROR worker.py:1714 -- A worker died or was killed while executing task 7b6954b4a601f280a6005513862c75e3.
2019-08-29 06:28:24,202 ERROR worker.py:1714 -- Traceback (most recent call last):
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 967, in _process_task
    self._store_outputs_in_object_store(return_object_ids, outputs)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 880, in _store_outputs_in_object_store
    self.put_object(object_ids[i], outputs[i])
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 398, in put_object
    self._try_store_and_register(object_id, value)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 418, in _try_store_and_register
    self.store_and_register(object_id, value)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 327, in store_and_register
    self.current_job_id))
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 256, in get_serialization_context
    _initialize_serialization(job_id)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 1187, in _initialize_serialization
    serialization_context = pyarrow.default_serialization_context()
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/pyarrow_files/pyarrow/serialization.py", line 361, in default_serialization_context
    register_default_serialization_handlers(context)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/pyarrow_files/pyarrow/serialization.py", line 356, in register_default_serialization_handlers
    _register_custom_pandas_handlers(serialization_context)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/pyarrow_files/pyarrow/serialization.py", line 150, in _register_custom_pandas_handlers
    import pandas as pd
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/pandas/__init__.py", line 23, in <module>
    from pandas.compat.numpy import *
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/pandas/compat/__init__.py", line 446, in <module>
    import typing
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/typing.py", line 1357, in <module>
    class Callable(extra=collections_abc.Callable, metaclass=CallableMeta):
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/typing.py", line 1005, in __new__
    self._abc_registry = extra._abc_registry
AttributeError: type object 'Callable' has no attribute '_abc_registry'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/workers/default_worker.py", line 98, in <module>
    ray.worker.global_worker.main_loop()
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 1088, in main_loop
    self._wait_for_and_process_task(task)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 1039, in _wait_for_and_process_task
    self._process_task(task, execution_info)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 971, in _process_task
    ray.utils.format_error_message(traceback.format_exc()))
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 980, in _handle_process_task_failure
    self._store_outputs_in_object_store(return_object_ids, failure_objects)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 880, in _store_outputs_in_object_store
    self.put_object(object_ids[i], outputs[i])
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 398, in put_object
    self._try_store_and_register(object_id, value)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 418, in _try_store_and_register
    self.store_and_register(object_id, value)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 327, in store_and_register
    self.current_job_id))
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 256, in get_serialization_context
    _initialize_serialization(job_id)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 1187, in _initialize_serialization
    serialization_context = pyarrow.default_serialization_context()
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/pyarrow_files/pyarrow/serialization.py", line 361, in default_serialization_context
    register_default_serialization_handlers(context)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/pyarrow_files/pyarrow/serialization.py", line 356, in register_default_serialization_handlers
    _register_custom_pandas_handlers(serialization_context)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/pyarrow_files/pyarrow/serialization.py", line 150, in _register_custom_pandas_handlers
    import pandas as pd
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/pandas/__init__.py", line 23, in <module>
    from pandas.compat.numpy import *
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/pandas/compat/__init__.py", line 446, in <module>
    import typing
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/typing.py", line 1357, in <module>
    class Callable(extra=collections_abc.Callable, metaclass=CallableMeta):
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/typing.py", line 1005, in __new__
    self._abc_registry = extra._abc_registry
AttributeError: type object 'Callable' has no attribute '_abc_registry'

  This error is unexpected and should not have happened. Somehow a worker
  crashed in an unanticipated way causing the main_loop to throw an exception,
  which is being caught in "python/ray/workers/default_worker.py".

2019-08-29 06:28:24,203 ERROR worker.py:1714 -- Traceback (most recent call last):
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 967, in _process_task
    self._store_outputs_in_object_store(return_object_ids, outputs)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 880, in _store_outputs_in_object_store
    self.put_object(object_ids[i], outputs[i])
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 398, in put_object
    self._try_store_and_register(object_id, value)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 418, in _try_store_and_register
    self.store_and_register(object_id, value)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 327, in store_and_register
    self.current_job_id))
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 256, in get_serialization_context
    _initialize_serialization(job_id)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 1187, in _initialize_serialization
    serialization_context = pyarrow.default_serialization_context()
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/pyarrow_files/pyarrow/serialization.py", line 361, in default_serialization_context
    register_default_serialization_handlers(context)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/pyarrow_files/pyarrow/serialization.py", line 356, in register_default_serialization_handlers
    _register_custom_pandas_handlers(serialization_context)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/pyarrow_files/pyarrow/serialization.py", line 150, in _register_custom_pandas_handlers
    import pandas as pd
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/pandas/__init__.py", line 23, in <module>
    from pandas.compat.numpy import *
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/pandas/compat/__init__.py", line 446, in <module>
    import typing
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/typing.py", line 1357, in <module>
    class Callable(extra=collections_abc.Callable, metaclass=CallableMeta):
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/typing.py", line 1005, in __new__
    self._abc_registry = extra._abc_registry
AttributeError: type object 'Callable' has no attribute '_abc_registry'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/workers/default_worker.py", line 98, in <module>
    ray.worker.global_worker.main_loop()
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 1088, in main_loop
    self._wait_for_and_process_task(task)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 1039, in _wait_for_and_process_task
    self._process_task(task, execution_info)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 971, in _process_task
    ray.utils.format_error_message(traceback.format_exc()))
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 980, in _handle_process_task_failure
    self._store_outputs_in_object_store(return_object_ids, failure_objects)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 880, in _store_outputs_in_object_store
    self.put_object(object_ids[i], outputs[i])
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 398, in put_object
    self._try_store_and_register(object_id, value)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 418, in _try_store_and_register
    self.store_and_register(object_id, value)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 327, in store_and_register
    self.current_job_id))
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 256, in get_serialization_context
    _initialize_serialization(job_id)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 1187, in _initialize_serialization
    serialization_context = pyarrow.default_serialization_context()
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/pyarrow_files/pyarrow/serialization.py", line 361, in default_serialization_context
    register_default_serialization_handlers(context)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/pyarrow_files/pyarrow/serialization.py", line 356, in register_default_serialization_handlers
    _register_custom_pandas_handlers(serialization_context)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/pyarrow_files/pyarrow/serialization.py", line 150, in _register_custom_pandas_handlers
    import pandas as pd
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/pandas/__init__.py", line 23, in <module>
    from pandas.compat.numpy import *
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/pandas/compat/__init__.py", line 446, in <module>
    import typing
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/typing.py", line 1357, in <module>
    class Callable(extra=collections_abc.Callable, metaclass=CallableMeta):
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/typing.py", line 1005, in __new__
    self._abc_registry = extra._abc_registry
AttributeError: type object 'Callable' has no attribute '_abc_registry'

  This error is unexpected and should not have happened. Somehow a worker
  crashed in an unanticipated way causing the main_loop to throw an exception,
  which is being caught in "python/ray/workers/default_worker.py".

2019-08-29 06:28:24,204 ERROR worker.py:1714 -- Traceback (most recent call last):
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 967, in _process_task
    self._store_outputs_in_object_store(return_object_ids, outputs)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 880, in _store_outputs_in_object_store
    self.put_object(object_ids[i], outputs[i])
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 398, in put_object
    self._try_store_and_register(object_id, value)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 418, in _try_store_and_register
    self.store_and_register(object_id, value)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 327, in store_and_register
    self.current_job_id))
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 256, in get_serialization_context
    _initialize_serialization(job_id)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 1187, in _initialize_serialization
    serialization_context = pyarrow.default_serialization_context()
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/pyarrow_files/pyarrow/serialization.py", line 361, in default_serialization_context
    register_default_serialization_handlers(context)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/pyarrow_files/pyarrow/serialization.py", line 356, in register_default_serialization_handlers
    _register_custom_pandas_handlers(serialization_context)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/pyarrow_files/pyarrow/serialization.py", line 150, in _register_custom_pandas_handlers
    import pandas as pd
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/pandas/__init__.py", line 23, in <module>
    from pandas.compat.numpy import *
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/pandas/compat/__init__.py", line 446, in <module>
    import typing
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/typing.py", line 1357, in <module>
    class Callable(extra=collections_abc.Callable, metaclass=CallableMeta):
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/typing.py", line 1005, in __new__
    self._abc_registry = extra._abc_registry
AttributeError: type object 'Callable' has no attribute '_abc_registry'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/workers/default_worker.py", line 98, in <module>
    ray.worker.global_worker.main_loop()
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 1088, in main_loop
    self._wait_for_and_process_task(task)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 1039, in _wait_for_and_process_task
    self._process_task(task, execution_info)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 971, in _process_task
    ray.utils.format_error_message(traceback.format_exc()))
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 980, in _handle_process_task_failure
    self._store_outputs_in_object_store(return_object_ids, failure_objects)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 880, in _store_outputs_in_object_store
    self.put_object(object_ids[i], outputs[i])
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 398, in put_object
    self._try_store_and_register(object_id, value)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 418, in _try_store_and_register
    self.store_and_register(object_id, value)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 327, in store_and_register
    self.current_job_id))
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 256, in get_serialization_context
    _initialize_serialization(job_id)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 1187, in _initialize_serialization
    serialization_context = pyarrow.default_serialization_context()
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/pyarrow_files/pyarrow/serialization.py", line 361, in default_serialization_context
    register_default_serialization_handlers(context)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/pyarrow_files/pyarrow/serialization.py", line 356, in register_default_serialization_handlers
    _register_custom_pandas_handlers(serialization_context)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/pyarrow_files/pyarrow/serialization.py", line 150, in _register_custom_pandas_handlers
    import pandas as pd
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/pandas/__init__.py", line 23, in <module>
    from pandas.compat.numpy import *
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/pandas/compat/__init__.py", line 446, in <module>
    import typing
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/typing.py", line 1357, in <module>
    class Callable(extra=collections_abc.Callable, metaclass=CallableMeta):
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/typing.py", line 1005, in __new__
    self._abc_registry = extra._abc_registry
AttributeError: type object 'Callable' has no attribute '_abc_registry'

  This error is unexpected and should not have happened. Somehow a worker
  crashed in an unanticipated way causing the main_loop to throw an exception,
  which is being caught in "python/ray/workers/default_worker.py".

2019-08-29 06:28:24,204 ERROR worker.py:1714 -- A worker died or was killed while executing task 20088ee16e4a4904f3c0c57eb2c1725e.
2019-08-29 06:28:24,206 ERROR worker.py:1714 -- A worker died or was killed while executing task cc699a38d46a9c81269fa406b567413b.
2019-08-29 06:28:24,208 ERROR worker.py:1714 -- A worker died or was killed while executing task c734fac15bca2349c314ffbca1fd1b88.
2019-08-29 06:28:24,209 ERROR worker.py:1714 -- Traceback (most recent call last):
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 967, in _process_task
    self._store_outputs_in_object_store(return_object_ids, outputs)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 880, in _store_outputs_in_object_store
    self.put_object(object_ids[i], outputs[i])
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 398, in put_object
    self._try_store_and_register(object_id, value)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 418, in _try_store_and_register
    self.store_and_register(object_id, value)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 327, in store_and_register
    self.current_job_id))
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 256, in get_serialization_context
    _initialize_serialization(job_id)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 1187, in _initialize_serialization
    serialization_context = pyarrow.default_serialization_context()
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/pyarrow_files/pyarrow/serialization.py", line 361, in default_serialization_context
    register_default_serialization_handlers(context)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/pyarrow_files/pyarrow/serialization.py", line 356, in register_default_serialization_handlers
    _register_custom_pandas_handlers(serialization_context)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/pyarrow_files/pyarrow/serialization.py", line 150, in _register_custom_pandas_handlers
    import pandas as pd
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/pandas/__init__.py", line 23, in <module>
    from pandas.compat.numpy import *
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/pandas/compat/__init__.py", line 446, in <module>
    import typing
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/typing.py", line 1357, in <module>
    class Callable(extra=collections_abc.Callable, metaclass=CallableMeta):
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/typing.py", line 1005, in __new__
    self._abc_registry = extra._abc_registry
AttributeError: type object 'Callable' has no attribute '_abc_registry'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/workers/default_worker.py", line 98, in <module>
    ray.worker.global_worker.main_loop()
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 1088, in main_loop
    self._wait_for_and_process_task(task)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 1039, in _wait_for_and_process_task
    self._process_task(task, execution_info)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 971, in _process_task
    ray.utils.format_error_message(traceback.format_exc()))
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 980, in _handle_process_task_failure
    self._store_outputs_in_object_store(return_object_ids, failure_objects)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 880, in _store_outputs_in_object_store
    self.put_object(object_ids[i], outputs[i])
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 398, in put_object
    self._try_store_and_register(object_id, value)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 418, in _try_store_and_register
    self.store_and_register(object_id, value)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 327, in store_and_register
    self.current_job_id))
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 256, in get_serialization_context
    _initialize_serialization(job_id)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 1187, in _initialize_serialization
    serialization_context = pyarrow.default_serialization_context()
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/pyarrow_files/pyarrow/serialization.py", line 361, in default_serialization_context
    register_default_serialization_handlers(context)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/pyarrow_files/pyarrow/serialization.py", line 356, in register_default_serialization_handlers
    _register_custom_pandas_handlers(serialization_context)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/pyarrow_files/pyarrow/serialization.py", line 150, in _register_custom_pandas_handlers
    import pandas as pd
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/pandas/__init__.py", line 23, in <module>
    from pandas.compat.numpy import *
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/pandas/compat/__init__.py", line 446, in <module>
    import typing
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/typing.py", line 1357, in <module>
    class Callable(extra=collections_abc.Callable, metaclass=CallableMeta):
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/typing.py", line 1005, in __new__
    self._abc_registry = extra._abc_registry
AttributeError: type object 'Callable' has no attribute '_abc_registry'

  This error is unexpected and should not have happened. Somehow a worker
  crashed in an unanticipated way causing the main_loop to throw an exception,
  which is being caught in "python/ray/workers/default_worker.py".

2019-08-29 06:28:24,210 ERROR worker.py:1714 -- Traceback (most recent call last):
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 967, in _process_task
    self._store_outputs_in_object_store(return_object_ids, outputs)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 880, in _store_outputs_in_object_store
    self.put_object(object_ids[i], outputs[i])
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 398, in put_object
    self._try_store_and_register(object_id, value)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 418, in _try_store_and_register
    self.store_and_register(object_id, value)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 327, in store_and_register
    self.current_job_id))
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 256, in get_serialization_context
    _initialize_serialization(job_id)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 1187, in _initialize_serialization
    serialization_context = pyarrow.default_serialization_context()
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/pyarrow_files/pyarrow/serialization.py", line 361, in default_serialization_context
    register_default_serialization_handlers(context)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/pyarrow_files/pyarrow/serialization.py", line 356, in register_default_serialization_handlers
    _register_custom_pandas_handlers(serialization_context)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/pyarrow_files/pyarrow/serialization.py", line 150, in _register_custom_pandas_handlers
    import pandas as pd
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/pandas/__init__.py", line 23, in <module>
    from pandas.compat.numpy import *
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/pandas/compat/__init__.py", line 446, in <module>
    import typing
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/typing.py", line 1357, in <module>
    class Callable(extra=collections_abc.Callable, metaclass=CallableMeta):
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/typing.py", line 1005, in __new__
    self._abc_registry = extra._abc_registry
AttributeError: type object 'Callable' has no attribute '_abc_registry'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/workers/default_worker.py", line 98, in <module>
    ray.worker.global_worker.main_loop()
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 1088, in main_loop
    self._wait_for_and_process_task(task)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 1039, in _wait_for_and_process_task
    self._process_task(task, execution_info)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 971, in _process_task
    ray.utils.format_error_message(traceback.format_exc()))
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 980, in _handle_process_task_failure
    self._store_outputs_in_object_store(return_object_ids, failure_objects)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 880, in _store_outputs_in_object_store
    self.put_object(object_ids[i], outputs[i])
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 398, in put_object
    self._try_store_and_register(object_id, value)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 418, in _try_store_and_register
    self.store_and_register(object_id, value)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 327, in store_and_register
    self.current_job_id))
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 256, in get_serialization_context
    _initialize_serialization(job_id)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 1187, in _initialize_serialization
    serialization_context = pyarrow.default_serialization_context()
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/pyarrow_files/pyarrow/serialization.py", line 361, in default_serialization_context
    register_default_serialization_handlers(context)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/pyarrow_files/pyarrow/serialization.py", line 356, in register_default_serialization_handlers
    _register_custom_pandas_handlers(serialization_context)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/pyarrow_files/pyarrow/serialization.py", line 150, in _register_custom_pandas_handlers
    import pandas as pd
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/pandas/__init__.py", line 23, in <module>
    from pandas.compat.numpy import *
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/pandas/compat/__init__.py", line 446, in <module>
    import typing
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/typing.py", line 1357, in <module>
    class Callable(extra=collections_abc.Callable, metaclass=CallableMeta):
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/typing.py", line 1005, in __new__
    self._abc_registry = extra._abc_registry
AttributeError: type object 'Callable' has no attribute '_abc_registry'

  This error is unexpected and should not have happened. Somehow a worker
  crashed in an unanticipated way causing the main_loop to throw an exception,
  which is being caught in "python/ray/workers/default_worker.py".

2019-08-29 06:28:24,212 ERROR worker.py:1714 -- A worker died or was killed while executing task f2a1dc94184103efd786254854270a12.
2019-08-29 06:28:24,219 ERROR worker.py:1714 -- A worker died or was killed while executing task 91a79a584cdc65d65e4aea8f58d3bc57.
2019-08-29 06:28:24,224 ERROR worker.py:1714 -- Traceback (most recent call last):
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 967, in _process_task
    self._store_outputs_in_object_store(return_object_ids, outputs)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 880, in _store_outputs_in_object_store
    self.put_object(object_ids[i], outputs[i])
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 398, in put_object
    self._try_store_and_register(object_id, value)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 418, in _try_store_and_register
    self.store_and_register(object_id, value)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 327, in store_and_register
    self.current_job_id))
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 256, in get_serialization_context
    _initialize_serialization(job_id)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 1187, in _initialize_serialization
    serialization_context = pyarrow.default_serialization_context()
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/pyarrow_files/pyarrow/serialization.py", line 361, in default_serialization_context
    register_default_serialization_handlers(context)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/pyarrow_files/pyarrow/serialization.py", line 356, in register_default_serialization_handlers
    _register_custom_pandas_handlers(serialization_context)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/pyarrow_files/pyarrow/serialization.py", line 150, in _register_custom_pandas_handlers
    import pandas as pd
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/pandas/__init__.py", line 23, in <module>
    from pandas.compat.numpy import *
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/pandas/compat/__init__.py", line 446, in <module>
    import typing
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/typing.py", line 1357, in <module>
    class Callable(extra=collections_abc.Callable, metaclass=CallableMeta):
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/typing.py", line 1005, in __new__
    self._abc_registry = extra._abc_registry
AttributeError: type object 'Callable' has no attribute '_abc_registry'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/workers/default_worker.py", line 98, in <module>
    ray.worker.global_worker.main_loop()
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 1088, in main_loop
    self._wait_for_and_process_task(task)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 1039, in _wait_for_and_process_task
    self._process_task(task, execution_info)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 971, in _process_task
    ray.utils.format_error_message(traceback.format_exc()))
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 980, in _handle_process_task_failure
    self._store_outputs_in_object_store(return_object_ids, failure_objects)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 880, in _store_outputs_in_object_store
    self.put_object(object_ids[i], outputs[i])
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 398, in put_object
    self._try_store_and_register(object_id, value)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 418, in _try_store_and_register
    self.store_and_register(object_id, value)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 327, in
williamma12 picture williamma12 on 29 Aug 2019

@simon-mo see result you asked for
issue-5316.pdf

wgong picture wgong on 30 Aug 2019

@williamma12 since you can reproduce this, can you try the solution in https://github.com/ray-project/ray/issues/5316#issuecomment-526040105?

robertnishihara picture robertnishihara on 1 Sep 2019

Closing as this should be fixed in Modin now. We had to change the order of imports for the typing library.

Please reopen if the issue persists.

devin-petersohn picture devin-petersohn on 3 Oct 2019
Was this page helpful?
0 / 5 - 0 ratings

Related issues

austinmw picture austinmw  ·  3Comments
robertnishihara picture robertnishihara  ·  3Comments
WangYiPengPeter picture WangYiPengPeter  ·  3Comments
AndreCNF picture AndreCNF  ·  3Comments
zplizzi picture zplizzi  ·  3Comments