Ray: Ray fails to initialize

Created on 1 Jul 2020  路  9Comments  路  Source: ray-project/ray

Problem:

2020-07-01 17:03:38,228 INFO resource_spec.py:212 -- Starting Ray with 6.93 GiB memory available for workers and up to 3.48 GiB for objects. You can adjust these settings with ray.init(memory=, object_store_memory=).
2020-07-01 17:03:39,822 INFO services.py:1165 -- View the Ray dashboard at localhost:8265
Traceback (most recent call last):
File "", line 549, in run_nodebug
File "", line 14, in
File "C:\Users\jrela\AppData\Local\Programs\Python\Python37\lib\site-packages\ray\worker.py", line 793, in init
job_id=job_id)
File "C:\Users\jrela\AppData\Local\Programs\Python\Python37\lib\site-packages\ray\worker.py", line 1091, in connect
faulthandler.enable(all_threads=False)
AttributeError: 'AsyncStream' object has no attribute 'fileno'

Reproduction (REQUIRED)

import ray

ray.init()

In the following system:

OS Name Microsoft Windows 10 Home
Version 10.0.18362 Build 18362
System Manufacturer HP
System Model HP EliteBook 830 G6
System Type x64-based PC
Processor Intel(R) Core(TM) i7-8565U CPU @ 1.80GHz, 1992 Mhz, 4 Core(s), 8 Logical Processor(s)
Installed Physical Memory (RAM) 16.0 GB
Total Physical Memory 15.8 GB
Available Physical Memory 10.6 GB
Total Virtual Memory 18.2 GB
Available Virtual Memory 9.38 GB
Page File Space 2.38 GB

Python 3.7-64 bit installation

Pip freeze:

aiohttp==3.6.2
altgraph==0.17
async-timeout==3.0.1
asyncio==3.4.3
attrs==19.3.0
beautifulsoup4==4.9.1
chardet==3.0.4
click==7.1.2
colorama==0.4.3
cycler==0.10.0
Cython==0.29.20
dask==2.16.0
filelock==3.0.12
future==0.18.2
google==2.0.3
grpcio==1.30.0
h5py==2.10.0
idna==2.10
importlib-metadata==1.7.0
jsonschema==3.2.0
kiwisolver==1.2.0
matplotlib==3.2.1
mpmath==1.1.0
msgpack==1.0.0
multidict==4.7.6
numpy==1.18.4
opencv-python==4.2.0.34
pandas==1.0.3
pefile==2019.4.18
Pillow==7.1.2
protobuf==3.12.2
py-spy==0.3.3
pycromanager==0.2.2
PyInstaller==3.6
pymmcore==10.0.0.1
pyparsing==2.4.7
PyQt5==5.14.2
PyQt5-sip==12.7.2
pyqtgraph==0.10.0
pyrsistent==0.16.0
pyserial==3.4
PySide2==5.14.2.1
python-dateutil==2.8.1
pytz==2020.1
pywin32-ctypes==0.2.0
PyYAML==5.3.1
pyzmq==19.0.1
qtmodern==0.2.0
QtPy==1.9.0
ray==0.8.6
redis==3.4.1
scipy==1.4.1
shiboken2==5.14.2.1
six==1.14.0
soupsieve==2.0.1
sympy==1.5.1
toolz==0.10.0
trackpy==0.4.2
yarl==1.4.2
zaber.serial==0.9.1
zipp==3.1.0
zmq==0.0.0

P2 bug core needs-repro-script

All 9 comments

@mehrdadn should we add this to https://github.com/ray-project/ray/issues/9114?

Thanks for the report. Which wheel have you installed? The latest release, or a nightly wheel? Also, what environment are you calling this from? Are you calling this from a console? A screenshot would be helpful.

@mehrdadn we could probably make this take a file descriptor, then always hardcode a stdout_fileno or stderr_fileno rather than grabbing it from sys.stdout.fileno. It's unclear whether changing sys.stdout will produce the desired effects within the environment.

https://github.com/ray-project/ray/blob/e0d8b58969728054d1800abc925aa5352882baa8/python/ray/worker.py#L898

I'd prefer not to hard-code stdout_fileno if possible since it's legitimate to try to replace sys.stdout. But I'll have to look at it in more detail to see how to make it work.

One missing test case that we'll need is a python module that prints from c/c++ (i.e. calls printf/cout), to make sure that output is properly redirected.

Hi guys!

Thanks so much for your quick reply.

Thanks to @mehrdadn for his questions. Indeed, the problem was calling it from PyScripter. When called from the Command Prompt, then it works fine.

Just in case, here are some questions answered:

1) this the release available for python 3.7 (pip installray). I installed it three days ago.

2) I'm calling it from the basic environment. Here's a list of modules in it:


Modules

aiohttp==3.6.2
altgraph==0.17
async-timeout==3.0.1
asyncio==3.4.3
attrs==19.3.0
beautifulsoup4==4.9.1
chardet==3.0.4
click==7.1.2
colorama==0.4.3
cycler==0.10.0
Cython==0.29.20
dask==2.16.0
filelock==3.0.12
future==0.18.2
google==2.0.3
grpcio==1.30.0
h5py==2.10.0
idna==2.10
importlib-metadata==1.7.0
jsonschema==3.2.0
kiwisolver==1.2.0
matplotlib==3.2.1
mpmath==1.1.0
msgpack==1.0.0
multidict==4.7.6
numpy==1.18.4
opencv-python==4.2.0.34
pandas==1.0.3
pefile==2019.4.18
Pillow==7.1.2
protobuf==3.12.2
py-spy==0.3.3
pycromanager==0.2.2
PyInstaller==3.6
pymmcore==10.0.0.1
pyparsing==2.4.7
PyQt5==5.14.2
PyQt5-sip==12.7.2
pyqtgraph==0.10.0
pyrsistent==0.16.0
pyserial==3.4
PySide2==5.14.2.1
python-dateutil==2.8.1
pytz==2020.1
pywin32-ctypes==0.2.0
PyYAML==5.3.1
pyzmq==19.0.1
qtmodern==0.2.0
QtPy==1.9.0
ray==0.8.6
redis==3.4.1
scipy==1.4.1
shiboken2==5.14.2.1
six==1.14.0
soupsieve==2.0.1
sympy==1.5.1
toolz==0.10.0
trackpy==0.4.2
yarl==1.4.2
zaber.serial==0.9.1
zipp==3.1.0
zmq==0.0.0

3) No, I was calling it from PyScripter.

Ok, so it stopped working just after a few successful tries. Now it fails when initializing from the Command Prompt. The code is:

import ray

if __name__ = "__main__":
ray.init()

I got the following output:


Output

py -3.7 RayTest.py
2020-07-03 15:34:18,276 INFO resource_spec.py:212 -- Starting Ray with 4.98 GiB memory available for workers and up to 2.5 GiB for objects. You can adjust these settings with ray.init(memory=<bytes>, object_store_memory=<bytes>).
2020-07-03 15:34:18,701 INFO services.py:1165 -- View the Ray dashboard at localhost:8265
E0703 15:34:23.979981   544  8392 raylet_client.cc:69] Retrying to connect to socket for pathname tcp://127.0.0.1:61794 (num_attempts = 1, num_retries = 10)
E0703 15:34:26.986989   544  8392 raylet_client.cc:69] Retrying to connect to socket for pathname tcp://127.0.0.1:61794 (num_attempts = 2, num_retries = 10)
E0703 15:34:29.991088   544  8392 raylet_client.cc:69] Retrying to connect to socket for pathname tcp://127.0.0.1:61794 (num_attempts = 3, num_retries = 10)
E0703 15:34:32.994232   544  8392 raylet_client.cc:69] Retrying to connect to socket for pathname tcp://127.0.0.1:61794 (num_attempts = 4, num_retries = 10)
E0703 15:34:36.001313   544  8392 raylet_client.cc:69] Retrying to connect to socket for pathname tcp://127.0.0.1:61794 (num_attempts = 5, num_retries = 10)
E0703 15:34:39.004927   544  8392 raylet_client.cc:69] Retrying to connect to socket for pathname tcp://127.0.0.1:61794 (num_attempts = 6, num_retries = 10)
E0703 15:34:42.006975   544  8392 raylet_client.cc:69] Retrying to connect to socket for pathname tcp://127.0.0.1:61794 (num_attempts = 7, num_retries = 10)
E0703 15:34:45.010505   544  8392 raylet_client.cc:69] Retrying to connect to socket for pathname tcp://127.0.0.1:61794 (num_attempts = 8, num_retries = 10)
E0703 15:34:48.013639   544  8392 raylet_client.cc:69] Retrying to connect to socket for pathname tcp://127.0.0.1:61794 (num_attempts = 9, num_retries = 10)
F0703 15:34:49.014528   544  8392 raylet_client.cc:78] Could not connect to socket tcp://127.0.0.1:61794
*** Check failure stack trace: ***
    @   00007FF94AFF3A8C  public: __cdecl google::LogMessage::~LogMessage(void) __ptr64
    @   00007FF94AE68954  public: virtual __cdecl google::NullStreamFatal::~NullStreamFatal(void) __ptr64
    @   00007FF94AEA351B  public: void __cdecl google::NullStreamFatal::`vbase destructor'(void) __ptr64
    @   00007FF94AEA5B5E  public: void __cdecl google::NullStreamFatal::`vbase destructor'(void) __ptr64
    @   00007FF94ADB3B98  public: class google::LogMessageVoidify & __ptr64 __cdecl google::LogMessageVoidify::operator=(class google::LogMessageVoidify const & __ptr64) __ptr64
    @   00007FF94ADB1C00  public: class google::LogMessageVoidify & __ptr64 __cdecl google::LogMessageVoidify::operator=(class google::LogMessageVoidify const & __ptr64) __ptr64
    @   00007FF94ADB00ED  public: class google::LogMessageVoidify & __ptr64 __cdecl google::LogMessageVoidify::operator=(class google::LogMessageVoidify const & __ptr64) __ptr64
    @   00007FF94ADAF9C3  public: class google::LogMessageVoidify & __ptr64 __cdecl google::LogMessageVoidify::operator=(class google::LogMessageVoidify const & __ptr64) __ptr64
    @   00007FF94AD2F179  public: virtual __cdecl google::LogSink::~LogSink(void) __ptr64
    @   00007FF95A8FF4AF  _PyObject_FastCallKeywords
    @   00007FF95A8FF2B6  _PyObject_FastCallKeywords
    @   00007FF95A8FDC3B  _PyFunction_FastCallDict
    @   00007FF95A9053B8  _PyEval_EvalFrameDefault
    @   00007FF95A8FE3AC  _PyEval_EvalCodeWithName
    @   00007FF95A8FDC0C  _PyFunction_FastCallDict
    @   00007FF95A90657B  _PyEval_EvalFrameDefault
    @   00007FF95AA015BD  _PyStack_UnpackDict
    @   00007FF95A94DD4A  PyErr_NoMemory
    @   00007FF95A9053B8  _PyEval_EvalFrameDefault
    @   00007FF95A8FE3AC  _PyEval_EvalCodeWithName
    @   00007FF95A90B573  PyEval_EvalCodeEx
    @   00007FF95A90B4D1  PyEval_EvalCode
    @   00007FF95A90B47B  PyArena_Free
    @   00007FF95AA74201  PyRun_FileExFlags
    @   00007FF95AA7499D  PyRun_SimpleFileExFlags
    @   00007FF95AA740DF  PyRun_AnyFileExFlags
    @   00007FF95A9C14AC  _Py_UnixMain
    @   00007FF95A9C1553  _Py_UnixMain
    @   00007FF95A931270  PyErr_NoMemory
    @   00007FF95A8D107B  Py_Main
    @   00007FF95A8D1056  Py_Main
    @   00007FF74BF7126D  (unknown)

image

@josepNTU Could you quit Python, double-check in Task Manager to make sure there's no Python or Ray process running, then try again?

Was this page helpful?
0 / 5 - 0 ratings

Related issues

WangYiPengPeter picture WangYiPengPeter  路  3Comments

robertnishihara picture robertnishihara  路  3Comments

AndreCNF picture AndreCNF  路  3Comments

zhaokang1228 picture zhaokang1228  路  3Comments

1beb picture 1beb  路  3Comments