Ray: `ray.init` on WSL does not work

[devin-petersohn]

What is the problem?

Ray version and other system information (Python version, TensorFlow version, OS): 0.8

This was reported in Modin. See discussion here: https://discuss.modin.org/t/problem-when-running-code-with-modin-1-time/90/2

This is related to, but not the same as #3059. See logs below

Reproduction (REQUIRED)

Please provide a script that can be run to reproduce the issue. The script should have no external library dependencies (i.e., use fake or mock data / environments):

import ray
ray.init()

Logs below:

2020-01-23 14:09:01,315 INFO resource_spec.py:216 – Starting Ray with 30.32 GiB memory available for workers and up to 15.16 GiB for objects. You can adjust these settings with ray.init(memory=, object_store_memory=).
F0123 14:09:07.020850 1052 redis_context.cc:215] Could not establish connection to redis 10.46.89.57:57714 (context.err = 1)
*** Check failure stack trace: ***
@ 0x7f9b27d3f98d (unknown)
@ 0x7f9b27d40dfc (unknown)

If we cannot run your script, we cannot fix your issue.

• [ ] I have verified my script runs in a clean environment and reproduces the issue.
• [ ] I have verified the issue also occurs with the latest wheels.

[mehrdadn]

Would you be able to check what the latest release/commit was that worked for you?

[devin-petersohn]

I have not reproduced the issue myself, I am reporting on behalf of a user. See the thread in my original comment for more information or feel free to reach out there.

[ninafiona]

I have the same problem, it's working on linux, but I get the error on my WSL.

[mehrdadn]

@ninafiona would you happen to know what the latest working version is?

[linjohn]

I've just encountered the same error on WSL:

• Windows 10 Pro Version 1909
• Ubuntu 18.04 LTS
• Python 3.6.9
• ray==0.8.1
• psutil==5.6.7
• setproctitle==1.1.10

Traceback (most recent call last):
  File "test.py", line 59, in <module>
    ray.init()
  File "/mnt/c/Users/User/x/.venv/lib/python3.6/site-packages/ray/worker.py", line 743, in init
    ray_params=ray_params)
  File "/mnt/c/Users/User/x/.venv/lib/python3.6/site-packages/ray/node.py", line 162, in __init__
    self.start_head_processes()
  File "/mnt/c/Users/User/x/.venv/lib/python3.6/site-packages/ray/node.py", line 594, in start_head_processes
    self.start_redis()
  File "/mnt/c/Users/User/x/.venv/lib/python3.6/site-packages/ray/node.py", line 434, in start_redis
    self.get_resource_spec(),
  File "/mnt/c/Users/User/x/.venv/lib/python3.6/site-packages/ray/node.py", line 225, in get_resource_spec
    self._ray_params.redis_max_memory).resolve(is_head=self.head)
  File "/mnt/c/Users/User/x/.venv/lib/python3.6/site-packages/ray/resource_spec.py", line 158, in resolve
    system_memory = ray.utils.get_system_memory()
  File "/mnt/c/Users/User/x/.venv/lib/python3.6/site-packages/ray/utils.py", line 438, in get_system_memory
    psutil_memory_in_bytes = psutil.virtual_memory().total
  File "/mnt/c/Users/User/x/.venv/lib/python3.6/site-packages/psutil/__init__.py", line 2088, in virtual_memory
    ret = _psplatform.virtual_memory()
  File "/mnt/c/Users/User/x/.venv/lib/python3.6/site-packages/psutil/_pslinux.py", line 398, in virtual_memory
    mems[fields[0]] = int(fields[1]) * 1024
ValueError: invalid literal for int() with base 10: b'kB'
Terminated

[mehrdadn]

Oh, this is exactly what I tried to fix in #6808. @pcmoritz thoughts?

[linjohn]

I've found a workaround. You need to downgrade psutil to at least version 4.3.1. Then ray.init() starts without difficulty.

pip install --upgrade psutil==4.3.1

[ninafiona]

@linjohn's suggestion unfortunately didn't fix it for me, I still get the same error as in the original post.

[linjohn]

@ninafiona Could you please check if you have installed Ray in this order?

pip install wheel
pip install --upgrade aiohttp==3.6.2
pip install --upgrade grpcio==1.26.0
pip install --upgrade psutil==4.3.1
pip install --upgrade setproctitle==1.1.10
pip install --upgrade ray==0.8.1

[KristoferHansson]

Hi, I commented on the original issue reported in modin, reported here by devin.
I now tried to uninstall all of the above and then install them in the proposed order.

Result is basically the same, still core dumping, with the addition of a new INFO print before:

Python 3.7.5 (default, Oct 25 2019, 15:51:11)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.11.1 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import ray
In [2]: ray.init()
2020-02-05 15:41:23,043 INFO resource_spec.py:212 -- Starting Ray with 24.8 GiB memory available for workers and up to 12.41 GiB for objects. You can adjust these settings with ray.init(memory=<bytes>, object_store_memory=<bytes>).
2020-02-05 15:41:23,698 INFO services.py:1093 -- View the Ray dashboard at localhost:8265
F0205 15:41:29.093791   279 redis_context.cc:215] Could not establish connection to redis 10.46.100.175:19384 (context.err = 1)
*** Check failure stack trace: ***
    @     0x7f0e337b8ded  (unknown)

[mehrdadn]

@KristoferHansson thanks for the info! Would you be able to find out what is the latest release of Ray that works for you?

[linjohn]

@KristoferHansson This could be a firewall/network issue. Could you please review your security settings if your firewall is blocking (e.g. temporarily disconnect from the internet and turn off the desktop firewall and re-run Ray locally)?

I have attached my Ray output for your reference. Despite the protocol error Ray successfully completes the processing.

2020-02-09 01:51:55,175 INFO resource_spec.py:212 -- Starting Ray with 34.57 GiB memory available for workers and up to 17.29 GiB for objects. You can adjust these settings with ray.init(memory=<bytes>, object_store_memory=<bytes>).
2020-02-09 01:51:55,538 INFO services.py:1093 -- View the Ray dashboard at localhost:8265
E0209 01:51:55.721890700     355 socket_utils_common_posix.cc:208] check for SO_REUSEPORT: {"created":"@1581209515.721819800","description":"Protocol not available","errno":92,"file":"external/com_github_grpc_grpc/src/core/lib/iomgr/socket_utils_common_posix.cc","file_line":185,"os_error":"Protocol not available","syscall":"getsockopt(SO_REUSEPORT)"}

[victor-ab]

Just ran into the same core dumping here.
No luck installing in this order:

pip install wheel
pip install --upgrade aiohttp==3.6.2
pip install --upgrade grpcio==1.26.0
pip install --upgrade psutil==4.3.1
pip install --upgrade setproctitle==1.1.10
pip install --upgrade ray==0.8.1

[futurepaycc]

same problem , any new progress?

[mehrdadn]

@futurepaycc to clarify, are you seeing ValueError: invalid literal for int() with base 10: b'kB' on WSL, or a different issue?

[futurepaycc]

just as "
F0123 14:09:07.020850 1052 redis_context.cc:215] Could not establish connection to redis 10.46.89.57:57714 (context.err = 1)
* Check failure stack trace: *
@ 0x7f9b27d3f98d (unknown)
@ 0x7f9b27d40dfc (unknown)
"
pyenv 3.7.7 ray0.8.4

[mehrdadn]

Ah okay, unfortunately I'm not sure at the moment, maybe someone else has found a workaround?

[futurepaycc]

bad luck, previous ideas not working for me

my case is :
head on arch linux Physical machine
with 3 wsl nodes

python 3.7.7 ray 0.8.5 same error

[MichaelRazum]

Hello,
I had the same issue. It is running now on windows using wsl.
Here is what I did (as far as I remember).

• updated windows to 1909
• executed some of these commands https://docs.microsoft.com/en-us/windows/wsl/install-win10
• uninstalled Ubuntu
• isntalled Ubuntu 18.04
• installed Anaconda3-5.2.0-Linux-x86_64.sh
• made python3.8 environment with conda
• conda install anaconda
• conda install tensorflow
• pip install ray[tune]

Hope this helps you guys.

[stefanbschneider]

It's working fine for me on:

• Ray 0.8.5
• Tensorflow 2.2.0
• Python 3.8.3
• OS: Ubuntu 20.04 on WSL (Win 10 version 1909)

[BrandonEscamilla]

The solution is to upgrade to that configuration? @stefanbschneider

[BrandonEscamilla]

@futurepaycc did you solved this problem?

[stefanbschneider]

The solution is to upgrade to that configuration? @stefanbschneider

I'm not sure - I never had the problem but also didn't try any other configuration. I just thought it might help in solving the issue.

[BrandonEscamilla]

What's the command you use to start a head node?

[stefanbschneider]

agent = ppo.PPOTrainer(config=config, env=MyEnv)
agent.train()

But it also works with tune.

[richardliaw]

Sounds like this is working now? Feel free to reopen if not.

[jlarsen-usgs]

This same issue is happening for me on windows subsystem for linux (ubuntu 18.04) with ray version 1.0.0. If I downgrade to ray 0.8.6, ray works fine.

[futurepaycc]

update windows probable the key