Ray: Make Ray work on Windows.

[robertnishihara]

Click here for instructions on how to use Ray on Windows (natively)

This is long term, but I'm creating this issue so we have a place to collect our thoughts. I'd like to have a better understanding of what's involved.

There has been substantial progress on this in #93 by @mehrdadn. The PR is outdated, but that should give a sense of what's involved.

One of the primary issues I believe was the heavy usage of system calls that don't have exact equivalents in Windows. Related to this, see code under src/common/shims/windows/.

Another issue had to do with thirdparty libraries (e.g., flatbuffers, Redis) that had issues on Windows. One approach is to fork these projects and make necessary modifications. There is a Windows port of Redis https://github.com/MSOpenTech/redis, but it's not clear how up-to-date it is, and we're relying on some pretty recent Redis features.

[mehrdadn]

For future reference: if my memory serves me right (it might not be), the main problem I recall having was the impedance mismatch between the networking I/O models. I was using Redis's compatibility layer to make the rest of Ray to work properly on Windows, so it wasn't just a matter of getting the Redis server itself to run. I ended up getting tons of hangs/crashes that were difficult to track down.

[pcmoritz]

Also, arrow is now officially supporting Windows. If there is a good portable C++ socket implementation that we could use to make the porting easier let us know.

[mehrdadn]

Arrow was never really a problem on Windows.

For sockets in C++, I believe the one and only consistent recommendation I've seen has been Boost.Asio. It's fairly heavyweight as you might expect. Apparently it has a non-Boost version too. Other recommendations for C++ can be found here but I don't really know anything about them.

[pcmoritz]

The standalone asio library seems the best candidate here. Boost::asio seems to be popular, a variant of it might even become part of the c++ standard in the future (see https://github.com/chriskohlhoff/networking-ts-impl) and it would also make it easier to support IPv6 in the future I think.

[flyingofftheshelves]

Any new developments with this?

[robertnishihara]

@flyingofftheshelves This is on hold for now until we find a Windows expert to help make it happen. In the meantime, some people have been successfully using Ray on the Linux subsystem for Windows (cc @devin-petersohn).

[richardliaw]

There's another open source fork of Redis for Windows that seems to be well-maintained here: https://github.com/tporadowski/redis

[robertnishihara]

Nice!

[jengelman]

For what it's worth, we use the MSOpenTech Redis port pretty heavily in production without major issues
Edit: we've switched over to the version @richardliaw linked, pretty close to the current version of Redis and big performance improvement

[felixludos]

How close are you currently to getting Ray to work on Windows?

[ErnstDinkelmann]

Hi there
I want to use Ray Tune for hyper parameter optimisation of a tensorflow graph. I work in windows.
I've been able to use a remote interpreter in Pycharm that sits on the Ubuntu (on WSL).
But now, even though Ray is able to execute, there is no support for CUDA/GPUs yet in WSL, the way I understand this.
I hate to ask this here, but are there good alternatives to Ray that do work on Windows. which make use of similar concepts to find the best hyper-parameters?

[eromoe]

Does windows support on progress ?

[iandanforth]

Any progress here? Seems like making cross-platform compatible architectural changes might be in order.

[robertnishihara]

There actually has been some progress, but there's still a bunch of work to do. @suquark has started replacing the underlying event loop in the Plasma shared memory object store (which uses the same event loop as Redis and which doesn't work on Windows) with the asio library. From there it will be easier to make the object store work with Windows. Once the shared memory details are worked out, then there are some more things to change (Unix domain sockets, some system calls, and using a Windows port of Redis).

[jengelman]

FWIW, you can run regular Redis on Windows 10 using Windows Subsystem for Linux (https://redislabs.com/blog/redis-on-windows-10/) with basically native performance, so no need to rely on the port. We recently switched from using the Windows port as a Celery backend and no complaints so far.

[robertnishihara]

@jengelman is there an easy way to programmatically start a Redis server on the Windows Linux Subsystem from a Python script running outside of the Subsystem?

[mehrdadn]

You might want to check if WSL Redis would have native performance—if it uses some kind of tmpfs such as for memory-mapped files (shared memory etc.) then on WSL that will hit the disk rather than occurring in RAM.

[jengelman]

@robertnishihara You can invoke wsl from cmd (ex: wsl redis-server start), so just need a call to subprocess.Popen

[gowthamnatarajan]

Since the windows support is being worked on, if I have to use RLLIB now, the only was is to install linux directly right? Installing linux as a VM inside Windows won't work because it cannot access my GPU?

[mehrdadn]

I have heard it is possible to directly link a GPU to a VM, but as I understand it is not for the faint-hearted... https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF

[mehrdadn]

Sorry, I should clarify I don't know if this feature is implemented in any VMs that run on Windows (it seems the host is Linux in that scenario), just that it isn't necessarily impossible in principle.

[robbertvc]

Hi, I'm also interested in making Ray work on Windows (without WSL). Is there any branch where we can follow the work being done? Or maybe help porting some dependencies?

[suquark]

@robbertvc Hi, I am replacing event loop with asio for plasma store (which Ray depends on) . The related PR is https://github.com/apache/arrow/pull/3704

The remaining tasks:

1. Implementing windows support for plasma store server/client using asio.
2. Shared memory support for windows.
3. Redis for windows.
4. Asio for Ray https://github.com/ray-project/ray/pull/3784

[robertnishihara]

@robbertvc, one possibility is to just start trying to compile Ray on Windows and see what issues you encounter. There will be a number of system calls that need to be changed.

[mehrdadn]

Just a heads up, there are also cases where platforms have similar APIs but different behavior (unless your wrapper handles it appropriately). The one I can think of off the top of my head is SO_REUSEADDR which can mean something different on each platform. There might also be others, not sure.

[suquark]

@robbertvc I agree with @robertnishihara that you could first trying to compiling Ray on windows, e.g. you can create a PR to implement a CI task for Ray on windows. And if you have experience in using valgrind, you could also help finding out the memory leaks in #3784.

[robbertvc]

Thanks for the status update and the pointers. I should mention that I'm not really a C++ dev though, but I can definitely try a windows build and see how far I can get. Should the windows build script include both CMake and bazel?

[robertnishihara]

@robbertvc just bazel. Cmake will be gone soon.

A good starting point like @suquark mentioned would be to try to set up Travis for Windows (we could also consider Appveyor, if there's a compelling reason).

[techvslife]

Thank you, I'm looking forward to seeing modin on Windows, as pandas is one of the most used statistics and data analysis programs on Windows. Until then, I would strongly recommend mentioning on the FIRST page of the docs and homepage that this package is currently limited to Linux. (It would save new users the time and headache researching and installing it, especially as the installation error message is not entirely obvious:
"Could not find a version that satisfies the requirement ray==0.6.2 (from modin)")

[Red-Eyed]

FWIW, you can run regular Redis on Windows 10 using Windows Subsystem for Linux (https://redislabs.com/blog/redis-on-windows-10/) with basically native performance, so no need to rely on the port. We recently switched from using the Windows port as a Celery backend and no complaints so far.

Why do people write above things?
WSL is kind of Virtual Machine, we ask here about Windows support, not linux.

A lot of people want to use GPU and WSL is not capable of doing this.

[sytelus]

Looking forward to this. As first step, let's just make minimal thing work so pip install works. Distributed and other things can wait. If we can just test things on single CPU and GPU on WIndows, that's a progress.

[aspenforest]

@suquark Go Ryans! You got this! We need Ray on Windows like a crack whore needs her crystal meth! cheerleading noises

[lrnv]

@mehrdadn I saw you were making some progress on this matter the past few months. Could we have an update on what' actualy done and what remains ?

More deeply, do you try to make Ray fully functional on Windows or do you first seek a port that mimick the API and allows to run code without doing much to performance ?

Thanks anyway for making this :)

[mehrdadn]

@lrnv So far most of my progress has been on "static" compatibility, like getting Ray to simply _compile_ on Windows, and using APIs that are actually cross-platform. There's been substantial progress on that, but it's still incomplete and I'm still discovering new issues as I go. Once that's done for at least the basic/core functionality, I plan to work on more "dynamic" compatibility issues that I discover for Python usage on 1 single node (like crashes/misbehaviors/other run-time differences), and hopefully multiple nodes afterward. Once I'm done with those, other parts of Ray (like the Java portions, and other components) will still remain and will probably take a longer time to port, and there's some refactoring that needs to be done to the codebase to avoid it becoming too messy in the process. ~There's also a good chance that at least one component of Ray (Redis, if not others) will need to run on WSL in the foreseeable future, since it might be unrealistic for me to try to port them in the near future.~ (Update 02/27/2020: Or maybe not, thanks to tporadowski/redis.) And tangential things, like compatibility with Visual Studio or platform-specific optimization, are low-priority for now and not likely to happen in the near future.

But all that said, I'm not attempting to fake or mimic any API; I'm trying to port Ray itself, so any functionality that works on Windows will be on the real thing. There's still a lot more work to be done (and I don't even _know_ everything that needs to be done), so I wouldn't hold my breath for sudden 100% codebase compatibility with Windows, but at the moment I'm fairly optimistic about some core parts of the codebase. :)

[dhirschfeld]

AFAIK you can run redis on Windows. Might not be as functional/performant but I'm pretty sure you can do it:
https://redislabs.com/ebook/appendix-a/a-3-installing-on-windows/a-3-1-drawbacks-of-redis-on-windows/

[mehrdadn]

MSOpenTech Redis is something that I did try to port to a long time ago, but I believe it doesn't have all the Redis features we use now. I've heard there might be another port that exists but I'm not sure. It's something I'll have to look into more deeply when I get to it.

[dhirschfeld]

Looks like the MSOpenTech repo is archived but I came across this memurai which might do the job.

[mehrdadn]

Thanks! I'll try to look into it when I get to Redis.

[AbhimanyuAryan]

Why do people write above things?
WSL is kind of Virtual Machine, we ask here about _Windows_ support, not

WSL2 is actual Linux kernel unlike WSL

[alfoa]

we are starting using ray for parallel computing in our code RAVEN (https://github.com/idaholab/raven). For now we are still using Parallel Python for Windows platforms and RAY for all Linux and MacOS. We would love to move to RAY completely. @mehrdadn if you need help or feedback on this issue, let us know. Thank you all for all this good work.

[mehrdadn]

Glad to hear, thanks! Hopefully I should be good for now but I'll let you know if I might need help. Looking forward to getting this working :)

[tjengbudi]

looks like there are some progress... what's next? can ray will be available in windows soon?

[mehrdadn]

That's the hope! There are some issues that we're still fixing, and there's likely going to be more work required before we get feature parity on Windows, but hopefully at least basic things will be working soon.

[javierabosch]

I am not sure if this is helpful, but CUDA and GPU support will be available on WSL for those who are a part of Insider's program and NVIDIA developer program for May 2020 Update. It should be released to the wider public soon after, likely the end of May or a little later.
references:
https://developer.nvidia.com/cuda/wsl
https://ubuntu.com/blog/new-gpu-and-gui-features-announced-for-wsl-at-build

[Red-Eyed]

Cool, now we (developers) who develop primary for linux, will not be worried about windows support, as now everything that works on linux automatically should work on windows (we can easily address such bugs aka: "add windows support" to use wsl2), don't we?

[dhirschfeld]

I'm a Windows user and I would love for Ray to work on Windows but I'm not pushing it because I know it's a huge burden on developers.

Nevertheless, telling Windows users to just use WSL2 is little better than telling them to just use Linux. This issue is specifically about supporting Windows so comments about WSL2 are irrelevant here - WSL2 is a Linux VM. If the decision is that it's too hard to support Windows then so be it - tell Windows users to switch to Linux or to spin up a Linux VM.

Commentary on this issue should be limited to what it takes to support Windows and if that effort is worthwhile.

As a Linux VM WSL2 requires Hyper-V and is unable to be run under any other hypervisor if you're already running on a VM and so is not an option for a lot of enterprise Window users.

[Red-Eyed]

So, it seems it works now on windows (experimental)
https://docs.ray.io/en/master/installation.html#latest-snapshots-nightlies

[mehrdadn]

Almost... we'll have an announcement very soon, stay tuned.

[juliusfrost]

Thank you for doing this! It really helps even when just developing on windows and running large scale experiments on a Linux cluster.

[mehrdadn]

Hi everyone! Exciting news:

Experimental Windows support is now officially released. :)

Just a heads up: it is alpha quality. It is currently incomplete and some things are known to be unstable/broken. Best results will come from running it in a typical, "friendly" environment. 🙂 Please read the release notes for important details.

If you encounter any issues, please check the latest nightly wheels since the release. If the issue still exists, check the known Windows issues page (#9114), as well as other issues (as some issues may not be Windows-specific). If you encounter any problems that are not yet being addressed, please let us know!

We'll have more information coming soon, including in the documentation. But most importantly, to run Ray on Windows, ensure you have the VC++ 2019 runtime, otherwise you may get errors about dependencies not being found.

I'll close this issue, so you can track the Known Issues page (#9114) for further updates on Windows development.

P.S. Shout-out to https://github.com/tporadowski/redis for making my life _much_ easier with Redis on Windows. Without it, we'd have probably needed to introduce a WSL dependency just for Redis. :)