Ray: Make Ray API thread safe

[raulchen]

The purpose of this issue to track the efforts of making Ray API fully thread safe.

Existing problems

Although currently it's not allowed for users to call Ray API from non-main threads, there're also some internal background threads in the Python worker, which causes thread-safety issues.

1) One example is that the local scheduler client is used in 3 threads (main thread, import thread, and flush-profile-data thread). And the access of its underlying connection isn't always protected (see https://github.com/ray-project/ray/pull/2306#pullrequestreview-135707250 for more detailed explanation).
2) Access of the buffer for profile data isn't protected.
3) worker.redis_client is also used in multiple threads. (Not sure if it's thread-safe, can anyone help confirm?)

Todos

Besides fixing the existing problems, we'd also like to make Ray API fully thread-safe. It'll allow users to run multi-thread programs on ray. Here's the list of what needs to be done:

• [x] Fix problem 1 by making the underlying c++ code thread-safe. (#2386)
• [x] Fix problem 2 by protecting the buffer with a lock
• [x] Lock other clients and stateful operations in the worker (#2248)

Java

TBD

Related PRs

2248

2349

2386

cc @alexbao

[raulchen]

@robertnishihara could you please take a look at #2349 and #2386 (or suggest someone else to take a look)?
I've done most of the work locally. Now I'm waiting for these 2 PRs to be merged so I can post the subsequent PRs.
Thank you very much!

[raulchen]

Python API should be multi-thread friendly now. If there's any other issue, please let me know.
cc @alexbao

[richardliaw]

Looks like we can close this for now and reopen if there are issues; thanks!

[alexbao]

This is HUGE! Thank you @raulchen !