Ray: Why is Redis mandatory for using Ray?

[thedrow]

What is your question?

It is not clear from the documentation why is Redis needed or that it is needed at all.

The installation guide only mentions that you need to specify a Redis port or one will be picked at random.
That implies that the Redis executable must be present alongside Ray but it isn't directly specified.

Can Redis be replaced with other backends or is there something unique that we use Redis for?

[thedrow]

@ray-project Can someone please address this question?

[sumanthratna]

It is not clear from the documentation why is Redis needed or that it is needed at all.

from the ray 1.0 whitepaper:

The GCS is currently implemented in Redis, and we rely on Redis for pub-sub. However, an effort is underway to remove the redis dependency and enable pluggable persistent storage (e.g., MySQL).

This holds the node locations of large shared memory objects. Raylets use redis pub-sub to get notified when an object they are interested in becomes available, and to pick a node to download object data from. When an object is created or deleted locally in shared memory, raylets are responsible for updating this table.

---

Can Redis be replaced with other backends or is there something unique that we use Redis for?

not yet, but as mentioned above, redis may eventually be removed as a dependency.

[rkooo567]

Just to add more explanation, Redis is the data plane of the Ray cluster which stores cluster-level metadata. For example, it stores the state or Ray nodes or actors (like where actors are).

Redis was hard dependency to Ray before, but as mentioned in the paper, we are moving towards removing it from a hard dependency. We built a new component gcs_server which can have pluggable backend (Redis by default). There are still more work to be done in order to remove Redis from hard dependency after 1.0.