Ray: Cluster : Saving persistent data to specific node's disk

[pitoupitou]

In an AWS cluster that I set up using "ray up _my-specs_.yaml"
How is it to checkpoint files (dumping files) in the head node's disk?

Since my worker_nodes' disks are meant to be destroyed automatically from time to time (using Spot).

Is there a best-practice for persistent & reusable data? is it possible using Ray directly?

Thank you

[rkooo567]

I believe using NFS is the clearest solution, but there could be other ways. @allenyin55 do you know any other good practice?

[surfingkaka]

I believe Amazon EFS fits well here and then point ray to write there.

Here's some links that may be helpful:

[https://docs.ray.io/en/master/autoscaling.html#using-amazon-efs]

[https://docs.aws.amazon.com/efs/latest/ug/how-it-works.html]