Pysyft: How the clientworker and serverworker load data and train model in real Federate learning scenario?

[StuLiu]

Question

How the clientworker and serverworker load data and train model in real federate learning scenario?

Further Information

Thank you for reading my question!

1. As we all know, the data holders(serverworkers) won't open their data to the model holder(clientworker) in federate learning architecture. However, the tutorials in this project teach us that clientworker creates data and sends to serverworkers. This is impossible in a real scenario. Is there any tutorials explaining the federate learning in a real scenario where the clientworks create a model (without creating data and sending data to serverworkers) and send it to serverworkers whose datasets are used to train the recieved model.

1. Can we set tensors with the following two characteristics：
2. one can search it by worker.search();
3. after searching it and get the pointer, one cannot to get the value of tensors by pointer.get().

If you know, please respond with me in this websit or send mail to [email protected]. Thank you very much!

[jmaunon]

Hi @StuLiu , i will try to help you.

1. I am not sure about if there is a tutorial for a more realistic scenario. In any case, I would recommend you to take a look to PyGrid. Here you a find a blog: https://blog.openmined.org/what-is-pygrid-demo/

2. Yes, it is possible!. Actually that is the functionality of PrivateTensors. Take a look (again) to the blog I recommend you

[StuLiu]

@jmaunon Thank you sincerely for your help! It will help me a lot.

[StuLiu]

Hi @StuLiu , i will try to help you.

1. I am not sure about if there is a tutorial for a more realistic scenario. In any case, I would recommend you to take a look to PyGrid. Here you a find a blog: https://blog.openmined.org/what-is-pygrid-demo/
2. Yes, it is possible!. Actually that is the functionality of PrivateTensors. Take a look (again) to the blog I recommend you

Thank you sincerely for your help! It will help me a lot.

[NeuZhangQiang]

@jmaunon The PyGrid may be a solution. However, I can not run the demo of pygrid, and #3974 describe the bug I have met.

@StuLiu Have you implement the clientworker and server workder data loading and model training in real federate learning scenario? Could you please provide me a demo?

Hi @StuLiu , i will try to help you.

1. I am not sure about if there is a tutorial for a more realistic scenario. In any case, I would recommend you to take a look to PyGrid. Here you a find a blog: https://blog.openmined.org/what-is-pygrid-demo/
2. Yes, it is possible!. Actually that is the functionality of PrivateTensors. Take a look (again) to the blog I recommend you

Thank you sincerely for your help! It will help me a lot.

[StuLiu]

@jmaunon The PyGrid may be a solution. However, I can not run the demo of pygrid, and #3974 describe the bug I have met.

@StuLiu Have you implement the clientworker and server workder data loading and model training in real federate learning scenario? Could you please provide me a demo?

Hi @StuLiu , i will try to help you.

1. I am not sure about if there is a tutorial for a more realistic scenario. In any case, I would recommend you to take a look to PyGrid. Here you a find a blog: https://blog.openmined.org/what-is-pygrid-demo/
2. Yes, it is possible!. Actually that is the functionality of PrivateTensors. Take a look (again) to the blog I recommend you

Thank you sincerely for your help! It will help me a lot.

您好！ @NeuZhangQiang
我也看了一下github上pygrid的项目，暂时没有一个完整的demo。正如你的问题[#3974]所说，需要先开启节点，具体怎么弄的我也不清楚。我用pysyft的serverworker和clientworker实现了数据指针获取与模型传输，能实现远程训练，但是没有保证数据隐私，感兴趣可以看一下这篇博客：https://blog.csdn.net/qq_26623993/article/details/106867566

[NeuZhangQiang]

@jmaunon The PyGrid may be a solution. However, I can not run the demo of pygrid, and #3974 describe the bug I have met.
@StuLiu Have you implement the clientworker and server workder data loading and model training in real federate learning scenario? Could you please provide me a demo?

Hi @StuLiu , i will try to help you.

1. I am not sure about if there is a tutorial for a more realistic scenario. In any case, I would recommend you to take a look to PyGrid. Here you a find a blog: https://blog.openmined.org/what-is-pygrid-demo/
2. Yes, it is possible!. Actually that is the functionality of PrivateTensors. Take a look (again) to the blog I recommend you

Thank you sincerely for your help! It will help me a lot.

您好！ @NeuZhangQiang
我也看了一下github上pygrid的项目，暂时没有一个完整的demo。正如你的问题[#3974]所说，需要先开启节点，具体怎么弄的我也不清楚。我用pysyft的serverworker和clientworker实现了数据指针获取与模型传输，能实现远程训练，但是没有保证数据隐私，感兴趣可以看一下这篇博客：https://blog.csdn.net/qq_26623993/article/details/106867566

@StuLiu Thank you very much! It help me a lot.

[NeuZhangQiang]

@jmaunon The PyGrid may be a solution. However, I can not run the demo of pygrid, and #3974 describe the bug I have met.
@StuLiu Have you implement the clientworker and server workder data loading and model training in real federate learning scenario? Could you please provide me a demo?

Hi @StuLiu , i will try to help you.

1. I am not sure about if there is a tutorial for a more realistic scenario. In any case, I would recommend you to take a look to PyGrid. Here you a find a blog: https://blog.openmined.org/what-is-pygrid-demo/
2. Yes, it is possible!. Actually that is the functionality of PrivateTensors. Take a look (again) to the blog I recommend you

Thank you sincerely for your help! It will help me a lot.

您好！ @NeuZhangQiang
我也看了一下github上pygrid的项目，暂时没有一个完整的demo。正如你的问题[#3974]所说，需要先开启节点，具体怎么弄的我也不清楚。我用pysyft的serverworker和clientworker实现了数据指针获取与模型传输，能实现远程训练，但是没有保证数据隐私，感兴趣可以看一下这篇博客：https://blog.csdn.net/qq_26623993/article/details/106867566

@StuLiu 另外，这个帖子中，是先收集所有的数据，然后顺序执行。在真实的场景中，能不能让不同的客户端（比如不同的医院）同时计算模型，然后在另一台电脑把所有的模型收集起来做平均？这种并行操作，pysyft可以实现吗？
另外，这里在载入数据的时候，是先把所有整体读入内存。假如我数据特别大，比如100G，没法一次性载入内存，怎么办？
感谢！！

[github-actions[bot]]

This issue has been marked stale because it has been open 30 days with no activity. Leave a comment or remove the stale label to unmark it. Otherwise, this will be closed in 7 days.