Pysyft: Implement Auto-Scaling on Google Cloud

[hericlesme]

Description

In this project, you will implement functionality necessary to automatically spin-up Google cloud machines, load a PyGrid instance, run a training job, and tear down the instance upon completion (depositing the results into another long-running instance). The primary feature will be the ability to run a “hyperparameter sweep”, as exemplified below.

parameters = {“alpha” : [0, 0.01, 0.02], “batch_size” : [32,64]}

gcloud = gr.GoogleCloud(credentials)

cluster = gcloud.LazyCluster(n_machines=10, 
                            type="n-series", 
                            priority="low",
                            eviction_policy="Delete",
                            max_price_usd=0.55,
                            reboot_max_price_usd=0.65)

cluster.sweep(model=model,
             parameters=parameters,
             optim=optim.SGD,
             results_node=grid['my node']
             )

Context

I will be mentoring this project as part of the Google Summer of Code

Also, this year we have some plans for PyGrid, along with our other projects, including a focus on production: Deployment on Cloud Providers (GCP, Amazon, Azure). You can check out the PyGrid Roadmap here.

Required Skills:

• Knowledge of PyTorch and Deep Learning
• Familiarity (or willingness to become familiar) with PySyft
• Familiarity (or willingness to become familiar) with Google Cloud’s Infrastructure

Difficulty: Beginner

While this is a sizeable project, the core functionality is relatively straightforward (provision machines, install and start PyGrid servers, and run training scripts). Furthermore, there are plenty of tutorials on how to run PyGrid servers and how to run Google Cloud machines.

If you are new to PyGrid and want to learn more about it, try out the tutorials. A comprehensive list of tutorials can be found here. These tutorials cover how to create a PyGrid node and what operations you can perform.

Useful Links

• PyTorch - https://pytorch.org/blog
• PySyft - https://github.com/OpenMined/PySyft
• PyGrid - https://github.com/OpenMined/PyGrid
• GCP Docs - https://cloud.google.com/docs
• GCloud APIs for Python - https://cloud.google.com/python/docs/reference
• Google APIs (Python Client) - https://github.com/googleapis/google-api-python-client
• Google Summer of Code - https://summerofcode.withgoogle.com
  
  
  
  • Timeline
  
  
  • Student Guide
  
  
  • OpenMined Org.
  
  

_List of all GSoC project ideas here._

[iamtrask]

Note: we want to allow for automatic node teardown after calling .sweep() as an option - where it deposits the results into a "main cluster node". In other words, if I spin up a cluster of 10 workers, this is ACTUALLY a cluster of 10 workers and 1 master, where the master can store resources produced by the 10 workers.

[carlodavid012]

I'd like to work on this project

[hericlesme]

Cool, @carlodavid012! I saw that you already know PySyft. I also think that basic PyGrid concepts can be useful. Feel free to chat with me if you need anything!

[MukulMaheshwari]

I am working on it.

[neeravjain24]

I have no experience related to Cloud but I am trying to learn PySyft .
I really want to contribute to this project.
Please can you help me .

[hericlesme]

Great! There are a lot of interested people. I think you could also talk to each other about the issue. :rocket: :nerd_face:

[neeravjain24]

Is there any particular channel for this project or should I message in gsoc slack group

[hericlesme]

Yess @neeravjain24! Don't worry if you're not familiar with GCloud. There's a lot of tutorials, docs and codelabs to learn. I also recommend taking a look at the PyGrid tutorial. The links are in the issue description.

GCloud links:
https://cloud.google.com/getting-started
https://codelabs.developers.google.com/cloud/

[hericlesme]

Is there any particular channel for this project or should I message in gsoc slack group

You can use the #gsoc channel :slightly_smiling_face:

[arghyadeep99]

I would like to work on this project and I am willing to learn PySyft. Also, I am learning about GCP already.

[Sasha1-0-0-9]

I'd like to contribute to this project

[hericlesme]

Great, guys. I'm thinking of creating a slack channel for this issue. It is even better to discuss it there too. What do you think? :slightly_smiling_face:

[sahiljit]

Hey guys,
I am pursuing B.Tech from University of Delhi and am fully interested in contributing to this project. I have also started working on this.

[github-actions[bot]]

This issue has been marked stale because it has been open 30 days with no activity. Leave a comment or remove the stale label to unmark it. Otherwise, this will be closed in 7 days.

[github-actions[bot]]

This issue has been marked stale because it has been open 30 days with no activity. Leave a comment or remove the stale label to unmark it. Otherwise, this will be closed in 7 days.

[iamtrask]

Wow! Really?