Pytorch-lightning: Add support for training on IPU's

[lezwon]

🚀 Feature

Graphcore IPU's are a new breed of processors for training machine learning models.
Pytorch support for training on Graphcore IPU's is available in Preview. Add support for training on IPU's.
Reference: https://docs.graphcore.ai/projects/poptorch-user-guide/en/latest/

Motivation

The IPU benchmarks seem pretty impressive, and with azure supporting them, they might soon grow in popularity. With Lightning's idea of the running the same code run on CPU, GPU, and TPU, it seems natural that it should also support IPU's.

Pitch

Add support for ipus in trainer just as gpus and tpu_cores.

Additional context

Here are the benchmarks: https://www.graphcore.ai/benchmarks
IPU's are available on Azure cloud: https://www.hpcwire.com/2019/11/15/microsoft-azure-adds-graphcores-ipu

[ananyahjha93]

@lezwon do you want to start working on this and submit a PR?

[rohitgr7]

this poptorch package is so good :tada:

[lezwon]

@ananyahjha93 I'd love to do that. I would however require access to IPU devices. I have applied for free access. If it gets through, i could take this up.

[AnthonyBarbier]

Glad to see some interest in Graphcore's IPUs :slightly_smiling_face:
Just so you know there will be a new version of PopTorch released as part of the SDK 1.3 (Around the end of the month), but Pytorch support in this release will still be experimental (Which means the public API changed quite a bit since the last release and might again in the next few months, especially the part related to distributed execution).

Support for up to 16 IPUs should work out of the box with PopTorch (It is possible to target more IPUs, but this is still being worked on).
Unfortunately we don't have any publicly available platforms you could use yet, but if you have questions feel free to reach out to me on Github or on Slack (I'm on the Lightning workspace)

[lezwon]

Hi @AnthonyBarbier, glad to know you're available on the lightning workspace. I am still awaiting a response to my request to access IPU's. Will definitely get back to you once I begin implementing them in lightning. :)