Pysyft: Create a separate comm channel for big objects

[LaRiffle]

Feature Description

I've been working a lot with sending and receiving huge objects, usually just serialized as bytes. And in my experience, lot of time is wasted in the compression and serialization part: does it make sense to compress ou msgpack 1GB of random bytes?

I was thinking we could maybe have another comm channel where we could send huge buffers, which could be intrepreted thanks to our classic messages send through the classic channel. I know nothing about communication but if this is smthg that make sense, it could be really helpful I guess.

[github-actions[bot]]

This issue has been marked stale because it has been open 30 days with no activity. Leave a comment or remove the stale label to unmark it. Otherwise, this will be closed in 7 days.

[tudorcebere]

Hello @LaRiffle,

Is this related to 0.2.X or 0.3.X? We are rethinking a bit a part of the serde in the current 0.3.X. Do you have any example from such channel in other projects?

[LaRiffle]

it's also valid for 0.3.X!
We could use pyarrow, which is dedicated to sending big object and comes together with a kind of network protocol called arrow flight
We have done a PoC of this for crypto purpose (in branches ryffel/ariaNN of Pysyft & PyGrid), it provides great speed-ups

[tudorcebere]

I really like the idea of using Arrow in our store/serde. I'll add the 0.3.X tag and create an issue, I'd be really interested in working on this, mostly because I am curious about how Arrow works.

@madhavajay :eyes: