Pysyft: websocket workers communication

[mari-linhares]

Encrypted matmul and move (and probably other operations involving a crypto provider) are not supported when using websocketworkers.

The following test should pass:

def test_websocket_workers_mult(hook, start_remote_worker):
    server, alice = start_remote_worker(id="alice-remote", hook=hook, port=8761)
    server, bob = start_remote_worker(id="bob-remote", hook=hook, port=8762)
    server, charlie = start_remote_worker(id="charlie-remote", hook=hook, port=8763)

    x = torch.tensor([[3, 4]])
    y = torch.tensor([[3, 4]])
    x_sh = x.share(alice, bob, crypto_provider=charlie)
    y_sh = x.share(alice, bob, crypto_provider=charlie)

    result = x_sh * y_sh
    assert torch.equal(result.get(), x * y) 

    alice.close()
    time.sleep(0.1)
    alice.remove_worker_from_local_worker_registry()

    bob.close()
    time.sleep(0.1)
    bob.remove_worker_from_local_worker_registry()

    charlie.close()
    time.sleep(0.1)
    charlie.remove_worker_from_local_worker_registry()
    server.terminate()

Current error:

/home/marianne/PySyft/syft/workers/websocket_server.py:91> exception=AttributeError("'str' object has no attribute '_recv_msg'",)>
Traceback (most recent call last):
  File "/home/marianne/PySyft/syft/workers/websocket_server.py", line 109, in _producer_handler
    response = self._recv_msg(message)
  File "/home/marianne/PySyft/syft/workers/websocket_server.py", line 120, in _recv_msg
    return self.recv_msg(message)
  File "/home/marianne/PySyft/syft/workers/base.py", line 270, in recv_msg
    response = self._message_router[msg_type](contents)
  File "/home/marianne/PySyft/syft/workers/base.py", line 388, in execute_command
    response = getattr(_self, command_name)(*args, **kwargs)
  File "/home/marianne/PySyft/syft/frameworks/torch/tensors/interpreters/native.py", line 533, in mid_get
    self.child.mid_get()
  File "/home/marianne/PySyft/syft/generic/object.py", line 107, in mid_get
    tensor = self.get()
  File "/home/marianne/PySyft/syft/frameworks/torch/tensors/interpreters/additive_shared.py", line 114, in get
    shares.append(share.get())
  File "/home/marianne/PySyft/syft/generic/pointers/pointer_tensor.py", line 273, in get
    tensor = ObjectPointer.get(self, deregister_ptr=deregister_ptr)
  File "/home/marianne/PySyft/syft/generic/pointers/object_pointer.py", line 145, in get
    obj = self.owner.request_obj(self.id_at_location, self.location)
  File "/home/marianne/PySyft/syft/workers/base.py", line 545, in request_obj
    obj = self.send_msg(messaging.ObjectRequestMessage(obj_id), location)
  File "/home/marianne/PySyft/syft/workers/base.py", line 236, in send_msg
    bin_response = self._send_msg(bin_message, location)
  File "/home/marianne/PySyft/syft/workers/virtual.py", line 10, in _send_msg
    return location._recv_msg(message)
AttributeError: 'str' object has no attribute '_recv_msg'

[LaRiffle]

From what I see, .location is a string because, get_worker(...) fails almost silently when retrieving workers, but emits those warnings:

WARNING:syft.workers.base:Worker alice-remote couldn't recognize worker charlie-remote
WARNING:syft.workers.base:Worker me couldn't recognize worker bob-remote

From my logs I have: alice-remote KNOWS {'me': <VirtualWorker id:me #objects:0>, 'alice': <VirtualWorker id:alice #objects:0>, 'bob': <VirtualWorker id:bob #objects:0>, 'james': <VirtualWorker id:james #objects:0>, 'alice-remote': <WebsocketServerWorker id:alice-remote #objects:7>}

Can you .move() data across WebsocketWorkers? I think this is what you need, and it requires end workers to know each others

[mari-linhares]

From what I see, .location is a string because, get_worker(...) fails almost silently when retrieving workers

Yes, that's exactly the case.

[mari-linhares]

Can you .move() data across WebsocketWorkers? I think this is what you need, and it requires end workers to know each others

I don't think so, this fails:

def test_websocket_workers_mult(hook, start_remote_worker):
    server, alice = start_remote_worker(id="alice-remote", hook=hook, port=8761)
    server, bob = start_remote_worker(id="bob-remote", hook=hook, port=8762)
    server, charlie = start_remote_worker(id="charlie-remote", hook=hook, port=8763)

    x = torch.tensor([[3, 4]])
    y = torch.tensor([[3, 4]])
    x_sh = x.share(alice, bob, crypto_provider=charlie)
    y_sh = x.share(alice, bob, crypto_provider=charlie)

    x_ptr = x.send(alice)
    x_move = x_ptr.move(bob)

Error:

Traceback (most recent call last):
  File "/home/marianne/PySyft/syft/workers/websocket_server.py", line 109, in _producer_handler
    response = self._recv_msg(message)
  File "/home/marianne/PySyft/syft/workers/websocket_server.py", line 120, in _recv_msg
    return self.recv_msg(message)
  File "/home/marianne/PySyft/syft/workers/base.py", line 274, in recv_msg
    response = self._message_router[msg_type](contents)
  File "/home/marianne/PySyft/syft/workers/base.py", line 392, in execute_command
    response = getattr(_self, command_name)(*args, **kwargs)
  File "/home/marianne/PySyft/syft/generic/object.py", line 107, in mid_get
    tensor = self.get()
  File "/home/marianne/PySyft/syft/generic/pointers/pointer_tensor.py", line 273, in get
    tensor = ObjectPointer.get(self, deregister_ptr=deregister_ptr)
  File "/home/marianne/PySyft/syft/generic/pointers/object_pointer.py", line 145, in get
    obj = self.owner.request_obj(self.id_at_location, self.location)
  File "/home/marianne/PySyft/syft/workers/base.py", line 549, in request_obj
    obj = self.send_msg(messaging.ObjectRequestMessage(obj_id), location)
  File "/home/marianne/PySyft/syft/workers/base.py", line 240, in send_msg
    bin_response = self._send_msg(bin_message, location)
  File "/home/marianne/PySyft/syft/workers/virtual.py", line 10, in _send_msg
    return location._recv_msg(message)
  File "/home/marianne/PySyft/syft/workers/websocket_client.py", line 103, in _recv_msg
    response = self._forward_to_websocket_server_worker(message)
  File "/home/marianne/PySyft/syft/workers/websocket_client.py", line 96, in _forward_to_websocket_server_worker
    self.ws.send(str(binascii.hexlify(message)))
AttributeError: 'NoneType' object has no attribute 'send'

[LaRiffle]

So, this test is equivalent to:

def test_websocket_workers_mult(hook, start_remote_worker):
    server, alice = start_remote_worker(id="alice-remote", hook=hook, port=8761)
    server, bob = start_remote_worker(id="bob-remote", hook=hook, port=8762)
    server, charlie = start_remote_worker(id="charlie-remote", hook=hook, port=8763)

    x = torch.tensor([[3, 4]])    
    x_ptr = x.send(alice)
    x_move = x_ptr.move(bob)
    x_back = x_move.get()

I think we need to get .move() to work, ie to allow remote workers to reference each other

[mari-linhares]

Yes, exactly.

[plkmo]

any updates on this?

[karlhigley]

@OpenMined/syft-core-team is working on some of the underlying communication primitives, which should make the comms depicted in @LaRiffle's diagram easier. In particular, we created CommunicationAction as a way to capture and transmit sending/receiving operations to be executed elsewhere. It's still work in progress, but we expect it to underlie a lot of the communications between workers (e.g. in Protocols.)

[github-actions[bot]]

This issue has been marked stale because it has been open 30 days with no activity. Leave a comment or remove the stale label to unmark it. Otherwise, this will be closed in 7 days.

[Zongshun96]

Hi,
how is the move() handled now?

I was trying to move some intermediate output parameters from one WebSocketServer to a local VirtualWorker with move() but I got error saying "Worker alice1 couldn't recognize worker bob".

Thanks.

[sug4ndh]

Hi, Is this issue still being worked on?

Thanks