Chapel: ZMQ string/record serialization incompatible with REQ/REP pattern

[nspark]

Summary of Problem

As identified by this Stack Overflow question and discussion, the ZMQ module does not correctly serialize Chapel strings (or other records) in a way that is compatible with the REQ/REP socket pattern or, likely, compatibility with other language bindings (e.g., Python).

Steps to Reproduce

Source Code:

Server code (server.chpl):

use ZMQ;
var context: Context;
var socket = context.socket(ZMQ.REP);
socket.bind("tcp://*:5555");

while ( 1 < 2) {
  var msg = socket.recv(string);
  writeln("got something");
  socket.send("back from chapel");
}

Client code (client.py):

import zmq

context = zmq.Context()
socket = context.socket(zmq.REQ)
socket.connect("tcp://localhost:5555")

for request in range(10):
    print("Sending request %s ..." % request)
    socket.send(str("Yo"))
    message = socket.recv()
    print("OMG!! He said %s" % message)

Compile command:
chpl -o server server.chpl

Execution command:

# in parallel terminals
./server
python client.py

Associated Future Test(s):

Add tests for:

• REQ/REP pattern
• Chapel-Python interoperability

Please assign to @nspark

[buddha314]

If it makes any kind of difference, the first use case I had in mind was passing JSON back and forth.

[nspark]

@buddha314 I have a rough fix for sending and receiving strings such that Python and Chapel can talk back and forth, as in your example. It's "rough" in the sense that the fix -- that is, to use zmq_msg_t objects in a Python-compatible way -- only works for the string type (and not other Chapel records or primitive) and is only tested with the Python example in this issue.

As a minor issue, it also leaks memory internal to ZeroMQ because I still need to figure out how to pass a C function pointer into an extern C call within a Chapel program. I'll poke @bradcray about this one.

[ben-albrecht]

I still need to figure out how to pass a C function pointer into an extern C call within a Chapel program.

@nspark - just in case you aren't aware, here are some helpful resources on this:

• c_fn_ptr() documentation
• fnPtrs tests
  
  
  
  • e.g. storeFnsThenPass.chpl
  
  

[buddha314]

I see activity on the other issues and I understand @nspark is going to add tests. When appropriate, could someone point to the tests from this ticket and/or respond to this SO Question so I can see a working example. I understand I'll have to run it from a branch at the moment.

[nspark]

@buddha314 Here's the test from PR #7049:
https://github.com/nspark/chapel/tree/zmq/test/modules/packages/ZMQ/interop-py

To run it manually, all you need is client.py and server.chpl, and you should remove everything above use ZMQ in server.chpl. The Spawn stuff, wrapper.sh, etc. are all there to enable the ZMQ tests to run with the Chapel start_test tool.

[buddha314]

I'm happy to try this out. I don't see your branch, though.

[buddha314]

@nspark any idea how this will inter-operate with C++? I'm not scope creeping, just wondering...

[nspark]

@buddha314 Not sure. Which ZeroMQ C++ bindings are you using? I'm happy to try something out.

[buddha314]

I don't have a preference, but right now I'm using . It's new to me so I'm open to suggestions.

From the ZMQ site, this is the "Hello, World!"

client.cpp

#include <zmq.hpp>
#include <string>
#include <iostream>

int request_update();

int main ()
{

    request_update();
    return 0;
}

int request_update()
{
     //  Prepare our context and socket
    zmq::context_t context (1);
    zmq::socket_t socket (context, ZMQ_REQ);

    std::cout << "Connecting to hello world server…" << std::endl;
    socket.connect ("tcp://localhost:5555");

    //  Do 10 requests, waiting each time for a response
    for (int request_nbr = 0; request_nbr != 10; request_nbr++) {
        std::string msg = "{\"type\":\"update_v\"}";
        zmq::message_t request (msg.length()+1);
        //memcpy (request.data (), "Hello", 5);
        memcpy (request.data (), (msg.c_str()), (msg.length()));
        std::cout << "Sending Hello " << request_nbr << "…" << std::endl;
        socket.send (request);

        //  Get the reply.
        zmq::message_t reply;
        socket.recv (&reply);
        std::cout << "Received World " << request_nbr << std::endl;
    }
    return 0;
}

server.cpp

#include <zmq.hpp>
#include <string>
#include <iostream>
#ifndef _WIN32
#include <unistd.h>
#else
#include <windows.h>

#define sleep(n)    Sleep(n)
#endif

int main () {
    //  Prepare our context and socket
    zmq::context_t context (1);
    zmq::socket_t socket (context, ZMQ_REP);
    socket.bind ("tcp://*:5555");

    while (true) {
        zmq::message_t request;

        //  Wait for next request from client
        socket.recv (&request);
        std::string msg_str(static_cast<char*>(request.data()), request.size());
        std::cout << "Received Hello: " << msg_str << std::endl;

        //  Do some 'work'
        sleep(1);

        //  Send reply back to client
        zmq::message_t reply (5);
        memcpy (reply.data (), "World", 5);
        socket.send (reply);
    }
    return 0;
}

Can we replace server.cpp with a Chapel version? AGAIN, not scope creeping, don't do it, but I was curious.

[nspark]

@buddha314 Yes, Chapel will work with C++ and zmq.hpp as in that example. (I actually replaced the client with the C++ source, but it works.) That C++ code uses the zmq_msg_t (or zmq::message_t in C++) based API and serializes strings the same way Python and, with this PR, Chapel do.

[buddha314]

Thanks, I'll give it a shot!

[buddha314]

Well, I'll be damned! On your branch this C++ / Chapel combination works

server.chpl

use ZMQ;

var context: Context;
var socket = context.socket(ZMQ.REP);
socket.bind("tcp://*:5555");

for i in 0..#10 {
  var msg = socket.recv(string);
  writeln("[Chapel] Received message: ", msg);
  socket.send("Hello %i from Chapel".format(i));
}

client.cpp

#include <zmq.hpp>
#include <string>
#include <iostream>

int request_update();

int main ()
{

    request_update();
    return 0;
}

int request_update()
{
     //  Prepare our context and socket
    zmq::context_t context (1);
    zmq::socket_t socket (context, ZMQ_REQ);

    std::cout << "Connecting to hello world server…" << std::endl;
    socket.connect ("tcp://localhost:5555");

    //  Do 10 requests, waiting each time for a response
    for (int request_nbr = 0; request_nbr != 10; request_nbr++) {
        std::string msg = "{\"type\":\"update_v\"}";
        zmq::message_t request (msg.length()+1);
        //memcpy (request.data (), "Hello", 5);
        memcpy (request.data (), (msg.c_str()), (msg.length()));
        std::cout << "Sending Hello " << request_nbr << "…" << std::endl;
        socket.send (request);

        //  Get the reply.
        zmq::message_t reply;
        socket.recv (&reply);
        std::cout << "Received World " << request_nbr << std::endl;
    }
    return 0;
}

Server output:

[Chapel] Received message: {"type":"update_v"}
[Chapel] Received message: {"type":"update_v"}
[Chapel] Received message: {"type":"update_v"}
[Chapel] Received message: {"type":"update_v"}
[Chapel] Received message: {"type":"update_v"}
[Chapel] Received message: {"type":"update_v"}
[Chapel] Received message: {"type":"update_v"}
[Chapel] Received message: {"type":"update_v"}
[Chapel] Received message: {"type":"update_v"}
[Chapel] Received message: {"type":"update_v"}

I'm going to try the Python/Chapel combination today. This is pretty exciting!

[buddha314]

Hell yes! With the same server.chpl as above and the following Python client on your branch:

client.py

import zmq

context = zmq.Context()
socket = context.socket(zmq.REQ)
socket.connect("tcp://localhost:5555")

for request in range(10):
    message = "Hello %i from Python" % request
    print("[Python] Sending request: %s" % message)
    socket.send_string(message)
    message = socket.recv_string()
    print("[Python] Received response: %s" % message)

I get the following server output

[Python] Sending request: Hello 0 from Python
[Python] Received response: Hello 0 from Chapel
[Python] Sending request: Hello 1 from Python
[Python] Received response: Hello 1 from Chapel
[Python] Sending request: Hello 2 from Python
[Python] Received response: Hello 2 from Chapel
[Python] Sending request: Hello 3 from Python
[Python] Received response: Hello 3 from Chapel
[Python] Sending request: Hello 4 from Python
[Python] Received response: Hello 4 from Chapel
[Python] Sending request: Hello 5 from Python
[Python] Received response: Hello 5 from Chapel
[Python] Sending request: Hello 6 from Python
[Python] Received response: Hello 6 from Chapel
[Python] Sending request: Hello 7 from Python
[Python] Received response: Hello 7 from Chapel
[Python] Sending request: Hello 8 from Python
[Python] Received response: Hello 8 from Chapel
[Python] Sending request: Hello 9 from Python
[Python] Received response: Hello 9 from Chapel

[buddha314]

Next question (probably for Stack Overflow) but I got scolded by an engineer for wanting to send text over 0MQ rather than bytestreams. I have no idea how to do this in a sparse array/vector context, but when I figure it out, do you expect the [Python, C++]/Chapel communication to work correctly?

[ben-albrecht]

@nspark -- should this be closed now?

[nspark]

Closed by #7049