Libzmq: Slow multicast data rate and ZMQ_RATE

[marcovolino]

I have created a simple PUB/SUB application to send small images using epgm multicast. I am hoping to be able to send the images at more than 20fps to many subscribers but so far it is taking about 40 seconds per image.

I have read that the data rate defaults to 100 kbit/sec so have tried to use the ZMQ_RATE socket option but with no success.This is being run on two separate computers over a local network with the publisher on linux and subscriber on windows.

I have enabled PGM logging by setting the PGM_MIN_LOG_LEVEL=TRACE environment variable.
My publisher regularly puts out to following message
Trace: Recv again on not-full

and the subscriber gives the following output:

Does anyone know of any issues with ZMQ_RATE?

Many Thanks,
Marco

[zhenglu0]

I believe increase ZMQ_RATE should help. 100 kbit/sec is really to small. You may try 1-100Mb/s

[iamthebot]

Have you set ZMQ_RATE on BOTH the subscribers and publishers? How large are the images? Are you verifying that zmq_set_sockopt is returning 0? Are you setting ZMQ_RATE _before_ binding/connecting?

Also please paste the output when you set PGM_MIN_LOG_LEVEL=TRACE on the publisher side.

[iamthebot]

For what it's worth, @marcovolino I've had nothing but trouble with OpenPGM. It buckles at around ~100mb/s, segfaults sometimes for no reason, etc. Doesn't help that openpgm hasn't seen active development in ages. If you need reliable multicast, I'd go with NORM. Otherwise, the UDP support in the new RADIO/DISH sockets is _much_ more robust.

[zhenglu0]

@iamthebot I have the same issue with OpenPGM, my receiver just crashed for no reason with data reached even 10mb/s, I am not sure why is that.

Also does ZMP support NORM now? According to http://zeromq.org/topics:norm-protocol-transport. In the reference it said that The current NORM code has been tested with the Python pyzmq API binding. How is C/C++ going there? What language did you use for NORM?

[iamthebot]

@RogerFederer03 honestly, while we love ZeroMQ for a lot of uses (eg; distributed systems, coordination, etc.) it's no longer the best choice for multicast. It works very well for PUB/SUB unicast, however. OpenPGM/NORM are particularly troubling choices because:

1. There's not an active developer community behind either of these protocols. They're hosted on google code (now deprecated) or on government websites. Neither mailing list has seen activity in ages.
2. Both protocols (PGM and NORM) don't handle fast streams (>400mb/s) well at all. OpenPGM straight up segfaults and NORM adds a lot of overhead. You can configure NORM to be lighter weight but I haven't seen any development in ZeroMQ to enable these features. Nor would it make sense to waste development cycles on adding these features seeing as NORM is somewhat dead as a project.

If you absolutely need wide language support I guess you're stuck with ZeroMQ using either of these transports. I suppose you could implement your own reliability using RADIO/DISH sockets with UDP multicast, message sequence numbers, and a separate unicast socket for message retransmissions. Seems like a lot of work.

By far the best multicast messaging protocol I've seen so far is aeron. Very nice design and a relatively high level API. The performance is very close to line-rate. I may consider submitting a PR to ZeroMQ to add aeron support to RADIO/DISH sockets in the future. For now, the aeron API is simple to use and supports Java and C++. There are working 3rd-party ports for Golang and C#/.Net. If you need another language, you'll either have to build a port or write bindings.

[zhenglu0]

@iamthebot Thanks for the information. That is very informative.

1. I found PGM and NORM are a little out of date too. Even OPEN PGM are not hosted on Github, it is still in Google Code. I have not tied NORM yet, but seem it is not worth trying according to your description.
2. How did you know that both PGM and NORM fast streams (>400mb/s) can not work well with those rate. Did you test with those rate?
3. I don't think implement a reliable multicast is a good idea. It is too much work as well as reinvent the wheel.
4. Did you try aeron before. Does it work reliably?
5. Also I found PGM may still lose data at even in 5mb/s rate. Have you ever experience that?

[iamthebot]

@RogerFederer03
Regarding poor performance with >400mbps streams I actually haven't even been able to get OpenPGM not to crash well below that. It's possible that if it were to remain stable, it might work OK. I've seen it lose data before... but that's when it freezes intermittently for some reason. Otherwise it does seem "reliable". I'm positive most of the issues are probably some OpenPGM incompatibility or bug. But given the lack of active development it's not like I can just submit an issue. As for NORM, at around 400mbps, the actual bandwidth consumption sometimes goes up to 500mbps+ measured at the switch. That's nearly a 25% overhead. When used with UDP the overhead is < 5%. NORM is great for use over a potentially unreliable network (eg; wireless). Otherwise the overhead is too much.

I agree re-implementing reliable multicast is not a great solution. If you don't immediately need the data (and can afford to retransmit it later) UDP with sequence numbers works fine (the clients just request the missing sequence numbers later). Still, not a great solution.

Aeron works _perfectly_. I've tested it with a 5gbps feed (~150 byte flatbuffer encoded messages) and it has low overhead, lower latency than ZeroMQ, and I haven't seen a single message lost. I still highly recommend using sequence numbers to detect data loss and react accordingly on the client side. Or in the rare event something arrives out of order you may want to drop stale messages. Plus they're adding native RDMA support (infiniband and RDMA over converged ethernet). It's completely focused on reliable multicast of small messages (< MTU). If you need to send large messages or use some other pattern, ZeroMQ with one of the other transports is still the best choice.

[eclazi]

@RogerFederer03 @iamthebot I believe this is the active development of OpenPGM, https://github.com/steve-o/openpgm

[zhenglu0]

@jossgray Thanks. But it looks like only one developer is maintaining the OpenPGM project now.

[stale[bot]]

This issue has been automatically marked as stale because it has not had activity for 365 days. It will be closed if no further activity occurs within 56 days. Thank you for your contributions.

[jackpimbert]

As a note on PGM for anyone who visits this issue in the future, I have tested sending dummy packages on a 1 Gbps connection successfully at around 800+ Mbps.

The main settings that are required:

• Set a sufficiently large ZMQ_RATE (e.g. 1Gbps)
• On the publisher, increase the ZMQ_SNDBUF (e.g. 20MB)
• On the subscriber, increase the ZMQ_RCVBUF (e.g. 20MB)

It's important to note that the ZMQ_SNDBUF and ZMQ_RCVBUF are limited by the underlying system OS! For example, in Linux the max is often set to around 0.2MB.

You can increase the system maximum on your OS (until reboot) with:

sudo sysctl -w net.core.rmem_max=2000000
sudo sysctl -w net.core.wmem_max=2000000

Hopefully this is useful for some.

[bluca]

@jackpimbert could you please send a PR to add some information to https://github.com/zeromq/libzmq/blob/master/doc/zmq_pgm.txt ?

[dxb-super]

@all, have you tested the performance of epgm with docker containers? I got some strange result and do you have any idea about that? my issue is here: https://github.com/zeromq/libzmq/issues/3961