Simulator: [cyber bridge] message size limit

[ntutangyun]

Hi team,

I've created a very big message (about 57 MB), at a frequency about 3 Hz, that's sending from Apollo to LGSVL (I've create a custom sensor to receive from this message channel)

However, when I enable the receiving sensor, the bridge keeps getting shut down and reopens again.

I wonder if there is a limit on the transfer size of the message? Or is it that there is no limit, it's just my network bus or network card cannot handle that transfer speed?

Note that I'm running LGSVL and apollo on the same machine.

I will try to dig into this but any help or advice is highly appreciated!

thanks

[rongguodong]

https://github.com/lgsvl/simulator/blob/cd0f0bfd421275b2c249971c019f11179517bd49/Assets/Scripts/Bridge/Cyber/CyberBridge.cs#L143
Current reader implementation uses a buffer with fixed size (1024), which may cause problems of your case. Can you change this line to the following and try?

var bytes = new List<byte>(1 + 4 + channelBytes.Length + 4 + typeBytes.Length);

[martins-mozeiko]

This part of C# code is not the problem. 1024 is only initial capacity of dynamic array. C# List expands as more data is added to it - just like std::vector in C++. 1024 there is like calling reserve(1024) on std::vector. Also function with line 143 is responsible only for registering reader with bridge on startup, not actually reading data - that happens in OnEndRead & BeginReceive functions.

Any size should be supported because it is just TCP socket - whatever you put it should come out same size on other end.
I don't think there is issue with simulator C# code - it just reads everything into big buffer dynamically expanding it, I can only assume there is some bug in sender side - in the cyber bridge.

[ntutangyun]

The unity shows error: SocketException: The socket has been shut down

and the cyber bridge terminal shows error: boost::asio::error::fault, meaning "Bad address" according to here

[ntutangyun]

for example, there is a repeated double field in the message definition.

when I add only 100 double values to the field, everything works fine. the total message size is about 1.8 KB.

But when I add all the data to the field, the message grows to over 3 Mb, and the error mentioned above happens..

[ntutangyun]

@rongguodong @martins-mozeiko hi, do you need anything from my side to reproduce the error ?

[ntutangyun]

I've modified the client handle_write function in cyber_bridge to show the bytes transferred:

void Client::handle_write(const boost::system::error_code& ec,
                          const std::size_t bytes_transferred) {
  AERROR << "[bridge client] bytes transferred: " << bytes_transferred;
...

and below is the result I got, every time the write error shows up.
E0903 21:34:58.617295 16271 client.cc:97] [cyber_bridge][bridge client] bytes transferred: 65536

[ntutangyun]

I think I might have found the solution to the large message size issue:

below is original code:

void Client::publish(const std::string& channel, const std::string& msg) {
  std::vector<uint8_t> data;
  data.reserve(sizeof(uint8_t) + sizeof(uint32_t) + channel.size() +
               sizeof(uint32_t) + msg.size());

  data.push_back(OP_PUBLISH);

  // ...

  boost::asio::async_write(
      socket, boost::asio::buffer(data.data(), data.size()),
      boost::bind(&Client::handle_write, shared_from_this(),
                  boost::asio::placeholders::error));
}

main reason (according to my understanding, please correct me if I'm wrong :smile:)

when the msg.size() is really big, say 100 Mb in my case :sweat_smile:, the async_write won't be able to transfer the data in one go.

But, since the function, async_write returns immediately, and as a result, Client::Publish function returns immediately afterward, there is no more reference to the std::vector<uint8_t> data as the it's in the local scope of the the Client::Publish function.

As a result, when the async_write resumes transmission, it could not find the remaining data anymore, so producing a fault (bad address) error.

my temporary fix

I created an std::vector<uint8_t> write_buffer; as a private member of the Client class, and use this to replace the previous data.

void Client::handle_write(const boost::system::error_code& ec,
                          const std::size_t bytes_transferred) {
  // whatever the result, reset the buffer for next write operation.
  write_buffer.clear();
  // ...

void Client::publish(const std::string& channel, const std::string& msg) {
  if (write_buffer.size() > 0) {
    // last operation not finished yet.
    return;
  }
  // ...

If you think the change's necessary,
please help create a pull request at ApolloAuto/Apollo, with a complete optimized fix. I'm really new to C++ ...
thanks.

[rongguodong]

It seems "boost::asio::async_write" does have problems of writing data larger than 65536 bytes. Somebody also asking this online: https://stackoverflow.com/questions/9432722/boostasioasync-write-writing-data-larger-than-65536-bytes

We will try to find a way to improve this issue soon. Thanks for bring up this to us.

[martins-mozeiko]

That's a good bug to find. It's not really a problem with async_write - it's just how it works by design, as it cannot pass arbitrary large writes to OS functions, otherwise it will block. asio::buffer keeps reference to existing container and writes pieces of it as long as OS allows, thus the data needs to persist during lifetime of async_write.
I guess we never did a test with receiving large messages on this bridge.

Your solution though will drop messages while send is in progress. I suggest doing following instead:
1) add two std::vector<uint8_t> buffers to class - writing and pending
2) in publish method - prepare data to write, and do the check - if writing buffer is empty, put data into it, and call async_write on it (writing buffer, not temporary data). In case writing buffer is non-empty, append prepared data to pending buffer.
3) in handle_write method - clear writing buffer, and if publish buffer is non-empty, move all data from pending to writing buffer and call async_write on it again.

This way all data for pending messages will be buffered up, and sent when current write finishes.

[rongguodong]

The fix is now merged into Apollo master (PR #12647)