Simulator: Do you consider communication with Apollo 3.5?

[frontw]

As you know, Apollo uses own Cyber RT system instead ROS.
Do you consider communication with Apollo 3.5?

[martins-mozeiko]

Yes, we are working on Apollo 3.5 support.
Currently we already have internal build with ability to communicate with Cyber RT. At this moment we are integrating this alongside ROS bridge and will be implementing all necessary changes in sensors code so Apollo 3.5 is able to drive virtual EGO vehicle in our simulator.

[krishnatoshniwal]

That's amazing! Would it be possible to share the timeline for when the bridge would be available. I'm working on the bridge between Apollo 3.5 and the simulator as well and would love to collaborate or contribute!

[martins-mozeiko]

I expect that we will push code this or next week that will include all changes in simulator to be able to talk to Cyber bridge. This will include code to send/receive messages and new Apollo 3.5 message types (all of them). Simulator will be publishing binary serialized protobuf messages directly over TCP socket. And bridge will only forward message to appropriate cyber node. Similar for receiving - bridge will get binary protobuf message and without any parsing will send it to simulator over TCP socket. There will be no changes in Cyber code required - just running an extra bridge process.

This push will probably not include necessary changes in simulator sensor code to actually publish new messages, which would be required for Apollo 3.5 to be able to control simulator vehicle. This will come later in next month or so so.

It is interesting that you are considering to work on this. Can you share what approach you were thinking/implementing? I would suggest waiting for our push with cyber bridge implementation as it is almost done. And continuing from there if you want to use simulator with Apollo Cyber.

[krishnatoshniwal]

Thanks for sharing the timeline, quite excited to try out the bridge! By TCP based bridge, do you mean modifying existing rosbridge to work with Cyber channels instead of ROS topics. I am currently familiarizing myself with the Cyber framework and was thinking to modify the previous bridge to work with Cyber channels. I can definitely continue working on it once you push the code!

This push will probably not include necessary changes in simulator sensor code to actually publish new messages, which would be required for Apollo 3.5 to be able to control simulator vehicle.

Transferring messages (images and point cloud) from the simulator to Cyber will not yet be supported, is that because they need to be converted to protobuf binary before being passed to the bridge?

[martins-mozeiko]

We created new bridge, because existing bridge is too ROS oriented and has extra overhead due to websocket/json protocol. New bridge is just raw TCP socket that sends protobuf binary blobs forward and back. No protobuf message unserialization happens on the bridge itself. The bridge will be integrated into Apollo buildsystem - it builds new cyber_bridge executable.

By ""not including" these change I mean that we need a time to implement them in existing code. It will take a few more days, and I'm not sure if this will be ready this week. Conversion will be done by Google protobuf library, we don't need to write any code there. We just need to put data into new Apollo 3.5 proto message data types, call protobuf serialization code and write the result to tcp socket.

[krishnatoshniwal]

So the bridge would directly publish protobuf data from the simulator to the channel it would be going to - from there on it cyber would convert to original format?
I am looking into the location where it is serializing and parsing in Cyber source code and if it is being done or not - I believe it has something to do with node and message Writer and Reader but I am unable to find the base script because of template specialization code, any pointers?

The second point makes sense, thanks!

[martins-mozeiko]

The conversion to C++ structure will happen on receiver said - in original cyber node.

Deserialization in Cyber happens here. ParseFrom is protobuf generated code for each message. Serialization code is in same file - SerializeToString function.

Btw, we have pushed simulator code that implements communication with cyber bridge. You can see code in CyberBridge.cs file. Serialization code is in CyberWriter.cs file. Nobody currently uses these classes in our github master, as we are currently working to add cyber writer and reader functionality to all sensors.

You can get cyber bridge we wrote in my apollo-cyber-bridge repo. We will push full Apollo 3.5 integration a bit later when we'll get all sensors working.

But currently using latest simulator and this bridge repo you should be able to talk to Apollo 3.5. All you need is to generate C# source file for proto message serialization. protoc can do this.

[krishnatoshniwal]

Thanks for sharing the source code of the bridge and for your helpful replies, can you please help me with the following questions.

1) I was able to start the server on Apollo side. But was unable to find a way to connect CyberBridge.cs client to the server. How can I run these C# files to connect to the server?

2) Is there any way I can read sensor information on Simulator's end so that I can send it to to the server?

3) Does Serialization and Parsing happen twice (Once when a message is published and read on Cyber channel and the second time when it is encoded and sent to server and decoded at other end)? Is that why you were asking to generate C# proto source files (Simulator side proto files)?

4) The following pattern of code is a bit confusing, is the length variable is being added in 4 different bytes? I know it has to do something with how bytes are arranged (little endian), but I couldn't find more on this online.

data.Add((byte)Op.AddWriter);
data.Add((byte)(channel.Length >> 0));
data.Add((byte)(channel.Length >> 8));
data.Add((byte)(channel.Length >> 16));
data.Add((byte)(channel.Length >> 24));

And I guess the following code stitches the data obtained. Is that correct?

buffer[offset + 0] | (buffer[offset + 1] << 8) | (buffer[offset + 2] << 16) | (buffer[offset + 3] << 24);

Thanks again!

[martins-mozeiko]

In our latest code push you can find answers to your questions.

1. Both bridge instances are created in RosBridgeConnector constructor. Each vehicle creates BridgeConnector that manages bridge connection (to reconnect if it drops). And each vehicle knows its type (Apollo, Apollo 3.5, Autoware, ...) so it can pass type for creating proper bridge.

2. All the sensors are set up for Cyber communications. For example, Lidar is creating Apollo 3.5 message and publishing in following location in LidarSensor.cs file.

3. Serialization and Parsing happens only once. Simulator serializes data to protobuf binary buffer. Sends it to cyber_bridge over TCP socket. Bridge only reads topic name and whole message as binary blob (no parsing happens). Then it publishes it to Cyber network. Only cyber node that receives it will do the parsing.

4. That codes writes length of topic (channel) to buffer, before putting topic itself in same buffer. This is needed because on receiver side where code reads topic bytes (as string) it needs to know how long is the topic string.

[krishnatoshniwal]

Thanks Martins! I was able to connect to cyber bridge on Apollo's side and it does receive messages published from the simulator.

The next thing I did was try viewing vehicle movement in Dreamview of Apollo's Master branch. As the map, vehicle and calibration files were needed, I copied them from lgsvl/apollo-simulator-map and lgsvl/apollo-vehicle. These were the locations that I had identified.

/apollo/modules/map/data/san_fransisco
/apollo/modules/calibration/data

However there were many more commits in lgsvl/apollo and changes to other modules calibrations and source code, so I believe there's more to it. What other files are needed to be added to visualize the vehicle movement in Apollo?

Update: I am able to see the location of the car on Baidu's map in Dreamview. Still working around to see if all the modules could work. Currently, I am able to toggle the Camera, Planning and Localization modules.

[luqiang21]

@krishnatoshniwal For apollo 3.5 to work with the simulator, you need to

• enable transform module by mainboard -d modules/transform/dag/static_transform.dag to publish tf_static
• enable localization module by using rtk localization: mainboard -d modules/localization/dag/dag_streaming_rtk_localization.dag to publish tf, or you can just change the configuration file to use rtk instead of msf. Then you can launch localization using the button in dreamview.
• decompress images from simulator by mainboard -d modules/drivers/tools/image_decompress/dag/image_decompress.dag

And also remember in Apollo 3.5, to enable planning, you need to make sure perception and predication are also enabled.

[krishnatoshniwal]

Thanks @luqiang21! While starting the Static Transform Component, I got an error saying that the yaml files for 128 and 16 channel lidars were absent. I started looking into other vehicles or in the vehicle params as provided by Apollo but I couldn't find any. Now I understand that these config files are supposed to belong to the vehicle sensors on the simulator vehicle, is there any way I can get access to them?

Here's the error
Extrinsic yaml file is noe exists:
/apollo/modules/drivers/velodyne/params/velodyne128_novatel_extrinsics.yaml
/apollo/modules/drivers/velodyne/params/lidar16_front_up_novatel_extrinsics.yaml
/apollo/modules/drivers/velodyne/params/lidar16_front_novatel_extrinsics.yaml
/apollo/modules/drivers/velodyne/params/lidar16_rear_left_novatel_extrinsics.yaml
/apollo/modules/drivers/velodyne/params/lidar16_rear_right_novatel_extrinsics.yaml
/apollo/modules/perception/data/params/radar_rear_extrinsics.yaml

[luqiang21]

Hi @krishnatoshniwal, you only need /apollo/modules/drivers/velodyne/params/velodyne128_novatel_extrinsics.yaml, which can be created by just changing corresponding 64 to 128 using the current vehicle calibration files. You can neglect other yaml files errors.

[martins-mozeiko]

Apollo 3.5 support is fully pushed to github master.
Instructions how to run Apollo 3.5 are available in our Apollo 3.5 fork: https://github.com/lgsvl/apollo-3.5

I'm closing this issue. For any new comments or questions please open new issue.