Airsim: Realistic Lidar modal

[tharindurmt]

I am currently looking at creating realistic Lidar measurements. It is my understanding that the current lidar model used in Airsim is based on UE ray casting and produce a perfect set of measurements (i.e. is't not motion-distorted - distortion that occur when the Lidar is on a moving vehicle, no noise etc). I was wondering what would be the best approach to do this.
My current idea is to generate a portion of the point cloud on a single tick of the physics engine and combine them after one Lidar sensor rotation. I was wondering whether there are any better approaches and whether anybody has worked on this.
Thank you.

[WouterJansen]

I've implemented a new LiDAR system. I pre-calculate the exact angles of the lasers based on a horizontal/vertical resolution scheme and based on the frame time decide which points should be calculated each frame. And I keep them in a array until a full scan is complete after which I send them through the API. That way you have a more fixed and stable LiDAR simulation.

The current LiDAR implementation is in fact not perfect and wrong due to the scan being arbitrary horizontally divided based on 'points per second'. And as such the horizontal angle distribution is incorrect and not the same each scan. And the amount of points in a pointcloud of the API is also not a fixed size which is also not handy.

The current AirSim implementation is also semi-motion distorted. Given it still generates partial scans each tick that it updates.

[tharindurmt]

@WouterJansen Thank you.
This is similar to what I am implementing at the moment. Let me take an example just to get this straight (and that I'm relatively new to AirSim).
Assume the following: N channel lidar with f Hz frequency, H degrees horizontal FoV and the simulation is running at S fps. Then for a given frame of the simulation, a single horizontal sweep will be (f/S)*H degrees. If we know the horizontal resolution r degrees then we have to shoot (f*H)/(S*r) number of lasers for each sweep (including all N channels) and return the point cloud after a completing a full rotation. Please correct me if I am wrong. I think this should work if a single full rotation of the lidar take more than 1 frame. Where I am struggling is how to invoke this function on per frame basis.
May I know whether your code is publicly available?

[WouterJansen]

As far as I know you cannot use AirSim in a fixed framerate that easily. Many things influence it and there is no guarantee it will ever be fixed. So I don't suggest to ever use a framerate as a variable in your argument but always look at the timedelta of the last frame to calculate the horizontal sweep of the next frame.

I would suggest to pre-calculate the horizontal angles and just calculate the amount of horizontal 'steps' to calculate that frame based on the frametime and the other parameters you mention. I cannot share my code directly from both a practical and office-rules related point of view but I can share some parts to give you an idea. (this is from my mess of a code, so I apologize if it looks bad, cause it is)

// Pre-calculate horizontal angles
float horizontal_delta = (params.horizontal_FOV_end - params.horizontal_FOV_start) / float(params.horizontal_resolution- 1);
for (uint32 i = 0; i < params.horizontal_resolution; i++) {
    horizontal_angles_.Add(params.horizontal_FOV_start + i * horizontal_delta);
}
current_horizontal_angle_index_ = horizontal_angles_.Num()-1;

getPointCloud(...., const msr::airlib::TTimeDelta delta_time, ...){
// calculate needed angle/distance between each point
const float angle_distance_of_tick = params.horizontal_rotation_frequency * 360.0f * delta_time;
const double angle_distance_of_laser_measure = 360.0f / params.horizontal_resolution;

// calculate number of points needed for each laser/channel
uint32 points_to_scan_with_one_laser_temp = FMath::RoundHalfFromZero(angle_distance_of_tick / angle_distance_of_laser_measure);
}

// To avoid absurd situations with a large number of lasers to shoot. 
constexpr float MAX_POINTS_IN_SCAN = 5000;
if (params.limit_points && points_to_scan_with_one_laser_temp * number_of_lasers > MAX_POINTS_IN_SCAN)
{
    UAirBlueprintLib::LogMessageString("Lidar Error: ", "Capping number of points to scan " + std::to_string(points_to_scan_with_one_laser_temp * number_of_lasers), LogDebugLevel::Failure);
    points_to_scan_with_one_laser_temp = MAX_POINTS_IN_SCAN / number_of_lasers;
}
const uint32 points_to_scan_with_one_laser = points_to_scan_with_one_laser_temp;

// normalize FOV start/end
float laser_start = std::fmod(360.0f + params.horizontal_FOV_start, 360.0f);
float laser_end = std::fmod(360.0f + params.horizontal_FOV_end, 360.0f);

float previous_horizontal_angle = horizontal_angles_[current_horizontal_angle_index_];

// shoot lasers
for (uint32 i = 1; i <= points_to_scan_with_one_laser; ++i)
{
    if (current_horizontal_angle_index_ == horizontal_angles_.Num() - 1) {
        current_horizontal_angle_index_ = 0;
    }
    else {
        current_horizontal_angle_index_ += 1;
    }       

    float horizontal_angle = horizontal_angles_[current_horizontal_angle_index_];

    // check if horizontal angle is a duplicate (could happen with framerate related issues i've noticed)
    if ((horizontal_angle - previous_horizontal_angle) <= 0.00005f && (horizontal_angle - 0) >= 0.00005f) {
        continue;
    }

    // check if the laser is outside the requested horizontal FOV
    if (!VectorMath::isAngleBetweenAngles(horizontal_angle, laser_start, laser_end)) {
        continue;
    }   

        // here would be the standard vertical for loop to shoot the actual lasers
        // here you should also make a check to make see if you have completed a full rotation to fill the pointcloud arrays correctly

       previous_horizontal_angle = horizontal_angles_[current_horizontal_angle_index_];
}
}

[tharindurmt]

Thank you @WouterJansen!
I did some tests with my approach and indeed the fps based solution is not viable. I was investigating how I can invoke a function that gets a partial point cloud on a single tick. I will look into this as well.

[tharindurmt]

@WouterJansen A quick question.
Shouldn't the vehicle pose and the lidar pose be updated every time before shooting a laser? Or at least before shooting a bunch of lasers corresponding to a certain time delta (i.e. before shooting lasers of a single horizontal step)?

[rajat2004]

@tharindurmt The vehicle & lidar pose are computed on every tick here - https://github.com/microsoft/AirSim/blob/master/AirLib/include/sensors/lidar/LidarSimple.hpp#L73-L106
Which calls getPointCloud with the updated pose

[tharindurmt]

@rajat2004 Thank you very much. I have overlooked it!
Will you be able to clarify one more thing please?
These lines mention two types of time deltas (i.e. tick delta time and the elapsed clock time). While I think I understand the difference between the two "time types", it's not clear to me why we need to cap the number of points to scan just because SensorBase is using elapsed clock time. Can you please clarify?

[rajat2004]

Sorry, missed this message
I haven't dived deep till now into how the Lidar sensor works, and the meaning of the statement isn't exactly apparent, others will most likely be able to answer better. Will post if there are any updates

[tharindurmt]

@rajat2004 Thank you.
I did some investigations in relation to your earlier comment on vehicle and lidar pose being computed on every tick and I think I have not explained myself properly. I'll try to elaborate it to the best I can.
The current lidar model first calculate the elapsed time (as far as I can see, this is the actual elapsed clock time - Code) since the last lidar update (Code).
Then get the current lidar and vehicle poses (Code) and call the getPointCloud function using the above collected parameters (Code).
Inside the getPointCloud function, calculate the lidar points corresponding to the time period between the last clock time that a lidar point was obtained and the current clock time (Code) - using the delta_time.

Now assume that a vehicle travelling at 5ms-2 is fitted with a lidar having a rotational frequency of 10Hz. Then, during a single rotation of the lidar, the distance the vehicle has travelled is 1.5m. Now, since the vehicle pose is not updated during a single rotation of the lidar (something similar is mentioned in this comment as well), this distortion is not captured in the current model.

As far as I understand, if we need to simulate a realistic set of lidar measurements, we need update the lidar and vehicle poses inside the getPointCloud function. Is my understanding correct?
Hope I have explained it well enough.

[WouterJansen]

Don't think what you seek is possible and also there might still be some confusion. The rotational frequency of the lidar just helps to decide how many points should be calculated over a time delta. A single execution of getPointCloud does not result in a full rotational scan. Only a partial one. It will calculate based on delta_time how many lasers should be calculated.

In that call getPointCloud indeed the simulation is 'paused'. It won't alter the state of the simulation (vehicles/sensors/world) and will simply see the simulation as static. I guess you could improve things by using perhaps physics substepping and somehow updating the position of the vehicle/sensor during the game thread but not sure how that would work from a practical point of view.

I think you what you are seeking is a (fixed step) simulation where you can update the simulation for each horizontal point on your lidar so that indeed all distortion is accounted for and accurate. This is as far as I know not feasible in a 'real-time' simulation such as AirSim. Where sensors such as LiDAR are heavily dependent on the framerate to indicate the accuracy.

[tharindurmt]

@WouterJansen That clears things up. Thank you!
Lidar model in Airsim is developed similar to that of Carla. This Carla project offer a motion distorted point cloud by setting an extremely small discrete timestep (0.05 milliseconds). So, I guess the easiest way (and with lengthy execution times) to achieve this is to set the ClockSpeed to a really small value. Otherwise, one would have to write a whole new set of codes starting from a new SensorBase class. I will try to see whether I can come up with any other solutions.
Thank you for your help.