Airsim: Effective way to save depth maps

[Mayankm96]

Hey all!

I am trying to record depth maps from the simulator in 1280 x 720 size. Since it is a large size, saving the depth maps in pfm format takes a lot of time. With other image types also being saved in that size, it takes 10-15 minutes for me to save one image request. I would like to know if it is possible to reduce this time by saving the depth maps in some other effective way.

One alternative is to save DepthVis and then convert it into DepthPerspective. However, I am concerned if this might lead to data accuracy. Other suggestions are also welcomed!

Thank you!

[sytelus]

The rendering performance greatly drops as you go to 720p. Even on high end cards like TitanX I would expect renderer to shutter significantly even if you don't save any files. So I would recommend using high resolution either at much lower rate or in computer vision mode where time doesn't matter.

The pfm format is useful to get exact numerical depth value for pixel. If you are ok with only 0-255 monochrome colors and artifacts of compression, you can retrieve image as png. This probably won't be very fast because I would think bottleneck is rendering (assuming you have SSD drive).

If you discover that file I/O is the real issue and you need exact floating point value of depth then you can probably still do compression using gzip in Python script and save pfm as zip file.

[Mayankm96]

@sytelus It roughly takes two minutes to save the pfm file, so I suppose a major lag is due to the image request. As you suggested, I am running the simulator in the computer vision mode.

Is it possible to have the depth visualization images in OpenNI standards i.e. as 16UC1 images? This would be in accordance with the REP-118 in ROS standard conventions.

[sytelus]

We chose pfm because some off the shelf DNNs seem to use this format and also pure C++ code for load/save without any dependency was fairly simple plus there are many viewers available for pfm.

Do you have any sample code to save/load images in OpenNI standard? What are the viewers available for Windows/Linux?

[Mayankm96]

I wrote a ROS package (here) to save data published using the Zed camera. The following snippet of code to process and save the depth camera images might serve the purpose here:

void DepthProcessor::processDepthData(cv_bridge::CvImagePtr depth_img_ptr)
{
  if (depth_img_ptr->encoding == "16UC1"){
    depth_vis_ = depth_img_ptr->image; // Visualization
    depth_raw_ = cv::Mat(depth_img_ptr->image.rows, depth_img_ptr->image.cols, CV_32FC1); // True depth

    // Fill in the depth image data, converting mm to m
    float bad_point = std::numeric_limits<float>::quiet_NaN ();
    for(size_t i = 0; i < depth_img_ptr->image.rows; i++)
    {
      for(size_t j = 0; j < depth_img_ptr->image.cols; j++)
      {
        uint16_t depth_vis_value = depth_img_ptr->image.at<uint16_t>(i,j);
        depth_raw_.at<float>(i,j) = (depth_vis_value == 0) ? bad_point : (float) depth_vis_value * 0.001f;
      }
    }
  }

  if (depth_img_ptr->encoding == "32FC1"){
    depth_raw_ = depth_img_ptr->image; // True depth
    depth_vis_ = cv::Mat(depth_img_ptr->image.rows, depth_img_ptr->image.cols, CV_16UC1);  // Visualization

    // Fill in the depth image data, converting m to mm
    uint16_t bad_point = 0;
    for(size_t i = 0; i < depth_img_ptr->image.rows; i++)
    {
      for(size_t j = 0; j < depth_img_ptr->image.cols; j++)
      {
        float depth_raw_value = depth_img_ptr->image.at<float>(i,j);
        depth_vis_.at<uint16_t>(i,j) = std::isnan(depth_raw_value) ? bad_point : (uint16_t)(depth_raw_value * 1000);
      }
    }
  }
}

void DepthProcessor::saveDepthData(const std::string &filename)
{
  cv::imwrite(filename + ".png", depth_vis_);
  writeFilePFM(filename + ".pfm", depth_raw_, 1/255.0);
}

The variable depth_vis_ is of type 16UC1. I am saving it using the OpenCV imwrite() function in png format. The saved image can be viewed by any standard image viewer. To load this image, I use the following line:

depth = cv::imread(IN_DIR + "depth_vis/depth_" + frame_name + ".png", CV_16UC1);