Pcl: [io] `pcl::io::loadPCDFile` corrupts data while reading over 268 million PointXYZRGB points

[mudskipper75]

When using pcl::io::loadPCDFile to load a large file (over 268 million PointXYZRGB points) the total number of points is read correctly but it appears the point data is partially corrupted. It appears that a buffer of some sort is filled during the read process and when the buffer has been filled completely, the earlier entries in the buffer are overwritten.

The code I am using is shown below.

---

filenames = pcl::console::parse_file_extension_argument (argc, argv, ".pcd");
// Load files | Works with PCD and PLY files
pcl::PointCloud::Ptr source_cloud (new pcl::PointCloud ());

pcl::PointCloud::Ptr register_cloud (new pcl::PointCloud ());

pcl::io::loadPCDFile (argv[filenames[0]], *source_cloud);

pcl::io::loadPCDFile (argv[filenames[1]], *register_cloud);

std::cout << "Files Loaded Time: " << currentDateTime() << std::endl;
//Test save of source file to check the point extents
pcl::io::savePCDFileASCII ("source-extents.pcd", *source_cloud);

---

When I examine the "source-extents.pcd" file I find that the number of points matches those from the original source cloud file but the earlier entries have been overwritten with points that should be at the end of the file. If I gradually reduce the number of points in the source file then the issue goes away. I've created a test source cloud file and placed the output from running head and tail on the generated "source-extents.pcd" file below.

:/head -20 source-extents.pcd

.PCD v0.7 - Point Cloud Data file format

VERSION 0.7
FIELDS x y z rgb
SIZE 4 4 4 4
TYPE F F F U
COUNT 1 1 1 1
WIDTH 268435460
HEIGHT 1
VIEWPOINT 0 0 0 1 0 0 0
POINTS 268435460
DATA ascii
82895.25 6.9710002 571.68903 16732288
82895.25 6.7969999 571.81097 16732288
82895.25 6.6230001 571.93201 16732288
82895.25 6.448 572.02301 16732288
616.25 50.410702 596.35449 16732288
616.25 50.2202 596.10089 16732288
616.25 50.034 595.9425 16732288
616.25 49.848499 595.79999 16732288
616.25 49.661701 595.62579 16732288**

:/tail source-extents.pcd
82895.25 8.0170002 571.12701 16732288
82895.25 7.842 571.21002 16732288
82895.25 7.6680002 571.31702 16732288
82895.25 7.494 571.38501 16732288
82895.25 7.3200002 571.52197 16732288
82895.25 7.145 571.59003 16732288
82895.25 6.9710002 571.68903 16732288
82895.25 6.7969999 571.81097 16732288
82895.25 6.6230001 571.93201 16732288
82895.25 6.448 572.02301 16732288

You can see that the last 4 lines of the file are also written to the start of the file. This behaviour occurs with ASCII and binary pcd files.

After reducing the number of points in the file by 10 I receive the following from running from running head on the "source-extents.pcd" file.

:/head -20 source-extents.pcd

.PCD v0.7 - Point Cloud Data file format

VERSION 0.7
FIELDS x y z rgb
SIZE 4 4 4 4
TYPE F F F U
COUNT 1 1 1 1
WIDTH 268435450
HEIGHT 1
VIEWPOINT 0 0 0 1 0 0 0
POINTS 268435450
DATA ascii
616.25 55.239399 603.50403 16732288
616.25 55.0453 603.22302 16732288
616.25 52.1297 598.65808 16732288
616.25 51.939098 598.41937 16732288
616.25 50.410702 596.35449 16732288
616.25 50.2202 596.10089 16732288
616.25 50.034 595.9425 16732288
616.25 49.848499 595.79999 16732288
616.25 49.661701 595.62579 16732288

I split the large source cloud into two sub-clouds and modified the code to perform an internal concatenation of these into a single point cloud with the code shown below.

---

filenames = pcl::console::parse_file_extension_argument (argc, argv, ".pcd");

// Load files | Works with PCD and PLY files
// Multi part pcd files to overcome filebuffer issue
pcl::PointCloud::Ptr source_cloud_a (new pcl::PointCloud ());
pcl::PointCloud::Ptr source_cloud_b (new pcl::PointCloud ());

pcl::PointCloud::Ptr source_cloud (new pcl::PointCloud ());

pcl::PointCloud::Ptr register_cloud (new pcl::PointCloud ());

pcl::io::loadPCDFile (argv[filenames[0]], *source_cloud_a);
pcl::io::loadPCDFile (argv[filenames[1]], *source_cloud_b);

pcl::io::loadPCDFile (argv[filenames[2]], *register_cloud);
*source_cloud += *source_cloud_a;
*source_cloud += *source_cloud_b;
source_cloud_a.reset(new pcl::PointCloud);
source_cloud_b.reset(new pcl::PointCloud);

std::cout << "Files Loaded Time: " << currentDateTime() << std::endl;
//Test save of source file to check the point extents
pcl::io::savePCDFileASCII ("source-extents.pcd", *source_cloud);

---

This gets around the problem as shown below. Additional output from my program is also included which shows that the extents of the points are correctly identified.

The sub-cloud test suggests there are no issues with the PointCloud objects themselves or the output IO and it is an input IO issue which may be an issue with the PCL loadPCDFile function but could also be an issue with the operating systems IO.

:/head -20 source-extents.pcd

.PCD v0.7 - Point Cloud Data file format

VERSION 0.7
FIELDS x y z rgb
SIZE 4 4 4 4
TYPE F F F U
COUNT 1 1 1 1
WIDTH 353936016
HEIGHT 1
VIEWPOINT 0 0 0 1 0 0 0
POINTS 353936016
DATA ascii
616.25 55.239399 603.50403 16732288
616.25 55.0453 603.22302 16732288
616.25 52.1297 598.65808 16732288
616.25 51.939098 598.41937 16732288
616.25 50.410702 596.35449 16732288
616.25 50.2202 596.10089 16732288
616.25 50.034 595.9425 16732288
616.25 49.848499 595.79999 16732288
616.25 49.661701 595.62579 16732288

:/tail source-extents.pcd
109102 -113.784 646.56 16732288
109102 -114.413 647.14099 16734080
109102 -114.631 647.422 16732288
109102 -114.85 647.71198 16734080
109102 -115.264 648.03802 16734080
109102 -115.697 648.55798 16734080
109102 -115.915 648.83099 16732288
109102 -116.353 649.38599 16732288
109102 -116.573 649.677 16733824
109102 -117.646 650.84399 16732288

./register ../../pcd/raw_laser_2019-10-30-10-51-53_00_0000594_1-24.pcd ../../pcd/raw_laser_2019-10-30-10-51-53_00_0000594_25-49.pcd 56E1-LowRes.pcd
Start Time: 15:19:30
Opened Database Successfully!
Files Loaded Time: 15:42:25
Max x: 109102
Max y: 57.2271
Max z: 673.147
Min x: 616.25
Min y: -118.898
Min z: 568.33
Total number of points: 353936016

I hope someone may be able to shed some light on this problem.

• OS: Ubuntu 20.04 (x86_64)
• Compiler: GCC 9.3.0
• PCL Versions 1.10.0 (Ubuntu Repo), 1.10.1, 1.11.1-rc1 (compiled from github/PointCloudLibrary source code)

[wz18207]

Hello,
I've read your problem.
Have you tried to change the file format. It's ASCII format.
I think it could be changed into binary part.

[mvieth]

@mudskipper75 Can you provide the pcd file you are using as input? That would help greatly to reproduce and debug this issue.

I noticed that 268435460-4=2^28, with 268435460 being the number of points in your cloud, and 4 lines being overwritten. This might give a hint where the problem lies.

Could you also try this:

filenames = pcl::console::parse_file_extension_argument (argc, argv, ".pcd");
// Load files | Works with PCD and PLY files
pcl::PCLPointCloud2 source_cloud;
pcl::io::loadPCDFile (argv[filenames[0]], source_cloud);
std::cout << "Files Loaded Time: " << currentDateTime() << std::endl;
//Test save of source file to check the point extents
pcl::io::savePCDFile ("source-extents.pcd", source_cloud);

This could show whether the internal conversion from PCLPointCloud2 to the templated point cloud is a problem.

Have you checked the point values in-program yet? Like

for(size_t i=0; i<10; ++i) {
    printf("point %lu is (%g,%g,%g)\n", i, cloud[i].x, cloud[i].y, cloud[i].z);
}

[SergioRAgostinho]

Most of the library should be able to cope all the way up to 2^31 points. Above that it will likely fail. It's an long-standing problem that is being addressed.

[mudskipper75]

Hello,
I've read your problem.
Have you tried to change the file format. It's ASCII format.
I think it could be changed into binary part.

Hi there

I have tried both ASCII and binary and I have the same problem.

[mudskipper75]

Most of the library should be able to cope all the way up to 2^31 points. Above that it will likely fail. It's an long-standing problem that is being addressed.

Thanks the information.

[mudskipper75]

@mudskipper75 Can you provide the pcd file you are using as input? That would help greatly to reproduce and debug this issue.

I noticed that 268435460-4=2^28, with 268435460 being the number of points in your cloud, and 4 lines being overwritten. This might give a hint where the problem lies.

Could you also try this:

filenames = pcl::console::parse_file_extension_argument (argc, argv, ".pcd");
// Load files | Works with PCD and PLY files
pcl::PCLPointCloud2 source_cloud;
pcl::io::loadPCDFile (argv[filenames[0]], source_cloud);
std::cout << "Files Loaded Time: " << currentDateTime() << std::endl;
//Test save of source file to check the point extents
pcl::io::savePCDFile ("source-extents.pcd", source_cloud);

This could show whether the internal conversion from PCLPointCloud2 to the templated point cloud is a problem.

Have you checked the point values in-program yet? Like

for(size_t i=0; i<10; ++i) {
    printf("point %lu is (%g,%g,%g)\n", i, cloud[i].x, cloud[i].y, cloud[i].z);
}

Hi There

I have used other large point clouds in my program and they work fine its just when they reach a certain size that I have a problem and the 2^28 value you have indicated would seem to be the likely barrier. I have done extensive testing with the program including printing cloud point values and have not found any issues with the cloud data or program output until I pass the 2^28 number of points. I can supply the point cloud for you to reproduce the problem but it is 2.2GB as tar.gz file.

I tried your code suggestion and have output the first 20 lines of the source-extents.pcd. The same issue persists.

.PCD v0.7 - Point Cloud Data file format

VERSION 0.7
FIELDS x y z rgb
SIZE 4 4 4 4
TYPE F F F U
COUNT 1 1 1 1
WIDTH 268435460
HEIGHT 1
VIEWPOINT 0 0 0 1 0 0 0
POINTS 268435460
DATA ascii
82895.25 6.9710002 571.68903 16732288
82895.25 6.7969999 571.81097 16732288
82895.25 6.6230001 571.93201 16732288
82895.25 6.448 572.02301 16732288
616.25 50.410702 596.35449 16732288
616.25 50.2202 596.10089 16732288
616.25 50.034 595.9425 16732288
616.25 49.848499 595.79999 16732288
616.25 49.661701 595.62579 16732288

[mvieth]

Considering that each of the points has 4 fields (x, y, z, rgb), with 4 bytes each, so 16 bytes per point, a cloud with 2^28 points would be 2^32 bytes large. Could you please find out the maximum size of a vector on your system (https://en.cppreference.com/w/cpp/container/vector/max_size)? Also, how much RAM do you have?
Ok, since your input pcd file is so big, we can first try to debug this without it. But please post the header (first 20 lines or so) of the input pcd file.

[mudskipper75]

Considering that each of the points has 4 fields (x, y, z, rgb), with 4 bytes each, so 16 bytes per point, a cloud with 2^28 points would be 2^32 bytes large. Could you please find out the maximum size of a vector on your system (https://en.cppreference.com/w/cpp/container/vector/max_size)? Also, how much RAM do you have?
Ok, since your input pcd file is so big, we can first try to debug this without it. But please post the header (first 20 lines or so) of the input pcd file.

Thanks for the help. I have added the following to the test code with the out put shown below.

std::vector p;
std::cout << "Maximum size of a 'vector' is " << p.max_size() << "\n";

'Maximum size of a 'vector' is 288230376151711743'

The system has 96GB of RAM

free -m
total used free shared buff/cache available
Mem: 96646 785 37754 1 58105 95014
Swap: 2047 118 1929

First 20 lines of the input pcd are shown below

head -20 raw_laser_2019-10-30-10-51-53_00_0000594_1-37_plus.pcd
VERSION .7
FIELDS x y z rgb
SIZE 4 4 4 4
TYPE F F F U
COUNT 1 1 1 1
WIDTH 268435460
HEIGHT 1
VIEWPOINT 0 0 0 1 0 0 0
POINTS 268435460
DATA ascii
616.2500 55.2394 603.5040 16732288
616.2500 55.0453 603.2230 16732288
616.2500 52.1297 598.6581 16732288
616.2500 51.9391 598.4194 16732288
616.2500 50.4107 596.3545 16732288
616.2500 50.2202 596.1009 16732288
616.2500 50.0340 595.9425 16732288
616.2500 49.8485 595.8000 16732288
616.2500 49.6617 595.6258 16732288
616.2500 49.4757 595.4675 16732288

[mvieth]

What sizes do int and std::size_t have on your system? Both 64 bit?
You could also try to add pcl::console::setVerbosityLevel(pcl::console::L_VERBOSE); at the start of your program to enable all logging. It might print some useful information.

[mudskipper75]

What sizes do int and std::size_t have on your system? Both 64 bit?
You could also try to add pcl::console::setVerbosityLevel(pcl::console::L_VERBOSE); at the start of your program to enable all logging. It might print some useful information.

I ran the test code with verbose set and additional int and size_t information.

./register ../../pcd/raw_laser_2019-10-30-10-51-53_00_0000594_1-37_plus.pcd 56E1-LowRes.pcd
Start Time: 18:15:02
Opened Database Successfully!
Maximum size of a pcl::PointXYZRGB 'vector' is 288230376151711743
Maximum size of an int 'vector' is 2305843009213693951
Maximum size of a size_t 'vector' is 1152921504606846975
[pcl::PCDReader::read] Loaded ../../pcd/raw_laser_2019-10-30-10-51-53_00_0000594_1-37_plus.pcd as a dense cloud in 991695 ms with 268435460 points. Available dimensions: x y z rgb.
Files Loaded Time: 18:31:34
[pcl::PCDWriter::setLockingPermissions] File source-extents.pcd locked successfully.

[mvieth]

Just to be sure, std::cout << "sizeof(int)=" << sizeof(int) << ", sizeof(std::size_t)=" << sizeof(std::size_t) << std::endl; returns 4 and 8, right?
The end of the saved pcd file is exactly as you would expect it to be? It is just that the first 4 points are overwritten?
My best guess right now is that the problem is here, that an overflow happens while computing the index in the data vector. But there is likely a similar problem while saving the pcd file, since you see good values at the end, not just 4 lines of zeros/nonsense.

[mudskipper75]

Just to be sure, std::cout << "sizeof(int)=" << sizeof(int) << ", sizeof(std::size_t)=" << sizeof(std::size_t) << std::endl; returns 4 and 8, right?
The end of the saved pcd file is exactly as you would expect it to be? It is just that the first 4 points are overwritten?
My best guess right now is that the problem is here, that an overflow happens while computing the index in the data vector. But there is likely a similar problem while saving the pcd file, since you see good values at the end, not just 4 lines of zeros/nonsense.

sizeof(int)=4, sizeof(std::size_t)=8 -> returned from the test program

The saved file is completely fine apart from the first four lines. There is nothing wrong with the end of the file. An issue with an index calculation during the reading/loading of the file makes a lot sense.

[mudskipper75]

Hello,
I've read your problem.
Have you tried to change the file format. It's ASCII format.
I think it could be changed into binary part.

I have tried both binary and ASCII files and achieve the same result.

[mvieth]

Ok, I am pretty sure that the problem is where I linked before: the overflow happens with point_index * cloud.point_step. A solution would be to either change the type of the parameter point_index to std::size_t or to cast it like so: static_cast<std::size_t>(point_index) * point_step. The solution has to be applied to 8 places in filo_io.h, I think.

[mudskipper75]

file_io.h.txt

I downloaded pcl 1.11.1, modified file_io.h replacing size_t instead of unsigned int for point_index and compiled and installed pcl. I recompiled and ran the test code but I still have the same issue with the first four lines being overwritten.

[mvieth]

Hm, that's strange. Are you sure that you are using the correct pcl when you are compiling your own code? Maybe try find_package(PCL 1.11.1 EXACT REQUIRED)?
Have you tried your original code or my code snippet with PCLPointCloud2? It is possible that there are more problems when the PCLPointCloud2 is converted to the templated PointCloud or back.

[mudskipper75]

Hm, that's strange. Are you sure that you are using the correct pcl when you are compiling your own code? Maybe try find_package(PCL 1.11.1 EXACT REQUIRED)?
Have you tried your original code or my code snippet with PCLPointCloud2? It is possible that there are more problems when the PCLPointCloud2 is converted to the templated PointCloud or back.

Apologies. I had uninstalled the OS provided libpcl-dev 1.10.0 but it appeared there were some libraries still installed which I have now removed and recompiled the test code with find_package(PCL 1.11.1 EXACT REQUIRED) in the CMakeLists.txt. The code now works fine and the output cloud does not have the first for lines overwritten.

head -20 source-extents.pcd

.PCD v0.7 - Point Cloud Data file format

VERSION 0.7
FIELDS x y z rgb
SIZE 4 4 4 4
TYPE F F F U
COUNT 1 1 1 1
WIDTH 268435460
HEIGHT 1
VIEWPOINT 0 0 0 1 0 0 0
POINTS 268435460
DATA ascii
616.25 55.239399 603.50403 16732288
616.25 55.0453 603.22302 16732288
616.25 52.1297 598.65808 16732288
616.25 51.939098 598.41937 16732288
616.25 50.410702 596.35449 16732288
616.25 50.2202 596.10089 16732288
616.25 50.034 595.9425 16732288
616.25 49.848499 595.79999 16732288
616.25 49.661701 595.62579 16732288

I will recompile my main program and leave it running over the weekend with a ~353 million point pcd file and see how it goes.

Many thanks for help.

[mvieth]

Many thanks for help.

You're very welcome. Could you make a pull request with the changes you made in file_io.h?

[mudskipper75]

I thought the issue was resolved but it is half the story. The solution works for ASCII files but the same issue persists with binary. I assume there is something else going on with binary conversion. The good new is that I have thrown an ASCII ~353 million point cloud at your test code and it works.

I have not made a pull request before but will follow the guidelines, probably Monday and issue a pull request.

[mvieth]

I thought the issue was resolved but it is half the story. The solution works for ASCII files but the same issue persists with binary. I assume there is something else going on with binary conversion.

Do you know whether the problem is with loading or with saving the files? Are the binary files compressed or uncompressed?

The good new is that I have thrown an ASCII ~353 million point cloud at your test code and it works.

That's great! I'm curious: what are you doing that you are processing that many points? Most people work with much fewer points, which is likely the reason why nobody had this problem before (that I am aware of).

I have not made a pull request before but will follow the guidelines, probably Monday and issue a pull request.

Great! If you have any questions, I am happy to help.

[mudskipper75]

I thought the issue was resolved but it is half the story. The solution works for ASCII files but the same issue persists with binary. I assume there is something else going on with binary conversion.

Do you know whether the problem is with loading or with saving the files? Are the binary files compressed or uncompressed?

I think the problem is with loading and saving files but I will do some more tests. The files are uncompressed binary. My main program uses pcl::PointCloud point clouds which also don't load or save correctly right now.

The good new is that I have thrown an ASCII ~353 million point cloud at your test code and it works.

That's great! I'm curious: what are you doing that you are processing that many points? Most people work with much fewer points, which is likely the reason why nobody had this problem before (that I am aware of).

I am working with point clouds associated with multiple laser scans of train rails which can be ~108 m in length. Initially I would like to consider them as one piece but it is not a deal breaker if I have to work with sub-sections of the rail. Each scan is 0.25 mm apart so 108 m gives 432000 scans of approximately 800 points. Owing to the amount of data I am dealing with I would like to work with binary files.

I have not made a pull request before but will follow the guidelines, probably Monday and issue a pull request.

Great! If you have any questions, I am happy to help.

I've not done this before but I assume I clone the master branch, make the modifications and then perform a pull request?

[mvieth]

I think the problem is with loading and saving files but I will do some more tests. The files are uncompressed binary. My main program uses pcl::PointCloudpcl::PointXYZRGB point clouds which also don't load or save correctly right now.

Does it work if you load to and save from PCLPointCloud2, like here? That would again be good to know to see whether the problem is with the actual loading/saving or the conversion between PCLPointCloud2 and the templated cloud.

[mudskipper75]

I think the problem is with loading and saving files but I will do some more tests. The files are uncompressed binary. My main program uses pcl::PointCloudpcl::PointXYZRGB point clouds which also don't load or save correctly right now.

Does it work if you load to and save from PCLPointCloud2, like here? That would again be good to know to see whether the problem is with the actual loading/saving or the conversion between PCLPointCloud2 and the templated cloud.

Loading and saving works from PCLPointCloud2 with an with a ~353 million point ASCII pcd file. Converting the same file from ASCII to binary and back to ASCII format using the freshly compiled pcl_convert_pcd_ascii_binary with the modified file_io.h works fine also. Loading the freshly converted binary pcd file with PCLPointCloud2 and saving as the default ASCII also works.

[mvieth]

Loading and saving works from PCLPointCloud2 with an with a ~353 million point ASCII pcd file. Converting the same file from ASCII to binary and back to ASCII format using the freshly compiled pcl_convert_pcd_ascii_binary with the modified file_io.h works fine also. Loading the freshly converted binary pcd file with PCLPointCloud2 and saving as the default ASCII also works.

So when exactly does it fail? When you use a templated point cloud?
The only place I can think of where it might fail during conversion is here.

[mvieth]

I think I found it. Same problem as before, in col * msg.point_step.
If you change col, and probably best also row, to std::size_t, it should work.

[mudskipper75]

Loading and saving works from PCLPointCloud2 with an with a ~353 million point ASCII pcd file. Converting the same file from ASCII to binary and back to ASCII format using the freshly compiled pcl_convert_pcd_ascii_binary with the modified file_io.h works fine also. Loading the freshly converted binary pcd file with PCLPointCloud2 and saving as the default ASCII also works.

So when exactly does it fail? When you use a templated point cloud?
The only place I can think of where it might fail during conversion is here.

It is taking some time to debug because the files are so large. It seems to fail during the use of/conversion to templated point clouds.

Loading from ASCII pcd into pcl::PointCloud and saving in ASCII seems OK
Loading from Binary pcd into pcl::PointCloud and saving in ASCII has problems. Writes the same number of points as the source cloud but after 2^28 points have been written, points from the beginning of source file rewritten instead of the later points in the source cloud. I know the binary file is good because I tested with your test code yesterday.
pcl::getMinMax3D() seems to have an indexing issue also.

[mudskipper75]

I think I found it. Same problem as before, in col * msg.point_step.
If you change col, and probably best also row, to std::size_t, it should work.

Hi, I've made the changes in conversions.h and my program appears to working OK. The issue with getMinMax3D(0 has gone as well. I'm going to leave it running until it finishes which will probably be 24 hours and I will let you know how it goes. Many thanks.

[kunaltyagi]

Another PR incoming? :smile: Thanks should be aimed at you @mudskipper75

[mudskipper75]

I think I found it. Same problem as before, in col * msg.point_step.
If you change col, and probably best also row, to std::size_t, it should work.

Hi, I've made the changes in conversions.h and my program appears to working OK. The issue with getMinMax3D(0 has gone as well. I'm going to leave it running until it finishes which will probably be 24 hours and I will let you know how it goes. Many thanks.

All seems fine, I was able to perform a successful registration with the ~353 million point cloud. Now I know that I can work with clouds of this size I can now start tuning my code. I'll put a pull request in for the changes in conversion.h and thanks again.

[mudskipper75]

Loading and saving issues have been resolved.