Opencv_contrib: Bug in sfm or data incorrect ?

[LaurentBerger]

Hi,

I am able to use sfm module on windows. I can run all examples. But I have got a problem using reconstruct function. I have got this exception

OpenCV Error: Assertion failed (data && dims <= 2 && (rows == 1 || cols == 1) && rows + cols - 1 == n && channels() == 1) in cv::Mat::operator class cv::Vec, file G:\Lib\opencv\modules\core\include\opencv2/core/mat.inl.hpp, line 1221

I think you can reproduce using this code :

    vector<Mat> pt2d;
    pt2d.push_back((Mat_<double>(1, 16) << 64., 298., 560., 405., 271., 298., 514., 417., 651., 435., 561, 373., 497., 369., 350., 241));
    pt2d.push_back((Mat_<double>(1,16)<< 62., 272., 5.1840002441406250e+02, 4.1817602539062500e+02, 268.,266., 513., 359., 649., 371., 559., 227., 495., 229., 346., 205.));
    pt2d.push_back((Mat_<double>(1, 16) << 362., 185., 650., 304., 160., 248., 364., 192., 78., 146., 357.,176., 247., 402., 76., 149));
    pt2d.push_back((Mat_<double>(1, 16) << 668., 296., 5.9040000915527344e+01, 3.8880001831054688e+02,6.5520001220703125e+02, 2.6496002197265625e+02,
        4.9447909545898438e+02, 2.9382092285156250e+02,4.6800003051757813e+01, 9.9600006103515625e+01,5.1120001220703125e+02, 4.1280001831054688e+02,
        1.3080000305175781e+02, 2.4600001525878906e+02,2.6873861694335938e+02, 1.1197442626953125e+02 ));
    for (int i=0;i<pt2d.size();i++)
        pt2d[i]= pt2d[i].reshape(2);
    vector<Mat> Rs_est, ts_est, points3d_estimated
    Mat  K = (cv::Mat_<float>(3, 3) << 350, 0, 240, 0, 350, 360, 0, 0, 1);
    reconstruct(pt2d, Rs_est, points3d_estimated, K, true);

In debugging problem is here and should be something like points2d[frame].col(track).at(0)

There is also this post . I don't understand this function

[LaurentBerger]

@sovrasov thanks. Label is bug. Have you got a bug when you replace line by Vec2d track_pt = pt2d[0].col(0).at(0); ?

[sovrasov]

In the provided code pt3d is undefined. Does bug happen on any set of 3d points?

[LaurentBerger]

Sorry I forgot to update my code. Now I don't understand thoses lines . How is it possible to copy a Vec3d in a Mat with only one columm and one channel?

[sovrasov]

Vec3d point3d is converted into Mat first and then deep copy to points3d[i] is performed. points3d is used here as OutputArrayOfArrays.

[LaurentBerger]

Thanks I don't know how it is possible but I will dig in source code. Now I think a PR is necessary with pt2d[0].col(0).at(0) and with thoses lines

[sovrasov]

Setting all indexes to zeros in pt2d[0].col(0) is strange approach. You will process the same data within the loop many times.

[LaurentBerger]

sorry I write
const Vec2d track_pt = points2d[frame].col(track).at<Vec2d>(0);
instead of
const Vec2d track_pt = points2d[frame].col(track);

I have build opencv with test. test_reconstruct call reconstruct function with a point2d cloud but only triangulate functions is called because there is only 2 views in point cloud.

Modifying nviews=3 gives:
```
OpenCV version: 3.2.0-dev
OpenCV VCS version: 3.2.0-571-g06b0fe3
Build type: debug
Parallel framework: ms-concurrency
CPU features: mmx sse sse2 sse3 ssse3
OpenCL Platforms:
NVIDIA CUDA
dGPU: GeForce GTX 970 (OpenCL 1.2 CUDA)
Intel(R) OpenCL
CPU: Intel(R) Core(TM) i7-5820K CPU @ 3.30GHz (OpenCL 1.2 (Build 8))
Experimental OpenCL 2.1 CPU Only Platform
CPU: Intel(R) Core(TM) i7-5820K CPU @ 3.30GHz (OpenCL 2.1 (Build 18))
Current OpenCL device:
Type = dGPU
Name = GeForce GTX 970
Version = OpenCL 1.2 CUDA
Driver version = 376.53
Compute units = 13
Max work group size = 1024
Local memory size = 48 kB
Max memory allocation size = 1 GB
Double support = Yes
Host unified memory = No
Has AMD Blas = No
Has AMD Fft = No
Preferred vector width char = 1
Preferred vector width short = 1
Preferred vector width int = 1
Preferred vector width long = 1
Preferred vector width float = 1
Preferred vector width double = 1
[==========] Running 21 tests from 8 test cases.
[----------] Global test environment set-up.
[----------] 2 tests from Sfm_conditioning
[ RUN ] Sfm_conditioning.normalizePoints
[ OK ] Sfm_conditioning.normalizePoints (0 ms)
[ RUN ] Sfm_conditioning.normalizeIsotropicPoints
[ OK ] Sfm_conditioning.normalizeIsotropicPoints (0 ms)
[----------] 2 tests from Sfm_conditioning (0 ms total)

[----------] 6 tests from Sfm_fundamental
[ RUN ] Sfm_fundamental.fundamentalFromProjections
[ OK ] Sfm_fundamental.fundamentalFromProjections (0 ms)
[ RUN ] Sfm_fundamental.normalizedEightPointSolver
[ OK ] Sfm_fundamental.normalizedEightPointSolver (3 ms)
[ RUN ] Sfm_fundamental.motionFromEssential
[ OK ] Sfm_fundamental.motionFromEssential (2 ms)
[ RUN ] Sfm_fundamental.fundamentalToAndFromEssential
[ OK ] Sfm_fundamental.fundamentalToAndFromEssential (0 ms)
[ RUN ] Sfm_fundamental.essentialFromFundamental
[ OK ] Sfm_fundamental.essentialFromFundamental (0 ms)
[ RUN ] Sfm_fundamental.motionFromEssentialChooseSolution
[ OK ] Sfm_fundamental.motionFromEssentialChooseSolution (1 ms)
[----------] 6 tests from Sfm_fundamental (8 ms total)

[----------] 2 tests from Sfm_numeric
[ RUN ] Sfm_numeric.meanAndVarianceAlongRows
[ OK ] Sfm_numeric.meanAndVarianceAlongRows (0 ms)
[ RUN ] Sfm_numeric.skewMat
[ OK ] Sfm_numeric.skewMat (0 ms)
[----------] 2 tests from Sfm_numeric (1 ms total)

[----------] 3 tests from Sfm_projection
[ RUN ] Sfm_projection.homogeneousToEuclidean
[ OK ] Sfm_projection.homogeneousToEuclidean (0 ms)
[ RUN ] Sfm_projection.euclideanToHomogeneous
[ OK ] Sfm_projection.euclideanToHomogeneous (0 ms)
[ RUN ] Sfm_projection.P_From_KRt
[ OK ] Sfm_projection.P_From_KRt (1 ms)
[----------] 3 tests from Sfm_projection (3 ms total)

[----------] 1 test from Sfm_reconstruct
[ RUN ] Sfm_reconstruct.twoViewProjectiveOutliers
unknown file: error: C++ exception with description "G:\Lib\opencv\modules\core\src\matrix.cpp:2945: error: (-215) k == STD_VECTOR_MAT || k == STD_ARRAY_MAT in function cv::_OutputArray::getMatRef
" thrown in the test body.
[ FAILED ] Sfm_reconstruct.twoViewProjectiveOutliers (29749 ms)
[----------] 1 test from Sfm_reconstruct (29751 ms total)

[----------] 4 tests from Sfm_robust
[ RUN ] Sfm_robust.fundamentalFromCorrespondences8PointRobust
[ OK ] Sfm_robust.fundamentalFromCorrespondences8PointRobust (19 ms)
[ RUN ] Sfm_robust.fundamentalFromCorrespondences8PointRealisticNoOutliers
[ OK ] Sfm_robust.fundamentalFromCorrespondences8PointRealisticNoOutliers (3 ms)
[ RUN ] Sfm_robust.fundamentalFromCorrespondences7PointRobust
[ OK ] Sfm_robust.fundamentalFromCorrespondences7PointRobust (11 ms)
[ RUN ] Sfm_robust.fundamentalFromCorrespondences7PointRealisticNoOutliers
[ OK ] Sfm_robust.fundamentalFromCorrespondences7PointRealisticNoOutliers (5 ms)
[----------] 4 tests from Sfm_robust (42 ms total)

[----------] 1 test from Sfm_simple_pipeline
[ RUN ] Sfm_simple_pipeline.backyard
F0511 11:32:55.878904 11012 utilities.cc:322] Check failed: !IsGoogleLoggingInitialized() You called InitGoogleLogging() twice!
* Check failure stack trace: *```

[LaurentBerger]

About first post finally datas are incorrect It must be a mat with two rows and N column (for N points). I close this comment.

[EnoahNetzach]

Sorry, is this still considered a bug?

It happens to me even when I simply copy the code from this official tutorial..

P.S. I'm running everything on a docker container with this Dockerfile:

FROM debian:stable-slim

COPY backports.list /etc/apt/sources.list.d/

RUN apt-get update && \
apt-get upgrade -y && \
apt-get install -y build-essential cmake git libgtk2.0-dev pkg-config libavcodec-dev \
libavformat-dev libswscale-dev python3-dev python3-numpy libtbb2 libtbb-dev \
libjpeg-dev libpng-dev libtiff-dev libjasper-dev libdc1394-22-dev ffmpeg

RUN apt-get install -y libboost-all-dev libeigen3-dev libgflags-dev libgoogle-glog-dev libatlas-base-dev \
libsuitesparse-dev libjpeg-dev libpng-dev libdc1394-22-dev

RUN git clone https://ceres-solver.googlesource.com/ceres-solver --depth=1 && \
git clone https://github.com/opencv/opencv.git --depth=1 && \
git clone https://github.com/opencv/opencv_contrib.git --depth=1

RUN apt-get install -y libvtk6-dev

RUN cd ceres-solver && \
mkdir build && \
cd build && \
cmake -D CXX11=ON \
-D EXPORT_BUILD_DIR=ON \
-D BUILD_TESTING=OFF .. && \
make -j8 && \
make install

RUN cd opencv && \
mkdir build && \
cd build && \
cmake -D CMAKE_BUILD_TYPE=Release \
-D CMAKE_CXX_FLAGS=-std=c++11 \
-D BUILD_TESTS=false \
-D BUILD_PERF_TESTS=false \
-D OPENCV_EXTRA_MODULES_PATH=../../opencv_contrib/modules  \
-D PYTHON3_EXECUTABLE=/usr/bin/python3 \
-D PYTHON_INCLUDE_DIR=/usr/include/python3.4m \
-D PYTHON_INCLUDE_DIR2=/usr/include/x86_64-linux-gnu/python3.4m \
-D PYTHON_LIBRARY=/usr/lib/x86_64-linux-gnu/libpython3.4m.so \
-D PYTHON3_NUMPY_INCLUDE_DIRS=/usr/lib/python3/dist-packages/numpy/core/include/ \
-D CMAKE_INSTALL_PREFIX=/usr/local .. && \
make -j8 && \
make install

COPY entrypoint.sh /entrypoint.sh
RUN chmod +x /entrypoint.sh

ENTRYPOINT ["/entrypoint.sh"]

[LaurentBerger]

My issue is not clear. I think you should forget it and not use it as a referrence. there was a bug in InitGoogleLogging() but if you clone opencv_contrib today (2017 june ) there is no problem. You should ask your question on answers.opencv.org before open a new issue

[EnoahNetzach]

@LaurentBerger thanks, but the last time I've rebuilt the docker container was yesterday, and it clones opencv_contrib every time.

I still have that exact message popping up...

[LaurentBerger]

What is exactly your programm and error message ?

[EnoahNetzach]

I just found out it was simply that this official tutorial is a bit outdated, this being the correct, and working, one.

@LaurentBerger thanks, and sorry for the trouble.

[LaurentBerger]

No problem and tell me if you can reconstruct 3d scene without problem :
./example_sfm_scene_reconstruction image_paths_file.txt 350 240 360

[atomoclast]

Is this still a bug?

I too am experiencing this same error.

I try to pass in a vector as the first argument for the reconstruct function.

I can run all the example code no issues.

[LaurentBerger]

@atomoclast there is no known bug in sfm module except if you write a new issue with a small example and opencv team can reproduce it.
If you have a question you should ask it on answers.opencv.org

[atomoclast]

@LaurentBerger Hi, I actually opened an issue on answers.opencv.org and I belive you replied.

I'm getting this exact error, when I try and use the reconstruct function when I pass it a vector<Mat>, where every element is a greyscale image opened and loaded into a Mat data type.

Perhaps I am using it wrong?

When I try and pass it that that vector, I get:

OpenCV(3.4.1) Error: Assertion failed (data && dims <= 2 && (rows == 1 || cols == 1) && rows + cols - 1 == n && channels() == 1) in operator cv::Vec<_Tp, m>, file /home/andrew/opencv_source/opencv/modules/core/include/opencv2/core/mat.inl.hpp, line 1282
terminate called after throwing an instance of 'cv::Exception'
  what():  OpenCV(3.4.1) /home/andrew/opencv_source/opencv/modules/core/include/opencv2/core/mat.inl.hpp:1282: error: (-215) data && dims <= 2 && (rows == 1 || cols == 1) && rows + cols - 1 == n && channels() == 1 in function operator cv::Vec<_Tp, m>

What is confusing me is that one of the arguments is:

InputArrayOfArrays points2d

and that's defined as:

points2d Input vector of vectors of 2d points (the inner vector is per image).

Does that mean we flatten the Mat data type of each image into a vector of vectors or something? I don't get it.

Do I need to use a feature matcher and find matching features, then put those features in this vector?

Either I'm not understanding what it means properly, or the documentation is a bit confusing. I'm rather new to OpenCV in C++.

[LaurentBerger]

Like in this sample?
if yes pt2d1 is vector < Mat > all mat size are equal and size of one Mat is 108 column and 2 rows. 108 means in each image you have 108 points with two coordinates (image coordinates colum and row)

[atomoclast]

@LaurentBerger Exactly like in that. I'm not sure what you're putting in that data type exactly?

I see in line 32 that it is defined as a vector<Mat>

[I am weak in C++ versus Python, so I'm trying to get better]

I may just post what I've done to modify the example and how I am trying to use it for some help.

[atomoclast]

This is the slightly modified version of the tutorial I am trying to work with.

I zipped it since it won't let me upload a C++ file directly.

SfM.cpp.zip

[LaurentBerger]

back to question