Mediapipe: AutoFlip is incompatible with OpenCV4

[thesofakillers]

Hi, I am trying to build and eventually run autoflip but I've been having issues all day. I am specifically trying to build autoflip for a GPU enabled linux desktop, running Ubuntu 16.04.6 LTS. Can you confirm this is possible?
I am attempting to do this by running

bazel build -c opt --copt -DMESA_EGL_NO_X11_HEADERS mediapipe/examples/desktop/autoflip:run_autoflip

The build process fails with the following output:

ERROR: /home/deployer/mediapipe/mediapipe/examples/desktop/autoflip/quality/BUILD:303:1: C++ compilation of rule '//mediapipe/examples/desktop/autoflip/quality:visual_scorer' failed (Exit 1) gcc failed: error executing command /usr/bin/gcc -U_FORTIFY_SOURCE -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -g0 -O2 '-D_FORTIFY_SOURCE=1' -DNDEBUG -ffunction-sections ... (remaining 71 argument(s) skipped)

Use --sandbox_debug to see verbose messages from the sandbox
mediapipe/examples/desktop/autoflip/quality/visual_scorer.cc: In member function 'mediapipe::Status mediapipe::autoflip::VisualScorer::CalculateColorfulness(const cv::Mat&, float*) const':
mediapipe/examples/desktop/autoflip/quality/visual_scorer.cc:118:34: error: 'CV_RGB2HSV' was not declared in this scope
   cv::cvtColor(image, image_hsv, CV_RGB2HSV);
                                  ^
At global scope:
cc1plus: warning: unrecognized command line option '-Wno-ignored-attributes'
Target //mediapipe/examples/desktop/autoflip:run_autoflip failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 1294.448s, Critical Path: 1018.07s
INFO: 863 processes: 862 linux-sandbox, 1 local.
FAILED: Build did NOT complete successfully

In deciphering this error output I notice two things:

1. cc1plus: warning: unrecognized command line option '-Wno-ignored-attributes' I assume this isn't the source of the error since this is just a warning, although somewhat weird nonetheless
2. The following:
   error: 'CV_RGB2HSV' was not declared in this scope cv::cvtColor(image, image_hsv, CV_RGB2HSV);
   This looks like an OpenCV error. I must be using the wrong version of OpenCV. However, I made sure I followed the instructions in install.md, making use of the setup_opencv.sh file you provide so I shouldn't be having OpenCV install issues right? The output of the setup script is:

OpenCV has been built. You can find the header files and libraries in /usr/local/include/opencv4/opencv2 and /usr/local/lib
Modifying MediaPipe opencv config
Done

I made sure to edit WORKSPACE and opencv_linux.BUILD as described in install.md. They now look like this:
WORKSPACE

[irrelevant stuff]

new_local_repository(
    name = "linux_opencv",
    build_file = "@//third_party:opencv_linux.BUILD",
    path = "/usr/local",
)

[irrelevant stuff]

and
opencv_linux.BUILD

[other stuff]
cc_library(
    name = "opencv",
    srcs = glob(
        [
            "lib/libopencv_core.so",
            "lib/libopencv_calib3d.so",
            "lib/libopencv_features2d.so",
            "lib/libopencv_optflow.so*",
            "lib/libopencv_highgui.so",
            "lib/libopencv_imgcodecs.so",
            "lib/libopencv_imgproc.so",
            "lib/libopencv_video.so",
            "lib/libopencv_videoio.so",
        ],
    ),
   hdrs = glob(["include/opencv4/**/*.h*"]),
   includes = ["include/opencv4/"],
    linkstatic = 1,
    visibility = ["//visibility:public"],
)
[other stuff]

Could any kind soul please enlighten me on what I am doing wrong? I think either:

1. autoflip is not GPU enabled
2. I am not specifying the correct path to the correct installation of openCV in WORKSPACE and opencv_linux.BUILD, I just need to figure out which one is the correct one.

Thank you very much and apologies for the long issue

PS I have tried simply using Docker for this but it appears that it does not support GPU for AutoFlip, although I suppose that is a separate issue.

[thesofakillers]

I've tried running the build process with GPU disabled i.e.

bazel build -c opt --define MEDIAPIPE_DISABLE_GPU=1 mediapipe/examples/desktop/autoflip:run_autoflip

but unfortunately got the same error as above

ERROR: /home/deployer/mediapipe/mediapipe/examples/desktop/autoflip/quality/BUILD:303:1: C++ compilation of rule '//mediapipe/examples/desktop/autoflip/quality:visual_scorer' failed (Exit 1) gcc failed: error executing command /usr/bin/gcc -U_FORTIFY_SOURCE -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -g0 -O2 '-D_FORTIFY_SOURCE=1' -DNDEBUG -ffunction-sections ... (remaining 70 argument(s) skipped)

Use --sandbox_debug to see verbose messages from the sandbox
mediapipe/examples/desktop/autoflip/quality/visual_scorer.cc: In member function 'mediapipe::Status mediapipe::autoflip::VisualScorer::CalculateColorfulness(const cv::Mat&, float*) const':
mediapipe/examples/desktop/autoflip/quality/visual_scorer.cc:118:34: error: 'CV_RGB2HSV' was not declared in this scope
   cv::cvtColor(image, image_hsv, CV_RGB2HSV);
                                  ^
At global scope:
cc1plus: warning: unrecognized command line option '-Wno-ignored-attributes'
Target //mediapipe/examples/desktop/autoflip:run_autoflip failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 1275.143s, Critical Path: 962.83s
INFO: 880 processes: 879 linux-sandbox, 1 local.
FAILED: Build did NOT complete successfully

[thesofakillers]

I've found this which suggests a change from CV_RGB2HSV to cv::COLOR_RGB2HSV, but I'm not sure if that's the actual solution.

If it is, then surely we would have to edit visual_scorer.cc. I would go ahead and submit a pull request for this but I don't see anyone else having the same issue so I am hesitant to do so.

If you could let me know if you spot any mistakes in what I'm doing above, or if anyone is getting the same issue, please let me know. Thanks guys

[thesofakillers]

right so out of curiosity I made the change from from CV_RGB2HSV to cv::COLOR_RGB2HSV which eliminated the original error but now I get an almost identical error:

ERROR: /home/deployer/mediapipe/mediapipe/examples/desktop/autoflip/quality/BUILD:112:1: C++ compilation of rule '//mediapipe/examples/desktop/autoflip/quality:scene_cropping_viz' failed (Exit 1) gcc failed: error executing command /usr/bin/gcc -U_FORTIFY_SOURCE -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -g0 -O2 '-D_FORTIFY_SOURCE=1' -DNDEBUG -ffunction-sections ... (remaining 72 argument(s) skipped)

Use --sandbox_debug to see verbose messages from the sandbox
mediapipe/examples/desktop/autoflip/quality/scene_cropping_viz.cc: In function 'mediapipe::Status mediapipe::autoflip::DrawFocusPointAndCropWindow(const std::vector<cv::Mat>&, const std::vector<mediapipe::autoflip::FocusPointFrame>&, float, int, int, mediapipe::ImageFormat::Format, std::vector<std::unique_ptr<mediapipe::ImageFrame> >*)':
mediapipe/examples/desktop/autoflip/quality/scene_cropping_viz.cc:169:55: error: 'CV_FILLED' was not declared in this scope
         cv::circle(viz_mat, cv::Point(x, y), 3, kRed, CV_FILLED);
                                                       ^
At global scope:
cc1plus: warning: unrecognized command line option '-Wno-ignored-attributes'
Target //mediapipe/examples/desktop/autoflip:run_autoflip failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 1399.618s, Critical Path: 501.93s
INFO: 761 processes: 761 linux-sandbox.
FAILED: Build did NOT complete successfully

It's complaining about CV_FILLED just as it was for CV_RGB2HSV. After some googling I found this page where the answer claims that most "CV_ constants were changed to cv:: ones in 3.0.0". If this is the case, and AutoFlip requires opencv >= 3, why does AutoFlip continue using CV_ constants which are causing the issues I am getting?

Again, any help is appreciated here. Thanks guys

[jiuqiant]

Hi, AutoFlip should work with OpenCV 3. We didn't try it with OpenCV 4. Unfortunately, CV_FILLED is removed in OpenCV 4. But, you still have find it in OpenCV 3.4.9.

You can either upgrade all such old APIs to be OpenCV 4 compatible or downgrade the OpenCV version to be OpenCV 3. Hope this helps.

[thesofakillers]

Hi, thank you for the response. I confirm that downgrading from OpenCV 4 to OpenCV 3.4.9 fixes the issues. I should note that I had to run the following after building opencv 3.4.9 to get the bazel build command to work

$ sudo /bin/bash -c 'echo "/usr/local/lib" > /etc/ld.so.conf.d/opencv.conf'
$ sudo ldconfig

This is because I was getting

error while loading shared libraries: libopencv_core.so.3.4: cannot open shared object file: No such file or directory

Since it seems that OpenCV 4 causes issues, should we perhaps update either install.md or autoflip.md noting that openCV 3 is required and OpenCV4 will cause issues?

[mcclanahoochie]

Related https://github.com/google/mediapipe/issues/435#issuecomment-591037436

[jiuqiant]

Hi, thank you for the response. I confirm that downgrading from OpenCV 4 to OpenCV 3.4.9 fixes the issues. I should note that I had to run the following after building opencv 3.4.9 to get the bazel build command to work

$ sudo /bin/bash -c 'echo "/usr/local/lib" > /etc/ld.so.conf.d/opencv.conf'
$ sudo ldconfig

This is because I was getting

error while loading shared libraries: libopencv_core.so.3.4: cannot open shared object file: No such file or directory

Since it seems that OpenCV 4 causes issues, should we perhaps update either install.md or autoflip.md noting that openCV 3 is required and OpenCV4 will cause issues?

Thanks for the feedback. We will have a note for this issue in the coming release.