Apollo: Missing osqp/osqp.h when BUILD

[Arzkaban]

System information

• Linux Ubuntu 20.04:
• Apollo installed from source:
• Apollo version 6.0:

Steps to reproduce the issue:

• In docker, "./apollo.sh build"

ERROR: /apollo/modules/common/math/BUILD:163:11: C++ compilation of rule '//modules/common/math:mpc_osqp' failed (Exit 1) crosstool_wrapper_driver_is_not_gcc failed: error executing command external/local_config_cuda/crosstool/clang/bin/crosstool_wrapper_driver_is_not_gcc -MD -MF bazel-out/k8-fastbuild/bin/modules/common/math/_objs/mpc_osqp/mpc_osqp.pic.d ... (remaining 54 argument(s) skipped)

Use --sandbox_debug to see verbose messages from the sandbox In file included from modules/common/math/mpc_osqp.cc:17:0: ./modules/common/math/mpc_osqp.h:26:10: fatal error: osqp/osqp.h: No such file or directory
#include "osqp/osqp.h"
^~~~~~~~~~~~~
compilation terminated. INFO: Elapsed time: 11.142s, Critical Path: 10.57s INFO: 64 processes: 64 processwrapper-sandbox.

Do I miss any Environment Variable, make these files can't be found. Cause osqp/osqp.h is not the only one that can't be found.

And here is the Configuration
[INFO] Apollo Environment Settings:
[INFO] APOLLO_ROOT_DIR: /apollo
[INFO] APOLLO_CACHE_DIR: /apollo/.cache
[INFO] APOLLO_IN_DOCKER: true
[INFO] APOLLO_VERSION: r6.0.0-2020-10-26-501e5f6a4b
[INFO] DOCKER_IMG: dev-x86_64-18.04-20200914_0742
[INFO] APOLLO_ENV: STAGE=dev USE_ESD_CAN=false
[INFO] USE_GPU: USE_GPU_HOST=1 USE_GPU_TARGET=1
[INFO] Configure .bazelrc in non-interactive mode
You have bazel 3.4.1 installed.
Found possible Python library paths:
/usr/lib/python3/dist-packages
/usr/local/lib/python3.6/dist-packages
Found CUDA 10.2 in:
/usr/local/cuda-10.2/targets/x86_64-linux/lib
/usr/local/cuda-10.2/targets/x86_64-linux/include
Found cuDNN 7 in:
/usr/lib/x86_64-linux-gnu
/usr/include
Found TensorRT 7 in:
/usr/lib/x86_64-linux-gnu
/usr/include/x86_64-linux-gnu

[ OK ] Successfully configured .bazelrc in non-interactive mode.

[Arzkaban]

any extra configuration or dependency needed?
osqp is not the only things missing.

#include "fastrtps/Domain.h"
^~~~~~~

[storypku]

Please type ./apollo.sh config and paste the output message below.

[Arzkaban]

Please type ./apollo.sh config and paste the output message below.

Here is configuration:

[INFO] Apollo Environment Settings:
[INFO] APOLLO_ROOT_DIR: /apollo
[INFO] APOLLO_CACHE_DIR: /apollo/.cache
[INFO] APOLLO_IN_DOCKER: true
[INFO] APOLLO_VERSION: r6.0.0-2020-10-26-501e5f6a4b
[INFO] DOCKER_IMG: dev-x86_64-18.04-20200914_0742
[INFO] APOLLO_ENV: STAGE=dev USE_ESD_CAN=false
[INFO] USE_GPU: USE_GPU_HOST=1 USE_GPU_TARGET=1
[INFO] Configure .bazelrc in non-interactive mode
You have bazel 3.4.1 installed.
Found possible Python library paths:
/usr/lib/python3/dist-packages
/usr/local/lib/python3.6/dist-packages
Found CUDA 10.2 in:
/usr/local/cuda-10.2/targets/x86_64-linux/lib
/usr/local/cuda-10.2/targets/x86_64-linux/include
Found cuDNN 7 in:
/usr/lib/x86_64-linux-gnu
/usr/include
Found TensorRT 7 in:
/usr/lib/x86_64-linux-gnu
/usr/include/x86_64-linux-gnu

[ OK ] Successfully configured .bazelrc in non-interactive mode.
[INFO] Usage: /apollo/scripts/apollo_config.sh [Options]
[INFO] Options:
[INFO] -i|--interactive Run in interactive mode
[INFO] -n|--noninteractive Run in non-interactive mode
[INFO] -h|--help Show this message and exit

[storypku]

Now that .bazelrc has been setup, does the build still fails? You can try to remove bazel cached output by running ./apollo.sh clean --bazel.

[Arzkaban]

Now that .bazelrc has been setup, does the build still fails? You can try to remove bazel cached output by running ./apollo.sh clean --bazel.

After ./apollo.sh clean --bazel, the build still fails with similar error.

./cyber/class_loader/utility/class_loader_utility.h:31:10: fatal error: Poco/SharedLibrary.h: No such file or directory

And I look into the folder, trying to find this file manully. In the folder ./.cache/bazel/540135163923.../external/poco

The file named 'SharedLibrary.h' doesn't exist.
Or any other diraction, should I look into to find this 'SharedLibrary.h'?

[storypku]

What after removing the whole /apollo/.cache/bazel directory ?

[storypku]

What after removing the whole /apollo/.cache/bazel directory ?
I have typed ./apollo.sh config --noninteractive and rerun ./apollo.sh build, everything seems OK?

[Arzkaban]

What after removing the whole /apollo/.cache/bazel directory ?
I have typed ./apollo.sh config --noninteractive and rerun ./apollo.sh build, everything seems OK?

It doesn't success. But I got some progress that it seems some downloading are failed, I highlight the part of ERROR massages below:

WARNING: Download from https://apollo-platform-system.cdn.bcebos.com/archive/6.0/3.3.7.tar.gz failed: class com.google.devtools.build.lib.bazel.repository.downloader.UnrecoverableHttpException Checksum was 980c7ad361efe6659f03038f24ed238029741a79fc5d969407f2cb45796a6670 but wanted a8d87c8df67b0404e97bcef37faf3b140ba467bc060e2b883192165b319cea8d
WARNING: Download from https://apollo-platform-system.cdn.bcebos.com/archive/6.0/v3.8.0.tar.gz failed: class com.google.devtools.build.lib.bazel.repository.downloader.UnrecoverableHttpException Checksum was 61f52125d27967b624fe1a095fe74f00b9d6768882f87cc7d815593505a151ee but wanted 7d0edf65f2ac7390af5e5a0b323b31202a6c11d744a74b588dc30f5a8c9865ba
INFO: Analyzed 4931 targets (606 packages loaded, 16625 targets configured).
INFO: Found 4931 targets...
ERROR: /apollo/modules/common/math/BUILD:163:11: C++ compilation of rule '//modules/common/math:mpc_osqp' failed (Exit 1) crosstool_wrapper_driver_is_not_gcc failed: error executing command external/local_config_cuda/crosstool/clang/bin/crosstool_wrapper_driver_is_not_gcc -MD -MF bazel-out/k8-fastbuild/bin/modules/common/math/_objs/mpc_osqp/mpc_osqp.pic.d ... (remaining 54 argument(s) skipped)

Use --sandbox_debug to see verbose messages from the sandbox
In file included from modules/common/math/mpc_osqp.cc:17:0:
./modules/common/math/mpc_osqp.h:26:10: fatal error: osqp/osqp.h: No such file or directory
#include "osqp/osqp.h"
^~~~~
compilation terminated.
INFO: Elapsed time: 1467.386s, Critical Path: 30.25s
INFO: 210 processes: 210 processwrapper-sandbox.

I tried a few times, it always failed to download these .tar.gz files.
How can I fix this, if the connection for download can not be reach

P.S.
I can download these '.tar.gz' with this link. but I don't know why ./apollo.sh always failed to downloading.
Is there any way to bypass this, just copy downloaded files to some specific direction?
here is the screensnap of these downloaded files:

[storypku]

@lx18233184051 Please help with the apollo-platform-system.cdn.bcebos.com issue in r6.0.0 release, since the URL may be invalidated after switching to apollo-system.cnd.bcebos.com.

[storypku]

What after removing the whole /apollo/.cache/bazel directory ?
I have typed ./apollo.sh config --noninteractive and rerun ./apollo.sh build, everything seems OK?

It doesn't success. But I got some progress that it seems some downloading are failed, I highlight the part of ERROR massages below:

Yep, in Apollo r6.0.0, you can put every failed package in /apollo/.cache/distdir. For example,

wget https://github.com/eigenteam/eigen-git-mirror/archive/3.3.7.tar.gz
mv 3.3.7.tar.gz /apollo/.cache/distdir/

[Arzkaban]

What after removing the whole /apollo/.cache/bazel directory ?
I have typed ./apollo.sh config --noninteractive and rerun ./apollo.sh build, everything seems OK?

It doesn't success. But I got some progress that it seems some downloading are failed, I highlight the part of ERROR massages below:

Yep, in Apollo r6.0.0, you can put every failed package in /apollo/.cache/distdir. For example,

wget https://github.com/eigenteam/eigen-git-mirror/archive/3.3.7.tar.gz
mv 3.3.7.tar.gz /apollo/.cache/distdir/

It rise another WARNING, and it seems should be the core reason.
After typing ./apollo.sh clean --bazel, the ./.bazelrc will be cleaned.
And maybe due to this issue: Bazel:Issue 6319
It give the following Error:

WARNING: The following rc files are no longer being read, please transfer their contents or import their path into one of the standard rc files:
/apollo/tools/bazel.rc
INFO: Analyzed 4931 targets (0 packages loaded, 0 targets configured).
INFO: Found 4931 targets...
ERROR: /apollo/cyber/service_discovery/BUILD:29:11: C++ compilation of rule '//cyber/service_discovery:participant_listener' failed (Exit 1) crosstool_wrapper_driver_is_not_gcc failed: error executing command external/local_config_cuda/crosstool/clang/bin/crosstool_wrapper_driver_is_not_gcc -MD -MF bazel-out/k8-fastbuild/bin/cyber/service_discovery/_objs/participant_listener/participant_listener.pic.d ... (remaining 53 argument(s) skipped)

Use --sandbox_debug to see verbose messages from the sandbox
In file included from cyber/service_discovery/communication/participant_listener.cc:17:0:
./cyber/service_discovery/communication/participant_listener.h:23:10: fatal error: fastrtps/Domain.h: No such file or directory
#include "fastrtps/Domain.h"
^~~~~~~

But everytime I run ./apollo.sh build the file './.bazelrc' will be covered, I can not use the method described in 'Bazel:Issue 6319' to fix it (copy the content of '/apollo/tools/bazel.rc' on the top line of './.bazelrc').
Any recommended solution?

[storypku]

It doesn't matter. .bazelrc file in apollo root directory "import" settings from tools/bazel.rc. As can be seen below:

https://github.com/ApolloAuto/apollo/blob/r6.0.0/.bazelrc

[Arzkaban]

It doesn't matter. .bazelrc file in apollo root directory "import" settings from tools/bazel.rc. As can be seen below:

https://github.com/ApolloAuto/apollo/blob/r6.0.0/.bazelrc

Thanks for your assistance. The Problem has solved.
My apollo source code was download in the Windows environment, then the GIT will automatically using CRLF at end of the line.
So I do dos2unix transform, for the whole APOLLO folder.
But obviously, there is some unknown error with this transform.
I fetching the repository again after git config --global core.autocrlf false.
Everything goes back to normal.

[storypku]

It doesn't matter. .bazelrc file in apollo root directory "import" settings from tools/bazel.rc. As can be seen below:
https://github.com/ApolloAuto/apollo/blob/r6.0.0/.bazelrc

Thanks for your assistance. The Problem has solved.
My apollo source code was download in the Windows environment, then the GIT will automatically using CRLF at end of the line.
So I do dos2unix transform, for the whole APOLLO folder.
But obviously, there is some unknown error with this transform.
I fetching the repository again after git config --global core.autocrlf false.
Everything goes back to normal.

Glad we can help. Please feel free to create issues if you have questions when using Apollo!