Apollo: Building apollo 3.5 or higher often causes the OS stuck

[davidhopper2003]

I downloaded the latest Apollo master branch and built it with the bash apollo.sh build command. I found that the OS is often stuck and it is useless to press the Ctrl + Alt + F1 key to open a text terminal. I had to power off the computer and restart it. After the restarting, there was still a stuck phenomenon. What's the matter with the apollo.sh file? Any suggestions will be appreciated.

The resource consumption built with 4 jobs (bash apollo.sh build -j 4) is shown as follows:

System information

• OS Platform and Distribution (e.g., Linux Ubuntu 14.04): Linux Ubuntu 16.04
• Apollo installed from (source or binary): source
• Apollo version (1.0, 1.5, 2.0, 2.5, 3.0): 3.5 or higher
• CPU cores: 8
• Memory: 8 GB

[martins-mozeiko]

It looks like you are running out of memory. Find --jobs=$(nproc) in apollo.sh file and replace it with --jobs=2. This will make build process to use only 2 cores. Building will be longer, but will use less memory.

[davidhopper2003]

@martins-mozeiko Thank you. Your approach is absolutely correct. My question is as follows:
It wouldn't run out of memory with the bash apollo.sh build command prior to apollo 3.0. Why does this problem occurred with apollo 3.5 or higher?

[davidhopper2003]

@martins-mozeiko I figured it out. Bazel consumed too much memory. After I add an 8 GB memory card, the GUI will never freeze. Therefore, it's best to build Apollo 3.5 or higher with two tasks on a computer with less than 16 GB of memory:

bash apollo.sh build -j 2

[gengqx]

To build apollo,it would be better use at least 8GB virtual memory, only 977M virtual memery may also cause a problem. Especially when use our own docker image.

[martins-mozeiko]

@daviduhm exactly. I have 64-core machine. And building Apollo with default settings hangs for me every time because it uses too much memory. Using swap file is terrible option because it slows down everything to many hours of building. I need to reduce how many cores it uses to 20 or so to be able to build Apollo successfully in reasonable time.

[davidhopper2003]

@gengqx Sure. Thanks.

[davidhopper2003]

@martins-mozeiko Thank you for detailed information.

[natashadsouza]

Closing this issue as it appears to be resolved. @martins-mozeiko @gengqx thank you for the explanation @davidhopper2003 thank you for posting your fix to help others developers who may encounter this issue. Added your suggestions to #7785

[Iqbalparvi]

I have the same issue but my system hangs out and no progress at all when running bash apollo.sh build at compiling stage looks like as ........
@in_dev_docker:/apollo$ bash apollo.sh build
System check passed. Build continue ...
[WARNING] ESD CAN library supplied by ESD Electronics does not exist. If you need ESD CAN, please refer to third_party/can_card_library/esd_can/README.md.
Running build under GPU mode. GPU is required to run the build.
[INFO] Start building, please wait ...
INFO: Reading 'startup' options from /apollo/tools/bazel.rc: --batch_cpu_scheduling --host_jvm_args=-XX:-UseParallelGC
____Loading package: modules/localization/msf/local_tool/local_visualization/online_visual
____Loading package: modules/planning/tasks/optimizers
____Loading package: modules/prediction/network/cruise_model
____Loading package: modules/perception/proto
____Loading package: modules/tools/prediction/fake_prediction
____Loading package: modules/planning/math
____Loading package: modules/third_party_perception/proto
____Loading package: modules/prediction/predictor
____Loading package: modules/monitor/hardware
____Loading package: modules/drivers/canbus
____Loading package: modules/perception/camera/tools/offline
____Loading package: modules/drivers/velodyne/parser
____Loading package: cyber
____Loading package: cyber/timer
____Loading package: cyber/transport
____Loading package: cyber/component
____Loading package: cyber/data
____Loading package: cyber/examples/common_component_example
[INFO] Building on x86_64...
[INFO] Building with --jobs=4 --ram_utilization_factor 80 for x86_64
INFO: Reading 'startup' options from /apollo/tools/bazel.rc: --batch_cpu_scheduling --host_jvm_args=-XX:-UseParallelGC
INFO: (04-15 05:26:16.751) Found 3240 targets...
Slow read: a 1729-byte read from /apollo/modules/control/common/pid_BC_controller.h took 14512ms.
[4,604 / 4,613] (04-15 05:51:27.122) Compiling modules/drivers/gnss/proto/gnss\
_best_pose.pb.cc

kindly any sugestions.......i cant make progress from here ...

[kttnk]

@Iqbalparvi --jobs=4 depends on your core(s).
If

$nproc

gives you 4, then --jobs=2 might work ( my guess).

[lemketron]

@martins-mozeiko wrote:

It looks like you are running out of memory. Find --jobs=$(nproc) in apollo.sh file and replace it with --jobs=2. This will make build process to use only 2 cores. Building will be longer, but will use less memory.

I have been experiencing this as well on a 12-core Razer Blade with 16GB of RAM, and have been experimenting with the build settings. I tried building with nproc/2 (6 cores) and also with 9 and with 10 and so far it seems to be doing ok. I'm inclined to think that (at least with 16GB of RAM) a better value is nproc-2. On my system this seems to avoid running out of memory, while taking advantage of 10 of my 12 cores.

Another issue that might have helped is that I decreased ram_utilitization_factor from 80 to 70. I'm not entirely sure whether dropping the cores or ram utilization helped more but together this lets my build succeed without failing.

I'm curious if @davidhopper2003 or anyone else with an 8GB system (or one that fails to build apollo successfully) could try nproc-2 (and ram_utilization_factor=70) and see how that goes?

My JOB_ARG line in apollo.sh now looks like this:
JOB_ARG="--jobs=$(expr $(nproc) - 2 ) --ram_utilization_factor 70"

It would be nice to change the apollo.sh script to be more generally successful but I wouldn't want to slow down builds that aren't crashing.