Apollo: Log Design in 3.5

[Fla3inH0tCheet0s]

There's been a lot of changes for logging from 3.0 to 3.5. I had a lot of questions regarding the design and configuration of logging in 3.5.

1. Was it intentional to have the minloglevel as part of the symlink name for latest log per module? The output from glog is /apollo/data/MODULE_NAME.MINLOGLEVEL, but I'm not sure that the MINLOGLEVEL part should be there.

2. I noticed that GLOG_alsologtostderr=0 in cyber/setup.bash. This means that DEBUG, INFO, and WARN will not be output to terminal or redirected to nohup.out. By specifying this glog parameter, there are essentially two log levels. The C++ cyber examples do not print anything to terminal, unless GLOG_alsologtostderr=1 is passed in before bazel-bin/cyber/examples/talker or bazel-bin/cyber/examples/listener. Was this intentional?

3. Most scripts and dreamview start programs through nohup and redirect the output to a single nohup.out file in the root directory. Was it meant to go in data/log? Also, was it meant for all of the output filenames to be the same?

4. I wasn't sure what module means in context of logging. It is set by copts = ["-DMODULE_NAME=\\\"module_name\\\""], in the BUILD file. It can be confusing, because common, compiled libraries will have the same module_name. In compiled libraries it is similar to __FUNCTION__, __LINE__, and __FILE__. Could those be used instead? If the header file has inline function that log, it will have different MODULE_NAME definitions. If a cyber program crashes, it is not obvious what the FATAL log file will be named, since it doesn't correlate to program, component, or dag name. If headers are precompiled, this macro will also slow down compilation. Also, the nohup.out will have text from other programs so a FATAL message will not be at the end of it. Should it use the dag name instead?

[baowanyu]

Answer the question No.1:
Yes, we followed the link name generation rules of glog.
In glog project: https://github.com/google/glog/blob/master/src/logging.cc#L932
In apollo project: https://github.com/ApolloAuto/apollo/blob/master/cyber/logger/log_file_object.cc#L171

[baowanyu]

Answer the question No.2:
The "cyber/setup.bash" file contains some default environment variables. In most cases, we don't want to log to standard error because this will increase the system load. So we just default output the error log. You can customize the "cyber/setup.bash" file and set GLOG_alsologtostderr to 0.

[baowanyu]

Answer the question No.4:
As you pointed out，the MODULE_NAME macro variable to split the log file is not a best practice.
We are developing new logging features, like split the log file according to component name or module name defined in dag file.

[xiaoxq]

3. Most scripts and dreamview start programs through nohup and redirect the output to a single nohup.out file in the root directory. Was it meant to go in data/log? Also, was it meant for all of the output filenames to be the same?

Nope, the nohup.out should be dummy and empty in most of the time, such as path not exist which is almost impossible if your docker container is clean. Actual program logs should be well managed by CyberRT and located at data/log/<module>.INFO, WARNING, ERROR, etc

[Fla3inH0tCheet0s]

Thank you for your responses.

Nope, the nohup.out should be dummy and empty in most of the time, such as path not exist which is almost impossible if your docker container is clean.

I'm not sure I understand why it should be empty. Because cyber/setup.bash sets GLOG_alsologtostderr=0 and GLOG_minloglevel=0, the programs should be still output AERROR and AFATAL messages to stderr. Then, dreamview uses nohup to redirect the output to nohup.out. Is this file meant to be ignored?

On a slightly different topic, why was the async logger glog sink developed and maintained? Will cyber implement its own logging stack? It seems that other loggers have this feature. For example, there is spdlog.

[baowanyu]

On a slightly different topic, why was the async logger glog sink developed and maintained? Will cyber implement its own logging stack? It seems that other loggers have this feature. For example, there is spdlog.

We just customized our Logger, glog's original LogSink function is not affected.
For forward compatibility considerations, we decided to customize our logger on glog instead of rewriting a log framework or using other frameworks like spdlog.