Apollo: cyber_recorder play bugs: error info

[westeast]

Describe the bug

root@in_dev_docker:/apollo# cyber_recorder play -f ./docs/demo_guide/demo_3.5.record
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0114 12:21:50.352372   833 global_data.cc:153] [cyber_recorder] host ip: 10.99.204.244
file: ./docs/demo_guide/demo_3.5.record, chunk_number: 3, begin_time: 1546888377338834894, end_time: 1546888422886740928, message_number: 61615
earliest_begin_time: 1546888377338834894, latest_end_time: 1546888422886740928, total_msg_num: 61615

Please wait for loading...
Hit Ctrl+C to stop, Space to pause, or 's' to step.

[RUNNING] Record Time: 0.000    Progress: 0.000 / 45.548E0114 12:21:53.566558   867 segment.cc:305] [cyber_recorder] remove shm failed, error code: No such file or directory
E0114 12:21:53.567346   867 segment.cc:305] [cyber_recorder] remove shm failed, error code: No such file or directory
E0114 12:21:53.567440   867 segment.cc:156] [cyber_recorder] create shm failed, error code: No such file or directory
E0114 12:21:53.567926   867 segment.cc:62] [cyber_recorder] segment update failed.
E0114 12:21:53.568365   867 shm_transmitter.h:110] [cyber_recorder] acquire block failed.
[RUNNING] Record Time: 1546888377.433    Progress: 0.094 / 45.548E0114 12:21:53.669898   867 segment.cc:305] [cyber_recorder] remove shm failed, error code: No such file or directory
E0114 12:21:53.670327   867 segment.cc:305] [cyber_recorder] remove shm failed, error code: No such file or directory
E0114 12:21:53.670640   867 segment.cc:156] [cyber_recorder] create shm failed, error code: No such file or directory
E0114 12:21:53.670897   867 segment.cc:47] [cyber_recorder] init failed, can't write now.
E0114 12:21:53.671157   867 shm_transmitter.h:110] [cyber_recorder] acquire block failed.

To Reproduce
Steps to reproduce the behavior:

1. cyber_recorder play -f ./docs/demo_guide/demo_3.5.record

Expected behavior
no error log

Screenshots

Desktop (please complete the following information):

• OS: [e.g. iOS]
• Browser [e.g. chrome, safari]
• Version [e.g. 22]

other log

data/log/cyber_recorder

E0114 19:43:03.312453 22561 segment.cc:47]  init failed, can't write now.
E0114 19:43:03.312475 22561 shm_transmitter.h:110]  acquire block failed.
I0114 19:43:03.413904 22561 segment.cc:143]  need larger space, recreate.
E0114 19:43:03.413960 22561 segment.cc:305]  remove shm failed, error code: No such file or directory
I0114 19:43:03.414022 22561 segment.cc:143]  need larger space, recreate.
E0114 19:43:03.414031 22561 segment.cc:305]  remove shm failed, error code: No such file or directory
E0114 19:43:03.414065 22561 segment.cc:156]  create shm failed, error code: No such file or directory
E0114 19:43:03.414096 22561 segment.cc:47]  init failed, can't write now.
E0114 19:43:03.414121 22561 shm_transmitter.h:110]  acquire block failed.
I0114 19:43:03.515285 22561 segment.cc:143]  need larger space, recreate.
E0114 19:43:03.515337 22561 segment.cc:305]  remove shm failed, error code: No such file or directory
I0114 19:43:03.515385 22561 segment.cc:143]  need larger space, recreate.
E0114 19:43:03.515393 22561 segment.cc:305]  remove shm failed, error code: No such file or directory
E0114 19:43:03.515422 22561 segment.cc:156]  create shm failed, error code: No such file or directory
E0114 19:43:03.515449 22561 segment.cc:47]  init failed, can't write now.
E0114 19:43:03.515475 22561 shm_transmitter.h:110]  acquire block failed.
I0114 19:43:03.604816 22561 segment.cc:143]  need larger space, recreate.
E0114 19:43:03.604858 22561 segment.cc:305]  remove shm failed, error code: No such file or directory
I0114 19:43:03.604898 22561 segment.cc:143]  need larger space, recreate.
E0114 19:43:03.604905 22561 segment.cc:305]  remove shm failed, error code: No such file or directory
E0114 19:43:03.604925 22561 segment.cc:156]  create shm failed, error code: No such file or directory
E0114 19:43:03.604948 22561 segment.cc:47]  init failed, can't write now.
E0114 19:43:03.604972 22561 shm_transmitter.h:110]  acquire block failed.
I0114 19:43:03.713138 22561 segment.cc:143]  need larger space, recreate.
E0114 19:43:03.713183 22561 segment.cc:305]  remove shm failed, error code: No such file or directory
I0114 19:43:03.713225 22561 segment.cc:143]  need larger space, recreate.
E0114 19:43:03.713233 22561 segment.cc:305]  remove shm failed, error code: No such file or directory
E0114 19:43:03.713259 22561 segment.cc:156]  create shm failed, error code: No such file or directory
E0114 19:43:03.713279 22561 segment.cc:47]  init failed, can't write now.
E0114 19:43:03.713296 22561 shm_transmitter.h:110]  acquire block failed.
I0114 19:43:03.800932 22561 segment.cc:143]  need larger space, recreate.
E0114 19:43:03.800969 22561 segment.cc:305]  remove shm failed, error code: No such file or directory
I0114 19:43:03.801066 22561 segment.cc:143]  need larger space, recreate.
E0114 19:43:03.801468 22561 segment.cc:305]  remove shm failed, error code: No such file or directory
E0114 19:43:03.801506 22561 segment.cc:156]  create shm failed, error code: No such file or directory
E0114 19:43:03.801558 22561 segment.cc:47]  init failed, can't write now.
E0114 19:43:03.801595 22561 shm_transmitter.h:110]  acquire block failed.
I0114 19:43:03.902107 22561 segment.cc:143]  need larger space, recreate.
E0114 19:43:03.902154 22561 segment.cc:305]  remove shm failed, error code: No such file or directory
I0114 19:43:03.902209 22561 segment.cc:143]  need larger space, recreate.
E0114 19:43:03.902216 22561 segment.cc:305]  remove shm failed, error code: No such file or directory
E0114 19:43:03.902247 22561 segment.cc:156]  create shm failed, error code: No such file or directory
E0114 19:43:03.902278 22561 segment.cc:47]  init failed, can't write now.
E0114 19:43:03.902302 22561 shm_transmitter.h:110]  acquire block failed.

[westeast]

I write some log code and print more logs and find at the time ,the id_ is negative, so maybe there is a bugs when play some record:

E0114 20:15:00.058220 14607 segment.cc:46] [cyber_recorder] id_-1027902123
E0114 20:15:00.058369 14607 segment.cc:306] [cyber_recorder] remove shm failed, error code: No such file or directory
E0114 20:15:00.058425 14607 segment.cc:306] [cyber_recorder] remove shm failed, error code: No such file or directory
E0114 20:15:00.058459 14607 segment.cc:157] [cyber_recorder] create shm failed, error code: No such file or directory
E0114 20:15:00.058492 14607 segment.cc:48] [cyber_recorder] init failed, can't write now.

[westeast]

dreamview error log:

[/apollo/bazel-bin/modules/dreamview/dreamview --flagfile=/apollo/modules/common/data/global_flagfile.txt]  E0114 14:17:29.248551 21861 hdmap_common.cc:184] [dreamview] lane[id = id: "11267dup0_1_-1"
[/apollo/bazel-bin/modules/dreamview/dreamview --flagfile=/apollo/modules/common/data/global_flagfile.txt]  ]. sampled_right_width_[0.976227] is too small. It should be larger than half vehicle width[1.05].
[/apollo/bazel-bin/modules/dreamview/dreamview --flagfile=/apollo/modules/common/data/global_flagfile.txt]  E0114 14:17:29.248565 21861 hdmap_common.cc:184] [dreamview] lane[id = id: "11267dup0_1_-1"
[/apollo/bazel-bin/modules/dreamview/dreamview --flagfile=/apollo/modules/common/data/global_flagfile.txt]  ]. sampled_right_width_[0.991181] is too small. It should be larger than half vehicle width[1.05].
[/apollo/bazel-bin/modules/dreamview/dreamview --flagfile=/apollo/modules/common/data/global_flagfile.txt]  E0114 14:17:29.248584 21861 hdmap_common.cc:184] [dreamview] lane[id = id: "11267dup0_1_-1"
[/apollo/bazel-bin/modules/dreamview/dreamview --flagfile=/apollo/modules/common/data/global_flagfile.txt]  ]. sampled_right_width_[1.00614] is too small. It should be larger than half vehicle width[1.05].
[/apollo/bazel-bin/modules/dreamview/dreamview --flagfile=/apollo/modules/common/data/global_flagfile.txt]  E0114 14:17:29.248598 21861 hdmap_common.cc:184] [dreamview] lane[id = id: "11267dup0_1_-1"
[/apollo/bazel-bin/modules/dreamview/dreamview --flagfile=/apollo/modules/common/data/global_flagfile.txt]  ]. sampled_right_width_[1.0211] is too small. It should be larger than half vehicle width[1.05].
[/apollo/bazel-bin/modules/dreamview/dreamview --flagfile=/apollo/modules/common/data/global_flagfile.txt]  E0114 14:17:29.248618 21861 hdmap_common.cc:184] [dreamview] lane[id = id: "11267dup0_1_-1"
[/apollo/bazel-bin/modules/dreamview/dreamview --flagfile=/apollo/modules/common/data/global_flagfile.txt]  ]. sampled_right_width_[1.03606] is too small. It should be larger than half vehicle width[1.05].

[westeast]

@quning78 can you give me some advice about fixing this error? thanks

[snuffysasa]

We have the same issue, happens when playing cyber_recorder play on the Apollo sample bags that have been converted to cyber .record files.

It does not happen on the smaller bags in mb, but does happen in the larger bags

[westeast]

We have the same issue, happens when playing cyber_recorder play on the Apollo sample bags that have been converted to cyber .record files.

It does not happen on the smaller bags in mb, but does happen in the larger bags

where can I found the smaller bags? I found only one cyber .record file in the repo

[zhongjunni]

@quning78 can you give me some advice about fixing this error? thanks

Can you run the following script to clear shared memory before playing record?

for id in `ipcs | grep 0x | awk '{print $1}'`; do ipcrm -M $id >/dev/null 2>&1; done

[westeast]

@quning78 can you give me some advice about fixing this error? thanks

Can you run the following script to clear shared memory before playing record?

for id in ipcs | grep 0x | awk '{print $1}'; do ipcrm -M $id >/dev/null 2>&1; done

why my all key is 0x00000

and the keys are invilad

root@in_dev_docker:/apollo# ipcrm -M 0x00000000
ipcrm: illegal key (0x00000000)

[zhongjunni]

@quning78 can you give me some advice about fixing this error? thanks

Can you run the following script to clear shared memory before playing record?
for id in ipcs | grep 0x | awk '{print $1}'; do ipcrm -M $id >/dev/null 2>&1; done

why my all key is 0x00000

and the keys are invilad

root@in_dev_docker:/apollo# ipcrm -M 0x00000000
ipcrm: illegal key (0x00000000)

Hi, the key is zero and the status is 'dest'.
Such information means that the shared memory segments have been marked to be destroyed.
However, there are still other threads attached to it. (nattach is 2)
So, maybe you need to kill these threads firstly.

[bjtulynn]

I have encountered the same problem, the following is my method to fix the issue:

1. check the size of kernel.shmmax and kernel.shmall, which limits separately single segment size of shared memory per process and all segments size of shared memory for all processes.
   sudo sysctl -a | grep -E "shmall|shmmax"
   we can get the output:
   kernel.shmall = xxxxxxxx
kernel.shmmax = xxxxxxxx

1. adjust/enlarge kernel.shmmax value, simply we can set same value for kernel.shmmax, according to kernel.shmall
   sudo vim /etc/sysctl.conf
   we can adjust/enlarge:
   kernel.shmmax = xxxxxxxx

2. reboot the machine, and try again

Actually, the error is not related with the size of the bag/record, it is caused when expected shared memory size is larger than limit shared memory size. Because when the size of the single message is large, the writer needs larger shared memory size to transport the message efficiently and smoothly.

[swdev1202]

I was able to change (or add) kernel.shmmax in the /etc/sysctl.conf.

However, it's still throwing the error.

Any other suggestions?

[bjtulynn]

I was able to change (or add) kernel.shmmax in the /etc/sysctl.conf.

However, it's still throwing the error.

Any other suggestions?

Can you post the result of the command "sudo sysctl -a | grep -E "shmall|shmmax""?

Sometimes kernel.shmmax the system is using is not in the /etc/sysctl.conf, especially when we set/unset or adjust our system variables environment.

[quning18]

@swdev1202

have you run "sysctl -p" after modifying the sysctl.conf file?

[mickeyouyou]

A reference output of the command sudo sysctl -a | grep -E "shmall|shmmax"

apollo@in_dev_docker:/apollo$ sudo sysctl -a | grep -E "shmall|shmmax"
kernel.shmall = 2097152
kernel.shmmax = 33554432

kernel.shmmax not in /etc/sysctl.conf, I need to add this field to this file.

after tried to restart docker container and machine , still don't work.

[zhongjunni]

A reference output of the command sudo sysctl -a | grep -E "shmall|shmmax"

apollo@in_dev_docker:/apollo$ sudo sysctl -a | grep -E "shmall|shmmax"
kernel.shmall = 2097152
kernel.shmmax = 33554432

kernel.shmmax not in /etc/sysctl.conf, I need to add this field to this file.

after tried to restart docker container and machine , still don't work.

Hi, the unit of kernel.shmmax is byte, if you use the value 33554432, it is only 32MB.
You can set kernel.shmmax to 2147483648 (2GB), then try again.
Thank you!

[mickeyouyou]

A reference output of the command sudo sysctl -a | grep -E "shmall|shmmax"

apollo@in_dev_docker:/apollo$ sudo sysctl -a | grep -E "shmall|shmmax"
kernel.shmall = 2097152
kernel.shmmax = 33554432

kernel.shmmax not in /etc/sysctl.conf, I need to add this field to this file.
after tried to restart docker container and machine , still don't work.

Hi, the unit of kernel.shmmax is byte, if you use the value 33554432, it is only 32MB.
You can set kernel.shmmax to 2147483648 (2GB), then try again.
Thank you!

It works! If you meet the same problems, refer this solution please.