Apollo: Planning module failed to update reference line after rerouting

[EricWu416]

System information

• OS Platform and Distribution (e.g., Linux Ubuntu 18.04): Linux Ubuntu 18.04
• Apollo installed from (source or binary): Source
• Apollo version (3.5, 5.0, 5.5, 6.0): 5.5

Steps to reproduce the issue:

I would like to test sim control (mainly the planning module) by playing the record file containing messages of type apollo.perception.PerceptionObstacles, apollo.perception.TrafficLightDetection, and apollo.prediction.PredictionObstacles. Before playing the record file, I already enabled sim control, the planning module, the perception module, and the routing module.
However, there is no guarantee that the planning module can always produce trajectories. The planning module can successfully produce trajectories initially most of the time, but it will start producing messages containing the error "Failed to updated reference line after rerouting" after several iterations, each sends a routing request and then plays the record file. Also, once this error occurred, the planning module will continue to produce messages with the same error for the subsequent iterations.

• enable sim control
• enable the planning, routing, and perception module
• repeat the following steps:
  
  
  
  • send routing request
  
  
  • play the record file that only contains apollo.perception.PerceptionObstacles, apollo.perception.TrafficLightDetection, and apollo.prediction.PredictionObstacles messages using cyber_recorder play
  
  

Any help or recommendation is much appreciated! Thanks.

Supporting materials (screenshots, command lines, code/script snippets):

• Planning Log
  The planning log contains the following messages repetitively
  E0918 04:41:33.150034 3764 reference_line_provider.cc:208] Fail to get reference line
  E0918 04:41:33.210606 3766 reference_line_provider.cc:543] Failed to extract segments from routing
  E0918 04:41:33.210633 3766 reference_line_provider.cc:585] Failed to create reference line from routing

[ntutangyun]

Does restarting the planning module helps ?

[jmtao]

@ntutangyun not sure how you ran into this error if there's no rerouting. Once it happens, you'll have to restart planning module (and resend routing for the planning module) to get over it. If you want to know the root cause whey you ran into this error, we'll have to look further on your set up.

[ntutangyun]

@ntutangyun not sure how you ran into this error if there's no rerouting. Once it happens, you'll have to restart planning module (and resend routing for the planning module) to get over it. If you want to know the root cause whey you ran into this error, we'll have to look further on your set up.

Hi, @jmtao i saw this error previous also. I believe it might be caused by the spawn location of the apollo vehicle. If it's not in the centre of the traffic lane, sometimes it would produce this error and the planning module would no longer work until i manually restart it..

I did a little hack modification of the planning module and solved this issue. Whenever it fails to find reference line, it will automatically hard reset everyting and calls the init function again. So it's like automatic restart. 😂

[ntutangyun]

[EricWu416]

@ntutangyun Thanks for the feedback! I tried restarting the planning module by running bash /apollo/scripts/planning.sh stop and then bash /apollo/scripts/planning.sh, which did not resolve the issue. I could only exit the container and run dev_start.sh and dev_into.sh again to make the planning module functional. I think I will try the hard reset approach you provided and post a follow-up comment if it works later on. Thank you!

[EricWu416]

@ntutangyun Sorry for the late response. I tried your approach of implementing a hard reset, and based on the current behavior I think this issue is resolved. Thanks for your help! 😄

[jmtao]

@ntutangyun thanks for trying this out. But, we design it this way on purpose. The planning module has to be restarted manually when it fails to create routing. It's considered a serious failure, and we hope to restart the module to get a clean and safe state, instead of automatically restarting from a bad state, which could be unsafe.