Openpilot: Understanding Lane Detection

[hlezki]

Hi,
I am trying to understand how OpenPilot detects lane for my master research. Can you please inform me about following subjects?

1. In the network that you calculate Rotation and translation, which method are you using? Are you calculate camera pose with respect to sky-view or are you trying to obtain global pose of camera with respect to initial point?

2. Why are you warping frames by using warpPerspective? Are you trying to obtain bird-eye-view of camera scene in order to remove perspective effect? Or are you intend to obtain something else? Which kind of output should I expect as a result of warpPerspective kernel? And what is resolution of this output frame?

3. In the code, there is a matrix which is "ground_from_medmodel_frame". What is that matrix?

4. After warpPerspective and modifying frames, which method are you using for lane detection? Again, I am trying to find out which kind of inputs the second network(DefaultRunModel) expects.

5. Is there any documentation or paper that explains algorithms you implemented for lane detection?
   Thanks in advance.

[dorkmo]

This would better be discussed on discord
https://discord.comma.ai/

[pd0wm]

1) The posenet takes in two frames and gives you relative translation/rotation between those two frames
2) Where do you see warpPerspective? The code from camera.py is not used live
3) This is a perspective to ensure the model always sees the road in the same way, even even the camera has slightly different pitch/yaw. Currently no roll correction is done
4) Lane detection is done by a neural network. It takes in a YUV420 image, and outputs points for each lane in car reference frame. 192 points are outputted, spaced 1m apart.
5) See 4. It's deep learning.

If you have more questions, you're always welcome in our discord like @dorkmo suggested.

[scyoyo]

I wrote an article recently in decoding the lane detection model: https://medium.com/@chengyao.shen/decoding-comma-ai-openpilot-the-driving-model-a1ad3b4a3612