Openpilot: How the visiond works?

[sunmoon9898]

Hi，
I am curious the internal of visiond, I want to do similar work but not in selfdrive domain, can anyone answer this questions:
1) If the car on the road is very fast, how can the visiond take a sharp photograph in a very short time?
2) Which deep learning framework does the visiond use now ? I want to optimize the backend with arm cpu, the tensorflow is too heavy.

thanks,

[felixhsu123]

hi,me too
I am also curious the internal of visiond. why not open source?

tks

[ghost]

They apparently use opencv.
https://github.com/commaai/openpilot/blob/master/selfdrive/visiond/visiond

Q: there a better forum for questions? Does anyone know where to discuss?

-Sorry to barge in.

[geohot]

1. The sun is very bright during the day. At night the picture is a little blurry but who cares.
2. visiond uses an internal framework to run the neural net. Tensorflow is way too slow on phones.

[ingenieroariel]

@geohot Is it possible for Comma to release a visiond skeleton that does the following:

1. Reads data from the camera
2. Has OpenCV libraries loaded
3. Returns no accel change, no brake change and no steering change.

i.e. visiond with no neural networks, the community can write other agents just using opencl.

[geohot]

You can already do this with the released visiond. See how UI reads the image from visiond, it's open source.

As for linking OpenCV, that should just work the same as on a desktop on NEOS.

[ingenieroariel]

The problem is that if I run visiond just to get the camera feed from it I am not sure I would have enough processing power to try to reproduce it. As I understand things it would continue to run the network, save the videos, etc.

[geohot]

If all you plan to use is OpenCV none of that will have any effect, since OpenCV is CPU only. Network and videos are done off CPU.

[ingenieroariel]

I was thinking of using opencl on the adreno GPU for now ...

... But after your comment I realized that I will have a full Nvidia GTX 1070 on the back of the car, so I can develop for tensorflow during the next few months and in the near future hopefully we all switch to the oneplus [1] that comes with a Snapdragon 835 for tensorflow support on CPU, GPU and DSP [2]

[1] https://oneplus4.net/oneplus-5-release-date/
[2] https://www.qualcomm.com/news/snapdragon/2017/01/09/tensorflow-machine-learning-now-optimized-snapdragon-835-and-hexagon-682

[ingenieroariel]

To answer the original poster and maybe close the issue.

1. If the car on the road is very fast, how can the visiond take a sharp photograph in a very short time?
   George answered. To add to it, due to how visiond works, sharp pictures are not needed, you can see in this other repo how neural networks are even used to increase resolution on low res pictures: https://github.com/alexjc/neural-enhance

2. Which deep learning framework does the visiond use now ? I want to optimize the backend with arm cpu, the tensorflow is too heavy.
   visiond uses a custom backend, I believe they are using OpenCL + OpenCV, and possibly a reverse engineered camera driver. They can train on the cloud with a lot of gpu power, and then port the right network and weights with a lot of dot products, trigonometrical operations, sum, etc on a simple c program to be run on the GPU. See a very basic example in python here: http://www.wildml.com/2015/09/implementing-a-neural-network-from-scratch/

[jjrv]

strings says it uses at least Qualcomm FastCV Library.

[petterreinholdtsen]

Issue #4 mention the API.

[ingenieroariel]

From objdump on visiond (running from an nvidia TX2):

Dynamic Section:
  NEEDED               libgnustl_shared.so
  NEEDED               libavformat.so.57
  NEEDED               libavcodec.so.57
  NEEDED               libswscale.so.4
  NEEDED               libavutil.so.55
  NEEDED               libz.so.1
  NEEDED               libOmxVenc.so
  NEEDED               libOmxCore.so
  NEEDED               libgsl.so
  NEEDED               libCB.so
  NEEDED               libOpenCL.so
  NEEDED               libcutils.so
  NEEDED               libm.so
  NEEDED               liblog.so
  NEEDED               libdl.so
  NEEDED               libc.so

[legonigel]

visiond is now open source.

[petterreinholdtsen]

Cool. I found it in https://github.com/commaai/openpilot/tree/devel/selfdrive/visiond and will definitely give it a go now. :)