Apollo: Any plan for supporting Nvidia RTX 2080?

Created on 20 Feb 2019  路  33Comments  路  Source: ApolloAuto/apollo

Hi,

I wonder if Apollo is going to support the new 20 series GPUs such as RTX 2080?

As indicated in another issue, Apollo comes with a pre-compiled version of Caffe which doesn't support new GPU architectures such as Volta. Is this still the case in version 3.5? If not, would Apollo provide support in next releases for 20 series GPUs?

Thanks,
Junjie

System information

  • OS Platform and Distribution (e.g., Linux Ubuntu 14.04): Ubuntu 14.04
  • Apollo installed from (source or binary): source
  • Apollo version (1.0, 1.5, 2.0, 2.5, 3.0): 3.0

Steps to reproduce the issue:

  • Please use bullet points and include as much details as possible:

Supporting materials (screenshots, command lines, code/script snippets):

Hardware Perception Question
Source
picture junjieshen

Most helpful comment

Hi,
Is NVIDIA RTX 2080 supported now for Apollo 5.0 ?

picture Avps1 on 12 Mar 2020
👍4

All 33 comments

@techoe Can you please explain if perception works on RTX 20-series (based on Turing architecture)? #4233 clarifies that it doesn't work on Volta, but what about Turing? Would perception have any issues with these GPUs?

deepaktalwardt picture deepaktalwardt on 15 Mar 2019

We cannot verify it at this moment since we do not have RTX_2080 in-house. I would appreciate if someone test with different GPUs and let us know. Thank you!

techoe picture techoe on 20 Mar 2019

@junjieshen @techoe I am using RTX2070, and succeed following the official build/install instructions.

michaelowenliu picture michaelowenliu on 21 Mar 2019
👍3 👎1

@michaelowenliu can you confirm that Perception module works properly as well?

deepaktalwardt picture deepaktalwardt on 21 Mar 2019

@deepaktalwardt could you tell me how to confirm it? I can try it. Currently I just finished the build/installation without hardware and see the dreamview demo.

michaelowenliu picture michaelowenliu on 21 Mar 2019

@michaelowenliu You can follow the step by step instructions here: apollo/docs/howto/how_to_run_perception_module_on_your_local_computer.md

Thank you for trying this out! Waiting to hear the results

deepaktalwardt picture deepaktalwardt on 21 Mar 2019

@deepaktalwardt, Hi

[perception] ]. sampled_right_width_[0.814828] is too small. It should be larger than half vehicle width[1.05].
[perception] E0321 14:17:33.081907 19244 hdmap_common.cc:183] [mainboard] lane[id = id: "11265_1_-1"
[perception] ]. sampled_right_width_[0.817991] is too small. It should be larger than half vehicle width[1.05].
[perception] E0321 14:17:33.081920 19244 hdmap_common.cc:183] [mainboard] lane[id = id: "11265_1_-1"

michaelowenliu picture michaelowenliu on 21 Mar 2019

@michaelowenliu Hi, have you tried to run "./apollo.sh test_gpu". I remember my build succeed, but the perception tests failed.

junjieshen picture junjieshen on 21 Mar 2019

@junjieshen Hi, what outputs do you expect if it fails or succeeds? I am just a beginner on Apollo.

michaelowenliu picture michaelowenliu on 22 Mar 2019

Hi @michaelowenliu,

It will report fails after you run ./apollo.sh test_gpu. Also you can find "Error" in /apollo/data/log/perception.out file if the perception module could not work properly.

I've a RTX2070 as well. Can you share the configuration of your host, includes OS version, Nvidia driver version, Cuda version and so on? It will be really appreciate.

fzd9752 picture fzd9752 on 22 Mar 2019

@michaelowenliu Hello, I tried 2070 with ubuntu 16.04, apollo 3.0, nvidia driver 430.14, doesn't work for me, could you please share some knowledge about how you made it? Thank you a lot!

penghuihuang picture penghuihuang on 28 May 2019

@penghuihuang Hi, it also failed. Now I don't have any plan on it.

michaelowenliu picture michaelowenliu on 28 May 2019

I have the same issues here,
Now , I can not buy new 1080 in market because it has stop production over half year.
the 2080 looks like conflict with apollo kernel
Anyone Run 2080 on real car successfully,please teach me

usasos000 picture usasos000 on 5 Jul 2019

Yep, running into a problem with an RTX 2080 Max-Q as well.

Kubuntu 19.04
nvidia-driver-418
cuda-10.1
apollo-5.0

The docker container gets shipped with cuda-8.0. This means that when you build apollo inside docker, you're building it with a version of cuda which is incompatible with your GPU. 

F0922 16:52:46.091820 31200 permute_layer.cu:78] Check failed: error == cudaSuccess (8 vs. 0)  invalid device function

Meaning of "Invalid Device Function"

image

_Note: RTX-series cards use the Turing architecture_

CUDA GPU architecture compatibility

Time to update the CUDA version used by Apollo?

Nikita240 picture Nikita240 on 23 Sep 2019

Yes indeed, an updated version of cuda would be much appreciated.
From what I see on NVIDIA's website, the most recent architecture for which there are specific automotive products is Volta - with their JETSON AGX XAVIER SERIES.

Autofoxsys picture Autofoxsys on 30 Sep 2019
👍1

Hi!
Just wonder if there is update on this issue or any plan to support the Turing architecture in the near future?

AIasd picture AIasd on 16 Feb 2020

Hi,
Is NVIDIA RTX 2080 supported now for Apollo 5.0 ?

Avps1 picture Avps1 on 12 Mar 2020
👍4

Will it be wise to buy RTX 2080 now and wait for the support from Apollo 5.0? In the meantime, try to build the Apollo using CPU?

faysalhossain2007 picture faysalhossain2007 on 7 Apr 2020

Hi,
Will NVIDIA RTX 2080 be supported in the near future?

AIasd picture AIasd on 28 May 2020

Will NVIDIA RTX 2080 be supported in the near future?

I believe current apollo/master supports the RTX2080.

lemketron picture lemketron on 28 May 2020

Will NVIDIA RTX 2080 be supported in the near future?

I believe current apollo/master supports the RTX2080.

That's great! Do you have any idea on if it will be supported by carla or lgsvl?

AIasd picture AIasd on 29 May 2020

Will NVIDIA RTX 2080 be supported in the near future?

I believe current apollo/master supports the RTX2080.

Not yet锛宎ctually. RTX 2080 cards will be supported in a month or two锛宒epending on our progress.

storypku picture storypku on 29 May 2020
👍3

Will NVIDIA RTX 2080 be supported in the near future?

I believe current apollo/master supports the RTX2080.

Not yet锛宎ctually. RTX 2080 cards will be supported in a month or two锛宒epending on our progress.

Could you inform in this issu when you solve this problem?

AlexandrZabolotny picture AlexandrZabolotny on 10 Jun 2020

Will NVIDIA RTX 2080 be supported in the near future?

I believe current apollo/master supports the RTX2080.

Not yet锛宎ctually. RTX 2080 cards will be supported in a month or two锛宒epending on our progress.

@storypku I'm confused... Doesn't apollo/master already support the RTX cards? I've used it on my 15" Razer Blade which has a RTX2070, along with the LGSVL Simulator as documented on the LGSVL Simulator site (though it maxes out all 12 cores and can exhaust the 8GB of GPU memory in my laptop which can cause apollo modules to crash).

A desktop workstation with a RTX 2080 Ti (with 11GB of GPU memory) would likely handle it more easily, but another option is to run Apollo and Simulator on separate machines which is my preferred setup.

Are you just saying that master has not hit "release" status yet for the next version of apollo? Or are there still known issues running apollo master on an RTX card? Thanks!

lemketron picture lemketron on 10 Jun 2020

has not hit "release" status yet fo

Sorry for my not-so-exact expression.
In tools/cuda_library.bzl:

cuda_arch = " ".join([
    "-arch=sm_30",
    "-gencode=arch=compute_30,code=sm_30",
    "-gencode=arch=compute_50,code=sm_50",
    "-gencode=arch=compute_52,code=sm_52",
    "-gencode=arch=compute_60,code=sm_60",
    "-gencode=arch=compute_61,code=sm_61",
    "-gencode=arch=compute_61,code=compute_61",
])

Compute capabilities specified here was for SMs=[3.0,6.1]. RTX 2080 Ti cards are running in a backward-compatible state.

storypku picture storypku on 10 Jun 2020

Compute capabilities specified here was for SMs=[3.0,6.1]. RTX 2080 Ti cards are running in a backward-compatible state.

@storypku That's interesting, thanks. I noticed that cuda_library.bzl has not changed since apollo 5.0 which crashed when running on the RTX2080 but now in apollo master it does seem to run without crashing. So I'm guessing something else was done to fix this since 5.0, though it sounds like you're saying it's perhaps not running as optimally as it could if it were fully supported rather than running in backward-compatible mode.

In any case it will be great to have full official support when it is ready. Thanks for the update!

lemketron picture lemketron on 11 Jun 2020

has not hit "release" status yet fo

Sorry for my not-so-exact expression.
In tools/cuda_library.bzl:

cuda_arch = " ".join([
    "-arch=sm_30",
    "-gencode=arch=compute_30,code=sm_30",
    "-gencode=arch=compute_50,code=sm_50",
    "-gencode=arch=compute_52,code=sm_52",
    "-gencode=arch=compute_60,code=sm_60",
    "-gencode=arch=compute_61,code=sm_61",
    "-gencode=arch=compute_61,code=compute_61",
])

Compute capabilities specified here was for SMs=[3.0,6.1]. RTX 2080 Ti cards are running in a backward-compatible state.

@storypku Can you tell me is RTX 2080 supported by pre6 version of Apollo? What about master branch? We plan to buy a GPU and would like to know what's the best GPU that is supported by Apollo at this time or a near future. Thank you!

ChihweiLHBird picture ChihweiLHBird on 6 Aug 2020

has not hit "release" status yet fo

Sorry for my not-so-exact expression.
In tools/cuda_library.bzl:

cuda_arch = " ".join([
    "-arch=sm_30",
    "-gencode=arch=compute_30,code=sm_30",
    "-gencode=arch=compute_50,code=sm_50",
    "-gencode=arch=compute_52,code=sm_52",
    "-gencode=arch=compute_60,code=sm_60",
    "-gencode=arch=compute_61,code=sm_61",
    "-gencode=arch=compute_61,code=compute_61",
])

Compute capabilities specified here was for SMs=[3.0,6.1]. RTX 2080 Ti cards are running in a backward-compatible state.

@storypku Can you tell me is RTX 2080 supported by pre6 version of Apollo? What about master branch? We plan to buy a GPU and would like to know what's the best GPU that is supported by Apollo at this time or a near future. Thank you!

RTX 2080 cards was fully supported by the master branch now. As for pre6, I am afraid it would keep RTX 2080 run in backward compatible state only ... There was no easy way for pre6 to take full advantage of RTX 2080 cards. And that's one of the top reasons we made great efforts to breaking upgrade this new master branch.

storypku picture storypku on 6 Aug 2020

has not hit "release" status yet fo

Sorry for my not-so-exact expression.
In tools/cuda_library.bzl:

cuda_arch = " ".join([
    "-arch=sm_30",
    "-gencode=arch=compute_30,code=sm_30",
    "-gencode=arch=compute_50,code=sm_50",
    "-gencode=arch=compute_52,code=sm_52",
    "-gencode=arch=compute_60,code=sm_60",
    "-gencode=arch=compute_61,code=sm_61",
    "-gencode=arch=compute_61,code=compute_61",
])

Compute capabilities specified here was for SMs=[3.0,6.1]. RTX 2080 Ti cards are running in a backward-compatible state.

@storypku Can you tell me is RTX 2080 supported by pre6 version of Apollo? What about master branch? We plan to buy a GPU and would like to know what's the best GPU that is supported by Apollo at this time or a near future. Thank you!

RTX 2080 cards was fully supported by the master branch now. As for pre6, I am afraid it would keep RTX 2080 run in backward compatible state only ... There was no easy way for pre6 to take full advantage of RTX 2080 cards. And that's one of the top reasons we made great efforts to breaking upgrade this new master branch.

Thank you for your response.

ChihweiLHBird picture ChihweiLHBird on 6 Aug 2020

As RTX 2080 is now fully supported by the master branch, I am going to close this issue. Thanks all for supporting Apollo. If you have any other questions, please file a new issue any time.

storypku picture storypku on 6 Aug 2020

As RTX 2080 is now fully supported by the master branch, I am going to close this issue. Thanks all for supporting Apollo. If you have any other questions, please file a new issue any time.

Does this mean perception is working again in master on RTX 2080? If so, can you say which commit that was? I'm investigating a report that it's still not working (as of yesterday).

If it is fully working then this means RTX users can use stop using the pre6 branch and switch to master? Thanks!

lemketron picture lemketron on 7 Aug 2020

Can we please re-open this issue until perception is fixed in master for the RTX-20x0 GPUs?

lemketron picture lemketron on 7 Aug 2020

Can we please re-open this issue until perception is fixed in master for the RTX-20x0 GPUs?

What about creating a new issue ?

storypku picture storypku on 7 Aug 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

Triangle001 picture Triangle001  路  3Comments
CCodie picture CCodie  路  3Comments
westeast picture westeast  路  3Comments
c-xyli picture c-xyli  路  3Comments
BenBaek picture BenBaek  路  3Comments