Openpose: CUDA 10 (RTX 20XX cards) compatibility with Windows - Check failed: error == cudaSuccess (8 vs. 0) invalid device function

[tech-gh]

Issue Summary

Cuda error with OpenPose / Nvidia 2080 Ti / CUDA 10 on Windows 10 pro (error == cudaSuccess (8 vs. 0) invalid device function)

Executed Command (if any)

OpenPoseDemo.exe --logging_level 0 --disable_multi_thread

OpenPose Output (if any)

Starting OpenPose demo...
{{{ full output that was in here is available at https://pastebin.com/ATu7amDD }}}
F1230 19:44:08.215404 15224 pooling_layer.cu:212] Check failed: error == cudaSuccess (8 vs. 0) invalid device function
* Check failure stack trace: *

Errors (if any)

Recompiled OpenPose with Cuda 10 on Windows 10 pro works well with 1080 ti, but with the 2080 ti it fails to open a window and the following message appears:
F1230 19:44:08.215404 15224 pooling_layer.cu:212] Check failed: error == cudaSuccess (8 vs. 0) invalid device function

Type of Issue

You might select multiple topics, delete the rest:

• Compilation/installation error
• Execution error
• Help wanted

Your System Configuration

1. Whole console output: https://pastebin.com/sdUKns74 (CMake output + CMakeCache.txt)
2. OpenPose version: commit 7a4fb5a2d8a0634659fa7ea6efddb23531e95df2
3. General configuration:
   
   
   
   • Installation mode: CMake-gui on Windows
   
   
   • Operating system: Windows 10 Professional x64 version 1809, version
   
   
   • Release or Debug mode? (by default: release):
   
   
   • Compiler: VS2017 community (build toolset v141, MSVC 19.16.27025.1)
   
   
4. Non-default settings:
   
   
   
   • 3-D Reconstruction module added?: no
   
   
   • Any other custom CMake configuration with respect to the default version? no
   
   
5. 3rd-party software:
   
   
   
   • Caffe version: Default from OpenPose
   
   
   • CMake version: CMake-gui 3.13.2
   
   
   • OpenCV version: OpenPose default -- 3.10 (Windows)
   
   
6. If GPU mode issue:
   
   
   
   • CUDA version: 10.0
   
   
   • cuDNN version: 7.4.2
   
   
   • GPU model: Nvidia 2080ti, driver 417.35
   
   
7. If Windows system:
   
   
   
   • Portable demo or compiled library? Compiled library, but also happens with release 1.4.0
   
   

[tech-gh]

Some additional things tried:

• Adding 'CUDA_FORCE_PTX_JIT=1' to the environmental variables only delayed the time it takes for the error to show up.
• After the above tried turning off display and rendering off and tried to save keypoints on disk. The output directory is generated but it's empty.
• Doesn't work when using the test video as an input either.
• Tried adding some flags (following the guidelines from https://docs.nvidia.com/cuda/turing-compatibility-guide/index.html#building-turing-compatible-apps-using-cuda-8-0): CUDA_NVCC_FLAGS (-gencode=arch=compute_50,code=sm_50 -gencode=arch=compute_52,code=sm_52 -gencode=arch=compute_60,code=sm_60 -gencode=arch=compute_61,code=sm_61 -gencode=arch=compute_70,code=sm_70 -gencode=arch=compute_70,code=compute_70)
• Deleted build directory and recompiled
• Tried VS2017 community both debug and release, and tried VS2015 community

[gineshidalgo99]

I am in the process of trying to install OpenPose on VS2017 with CUDA 10 (note that we officially only support VS2015 for CUDA 8, so 20XX graphics cards would not be compatible there).

Hopefully this should solve the issue for the newer GPUs. But it will take me some time (hopefully between 1-2 weeks). I will keep you updated here.

Note: I deleted your comment from #816 given that the reason is different.

[A7ocin]

Interesting question, I'm trying to make it work as well on Ubuntu with two RTX 2070. Let's keep ourselves updated

[gineshidalgo99]

Ubuntu should run with no issue. The problem is that I have to compile Caffe for CUDA 10 on Windows... and Caffe does not have any support in there. I will fight with it this weekend.

[nchauvin1]

Any progress with compiling Caffe for CUDA 10 on Windows?

[gineshidalgo99]

I could not compile Caffe following their Windows instructions (lots of errors which I need to figure out how to solve)... If somebody is able to compile CUDA version of Caffe for VS 2017, please forward with me the instructions. Otherwise, I will try more deeply whenever I have a change, but right now I wont be able in the following 1-2 weeks at least.

[nchauvin1]

Nvidia have their own branch of Caffe : NVCaffe
The latest version is based on Cuda 10.
https://docs.nvidia.com/deeplearning/dgx/caffe-release-notes/rel_18.09.html

"xlert" has apparently been able to compile Caffe with CUDA 10:
https://github.com/BVLC/caffe/issues/6666

[gineshidalgo99]

The problem is Windows. xlert has most probably done it on Ubuntu, where it is as simple as in previous CUDA versions.

For the NVCaffe, could you forward me where I can download it? I could only find doc but not the actual download link.

[hw07216]

https://github.com/NVIDIA/caffe/

[kgiroux]

Is there is any new about the RTX support ? In our lab, we just received our newer computer with RTX card, and we would like to use it with OpenPose

[JakeMalis]

I am trying to get it to work with my new RTX 2070. Any updates?

[pablovicentem]

I am having the same problem with the new RTX2080. Any updates?

[rathaROG]

So RTX 2080 should not be a problem on Ubuntu, right?

[JakeMalis]

I was having issues with 2070 on Ubuntu, so I presume the same would be with the 2080. It seems that caffe isn't compatible with RTX yet, but if you can figure out how to replace caffe with NVCaffe, it may work.

[nchauvin1]

I've discussed with deep learning team at Nvidia.
They definitely recommend to upgrade to NVCaffe as Nvidia is supporting and maintaining the compatibility between their Caffe implementation (as well as other frameworks) and their hardware:
https://docs.nvidia.com/deeplearning/dgx/caffe-release-notes/overview_relnot.html

Some additional info on the release notes:
https://docs.nvidia.com/deeplearning/dgx/caffe-release-notes/index.html

However, their still seems to be some issue to compile NVCaffe on Windows.

[JakeMalis]

@nchauvin1 Do you know how to install OpenPose using NVCaffe instead of Caffe? Thanks!

[nchauvin1]

Otherwise, there are several branches of Caffe for Windows:
https://github.com/BVLC/caffe/tree/windows
https://github.com/happynear/caffe-windows
https://github.com/Microsoft/caffe
https://github.com/willyd/caffe

But none of them seems to support CUDA 10 yet.

[JakeMalis]

@nchauvin1 do you know how to tell CMake to use NVCaffe instead of Caffe?

[gineshidalgo99]

2070 and 2080 (or any graphic card) is no issue in Ubuntu. The problem is Windows. As @nchauvin1 points out, there are many Windows branches, but none supports CUDA 10 (so no 20XX cards compatible). If anybody finds a Windows Caffe branch that supports CUDA 10, please let me know!

@JakeMalis So far, I can only find NVCaffe for Ubuntu, I have not found successful instructions for Windows either.

[JakeMalis]

@gineshidalgo99 I have both Ubuntu and Windows, so if you could send me Ubuntu instructions that would be awesome!

[gineshidalgo99]

@JakeMalis There is no particular instructions for 20XX cards. As long as you have CUDA >= 10, OpenPose will simply work on Ubuntu with 20XX cards (I tried myself a couple days ago in a new one we just got). Use the latest GitHub code but that's it.

[rathaROG]

2070 and 2080 (or any graphic card) is no issue in Ubuntu. The problem is Windows. As @nchauvin1 points out, there are many Windows branches, but none supports CUDA 10 (so no 20XX cards compatible). If anybody finds a Windows Caffe branch that supports CUDA 10, please let me know!

@JakeMalis So far, I can only find NVCaffe for Ubuntu, I have not found successful instructions for Windows either.

Thank you so much! I need to buy a new laptop in this week, so now I'll look for an RTX2080/RTX2070 laptop instead of GTX 1070.

[JakeMalis]

@gineshidalgo99 I was having lots of issues with it. Did you use NVCaffe?

[JakeMalis]

@gineshidalgo99 I was getting lots of errors when trying to build OpenPose

[gineshidalgo99]

Default OpenPose, as it is today and following the , works on a newly formatted Ubuntu 16 machine with any CUDA version >= 8. As mentioned in the documentation, we do not (officially) support Ubuntu 18, nor CUDA >= 9 on Windows 10, nor NVCaffe. I have no clue how hard it would be to make them work on those.

[JakeMalis]

@gineshidalgo99 if my machine is Ubuntu 18, would that be why it is not working?

[gineshidalgo99]

@JakeMalis I have been able to install OpenPose on Ubuntu 18. From now on, the documentation includes a new section about how to install it on Ubuntu 18 (I tried CUDA 10):

https://github.com/CMU-Perceptual-Computing-Lab/openpose/blob/master/doc/installation.md

Updating the previous message:

2070 and 2080 (or any 7XX, 8XX, 9XX, 10XX, 20XX, V100, Titan V, etc.) is no issue in Ubuntu. The problem is Windows. As @nchauvin1 points out, there are many Windows branches, but none supports CUDA 10 (so no 20XX cards compatible nor V100 nor Titan V). If anybody finds a Windows Caffe branch that supports CUDA 10, please let me know!

We do not (officially) support CUDA >= 9 (for Windows) nor NVCaffe (for any OS). I have no clue how hard it would be to make them work on those. So far I was not able to install Caffe with CUDA 10 on Windows.

[JakeMalis]

@gineshidalgo99 thank you so much for added that to the documentation! When I get home tonight I will definitely try to get it working. I'll let you know how it goes.

[JakeMalis]

@gineshidalgo99 when I try to build using make -j nproc but it fails with the following output:

CMake Error: The following variables are used in this project, but they are set to NOTFOUND.
Please set them or make sure they are set and tested correctly in the CMake files:
CUDA_cublas_device_LIBRARY (ADVANCED)
    linked by target "caffe" in directory /home/jakebmalis/Documents/openpose/3rdparty/caffe/src/caffe

-- Configuring incomplete, errors occurred!
See also "/home/jakebmalis/Documents/openpose/build/caffe/src/openpose_lib-build/CMakeFiles/CMakeOutput.log".
See also "/home/jakebmalis/Documents/openpose/build/caffe/src/openpose_lib-build/CMakeFiles/CMakeError.log".
CMakeFiles/openpose_lib.dir/build.make:105: recipe for target 'caffe/src/openpose_lib-stamp/openpose_lib-configure' failed
make[2]: *** [caffe/src/openpose_lib-stamp/openpose_lib-configure] Error 1
CMakeFiles/Makefile2:67: recipe for target 'CMakeFiles/openpose_lib.dir/all' failed
make[1]: *** [CMakeFiles/openpose_lib.dir/all] Error 2
Makefile:83: recipe for target 'all' failed
make: *** [all] Error 2

[gineshidalgo99]

Please, in order for me to help you better, open a new post where you include all your software/hw configuration as the GitHub issue template specifies.

Let's leave this issue for compatibility between CUDA 10 and Caffe on Windows.

[JakeMalis]

@gineshidalgo99 I created a new issue with all the info per the template. Thanks!

[tompollok]

In this website an explanation about how to make Caffe work with CUDA 9 is documented. Maybe thats transferable to CUDA 10, too.

https://tharindumathew.com/2018/04/13/using-caffe-on-windows-with-cuda-9-0-for-vs-2013-2015-and-2017

[gineshidalgo99]

VS2017 and CUDA 10 support added in the latest commit (for Windows)! Please, let me know if it finally works on your machines. I will consider this issue closed then, but please post if you find any errors.

VERY IMPORTANT: Manually remove OpenPose and re-clone it again (to remove VS 2015 files)

[chrisrogers3d]

I still cannot build on Windows 10 with VS2017 and Cuda 10.
Visual studio says it cant find openpose.lib, it does not say why it wont build it.

The only other error is
{code}>CMake Error at openpose_generated_renderFace.cu.obj.Release.cmake:219 (message):
1> Error generating
1> D:/code/openpose/build/src/openpose/CMakeFiles/openpose.dir/face/Release/openpose_generated_renderFace.cu.obj
1>
1>
1>C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\Common7\IDE\VC\VCTargets\Microsoft.CppCommon.targets(209,5): error MSB6006: "cmd.exe" exited with code 1.{code}

[UnNeill]

Following up on last comment. (Which is mine under different name)
I tried running the portable demo with cuda 8, 9, and 10 and got the same cuda error as the others using an rtx 2080.
I also tried building with 8 and 10 and got the cmake error listed above.

[rathaROG]

Following up on last comment. (Which is mine under different name)
I tried running the portable demo with cuda 8, 9, and 10 and got the same cuda error as the others using an rtx 2080.
I also tried building with 8 and 10 and got the cmake error listed above.

The portable demo won't run on RTX card even though it runs on GTX 10xx with CUDA 10.

You have to clone the master repo and build your own openpose for your RTX card. It's working fine on my RTX 2080 Max-Q laptop.

[chrisrogers3d]

thanks @rathaROG
I was not able to build (brand new win10, vs2017, cuda10) because of the cmake error I posted in this thread.

CMake Error at openpose_generated_renderFace.cu.obj.Release.cmake:219 (message):
1> Error generating
1> D:/code/openpose/build/src/openpose/CMakeFiles/openpose.dir/face/Release/openpose_generated_renderFace.cu.obj

Ill try again

[rathaROG]

thanks @rathaROG
I was not able to build (brand new win10, vs2017, cuda10) because of the cmake error I posted in this thread.

CMake Error at openpose_generated_renderFace.cu.obj.Release.cmake:219 (message):
1> Error generating
1> D:/code/openpose/build/src/openpose/CMakeFiles/openpose.dir/face/Release/openpose_generated_renderFace.cu.obj

Ill try again

I tried to build again based on the latest master repo, and it was successfully built without any error. Please take a look at my system and my procedure below. Good luck!

The details of my system:

• Windows 10 Pro fo Workstations (RS5 v1809 b17763.404) 64-bit
• CUDA 10.0 with cuDNN 7.4.2
• Visual Studio Enterprise 2017 (15.9.11)
• CMake GUI 3.14.1
• NVIDIA RTX 2080 Max-Q

Procedure:

1. Clone master
2. Config CMake GUI (Internet connection required):
   
   
   
   • Compiler: Microsoft Visual Studio 2017 64-bit
   
   
   • CMake always fails to set CUDA_SDK_ROOT_DIR, so manually set it! Mine is C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v10.0 [screenshot]
   
   
   • You may also tick to download other models as you require
   
   
   • Finally generate the solution
   
   

1. Open the .sln and build (in Release not Debug) the sub OpenPoseDemo [screenshot]
2. After successfully built, you can find the openpose.lib in openpose\build\src\openpose\Release and openpose.dll in openpose\build\x64\Release .

[thiravet]

hi @chrisrogers3d ,

i had the same problem. I'm not sure totally to understand this issue but to solve it, I changed the default value in cmake for CUDA_HOST_COMPILER by "C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.16.27023\bin\HostX86\x64" (path for a default community vs2017). I found this path in cuda example for the -ccbin parameter in nvcc.
if it can help you...

[chrisrogers3d]

Thanks Everyone!
I tried again and got it to work! Setting x64 in Cmake might need to be in the directions.
Building the 3d viewer did not work.
Win10 twin rtx2080 cuda 10.1 cudnn 7.5 vs2017

[thomasYiPP]

Hey everyone, hopefully someone is still active here.
We're trying to get openpose on an RTX2070 working as well and I tried compiling it myself as well but after the first few errors and endless googles I gave up. Is it possible to share your compiled binaries?