Virtual-environments: Add Cuda SDK

[sergeneren]

Tool information

• Tool name: Cuda SDK
• Add or update? Add
• Desired version: 10.2
• Approximate size: 2.6 GB
• If this is an add request:
  
  
  
  • Brief description of tool: Nvidia Cuda SDK for parallel computing
  
  
  • URL for tool's homepage: https://developer.nvidia.com/cuda-downloads
  
  

Virtual environments affected

• [x] macOS 10.15
• [x] Ubuntu 16.04 LTS
• [x] Ubuntu 18.04 LTS
• [x] Windows Server 2016 R2
• [x] Windows Server 2019

Can this tool be installed during the build?
No. There is no headless installer

Are you willing to submit a PR?
yes

[bek0s]

Hello,

I wonder if there were any updates on this. I am developing the CICD pipeline for a project that needs CUDA. Unfortunately, the CUDA toolkit is quite large. It takes very long to download and install on the available Linux virtual machines. Having the CUDA toolkit pre-installed on the available virtual machines would be fantastic!

Thank you.

[sergeneren]

Hello,

Is there an update on this?

Thank you.

[jrood-nrel]

Would be nice to have the SDK available to test compiling, even if GPUs don't exist on the nodes.

[maxim-lobanov]

Hello everyone, thank you fro feature request and your comments.
Could you please help us to understand versioning approach for Cuda SDK?
Will it work for you to have only the latest stable Cuda SDK release on image (10.2 for now)? Will it be a breaking change for you when we update it to 10.3 in future (when it is released)?
We are trying to avoid adding tools for which we will have to keep several versions on image in future. So we would be very appreciate for your thoughts on this point.

@alepauly , Installation in runtime is not trivial, so may be Cuda SDK is good candidate for setup task.
But installation time is pretty long. ~4 minutes on Windows / Ubuntu. A bit faster for macOS.

[sergeneren]

Hello @maxim-lobanov,

Thank you for looking into this. For me, the latest stable version works. Cuda sdk doesn't introduce breaking changes very often so I don't think it will be a problem t least until cuda 11.

I also think keeping several versions of cuda sdk is unnecessary and it should be updated in regular intervals.

Of course this is my personal opinion and I would love to hear other peoples input on this.

[simogasp]

I agree with @maxim-lobanov , having the latest version available would be great.
For Ubuntu, the workaround is to use Nvidia docker images that have all kind of combinations of Linux distributions versions and sdk versions. But I'm not aware of a similar thing for the other OS.

[bek0s]

I agree with the gentlemen above. Having the latest version should be fine.

[alexfrasson]

Same for me. Latest version would be perfect.

[alexfrasson]

Question: would a full installation be required? Some space could be saved if only development subpackages are installed (see the list under Silent Installation). The same probably applies to debian packages. I guess that subpackages such as demo_suite_10.2, samples_10.2 and fortran_examples_10.2 could be safely left out. Maybe a more experienced cuda developer could give his/hers opinion about this.

[yhmtsai]

I use choco install cuda for Windows in the Github action, and I can check the cuda compilation.
Everything works as expected until yesterday. For some reason, Github action can not find the cuda static library after the updated windows-latest.

• Previous pass action
• Current failed action

[maxim-lobanov]

Hello @yhmtsai , thank you for report.
Can the issue come from any updates of choco formula?

Yesterday, we have finished deployment of new image that brings the following major changes:

• VisualStudio/16.5.0+29911.84 -> VisualStudio/16.5.4+30011.22
• Cmake 3.16.5 -> Cmake 3.17.1

I have found the following interesting notes in your logs:
1) Updated compiler version (probably, it is the result of VS update)

The C compiler identification is MSVC 19.25.28610.4 -> 19.25.28614.0
The CXX compiler identification is MSVC 19.25.28610.4 -> 19.25.28614.0

2) CMake errors (can come from Cmake update)

CMake Error: The following variables are used in this project, but they are set to NOTFOUND.
Please set them or make sure they are set and tested correctly in the CMake files:
CUBLAS
    linked by target "ginkgo_cuda" in directory D:/a/ginkgo/ginkgo/cuda
    linked by target "solver" in directory D:/a/ginkgo/ginkgo/benchmark/solver
    linked by target "spmv" in directory D:/a/ginkgo/ginkgo/benchmark/spmv
CUDA_RUNTIME_LIBS_STATIC
    linked by target "ginkgo_cuda" in directory D:/a/ginkgo/ginkgo/cuda
    linked by target "solver" in directory D:/a/ginkgo/ginkgo/benchmark/solver
    linked by target "spmv" in directory D:/a/ginkgo/ginkgo/benchmark/spmv
CUSPARSE
    linked by target "ginkgo_cuda" in directory D:/a/ginkgo/ginkgo/cuda
    linked by target "solver" in directory D:/a/ginkgo/ginkgo/benchmark/solver
    linked by target "spmv" in directory D:/a/ginkgo/ginkgo/benchmark/spmv

Hope it helps!
If it doesn't fix your issue, feel free to log the separate issue for investigation

[yhmtsai]

@maxim-lobanov thanks for hint!

The problem is from updated CMake.
CMake (>=3.17.0) can find CMAKE_CUDA_TOOLKIT_INCLUDE_DIRECTORIES which is not found in older version.
Thus, I can not write some command which depends on empty CMAKE_CUDA_TOOLKIT_INCLUDE_DIRECTORIES condition.
After fixing it, cuda compile in windows is passed again.

By the way, CMake 3.17.0 introduces FindCUDAToolkit which should be useful for windows.

[AlenaSviridenko]

Hi there 👋,
sorry for no activity for a while.

After a long internal discussion we decided not add Cuda SDK on our images and there are several reasons for it:

1. _Tool size._ 2.5Gb for Ubuntu, 3.5Gb for Windows which is a very significant size. Even though there is enough space for a single Cuda SDK version on Windows and macOS, we have disk space issues on Ubuntu, so we have to be very careful when choosing tools for pre-installation.
2. _Multiple versions support._ According to Cuda SDK compatibility docs, starting with Cuda 11, the toolkit versions are based on an industry-standard semantic versioning scheme: X.Y.Z, where each X.Y (major.minor) update can contain breaking changes, and in order to reduce possible impact we'd have to have at least two latest Cuda SDK versions, which increases the disk space requirements x2 times.

I am going to close this issue, please feel free to create a new one in case of any questions.
Thank you!