Singularity: Feature: Allow --nv during build

[yangyang-zhang]

Version of Singularity:

2.4-dist

singularity build ,No --nv parameter, When I want to create a image, it is not supported to enable GPU, when i want to neon，this is a deep learning framework,It reads whether there is a GPU in the environment to determine whether the GPU version is installed.I feel can add this part of the function

[GodloveD]

Hi @yangyang-zhang. Several people have asked for the --nv option during build. It should be easy to implement. Especially in the context of #1082. I could probably tack this change onto that PR without much trouble.

[dtrudg]

@cclerget with #5122 in place can we do this more simply now, or is the plumbing in the wrong place, too high up in the CLI at present?

[cclerget]

@dctrud Yep should be pretty easy

[calebho]

I'm currently on version 3.6.1. What's the status of this? I need access to driver libraries at build time.

[dtrudg]

This is not in 3.6.1 and isn't in the immediate roadmap for 3.7 which is under discussion now. Generally building a container from a CUDA base image you can link against the stub libraries at /usr/local/cuda/lib64/stubs.

Also, if there is a strong need for this and a wish to become involved in the project then PRs are always gratefully received.

[torrance]

My use case for this is with Julia lang and using their CuArrays package. Julia needs to able know the CUDA driver version during build to be able to download it's own local files.

This needs to happen during the build process and the filesystem isn't writable at any other point and it's not feasible to download these files onto a temporary filesystem on each run.

[dtrudg]

To work around this, and follow the same 'delayed build' of the Julia GPU dependent components as is prescribed in the Julia docs for nvidia-docker, you can:

• build to a sandbox excluding the GPU driver dependent components
• use --writable with the sandbox to add the GPU driver dependent components
• build from the sandbox into an immutable SIF

.. or ...

Look at the persistent overlay support at: https://sylabs.io/guides/3.6/user-guide/persistent_overlays.html

[torrance]

Thanks for your reply dctrud. Your suggestion is exactly what we tried doing but unfortunately —nv and —writable aren’t allowed to be passed together it seems.

We will look into your second option (and petition our SC provider to upgrade to 3.6!)

[dtrudg]

@torrance - apologies, that slipped my mind - I'm not at a box with GPU right now... but I thought it was just a warning for some situations and would actually work.

~@cclerget - is there a reason that we don't allow --writable with --nv and a sandbox?~

[dtrudg]

@torrance - actually, now I'm sat at a machine with GPU I think I might have been initially right and it does work as would be expected...

dave@piran~> singularity version
3.6.1+4-g5417e6b82

dave@piran~> singularity shell --nv --writable writable_sandbox/
WARNING: Skipping mount /etc/localtime [binds]: /etc/localtime doesn't exist in container
Singularity> ls /.singularity.d/libs
libEGL.so                             libGLESv2.so.2                        libOpenGL.so.0                        libnvidia-encode.so.1                 libnvidia-opencl.so.1
libEGL.so.1                           libGLESv2_nvidia.so.2                 libcuda.so                            libnvidia-fatbinaryloader.so.440.100  libnvidia-opticalflow.so.1
libEGL_nvidia.so.0                    libGLX.so                             libcuda.so.1                          libnvidia-fbc.so.1                    libnvidia-ptxjitcompiler.so.1
libGL.so                              libGLX.so.0                           libnvcuvid.so.1                       libnvidia-glcore.so.440.100           libnvidia-rtcore.so.440.100
libGL.so.1                            libGLX_nvidia.so.0                    libnvidia-cbl.so.440.100              libnvidia-glsi.so.440.100             libnvidia-tls.so.440.100
libGLESv1_CM.so                       libGLdispatch.so                      libnvidia-cfg.so.1                    libnvidia-glvkspirv.so.440.100        libnvoptix.so.1
libGLESv1_CM.so.1                     libGLdispatch.so.0                    libnvidia-compiler.so.440.100         libnvidia-gtk3.so.440.100
libGLESv1_CM_nvidia.so.1              libOpenCL.so.1                        libnvidia-egl-wayland.so.1            libnvidia-ifr.so.1
libGLESv2.so                          libOpenGL.so                          libnvidia-eglcore.so.440.100          libnvidia-ml.so.1
Singularity> echo date > /writing_here
Singularity> cat /writing_here 
date

[dtrudg]

This is a CtrlIQ roadmap item: https://docs.google.com/document/d/1mEjRmwOfFXiz4U5uPQ6paa-h3jDgMeGoxM5PPs6r8SI