Nvidia-docker: Why a volume driver?

[sargun]

Why is nVidia Docker a volume driver? Wouldn't it suffice just to give a bind mount?

[flx42]

It could, but you would have to search for the libraries and the binaries every single time. With a volume you only need to do the search once.
Our volumes are versioned following the NVIDIA driver version itself, so you can switch drivers easily and reuse the existing volumes.
Since we use hard-links inside the volume, we are not wasting space (but it does cause other issues...).

It's also useful for remote deployments: you can ask a remote nvidia-docker-plugin to provision the volume corresponding to the driver version on this other machine.
Finally, it makes the command-line more manageable when looking at it:

$ curl -s localhost:3476/docker/cli
--volume-driver=nvidia-docker --volume=nvidia_driver_375.26:/usr/local/nvidia:ro --device=/dev/nvidiactl --device=/dev/nvidia-uvm --device=/dev/nvidia-uvm-tools --device=/dev/nvidia1 --device=/dev/nvidia0

[sanjeevm0]

I am still a little confused. Why not just do?

docker run --volume=nvidia_driver_375.26:/usr/local/nvidia:ro --device=/dev/nvidiactl --device=/dev/nvidia-uvm --device=/dev/nvidia-uvm-tools --device=/dev/nvidia1 --device=/dev/nvidia0 ...

What value does the following option added to the "docker run" command line provide to docker?
--volume-driver=nvidia-docker

The library search etc., is handled by the nvidia-docker-plugin (the Docker plugin).

[flx42]

Without --volume-driver, you will use the local driver and then you are just creating an empty named volume, our nvidia-docker-plugin will not be called.

[sanjeevm0]

I see, makes sense, so inside docker it will utilize the nvidia-docker volume driver which will subsequently call the plugin to find the correct files.

Another related question would be, it seems volume-driver can only be called once, and it seems --volume-driver has been depcrated, https://github.com/docker/docker/pull/16595. Would this cause any issues if different drivers are needed (for example if other volumes are also being added)? Thanks!

[flx42]

It doesn't look like --volume-driver is deprecated, there was indeed a proposal to do so, but it was denied.

You are right that you can only use one volume-driver, but there is a trick: if the volume is already created, you don't need to use volume-driver afterwards. See below:

# This will create an empty named volume:
$ docker run --rm -ti --volume=nvidia_driver_375.26:/usr/local/nvidia:ro ubuntu ls -l /usr/local/nvidia
total 0

$ docker volume rm -f nvidia_driver_375.26
nvidia_driver_375.26

# Create the right volume through our plugin, manually.
$ docker volume create --driver=nvidia-docker nvidia_driver_375.26
nvidia_driver_375.26

# Same command as above, but now it works
docker run --rm -ti --volume=nvidia_driver_375.26:/usr/local/nvidia:ro ubuntu ls -l /usr/local/nvidia
total 12
drwxr-xr-x 2 999 998 4096 Mar 23 20:55 bin
drwxr-xr-x 2 999 998 4096 Mar 23 20:55 lib
drwxr-xr-x 2 999 998 4096 Mar 23 20:55 lib64