Nvidia-docker: VolumeDriver.Create: internal error on Ubuntu 16.04

[dpatschke]

I am unable to run the example given on the front page:
sudo nvidia-docker run --rm nvidia/cuda nvidia-smi

I receive the following error:
docker: Error response from daemon: create nvidia_driver_367.35: VolumeDriver.Create: internal error, check logs for details.

Details surrounding the attempted problem resolution are documented in issue #133. My nvidia drivers are installed in '/usr' but my '/var' directory is sitting on a separate partition. I have reinstalled nvidia-docker, created an override.conf file as recommended in the comments of the above issue, and rebooted.

Below is the output from the journalctl command:

Jul 21 08:52:56 Precision-Tower-7910 systemd[1]: Starting NVIDIA Docker plugin...
Jul 21 08:52:56 Precision-Tower-7910 systemd[1]: Started NVIDIA Docker plugin.
Jul 21 08:52:56 Precision-Tower-7910 nvidia-docker-plugin[1142]: /usr/bin/nvidia-docker-plugin | 2016/07/21 08:52:56 Loading NVIDIA unified memory
Jul 21 08:52:56 Precision-Tower-7910 nvidia-docker-plugin[1142]: /usr/bin/nvidia-docker-plugin | 2016/07/21 08:52:56 Loading NVIDIA management library
Jul 21 08:52:56 Precision-Tower-7910 nvidia-docker-plugin[1142]: /usr/bin/nvidia-docker-plugin | 2016/07/21 08:52:56 Discovering GPU devices
Jul 21 08:52:57 Precision-Tower-7910 nvidia-docker-plugin[1142]: /usr/bin/nvidia-docker-plugin | 2016/07/21 08:52:57 Provisioning volumes at /usr/local/nvidia-driver
Jul 21 08:52:57 Precision-Tower-7910 nvidia-docker-plugin[1142]: /usr/bin/nvidia-docker-plugin | 2016/07/21 08:52:57 Serving plugin API at /var/lib/nvidia-docker
Jul 21 08:52:57 Precision-Tower-7910 nvidia-docker-plugin[1142]: /usr/bin/nvidia-docker-plugin | 2016/07/21 08:52:57 Serving remote API at localhost:3476

[flx42]

After the nvidia-docker run command, you have nothing else showing up in journalctl? That would be surprising.

[dpatschke]

Yes!!! That did it ... well ... your suggestion led me to figure out what was going on.

You are correct ... I did not execute the 'run' command. However, when I did, I noticed it was failing on the 'mkdir' command. Once I gave the folder the correct permission, it worked liked a charm.

Thanks again for your help! I will close this issue and attempt to remember all the various commands to help diagnose potential issues in the future :-).

[flx42]

Ah yes, it's a little tricky since the plugin runs under user nvidia-docker. Sorry for the trouble!

[jingpengw]

This works for me too, Thanks 💯

To be more clear:

systemctl edit nvidia-docker

add following:

[Service]
ExecStart=
ExecStart=/usr/bin/nvidia-docker-plugin -s $SOCK_DIR -d /usr/local/nvidia-docker-test

sudo mkdir /usr/local/nvidia-docker-test
sudo chown nvidia-docker /usr/local/nvidia-docker-test
sudo systemctl restart docker nvidia-docker

[ferrouswheel]

I got this after an update from 14.04 to 16.04, and after purging and reinstalling nvidia-docker. To fix I just had to do:

sudo chown -R nvidia-docker:nvidia-docker /var/lib/nvidia-docker/volumes/

[lxzh]

create the volume mamually
nvidia-docker volume create nvidia_driver_375.66

[zhangzengli]

随意创建体积
nvidia-docker volume create nvidia_driver_375.66

this is ok,thanks!