Keras: modprobe: ERROR: could not insert 'nvidia_361_uvm': Invalid argument

[ShuaiW]

• GPU: GTX1080
• GPU driver: 367.27
• OS: Ubuntu 16.04
• CUDA: 8.0 rc

When I run
THEANO_FLAGS=device=gpu,floatX=float32 python *.py

I got

Using Theano backend.
modprobe: ERROR: could not insert 'nvidia_361_uvm': Invalid argument
WARNING (theano.sandbox.cuda): CUDA is installed, but device gpu is not available (error: Unable to get the number of gpus available: unknown error)

EDIT:
For a step by step setting of using GTX 1070/1080 and Ubuntu 16.04 for deep learning, see this markdown my friend and I put together

[aymericdamien]

That should solve your error:

sudo apt-get remove nvidia-361

You got both drivers 361and 367, for some reason it is trying to use 361 which is not compatible with the GTX1080.

[StevenLOL]

Same error here, the problem is if

sudo apt-get remove nvidia-361

then

The following packages will be REMOVED:
  cuda cuda-8-0 cuda-demo-suite-8-0 cuda-drivers cuda-runtime-8-0 nvidia-361 nvidia-361-dev nvidia-361-updates
0 upgraded, 0 newly installed, 8 to remove and 1 not upgraded.

[jonathanstrong]

for anyone finding this: just spent a day trying to get a gtx 1070 to run on ubuntu 16.04 w/ CUDA 8 rc and Theano. Here are some guidelines:

• gtx 1070 and 1080 require a nvidia 367 driver. You can't currently get this from apt-get, so you need to download a runfile and execute it w/ the ubuntu gui shut down. here's a tutorial: http://www.yourownlinux.com/2016/06/how-to-install-nvidia-367-27-stable-graphics-drivers-in-linux.html
• cuda 7.5 did not seem to want to work w/ the 367 driver. Potentially I could have fixed this. Instead I decided to install cuda 8.0 rc. However, using the deb file installation approach automatically installed the nvidia 361 drivers, which caused a conflict that was very difficult to remove. This is the source of the "could not insert 'nvidia_361_uvm'" error.
• Also, cuda 8.0 rc requires you to reinstall nvidia drivers anyway, which I did not realize.
• To repair:
  
  
  
  • sudo apt-get purge nvidia* to remove 361 and its assorted packages.
  
  
  • get to pure shell w/ ctrl+alt+f2, shut down lightdm, and use the 367 runfile w/ the --uninstall argument to uninstall the 367 drivers. i.e. sudo sh nvidia367.xx.run --uninstall
  
  
  • w/ lightdm shutdown, reinstall 367 drivers with the runfile. i.e. sudo sh nvidia367.xx.run
  
  
  • install cuda-8.0 rc w/ its runfile, which gives you the _option_ to install the 361 drivers. When presented with it, choose no. i.e. sudo sh cuda-8.0.x.x.run
  
  
  • following this I needed to update some PATH stuff that was pointing to the cuda 7.5 folder. Also I had copied cuDNN files into 7.5 include and lib64 folders, so those files needed to be copied to the 8.0 equivalents.
  
  

Hope this helps someone trying to fix this stuff. it was a huge headache.

[ibujanov]

Great help "jonathanstrong", saved me the day ;-)

[ahundt]

To shutdown lightdm in the instructions of @jonathanstrong sudo service lightdm stop

[joshuacv]

I had a similar issue, only that it was conflict with a nvidia-352 against the automatically installed nvidia-367 along with cuda-8.0. I did not have to do much except to switch to pure shell, stop lightdm, restart to let nvidia 367 be the driver and then it worked! Thanks for the inputs on this thread. Having fun cud-ing.

PS: I am using an old 860M as the GPU not 1070 or 1080.

[DoubleXiJin]

The solution from aymericdamien 'sudo apt-get remove nvidia-xxx' is helpful for me to solve the problem: modprobe: ERROR: could not insert 'nvidia_340_uvm': Invalid argument. Thanks.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 30 days if no further activity occurs, but feel free to re-open a closed issue if needed.