Deeplearning4j: nd4j-cuda-10.1-platform failure due to jnicusparse.dll issue

Created on 24 May 2019  路  20Comments  路  Source: eclipse/deeplearning4j

I setup nd4j-cuda-10.1-platform in my pom.xml but when I run my program. It threw an exception with following root cause:

_Caused by: java.lang.UnsatisfiedLinkError: C:Userszollen.javacppcachecuda-10.1-7.5-1.5-windows-x86_64.jarorgbytedecocudawindows-x86_64jnicusparse.dll: The specified procedure could not be found_

System Information

<dependency>
    <groupId>org.deeplearning4j</groupId>
    <artifactId>deeplearning4j-core</artifactId>
    <version>1.0.0-beta4</version>
</dependency>
 <dependency>
          <groupId>org.nd4j</groupId>
          <artifactId>nd4j-cuda-10.1-platform</artifactId>
           <version>1.0.0-beta4</version>
</dependency>

C:\Users\zollen>nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Wed_Apr_24_19:11:20_Pacific_Daylight_Time_2019
Cuda compilation tools, release 10.1, V10.1.168



md5-eef807a38b08fd3589c5325b5160543e



C:\Users\zollen>systeminfo | findstr /B /C:"OS Name" /C:"OS Version"
OS Name:                   Microsoft Windows 10 Home
OS Version:                10.0.17763 N/A Build 17763



md5-eef807a38b08fd3589c5325b5160543e



Nvidia GeForce GTX 1070
Driver Version: 430.64
ND4J Question
Source
picture zollen

Most helpful comment

Yeah, CUDA updates have been getting really strange since 10.1. Just use the bundled binaries as per this pom.xml file here instead of trying to figure out which version of CUDA you need:
https://github.com/bytedeco/javacpp-presets/tree/1.5/cuda#the-pomxml-build-file

picture saudet on 24 May 2019
👍5

All 20 comments

Do you have cuDNN set up also (i.e., cudnn DLL on the system PATH environment variable)? IIRC I've seen "The specified procedure could not be found" before with wrong cuDNN version, though you'd probably need deeplearning4j-cuda-10.1 dependency also for that to be a cause of this...

AlexDBlack picture AlexDBlack on 24 May 2019

May I ask what are the deeplearning4j-cuda-10.1 dependencies?

zollen picture zollen on 24 May 2019

May I ask what are the deeplearning4j-cuda-10.1 dependencies?

https://deeplearning4j.org/docs/latest/deeplearning4j-config-cudnn

So literally,

<dependency>
    <groupId>org.deeplearning4j</groupId>
    <artifactId>deeplearning4j-cuda-10.1</artifactId>
    <version>1.0.0-beta4</version>
</dependency>

To confirm - you don't also have that in your pom.xml? (If not, that's fine, we can rule out one possible cause).

AlexDBlack picture AlexDBlack on 24 May 2019

I don't have the following in my pom.xml.

<dependency>
    <groupId>org.deeplearning4j</groupId>
    <artifactId>deeplearning4j-cuda-10.1</artifactId>
    <version>1.0.0-beta4</version>
</dependency>
zollen picture zollen on 24 May 2019

This is my latest pom.xml and I have also installed Nvidia Cudnn, but I am still getting the same error.

<dependency>
    <groupId>org.deeplearning4j</groupId>
    <artifactId>deeplearning4j-core</artifactId>
    <version>1.0.0-beta4</version>
</dependency>
 <dependency>
          <groupId>org.nd4j</groupId>
          <artifactId>nd4j-cuda-10.1-platform</artifactId>
           <version>1.0.0-beta4</version>
</dependency>
<dependency>
    <groupId>org.deeplearning4j</groupId>
    <artifactId>deeplearning4j-cuda-10.1</artifactId>
    <version>1.0.0-beta4</version>
</dependency>
zollen picture zollen on 24 May 2019

Show nvidia-smi output please.

raver119 picture raver119 on 24 May 2019

Do you have cuDNN set up also (i.e., cudnn DLL on the system PATH environment variable)? IIRC I've seen "The specified procedure could not be found" before with wrong cuDNN version, though you'd probably need deeplearning4j-cuda-10.1 dependency also for that to be a cause of this...

is cudnn a mandatory library ? (just because the documentation doesn't specify it at all in the set-up phase)

Aelentel picture Aelentel on 24 May 2019

No, cuDNN is not mandatory. Just recommended if you're working with convolutional/recurrent stuff.

raver119 picture raver119 on 24 May 2019
👍1

No, cuDNN is not mandatory. Just recommended if you're working with convolutional/recurrent stuff.

thanks, but i've exactly the same problem, even after installing CuDNN (i checked the version against Cuda 10.1 to be sure)
i'm working on Word2Vec and TSNE to reduce dimensionality, the final goal is to train a text categorisation model.

i'll put all the information i can collect, please if something is wrong, doesn't hesitate to bash me :)

here's the actual stacktace on my side when doing Word2Vec with Cuda :

Caused by: java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
    at org.nd4j.linalg.factory.Nd4j.initWithBackend(Nd4j.java:5900)
    at org.nd4j.linalg.factory.Nd4j.initContext(Nd4j.java:5766)
    at org.nd4j.linalg.factory.Nd4j.<clinit>(Nd4j.java:202)
    ... 4 more
Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:408)
    at org.nd4j.linalg.factory.Nd4j.initWithBackend(Nd4j.java:5860)
    ... 6 more
Caused by: java.lang.UnsatisfiedLinkError: no jnicusparse in java.library.path
    at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1857)
    at java.lang.Runtime.loadLibrary0(Runtime.java:870)
    at java.lang.System.loadLibrary(System.java:1119)
    at org.bytedeco.javacpp.Loader.loadLibrary(Loader.java:1336)
    at org.bytedeco.javacpp.Loader.load(Loader.java:1077)
    at org.bytedeco.javacpp.Loader.load(Loader.java:947)
    at org.bytedeco.cuda.global.cusparse.<clinit>(cusparse.java:15)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:340)
    at org.bytedeco.javacpp.Loader.load(Loader.java:1006)
    at org.bytedeco.javacpp.Loader.load(Loader.java:947)
    at org.bytedeco.cuda.global.cusolver.<clinit>(cusolver.java:19)
    at org.nd4j.jita.allocator.context.impl.BasicContextPool.createNewSolverHandle(BasicContextPool.java:220)
    at org.nd4j.jita.allocator.context.impl.LimitedContextPool.fillPoolWithResources(LimitedContextPool.java:115)
    at org.nd4j.jita.allocator.context.impl.LimitedContextPool.<init>(LimitedContextPool.java:80)
    at org.nd4j.jita.handler.impl.CudaZeroHandler.<init>(CudaZeroHandler.java:145)
    at org.nd4j.jita.allocator.impl.AtomicAllocator.<init>(AtomicAllocator.java:145)
    at org.nd4j.jita.allocator.impl.AtomicAllocator.<clinit>(AtomicAllocator.java:93)
    at org.nd4j.linalg.jcublas.JCublasNDArrayFactory.<init>(JCublasNDArrayFactory.java:85)
    ... 11 more
Caused by: java.lang.UnsatisfiedLinkError: C:\Users\Sandro\.javacpp\cache\cuda-10.1-7.5-1.5-windows-x86_64.jar\org\bytedeco\cuda\windows-x86_64\jnicusparse.dll: The specified procedure could not be found
    at java.lang.ClassLoader$NativeLibrary.load(Native Method)
    at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1929)
    at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1814)
    at java.lang.Runtime.load0(Runtime.java:809)
    at java.lang.System.load(System.java:1083)
    at org.bytedeco.javacpp.Loader.loadLibrary(Loader.java:1316)
    ... 26 more

my POM deps and props :

<dl4j.version>1.0.0-beta4</dl4j.version>

        <dependency>
            <groupId>org.deeplearning4j</groupId>
            <artifactId>deeplearning4j-ui_2.11</artifactId>
            <version>${dl4j.version}</version>
        </dependency>
        <dependency>
            <groupId>org.deeplearning4j</groupId>
            <artifactId>deeplearning4j-modelimport</artifactId>
            <version>${dl4j.version}</version>
        </dependency>
        <dependency>
            <groupId>org.deeplearning4j</groupId>
            <artifactId>deeplearning4j-nn</artifactId>
            <version>${dl4j.version}</version>
        </dependency>
        <dependency>
            <groupId>org.deeplearning4j</groupId>
            <artifactId>deeplearning4j-cuda-10.1</artifactId>
            <version>${dl4j.version}</version>
        </dependency>
        <dependency>
            <groupId>org.nd4j</groupId>
            <artifactId>nd4j-cuda-10.0-platform</artifactId>
            <version>${dl4j.version}</version>
        </dependency>
        <dependency>
            <groupId>org.nd4j</groupId>
            <artifactId>nd4j-native-platform</artifactId>
            <version>${dl4j.version}</version>
        </dependency>

my nvcc and device query .

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Wed_Apr_24_19:11:20_Pacific_Daylight_Time_2019
Cuda compilation tools, release 10.1, V10.1.168

deviceQuery.exe Starting...

 CUDA Device Query (Runtime API) version (CUDART static linking)

Detected 1 CUDA Capable device(s)

Device 0: "GeForce GTX 970"
  CUDA Driver Version / Runtime Version          10.1 / 10.1
  CUDA Capability Major/Minor version number:    5.2
  Total amount of global memory:                 4096 MBytes (4294967296 bytes)
  (13) Multiprocessors, (128) CUDA Cores/MP:     1664 CUDA Cores
  GPU Max Clock rate:                            1329 MHz (1.33 GHz)
  Memory Clock rate:                             3505 Mhz
  Memory Bus Width:                              256-bit
  L2 Cache Size:                                 1835008 bytes
  Maximum Texture Dimension Size (x,y,z)         1D=(65536), 2D=(65536, 65536), 3D=(4096, 4096, 4096)
  Maximum Layered 1D Texture Size, (num) layers  1D=(16384), 2048 layers
  Maximum Layered 2D Texture Size, (num) layers  2D=(16384, 16384), 2048 layers
  Total amount of constant memory:               zu bytes
  Total amount of shared memory per block:       zu bytes
  Total number of registers available per block: 65536
  Warp size:                                     32
  Maximum number of threads per multiprocessor:  2048
  Maximum number of threads per block:           1024
  Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
  Max dimension size of a grid size    (x,y,z): (2147483647, 65535, 65535)
  Maximum memory pitch:                          zu bytes
  Texture alignment:                             zu bytes
  Concurrent copy and kernel execution:          Yes with 2 copy engine(s)
  Run time limit on kernels:                     Yes
  Integrated GPU sharing Host Memory:            No
  Support host page-locked memory mapping:       Yes
  Alignment requirement for Surfaces:            Yes
  Device has ECC support:                        Disabled
  CUDA Device Driver Mode (TCC or WDDM):         WDDM (Windows Display Driver Model)
  Device supports Unified Addressing (UVA):      Yes
  Device supports Compute Preemption:            No
  Supports Cooperative Kernel Launch:            No
  Supports MultiDevice Co-op Kernel Launch:      No
  Device PCI Domain ID / Bus ID / location ID:   0 / 2 / 0
  Compute Mode:
     < Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) >

deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 10.1, CUDA Runtime Version = 10.1, NumDevs = 1, Device0 = GeForce GTX 970
Result = PASS
Aelentel picture Aelentel on 24 May 2019

I did a system search for jnicusparse.dll:

C:Userszollen.javacppcachecuda-10.1-7.5-1.5-windows-x86_64.jarorgbytedecocudawindows-x86_64>dir
Volume in drive C has no label.
Volume Serial Number is 945B-4DD2

Directory of C:Userszollen.javacppcachecuda-10.1-7.5-1.5-windows-x86_64.jarorgbytedecocudawindows-x86_64

2019-05-22 07:50 PM

.
2019-05-22 07:50 PM ..
2019-04-09 09:03 AM 894,464 jnicublas.dll
2019-04-09 09:03 AM 1,948,672 jnicudart.dll
2019-04-09 09:04 AM 1,940,992 jnicusparse.dll
3 File(s) 4,784,128 bytes
2 Dir(s) 160,367,009,792 bytes free

zollen picture zollen on 24 May 2019

Curious.... It displays CUDA Version 10.2...

C:\Program Files\NVIDIA Corporation\NVSMI>nvidia-smi
Fri May 24 05:35:58 2019
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 430.64       Driver Version: 430.64       CUDA Version: 10.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name            TCC/WDDM | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 1070   WDDM  | 00000000:01:00.0  On |                  N/A |
|  0%   35C    P8    10W / 185W |    399MiB /  8192MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0      1200    C+G   Insufficient Permissions                   N/A      |
|    0      3916    C+G   ...oftEdge_8wekyb3d8bbwe\MicrosoftEdge.exe N/A      |
|    0      6784    C+G   ...)\Microsoft\Skype for Desktop\Skype.exe N/A      |
|    0      6832    C+G   ...dows.Cortana_cw5n1h2txyewy\SearchUI.exe N/A      |
|    0      7352    C+G   C:\Windows\explorer.exe                    N/A      |
|    0      8116    C+G   ...t_cw5n1h2txyewy\ShellExperienceHost.exe N/A      |
|    0      8980    C+G   ....481.0_x64__8wekyb3d8bbwe\YourPhone.exe N/A      |
|    0     10452    C+G   ...)\Microsoft\Skype for Desktop\Skype.exe N/A      |
|    0     11044    C+G   ...DIA GeForce Experience\NVIDIA Share.exe N/A      |
|    0     11628    C+G   ...\bin\cef\cef.win7x64\steamwebhelper.exe N/A      |
|    0     12560    C+G   ...6)\Google\Chrome\Application\chrome.exe N/A      |
|    0     14092    C+G   C:\Windows\System32\MicrosoftEdgeCP.exe    N/A      |
|    0     15244    C+G   ... Files (x86)\Dropbox\Client\Dropbox.exe N/A      |
|    0     15484    C+G   ...mmersiveControlPanel\SystemSettings.exe N/A      |
|    0     16152    C+G   ...Corsair\Corsair Utility Engine\iCUE.exe N/A      |
|    0     17360    C+G   ...hell.Experiences.TextInput.InputApp.exe N/A      |
+-----------------------------------------------------------------------------+
zollen picture zollen on 24 May 2019

CUDA Version: 10.2

See this line in your nvidia-smi output? That one looks strange, since latest CUDA is 10.1

Also, you're showing me two different systems here. One is cc 52, other one is cc 61.

raver119 picture raver119 on 24 May 2019
👍1
  1. That makes no senses. If the official Nvidia CUDA latest release 10.1. How did I managed to get a 10.2 CUDA? Is there any difference between CUDA version and CUDA driver version?
  2. What is cc52 and cc61? Where did you spot those?
  3. I have only a single Windows 10. What do you mean two different systems?
zollen picture zollen on 24 May 2019

2/3:

Detected 1 CUDA Capable device(s)

Device 0: "GeForce GTX 970"
  CUDA Driver Version / Runtime Version          10.1 / 10.1
  CUDA Capability Major/Minor version number:    5.2

and 

 GeForce GTX 1070   WDDM

that's two different GPUs, with different compute capabilities.

raver119 picture raver119 on 24 May 2019

Well, anyway, the problem looks like binary incompatibility between what we've used to build and what you use. You're probably using CUDA 10.1 Update 1. Please downgrade to regular CUDA 10.1 (there are previous versions available on nvidia website), and everything should work fine.

And dont forget to reboot your box after downgrade.

raver119 picture raver119 on 24 May 2019
👍1

I shall downgrade my CUDA SDK and see how it goes. Thanks for the tips.

C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1\extras\demo_suite>deviceQuery.exe
deviceQuery.exe Starting...

 CUDA Device Query (Runtime API) version (CUDART static linking)

Detected 1 CUDA Capable device(s)

Device 0: "GeForce GTX 1070"
  CUDA Driver Version / Runtime Version          10.2 / 10.1
  CUDA Capability Major/Minor version number:    6.1
zollen picture zollen on 24 May 2019

Yeah, CUDA updates have been getting really strange since 10.1. Just use the bundled binaries as per this pom.xml file here instead of trying to figure out which version of CUDA you need:
https://github.com/bytedeco/javacpp-presets/tree/1.5/cuda#the-pomxml-build-file

saudet picture saudet on 24 May 2019
👍5

It works!! Thanks for the quick help!!

zollen picture zollen on 25 May 2019
👍1

Yeah, CUDA updates have been getting really strange since 10.1. Just use the bundled binaries as per this pom.xml file here instead of trying to figure out which version of CUDA you need:
https://github.com/bytedeco/javacpp-presets/tree/1.5/cuda#the-pomxml-build-file

thanks a bunch ! worked on my side too, this should really go on the official documentation if problem arise !

Aelentel picture Aelentel on 25 May 2019
👍2

it works. just add few lines to pom.xml 

<dependency>
    <groupId>org.bytedeco</groupId>
    <artifactId>cuda</artifactId>
    <version>10.1-7.5-1.5</version>
    <classifier>windows-x86_64-redist</classifier>
</dependency>
BourneSuper picture BourneSuper on 14 Nov 2019
👍1
Was this page helpful?
0 / 5 - 0 ratings

Related issues

hanbaoan123 picture hanbaoan123  路  4Comments
maxgfr picture maxgfr  路  4Comments
Paranormaly picture Paranormaly  路  5Comments
AlexanderSerbul picture AlexanderSerbul  路  3Comments
tintinxue1 picture tintinxue1  路  5Comments