Cupy: Nothing works, with error: CUDADriverError: CUDA_ERROR_INVALID_PTX: a PTX JIT compilation failed

[KShutter]

I installed CuPy into a new machine of mine. I did install using pip install cupy-cuda91, since my CUDA version is 9.1. However, whenever I try to use anything form CuPy, I get an error message saying "CUDADriverError: CUDA_ERROR_INVALID_PTX: a PTX JIT compilation failed".

Below I give details.

• Conditions (from python -c 'import cupy; cupy.show_config()'):

  CuPy Version : 5.4.0
  CUDA Root : /usr
  CUDA Build Version : 9010
  CUDA Driver Version : 10010
  CUDA Runtime Version : 9010
  cuDNN Build Version : 7102
  cuDNN Version : 7102
  NCCL Build Version : 2115
  NCCL Runtime Version : (unknown)

• Code to reproduce (obviously, same code will work normally in computers without the problem):

  import cupy as cp
  x = cp.array([[1,2],[3,4]])
  x.dot(x)

• Error messages, stack traces, or logs:

CUDADriverError Traceback (most recent call last)
in ()
----> 1 x.dot(x)

cupy/core/core.pyx in cupy.core.core.ndarray.dot()

cupy/core/core.pyx in cupy.core.core.dot()

cupy/core/core.pyx in cupy.core.core.tensordot_core()

cupy/core/core.pyx in cupy.core.core.ndarray.astype()

cupy/core/core.pyx in cupy.core.core.ndarray.astype()

cupy/core/_kernel.pyx in cupy.core._kernel.ufunc.__call__()

cupy/util.pyx in cupy.util.memoize.decorator.ret()

cupy/core/_kernel.pyx in cupy.core._kernel._get_ufunc_kernel()

cupy/core/_kernel.pyx in cupy.core._kernel._get_simple_elementwise_kernel()

cupy/core/carray.pxi in cupy.core.core.compile_with_cache()

~/Programs/anaconda3/lib/python3.6/site-packages/cupy/cuda/compiler.py in compile_with_cache(source, options, arch, cache_dir, extra_source)
164 ptx = compile_using_nvrtc(source, options, arch)
165 ls = function.LinkState()
--> 166 ls.add_ptr_data(ptx, six.u('cupy.ptx'))
167 cubin = ls.complete()
168 cubin_hash = six.b(hashlib.md5(cubin).hexdigest())

cupy/cuda/function.pyx in cupy.cuda.function.LinkState.add_ptr_data()

cupy/cuda/function.pyx in cupy.cuda.function.LinkState.add_ptr_data()

cupy/cuda/driver.pyx in cupy.cuda.driver.linkAddData()

cupy/cuda/driver.pyx in cupy.cuda.driver.check_status()

CUDADriverError: CUDA_ERROR_INVALID_PTX: a PTX JIT compilation failed

[kmaehashi]

Could you try specifying CUDA_PATH and LD_LIBRARY_PATH?
https://docs-cupy.chainer.org/en/latest/install.html#cupy-always-raises-cupy-cuda-compiler-compileexception

[HesseSummer]

I tried @kmaehashi's advice, and it works for me, thank you. The following is my practice:

1. ls /usr/local/cuda*

/usr/local/cuda-10.0:
bin      lib64      NsightCompute-1.0  README   targets
doc      libnsight  nsightee_plugins   samples  tools
extras   libnvvp    nvml               share    version.txt
include  LICENSE    nvvm               src

1. gedit ~/.profile
   Copy the following to the end of the file

# set PATH for cuda 10.0 installation
if [ -d "/usr/local/cuda-10.0/bin/" ]; then
    export PATH=/usr/local/cuda-10.0/bin${PATH:+:${PATH}}
    export LD_LIBRARY_PATH=/usr/local/cuda-10.0/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
fi

1. source ~/.profile
2. nvcc --version

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2018 NVIDIA Corporation
Built on Sat_Aug_25_21:08:01_CDT_2018
Cuda compilation tools, release 10.0, V10.0.130

1. pip uninstall cupy
2. pip install cupy-cuda100

[pfn-ci-bot]

  [INVALID_ARGUMENT] unknown command: usr/local/cuda-10.0: (7573722f6c6f63616c2f637564612d31302e303a)
  2019-09-04 21:49:06.676884 github_issue_comment.go:99] unknown command: usr/local/cuda-10.0: (7573722f6c6f63616c2f637564612d31302e303a)

  Stack trace:
    github.com/pfnet/imosci/util/frontend/handler/apihandler.(*githubWebhookIssueCommentFlow).Do (github_issue_comment.go:99)
    github.com/pfnet/imosci/util/frontend/handler/apihandler.githubIssueCommentHandler (github_issue_comment.go:45)
    runtime.call64 (asm_amd64.s:523)
    reflect.Value.call (value.go:447)
    reflect.Value.Call (value.go:308)
    github.com/pfnet/imosci/util/frontend/core.RegisterAPIHandlerInternal.func1 (handler.go:417)
    github.com/pfnet/imosci/util/frontend/core.RegisterHandler.func1 (handler.go:173)
    github.com/pfnet/imosci/util/frontend/core.RegisterHandler.func2.1 (handler.go:275)
    github.com/pfnet/imosci/util/frontend/core.RegisterHandler.func2 (handler.go:280)
    net/http.HandlerFunc.ServeHTTP (server.go:1964)
    net/http.(*ServeMux).ServeHTTP (server.go:2361)
    github.com/pfnet/imosci/util/api.callInternal.func2 (call.go:196)
    github.com/pfnet/imosci/util/api.callInternal (call.go:204)
    github.com/pfnet/imosci/util/api.Call (call.go:120)
    github.com/pfnet/imosci/util/api.GithubIssueComment (call.go:492)
    github.com/pfnet/imosci/util/frontend/handler/xternalhandler.githubWebhookHandler (github_webhook.go:118)
    github.com/pfnet/imosci/util/frontend/core.RegisterHandler.func1 (handler.go:173)
    github.com/pfnet/imosci/util/frontend/core.RegisterHandler.func2.1 (handler.go:275)
    github.com/pfnet/imosci/util/frontend/core.RegisterHandler.func2 (handler.go:280)
    net/http.HandlerFunc.ServeHTTP (server.go:1964)
    net/http.(*ServeMux).ServeHTTP (server.go:2361)
    google.golang.org/appengine/internal.executeRequestSafely (api.go:162)
    google.golang.org/appengine/internal.handleHTTP (api.go:121)
    net/http.HandlerFunc.ServeHTTP (server.go:1964)
    net/http.serverHandler.ServeHTTP (server.go:2741)
    net/http.(*conn).serve (server.go:1847)
    runtime.goexit (asm_amd64.s:1333)