hello,all
when using mx.nd function, sometimes it output the error like this:
./dmlc-core/include/dmlc/logging.h:208: [17:19:40] ./mshadow/mshadow/./stream_gpu-inl.h:98: Check failed: (err) == (CUBLAS_STATUS_SUCCESS) Create cublas handle failed
terminate called after throwing an instance of 'dmlc::Error'
terminate called recursively
it is strange for me that all the example and test is correct.
so the simple code which outputs error above is(test.py here):
import mxnet as mx
import numpy as np
import logging
logger = logging.getLogger()
logger.setLevel(logging.INFO)
ctx = mx.gpu(0)
if __name__ == "__main__":
mx.nd.zeros((3L, 1L, 5L, 5L), ctx)
and run in shell: python -u test.py, it will error, but when run line by line in python terminal, no erroe there, why?
ths~
there are issues with singleton destruction in mxnet when your program shutdown before mxnet is fully initialized. Just add a time.sleep(5) at the end and it should be find.
@piiswrong thanks, i'll try to do this,
I confirm the solution worked. If not, try a larger number, like 35 seconds.
I met with the similar problem and the solution provided here solves my problem.
Thank you.
or just put a mx.nd.waitall()
Most helpful comment
there are issues with singleton destruction in mxnet when your program shutdown before mxnet is fully initialized. Just add a time.sleep(5) at the end and it should be find.