Dali: Issue with VideoReader and libnvcuvid.so

[LightnessOfBeing]

I've been using a Paperspace notebook instance which is based on some default Pytorch image.

Running nvidia-smi gives:

Tue Aug 25 14:11:37 2020       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 450.36.06    Driver Version: 450.36.06    CUDA Version: 11.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Quadro M4000        On   | 00000000:00:05.0 Off |                  N/A |
| 46%   27C    P8    12W / 120W |      1MiB /  8126MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

and pip show yields:

absl-py==0.10.0
astunparse==1.6.3
attrs==19.1.0
backcall==0.1.0
bleach==3.1.0
cachetools==4.1.1
certifi==2020.6.20
chardet==3.0.4
cloudpickle==1.2.1
cycler==0.10.0
Cython==0.29.13
decorator==4.4.0
defusedxml==0.6.0
entrypoints==0.3
enum34==1.1.6
future==0.17.1
gast==0.3.3
google-auth==1.20.1
google-auth-oauthlib==0.4.1
google-pasta==0.2.0
grpcio==1.31.0
h5py==2.10.0
idna==2.10
importlib-metadata==1.7.0
ipykernel==5.1.1
ipython==7.7.0
ipython-genutils==0.2.0
ipywidgets==7.5.1
jedi==0.14.1
Jinja2==2.10.1
joblib==0.13.2
json5==0.8.5
jsonschema==3.0.2
jupyter==1.0.0
jupyter-client==5.3.1
jupyter-console==6.0.0
jupyter-core==4.5.0
jupyterlab==1.0.4
jupyterlab-server==1.0.0
Keras-Preprocessing==1.1.2
kiwisolver==1.1.0
Markdown==3.2.2
MarkupSafe==1.1.1
matplotlib==3.1.1
mistune==0.8.4
nbconvert==5.5.0
nbformat==4.4.0
notebook==6.0.0
numpy==1.17.0
oauthlib==3.1.0
opt-einsum==3.3.0
pandas==0.25.0
pandocfilters==1.4.2
parso==0.5.1
pexpect==4.7.0
pickleshare==0.7.5
Pillow==6.1.0
prometheus-client==0.7.1
prompt-toolkit==2.0.9
protobuf==3.13.0
ptyprocess==0.6.0
pyasn1==0.4.8
pyasn1-modules==0.2.8
Pygments==2.4.2
pygobject==3.26.1
pyparsing==2.4.2
pyrsistent==0.15.4
python-apt==1.6.4
python-dateutil==2.8.0
python-distutils-extra==2.39
pytz==2019.2
PyYAML==5.1.2
pyzmq==18.0.2
qtconsole==4.5.2
requests==2.24.0
requests-oauthlib==1.3.0
rsa==4.6
scikit-learn==0.21.3
scipy==1.4.1
Send2Trash==1.5.0
six==1.12.0
tensorboard==2.3.0
tensorboard-plugin-wit==1.7.0
tensorflow-estimator==2.3.0
tensorflow-gpu==2.3.0
termcolor==1.1.0
terminado==0.8.2
testpath==0.4.2
torch==1.1.0
torch-nightly==1.2.0.dev20190805
torchvision==0.4.0a0+d31eafa
tornado==6.0.3
traitlets==4.3.2
typing==3.7.4
urllib3==1.25.10
wcwidth==0.1.7
webencodings==0.5.1
Werkzeug==1.0.1
widgetsnbextension==3.5.1
wrapt==1.12.1
zipp==3.1.0

(I've actually manually installed latest tensorflow-gpu package from pip).

So, the Cuda version is 11. Then I go to the installation page and run the following command:

pip install --extra-index-url https://developer.download.nvidia.com/compute/redist nvidia-dali-cuda110

And everything is great and it shows that Successfully installed nvidia-dali-cuda110-0.23.0.

Then I have the following script which has a very minimal changes (just paths and file reading) from your example for VideoReader.

from __future__ import division
import os
import numpy as np

from nvidia.dali.pipeline import Pipeline
import nvidia.dali.ops as ops
import nvidia.dali.types as types

batch_size=2
sequence_length=8

initial_prefetch_size=16

video_directory = "./videos_test"

video_files=[video_directory + '/' + f for f in os.listdir(video_directory)]

shuffle=True

n_iter=6

class VideoPipe(Pipeline):
    def __init__(self, batch_size, num_threads, device_id, data, shuffle):
        super(VideoPipe, self).__init__(batch_size, num_threads, device_id, seed=16)
        self.input = ops.VideoReader(device="gpu", filenames=data, sequence_length=sequence_length,
                                     shard_id=0, num_shards=1,
                                     random_shuffle=shuffle, initial_fill=initial_prefetch_size)


    def define_graph(self):
        output = self.input(name="Reader")
        return output

pipe = VideoPipe(batch_size=batch_size, num_threads=2, device_id=0, data=video_files, shuffle=shuffle)
pipe.build()
for i in range(n_iter):
    pipe_out = pipe.run()
    sequences_out = pipe_out[0].as_cpu().as_array()
    print(sequences_out.shape)

Running python script.py gives:

dlopen "libnvcuvid.so" failed!
[/opt/dali/dali/operators/reader/video_reader_op.h:66] [/opt/dali/dali/operators/reader/loader/video_loader.h:182] Assert on "lib_handle_" failed: Failed to load libnvcuvid.so, needed by the VideoReader operator. If you are running in a Docker container, please refer to https://github.com/NVIDIA/nvidia-docker/wiki/Usage
Stacktrace (27 entries):
[frame 0]: /usr/local/lib/python3.6/dist-packages/nvidia/dali/libdali_operators.so(+0x39790e) [0x7f8ea60d290e]
[frame 1]: /usr/local/lib/python3.6/dist-packages/nvidia/dali/libdali_operators.so(+0x1541ec9) [0x7f8ea727cec9]
[frame 2]: /usr/local/lib/python3.6/dist-packages/nvidia/dali/libdali_operators.so(+0x15430b6) [0x7f8ea727e0b6]
[frame 3]: /usr/local/lib/python3.6/dist-packages/nvidia/dali/libdali_operators.so(+0x1544942) [0x7f8ea727f942]
[frame 4]: /usr/local/lib/python3.6/dist-packages/nvidia/dali/libdali_operators.so(std::_Function_handler<std::unique_ptr<dali::OperatorBase, std::default_delete<dali::OperatorBase> > (dali::OpSpec const&), std::unique_ptr<dali::OperatorBase, std::default_delete<dali::OperatorBase> > (*)(dali::OpSpec const&)>::_M_invoke(std::_Any_data const&, dali::OpSpec const&)+0xc) [0x7f8ea60ca92c]
[frame 5]: /usr/local/lib/python3.6/dist-packages/nvidia/dali/libdali.so(+0x17a364) [0x7f8e90900364]
[frame 6]: /usr/local/lib/python3.6/dist-packages/nvidia/dali/libdali.so(dali::InstantiateOperator(dali::OpSpec const&)+0x32f) [0x7f8e908ffcbf]
[frame 7]: /usr/local/lib/python3.6/dist-packages/nvidia/dali/libdali.so(dali::OpGraph::InstantiateOperators()+0xb1) [0x7f8e908bd641]
[frame 8]: /usr/local/lib/python3.6/dist-packages/nvidia/dali/libdali.so(dali::Pipeline::Build(std::vector<std::pair<std::string, std::string>, std::allocator<std::pair<std::string, std::string> > >)+0xa48) [0x7f8e9091a2b8]
[frame 9]: /usr/local/lib/python3.6/dist-packages/nvidia/dali/backend_impl.cpython-36m-x86_64-linux-gnu.so(+0x5164f) [0x7f8ec0de664f]
[frame 10]: /usr/local/lib/python3.6/dist-packages/nvidia/dali/backend_impl.cpython-36m-x86_64-linux-gnu.so(+0x59d53) [0x7f8ec0deed53]
[frame 11]: python(_PyCFunction_FastCallDict+0x35c) [0x56204c]
[frame 12]: python() [0x4f88ba]
[frame 13]: python(_PyEval_EvalFrameDefault+0x467) [0x4f98c7]
[frame 14]: python() [0x4f6128]
[frame 15]: python() [0x4f7d60]
[frame 16]: python() [0x4f876d]
[frame 17]: python(_PyEval_EvalFrameDefault+0x467) [0x4f98c7]
[frame 18]: python() [0x4f6128]
[frame 19]: python(PyEval_EvalCode+0x23) [0x4f9023]
[frame 20]: python() [0x6415b2]
[frame 21]: python(PyRun_FileExFlags+0x9a) [0x64166a]
[frame 22]: python(PyRun_SimpleFileExFlags+0x190) [0x643730]
[frame 23]: python(Py_Main+0x59e) [0x62b26e]
[frame 24]: python(main+0xe0) [0x4b4cb0]
[frame 25]: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7) [0x7f8ed9d9ab97]
[frame 26]: python(_start+0x2a) [0x5bdf6a]

Traceback (most recent call last):
  File "script.py", line 35, in <module>
    pipe.build()
  File "/usr/local/lib/python3.6/dist-packages/nvidia/dali/pipeline.py", line 437, in build
    self._pipe.Build(self._names_and_devices)
RuntimeError: [/opt/dali/dali/operators/reader/loader/video_loader.h:182] Assert on "lib_handle_" failed: Failed to load libnvcuvid.so, needed by the VideoReader operator. If you are running in a Docker container, please refer to https://github.com/NVIDIA/nvidia-docker/wiki/Usage
Stacktrace (27 entries):
[frame 0]: /usr/local/lib/python3.6/dist-packages/nvidia/dali/libdali_operators.so(+0x39790e) [0x7f8ea60d290e]
[frame 1]: /usr/local/lib/python3.6/dist-packages/nvidia/dali/libdali_operators.so(+0x1541ec9) [0x7f8ea727cec9]
[frame 2]: /usr/local/lib/python3.6/dist-packages/nvidia/dali/libdali_operators.so(+0x15430b6) [0x7f8ea727e0b6]
[frame 3]: /usr/local/lib/python3.6/dist-packages/nvidia/dali/libdali_operators.so(+0x1544942) [0x7f8ea727f942]
[frame 4]: /usr/local/lib/python3.6/dist-packages/nvidia/dali/libdali_operators.so(std::_Function_handler<std::unique_ptr<dali::OperatorBase, std::default_delete<dali::OperatorBase> > (dali::OpSpec const&), std::unique_ptr<dali::OperatorBase, std::default_delete<dali::OperatorBase> > (*)(dali::OpSpec const&)>::_M_invoke(std::_Any_data const&, dali::OpSpec const&)+0xc) [0x7f8ea60ca92c]
[frame 5]: /usr/local/lib/python3.6/dist-packages/nvidia/dali/libdali.so(+0x17a364) [0x7f8e90900364]
[frame 6]: /usr/local/lib/python3.6/dist-packages/nvidia/dali/libdali.so(dali::InstantiateOperator(dali::OpSpec const&)+0x32f) [0x7f8e908ffcbf]
[frame 7]: /usr/local/lib/python3.6/dist-packages/nvidia/dali/libdali.so(dali::OpGraph::InstantiateOperators()+0xb1) [0x7f8e908bd641]
[frame 8]: /usr/local/lib/python3.6/dist-packages/nvidia/dali/libdali.so(dali::Pipeline::Build(std::vector<std::pair<std::string, std::string>, std::allocator<std::pair<std::string, std::string> > >)+0xa48) [0x7f8e9091a2b8]
[frame 9]: /usr/local/lib/python3.6/dist-packages/nvidia/dali/backend_impl.cpython-36m-x86_64-linux-gnu.so(+0x5164f) [0x7f8ec0de664f]
[frame 10]: /usr/local/lib/python3.6/dist-packages/nvidia/dali/backend_impl.cpython-36m-x86_64-linux-gnu.so(+0x59d53) [0x7f8ec0deed53]
[frame 11]: python(_PyCFunction_FastCallDict+0x35c) [0x56204c]
[frame 12]: python() [0x4f88ba]
[frame 13]: python(_PyEval_EvalFrameDefault+0x467) [0x4f98c7]
[frame 14]: python() [0x4f6128]
[frame 15]: python() [0x4f7d60]
[frame 16]: python() [0x4f876d]
[frame 17]: python(_PyEval_EvalFrameDefault+0x467) [0x4f98c7]
[frame 18]: python() [0x4f6128]
[frame 19]: python(PyEval_EvalCode+0x23) [0x4f9023]
[frame 20]: python() [0x6415b2]
[frame 21]: python(PyRun_FileExFlags+0x9a) [0x64166a]
[frame 22]: python(PyRun_SimpleFileExFlags+0x190) [0x643730]
[frame 23]: python(Py_Main+0x59e) [0x62b26e]
[frame 24]: python(main+0xe0) [0x4b4cb0]
[frame 25]: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7) [0x7f8ed9d9ab97]
[frame 26]: python(_start+0x2a) [0x5bdf6a]

If I manually copy paste libnvcuvid.so and libnvcuvid.so.1 to /usr/lib/x86_64-linux-gnu/ and run the same script again the result is:

/opt/dali/dali/operators/reader/nvdecoder/nvdecoder.cc:158: Unable to decode file ./videos_test/video.mp4
Traceback (most recent call last):
  File "script.py", line 37, in <module>
    pipe_out = pipe.run()
  File "/usr/local/lib/python3.6/dist-packages/nvidia/dali/pipeline.py", line 612, in run
    return self.outputs()
  File "/usr/local/lib/python3.6/dist-packages/nvidia/dali/pipeline.py", line 511, in outputs
    return self._outputs()
  File "/usr/local/lib/python3.6/dist-packages/nvidia/dali/pipeline.py", line 595, in _outputs
    return self._pipe.Outputs()
RuntimeError: Critical error in pipeline: CUDA driver API error CUDA_ERROR_NO_DEVICE (100):
no CUDA-capable device is detected
Current pipeline object is no longer valid.

I've tried using nightly builds and even installation from WML CE, but nothing works.

Any suggestions?

[JanuszL]

Hi,
Do you run everything inside the docker? If so please check this issue https://github.com/NVIDIA/DALI/issues/2091, by default docker doesn't expose GPU video capabilities and additional options need to be added when launched.

[LightnessOfBeing]

No, I run everything as-is without docker.

[JanuszL]

If I manually copy paste libnvcuvid.so and libnvcuvid.so.1 to /usr/lib/x86_64-linux-gnu/

Where did you get the libraries? They should be installed there together with the driver.

[LightnessOfBeing]

Firstly, I decided to test that the script works in Google Colab. Colab has Cuda 10.1, I installed Dali in Colab and the script woked fine. Then I'd switched to Paperspace, encountered error described above and decided to check the result of ldconfig -p | grep libnvcuvid. In Colab two files appeared, namely libnvcuvid.so and libnvcuvid.so.1, but for paperspace instance the output was empty. So I just copy-pasted these two files from Colab to Paperspace instance.

[JanuszL]

Then it is the question for the Paperspace tech support, why their instances don't support video decoding.

[tgajewski]

Firstly, I decided to test that the script works in Google Colab. Colab has Cuda 10.1, I installed Dali in Colab and the script woked fine. Then I'd switched to Paperspace, encountered error described above and decided to check the result of ldconfig -p | grep libnvcuvid. In Colab two files appeared, namely libnvcuvid.so and libnvcuvid.so.1, but for paperspace instance the output was empty. So I just copy-pasted these two files from Colab to Paperspace instance.

Thanks for identifying this @LightnessOfBeing. We've pushed a hotfix this evening which will allow any GPU capable container to process video.

[JanuszL]

Thanks for identifying this @LightnessOfBeing. We've pushed a hotfix this evening which will allow any GPU capable container to process video.

I considered this as fixed. Thanks for the fast response @tgajewski .