Nvidia-docker: Fail to run with Cuda 9.2: docker: Error response from daemon: OCI runtime create failed
1. Issue or feature description
Duplicate of #746.
2. Steps to reproduce the issue
- Install
nvidia-docker2following the regular instructions. - Run the following command:
docker run --runtime=nvidia --rm nvidia/cuda:9.2-runtime-ubuntu16.04 nvidia-smi
docker: Error response from daemon: OCI runtime create failed: container_linux.go:348: starting container process caused "process_linux.go:402: container init caused \"process_linux.go:385: running prestart hook 1 caused \\\"error running hook: exit status 1, stdout: , stderr: exec command: [/usr/bin/nvidia-container-cli --load-kmods configure --ldconfig=/sbin/ldconfig.real --device=all --compute --utility --require=cuda>=9.2 --pid=15437 /home/hadim/.local/docker/overlay2/33e071b602a668ec4d7b0761116a59d35c9cf3d314bfb9afaa0f10ee53e7bd57/merged]\\\\nnvidia-container-cli: requirement error: unsatisfied condition: cuda >= 9.2\\\\n\\\"\"": unknown.
nvidia-smi -a
==============NVSMI LOG==============
Timestamp : Mon Jun 4 17:42:35 2018
Driver Version : 390.48
Attached GPUs : 1
GPU 00000000:01:00.0
Product Name : GeForce GTX 1050 Ti
Product Brand : GeForce
Display Mode : Disabled
Display Active : Disabled
Persistence Mode : Disabled
Accounting Mode : Disabled
Accounting Mode Buffer Size : 4000
Driver Model
Current : N/A
Pending : N/A
Serial Number : N/A
GPU UUID : GPU-2702064b-7901-52d3-7ba0-3a481c59eacb
Minor Number : 0
VBIOS Version : 86.07.3C.00.BE
MultiGPU Board : No
Board ID : 0x100
GPU Part Number : N/A
Inforom Version
Image Version : N/A
OEM Object : N/A
ECC Object : N/A
Power Management Object : N/A
GPU Operation Mode
Current : N/A
Pending : N/A
GPU Virtualization Mode
Virtualization mode : None
PCI
Bus : 0x01
Device : 0x00
Domain : 0x0000
Device Id : 0x1C8C10DE
Bus Id : 00000000:01:00.0
Sub System Id : 0x34651458
GPU Link Info
PCIe Generation
Max : 3
Current : 3
Link Width
Max : 16x
Current : 16x
Bridge Chip
Type : N/A
Firmware : N/A
Replays since reset : 0
Tx Throughput : 1621000 KB/s
Rx Throughput : 0 KB/s
Fan Speed : N/A
Performance State : P3
Clocks Throttle Reasons
Idle : Active
Applications Clocks Setting : Not Active
SW Power Cap : Not Active
HW Slowdown : Not Active
HW Thermal Slowdown : Not Active
HW Power Brake Slowdown : Not Active
Sync Boost : Not Active
SW Thermal Slowdown : Not Active
Display Clock Setting : Not Active
FB Memory Usage
Total : 4042 MiB
Used : 522 MiB
Free : 3520 MiB
BAR1 Memory Usage
Total : 256 MiB
Used : 3 MiB
Free : 253 MiB
Compute Mode : Default
Utilization
Gpu : 3 %
Memory : 2 %
Encoder : 0 %
Decoder : 0 %
Encoder Stats
Active Sessions : 0
Average FPS : 0
Average Latency : 0
Ecc Mode
Current : N/A
Pending : N/A
ECC Errors
Volatile
Single Bit
Device Memory : N/A
Register File : N/A
L1 Cache : N/A
L2 Cache : N/A
Texture Memory : N/A
Texture Shared : N/A
CBU : N/A
Total : N/A
Double Bit
Device Memory : N/A
Register File : N/A
L1 Cache : N/A
L2 Cache : N/A
Texture Memory : N/A
Texture Shared : N/A
CBU : N/A
Total : N/A
Aggregate
Single Bit
Device Memory : N/A
Register File : N/A
L1 Cache : N/A
L2 Cache : N/A
Texture Memory : N/A
Texture Shared : N/A
CBU : N/A
Total : N/A
Double Bit
Device Memory : N/A
Register File : N/A
L1 Cache : N/A
L2 Cache : N/A
Texture Memory : N/A
Texture Shared : N/A
CBU : N/A
Total : N/A
Retired Pages
Single Bit ECC : N/A
Double Bit ECC : N/A
Pending : N/A
Temperature
GPU Current Temp : 48 C
GPU Shutdown Temp : 102 C
GPU Slowdown Temp : 97 C
GPU Max Operating Temp : 94 C
Memory Current Temp : N/A
Memory Max Operating Temp : N/A
Power Readings
Power Management : N/A
Power Draw : N/A
Power Limit : N/A
Default Power Limit : N/A
Enforced Power Limit : N/A
Min Power Limit : N/A
Max Power Limit : N/A
Clocks
Graphics : 885 MHz
SM : 885 MHz
Memory : 2504 MHz
Video : 797 MHz
Applications Clocks
Graphics : N/A
Memory : N/A
Default Applications Clocks
Graphics : N/A
Memory : N/A
Max Clocks
Graphics : 1911 MHz
SM : 1911 MHz
Memory : 3504 MHz
Video : 1708 MHz
Max Customer Boost Clocks
Graphics : N/A
Clock Policy
Auto Boost : N/A
Auto Boost Default : N/A
Processes
Process ID : 1513
Type : G
Name : /usr/lib/xorg/Xorg
Used GPU Memory : 40 MiB
Process ID : 1908
Type : G
Name : /usr/bin/gnome-shell
Used GPU Memory : 50 MiB
Process ID : 3083
Type : G
Name : /usr/lib/xorg/Xorg
Used GPU Memory : 163 MiB
Process ID : 4246
Type : G
Name : /usr/bin/gnome-shell
Used GPU Memory : 140 MiB
Process ID : 4764
Type : G
Name : /usr/lib/chromium-browser/chromium-browser --type=gpu-process --field-trial-handle=14652321548495661849,6913534254963283766,131072 --gpu-preferences=KAAAAAAAAACAAACAAQAAAAAAAAAAAGAAEAAAAAAAAAAAAAAAAAAAAAgAAAAAAAAA --service-request-channel-token=ACBF9C417B676E310771677F9B1C15A4
Used GPU Memory : 127 MiB
docker version
Client: Version: 17.12.1-ce API version: 1.35 Go version: go1.10.1 Git commit: 7390fc6 Built: Wed Apr 18 01:23:11 2018 OS/Arch: linux/amd64 Server: Engine: Version: 17.12.1-ce API version: 1.35 (minimum version 1.12) Go version: go1.10.1 Git commit: 7390fc6 Built: Wed Feb 28 17:46:05 2018 OS/Arch: linux/amd64 Experimental: false
dpkg -l 'nvidia'
Desired=Unknown/Install/Remove/Purge/Hold | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad) ||/ Name Version Architecture Description +++-========================-=================-=================-===================================================== un libgldispatch0-nvidia(no description available) ii libnvidia-cfg1-390:amd64 390.48-0ubuntu3 amd64 NVIDIA binary OpenGL/GLX configuration library un libnvidia-cfg1-any (no description available) un libnvidia-common (no description available) ii libnvidia-common-390 390.48-0ubuntu3 all Shared files used by the NVIDIA libraries ii libnvidia-compute-390:am 390.48-0ubuntu3 amd64 NVIDIA libcompute package ii libnvidia-compute-390:i3 390.48-0ubuntu3 i386 NVIDIA libcompute package ii libnvidia-container-tool 1.0.0~rc.1-1 amd64 NVIDIA container runtime library (command-line tools) ii libnvidia-container1:amd 1.0.0~rc.1-1 amd64 NVIDIA container runtime library un libnvidia-decode (no description available) ii libnvidia-decode-390:amd 390.48-0ubuntu3 amd64 NVIDIA Video Decoding runtime libraries ii libnvidia-decode-390:i38 390.48-0ubuntu3 i386 NVIDIA Video Decoding runtime libraries un libnvidia-encode (no description available) ii libnvidia-encode-390:amd 390.48-0ubuntu3 amd64 NVENC Video Encoding runtime library ii libnvidia-encode-390:i38 390.48-0ubuntu3 i386 NVENC Video Encoding runtime library un libnvidia-fbc1 (no description available) ii libnvidia-fbc1-390:amd64 390.48-0ubuntu3 amd64 NVIDIA OpenGL-based Framebuffer Capture runtime libra ii libnvidia-fbc1-390:i386 390.48-0ubuntu3 i386 NVIDIA OpenGL-based Framebuffer Capture runtime libra un libnvidia-gl (no description available) ii libnvidia-gl-390:amd64 390.48-0ubuntu3 amd64 NVIDIA OpenGL/GLX/EGL/GLES GLVND libraries and Vulkan ii libnvidia-gl-390:i386 390.48-0ubuntu3 i386 NVIDIA OpenGL/GLX/EGL/GLES GLVND libraries and Vulkan un libnvidia-ifr1 (no description available) ii libnvidia-ifr1-390:amd64 390.48-0ubuntu3 amd64 NVIDIA OpenGL-based Inband Frame Readback runtime lib ii libnvidia-ifr1-390:i386 390.48-0ubuntu3 i386 NVIDIA OpenGL-based Inband Frame Readback runtime lib un nvidia-304 (no description available) un nvidia-340 (no description available) un nvidia-384 (no description available) in nvidia-390 amd64 (no description available) un nvidia-common (no description available) ii nvidia-compute-utils-390 390.48-0ubuntu3 amd64 NVIDIA compute utilities ii nvidia-container-runtime 2.0.0+docker17.12 amd64 NVIDIA container runtime ii nvidia-container-runtime 1.3.0-1 amd64 NVIDIA container runtime hook ii nvidia-dkms-390 390.48-0ubuntu3 amd64 NVIDIA DKMS package un nvidia-dkms-kernel (no description available) un nvidia-docker (no description available) ii nvidia-docker2 2.0.3+docker17.12 all nvidia-docker CLI wrapper ii nvidia-driver-390 390.48-0ubuntu3 amd64 NVIDIA driver metapackage un nvidia-driver-binary (no description available) un nvidia-driver-libs-i386 (no description available) un nvidia-kernel-common (no description available) ii nvidia-kernel-common-390 390.48-0ubuntu3 amd64 Shared files used with the kernel module un nvidia-kernel-source (no description available) ii nvidia-kernel-source-390 390.48-0ubuntu3 amd64 NVIDIA kernel source package un nvidia-legacy-340xx-vdpa (no description available) un nvidia-libopencl1-390 (no description available) un nvidia-opencl-icd (no description available) rc nvidia-opencl-icd-390 390.30-0ubuntu1 amd64 NVIDIA OpenCL ICD un nvidia-persistenced (no description available) ii nvidia-prime 0.8.8 all Tools to enable NVIDIA's Prime ii nvidia-settings 390.42-0ubuntu1 amd64 Tool for configuring the NVIDIA graphics driver un nvidia-settings-binary (no description available) un nvidia-smi (no description available) un nvidia-utils (no description available) ii nvidia-utils-390 390.48-0ubuntu3 amd64 NVIDIA driver support binaries un nvidia-vdpau-driver (no description available) ii xserver-xorg-video-nvidi 390.48-0ubuntu3 amd64 NVIDIA binary Xorg driver
nvidia-container-cli -V
version: 1.0.0 build date: 2018-04-26T23:03+00:00 build revision: 163054a04b21c4455c8cae7e47873d9f2a091f55 build compiler: x86_64-linux-gnu-gcc-7 7.3.0 build platform: x86_64 build flags: -D_GNU_SOURCE -D_FORTIFY_SOURCE=2 -DNDEBUG -std=gnu11 -O2 -g -fdata-sections -ffunction-sections -fstack-protector -fno-strict-aliasing -fvisibility=hidden -Wall -Wextra -Wcast-align -Wpointer-arith -Wmissing-prototypes -Wnonnull -Wwrite-strings -Wlogical-op -Wformat=2 -Wmissing-format-attribute -Winit-self -Wshadow -Wstrict-prototypes -Wunreachable-code -Wconversion -Wsign-conversion -Wno-unknown-warning-option -Wno-format-extra-args -Wno-gnu-alignof-expression -Wl,-zrelro -Wl,-znow -Wl,-zdefs -Wl,--gc-sections
hadim
All 2 comments
Nope, it's a different problem :) You are getting:
requirement error: unsatisfied condition: cuda >= 9.2
You need a driver 396+ to run CUDA 9.2, and you have 390.
You can take a look at this table:
https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/index.html
flx42
on 4 Jun 2018
👍3
😄1
That makes sense indeed. Thank you.
hadim
on 4 Jun 2018
Was this page helpful?
0 / 5 - 0 ratings
Related issues
DimanNe
路
3Comments
mmitterma
路
4Comments
SpotCrowdTech
路
3Comments
Wutever
路
3Comments
jonghwanhyeon
路
4Comments
Most helpful comment
Nope, it's a different problem :) You are getting:
You need a driver 396+ to run CUDA 9.2, and you have 390.
You can take a look at this table:
https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/index.html