Nvidia-docker: libnvidia-container-tools >= 1.1.1 for package: nvidia-container-toolkit-1.1.2-2.amzn2.x86_64

[x86nick]

Getting dependency error:
Processing Dependency: nvidia-container-toolkit >= 1.1.0 for package: nvidia-container-runtime-3.2.0-1.x86_64

nvidia-container-toolkit >= 1.1.0 is not available in nvidia pkg repo.

OS
NAME="Amazon Linux"
VERSION="2"

ID="amzn"

$uname -a
Linux ip-10-110-106-155.vpc.internal 4.14.181-140.257.amzn2.x86_64 #1 SMP Wed May 27 02:17:36 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Docker version
19.03.6-ce

====
yum search --showduplicates nvidia-container-toolkit

nvidia-container-toolkit-1.0.5-2.amzn2.x86_64 : NVIDIA container runtime hook

[klueska]

Can you try again? It’s definitely there, but we were resyncing the repos earlier today and you might have hit a glitch because of it.

[jweber-mimsoftware]

I'm running into the same (or similar sounding) issue trying to install nvidia-docker-container or nvidia-docker2 on Amazon Linux 2 (p2.xlarge if that matters):

==> amazon-ebs: Error: Package: nvidia-container-toolkit-1.1.2-2.amzn2.x86_64 (nvidia-container-runtime)
==> amazon-ebs:            Requires: libnvidia-container-tools >= 1.1.1
==> amazon-ebs:            Installing: libnvidia-container-tools-1.0.0-1.amzn2.x86_64 (amzn2-graphics)
==> amazon-ebs:                libnvidia-container-tools = 1.0.0-1.amzn2

[ewertonsilveira]

Same here!

amznz2
AMI: amzn2-ami-ecs-gpu-hvm-2.0.20200623-x86_64-ebs (ami-0d5fad86866a3a449)

[root@ip-10-1-15-92 ec2-user]#
[root@ip-10-1-15-92 ec2-user]# nvidia-smi
Mon Jul  6 02:38:47 2020
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 418.87.00    Driver Version: 418.87.00    CUDA Version: 10.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla K80           On   | 00000000:00:1E.0 Off |                    0 |
| N/A   38C    P8    31W / 149W |      0MiB / 11441MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+
[root@ip-10-1-15-92 ec2-user]# yum install nvidia-docker2
Loaded plugins: dkms-build-requires, priorities, update-motd, upgrade-helper
Repository libnvidia-container is listed more than once in the configuration
Repository libnvidia-container-experimental is listed more than once in the configuration
24 packages excluded due to repository priority protections
Resolving Dependencies
--> Running transaction check
---> Package nvidia-docker2.noarch 0:2.3.0-1 will be installed
--> Processing Dependency: nvidia-container-runtime >= 3.2.0 for package: nvidia-docker2-2.3.0-1.noarch
--> Running transaction check
---> Package nvidia-container-runtime.x86_64 0:3.2.0-1 will be installed
--> Processing Dependency: nvidia-container-toolkit < 2.0.0 for package: nvidia-container-runtime-3.2.0-1.x86_64
--> Processing Dependency: nvidia-container-toolkit >= 1.1.0 for package: nvidia-container-runtime-3.2.0-1.x86_64
--> Running transaction check
---> Package nvidia-container-toolkit.x86_64 0:1.1.2-2.amzn2 will be installed
--> Processing Dependency: libnvidia-container-tools >= 1.1.1 for package: nvidia-container-toolkit-1.1.2-2.amzn2.x86_64
--> Finished Dependency Resolution
Error: Package: nvidia-container-toolkit-1.1.2-2.amzn2.x86_64 (nvidia-container-runtime)
           Requires: libnvidia-container-tools >= 1.1.1
           Installed: libnvidia-container-tools-1.0.0-1.amzn2.x86_64 (@amzn2-graphics)
               libnvidia-container-tools = 1.0.0-1.amzn2
 You could try using --skip-broken to work around the problem
 You could try running: rpm -Va --nofiles --nodigest

Linux ip-10-1-15-92 4.14.181-140.257.amzn2.x86_64 #1 SMP Wed May 27 02:17:36 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

[klueska]

Sorry for the delayed reply.

It seems that your system is pulling libraries from a yum repository called @amzn2-graphics that has higher priority over the nvidia repos.

The latest libnvidia-container-tools library available from the nvidia repos is, in fact, 1.1.1 and can be seen here:
https://github.com/NVIDIA/libnvidia-container/tree/gh-pages/stable/amzn2/x86_64/

I'm not sure what all is included in the @amzn2-graphics repo (and if you need it at all), but you will need to force the install of libnvidia-container-tools and libnvidia-container1 from the nvidia repos in order to run the latest version of nvidia-docker.

Otherwise, you can downgrade to nvidia-docker version 2.2.2 and that should fix your problem as well.

[tomercagan]

Thanks for the suggestion.
On amazon linux 2 - the following did the trick

sudo yum-config-manager --disable amzn2-graphics

Then I could install the container tools and get the nvidia-smi from a container.

I guess you should enable the amzn2-graphics back using

sudo yum-config-manager --enable amzn2-graphics

[opptimus]

force the install of libnvidia-container-tools and libnvidia-container

@klueska
How force the install libnvidia-container-tools and libnvidia-container？
How should I type the command?

[klueska]

Just specify their versions explicitly, i.e. on amazon linux (for the latest packages) this would be:

yum install -y libnvidia-container1-1.3.1-1
yum install -y libnvidia-container-tools-1.3.1-1
yum install -y nvidia-container-toolkit-1.4.0-2