Nvidia-docker: Failed to install nvidia-container-toolkit on Ubuntu 18.04 with ppc64le

Created on 7 Feb 2020 · 18Comments · Source: NVIDIA/nvidia-docker

1. Issue or feature description

(Different from #1064 and #1126)

apt-get update and apt-get install nvidia-container-toolkit fail on ubuntu18.04 ppc64le

sudo apt-get install -y nvidia-container-toolkit

Reading package lists... Done
Building dependency tree       
Reading state information... Done
E: Unable to locate package nvidia-container-toolkit

2. Steps to reproduce the issue

DIST=$(. /etc/os-release; echo $ID$VERSION_ID)
curl -s -L https://nvidia.github.io/libnvidia-container/gpgkey | \
  sudo apt-key add -
curl -s -L https://nvidia.github.io/libnvidia-container/$DIST/libnvidia-container.list | \
  sudo tee /etc/apt/sources.list.d/libnvidia-container.list
sudo apt-get update
sudo apt-get install nvidia-container-toolkit

3. Information to attach (optional if deemed irrelevant)

[x] Kernel version from uname -a: Linux tuleta 4.15.0-70-generic #79-Ubuntu SMP Tue Nov 12 10:34:21 UTC 2019 ppc64le ppc64le ppc64le GNU/Linux
[x] Driver information from nvidia-smi:

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 440.33.01    Driver Version: 440.33.01    CUDA Version: 10.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla K80           On   | 00000006:03:00.0 Off |                    0 |
| N/A   27C    P8    25W / 149W |      0MiB / 11441MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla K80           On   | 00000006:04:00.0 Off |                    0 |
| N/A   25C    P8    30W / 149W |      0MiB / 11441MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

[x] Docker version from docker version

Client: Docker Engine - Community
 Version:           19.03.5
 API version:       1.40
 Go version:        go1.12.12
 Git commit:        
 Built:             Wed Feb  5 20:28:52 2020
 OS/Arch:           linux/ppc64le
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.5
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.12
  Git commit:       633a0ea838
  Built:            Wed Feb  5 20:35:54 2020
  OS/Arch:          linux/ppc64le
  Experimental:     false
 containerd:
  Version:          v1.2.10
  GitCommit:        b34a5c8af56e510852c35414db4c1f4fa6172339
 runc:
  Version:          spec: 1.0.1-dev
  GitCommit:        
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

bug platform

Source

James-QiuHaoran

Most helpful comment

Hello!

In README document Ubuntu 16.04/18.04/20.04, Debian Jessie/Stretch/Buster section, there is commands are listing for installation.

When I run these commands respectively, they were running and no one is failing except the latest.
Because of this commands returning successly, I didn't consider warnings.

Then, I inspect commands. First command is:
~sh
distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
~

This command generates a variable, and it's value is as this:
~sh
$ echo distribution
debian
~

Then, I visited Nvidia Docker documents and inspected supported OS list. My distribution is Debian 10 and its in the supported distributions list. So, I should be able to install. I inspected next steps.

~sh
curl -s -L | sudo apt-key add -
~

This command is working properly, no problem. Next command is:

~sh
curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | sudo tee /etc/apt/sources.list.d/nvidia-docker.list
~

This command is not generating an error, instead generating an output:

~~~

Unsupported distribution!

Check https://nvidia.github.io/nvidia-docker

~~~

My distribution ( Debian 10 ) is listing as supported in Nvidia Docker documents but command outputs Unsupported distribution! .

I inspected the first command and see that it should contain $ID and $VERSION_ID to distribution variable but mine was only debian. So, when I printed content of /etc/os-release file, my output as this:

~
PRETTY_NAME="Debian GNU/Linux bullseye/sid"
NAME="Debian GNU/Linux"
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
~

So, when I run . /etc/os-release command, there is $ID variable exists but $VERSION_ID not exists.

I tried to check https://nvidia.github.io/nvidia-docker/debian/nvidia-docker.list address, and it's output was:

~~~

Unsupported distribution!

Check https://nvidia.github.io/nvidia-docker

~~~

Then , i add my VERSION_ID to my ID as: debian -> debian10 , then I tried to check https://nvidia.github.io/nvidia-docker/debian10/nvidia-docker.list address, and it's output was:

~
deb https://nvidia.github.io/libnvidia-container/debian10/$(ARCH) /
deb https://nvidia.github.io/nvidia-container-runtime/debian10/$(ARCH) /
deb https://nvidia.github.io/nvidia-docker/debian10/$(ARCH) /
~

Gotcha!

Because of distribution variable didn't generated properly, I wasn't able to add package lists to my system.

TL;DR

Set distribution variable's value as

Ubuntu 18.04:
~sh
distribution=ubuntu18.04
~
Ubuntu 16.04:
~sh
distribution=ubuntu16.04
~
Ubuntu 14.04:
~sh
distribution=ubuntu14.04
~
Debian 10:
~sh
distribution=debian10
~
Debian 9:
~sh
distribution=debian9
~
Debian 8:
~
distribution=debian8
~

Then continue other commands.

~~~sh

Add the package repositories

curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add -
curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | sudo tee /etc/apt/sources.list.d/nvidia-docker.list

sudo apt-get update && sudo apt-get install -y nvidia-container-toolkit
sudo systemctl restart docker
~~~

ileri on 13 May 2020

👍12 ❤4 🚀2 🎉2 😄2

All 18 comments

I was trying to install only the nvidia-container-toolkit instead of the nvidia-docker2. The reason is that I want to use the native support for GPU (this is enabled since Docker 19.03).

On Fri, Feb 7, 2020 at 9:16 PM Koji notifications@github.com wrote:

Try the following commands.
They will work since my env is almost same as yours.

$ curl -sL https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add -
$ distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
$ curl -sL https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | sudo tee /etc/apt/sources.list.d/nvidia-docker.list
$ sudo apt update
$ sudo apt install nvidia-docker2 -y
$ sudo pkill -SIGHUP dockerd
$ sudo docker run --runtime=nvidia --rm nvidia/cuda nvidia-smi

Probably you will see like this.

7ddbc47eeb70: Pull complete
c1bbdc448b72: Pull complete
8c3b70e39044: Pull complete
45d437916d57: Pull complete
d8f1569ddae6: Pull complete
85386706b020: Pull complete
ee9b457b77d0: Pull complete
be4f3343ecd3: Pull complete
30b4effda4fd: Pull complete
Digest: sha256:31e2a1ca7b0e1f678fb1dd0c985b4223273f7c0f3dbde60053b371e2a1aee2cd
Status: Downloaded newer image for nvidia/cuda:latest
Sat Feb 8 03:11:46 2020
+-----------------------------------------------------------------------------+| NVIDIA-SMI 440.33.01 Driver Version: 440.33.01 CUDA Version: 10.2 ||-------------------------------+----------------------+----------------------+| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC || Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. ||===============================+======================+======================|| 0 GeForce GTX 1070 On | 00000000:01:00.0 On | N/A || N/A 52C P8 4W / N/A | 349MiB / 8111MiB | 2% Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+| Processes: GPU Memory || GPU PID Type Process name Usage ||=============================================================================|
+-----------------------------------------------------------------------------+

James-QiuHaoran on 8 Feb 2020

Hello!

Sorry for the delay, we are in the process of building PPC packages, they should be available next week.

RenaudWasTaken on 10 Feb 2020

👍2

Are this PPC packages already available? I have still that error : E: Unable to locate package nvidia-container-toolkit
Ubuntu 18.04, upgraded all other packages.
root@uinference-115:/etc/apt/sources.list.d# apt-get install nvidia-container-toolkit
Reading package lists... Done
Building dependency tree
Reading state information... Done
E: Unable to locate package nvidia-container-toolkit
root@uinference-115:/etc/apt/sources.list.d# nvidia-smi
Wed Feb 19 11:40:55 2020
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 418.87.01 Driver Version: 418.87.01 CUDA Version: 10.1 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce RTX 2070 On | 00000001:00:01.0 Off | N/A |
| 0% 49C P8 13W / 175W | 0MiB / 7952MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 1 GeForce RTX 2070 On | 00000002:00:01.0 Off | N/A |
| 0% 53C P8 8W / 175W | 0MiB / 7952MiB | 0% Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| No running processes found |
+-----------------------------------------------------------------------------+

PeS-hkfree on 19 Feb 2020

Are this PPC packages already available? I have still that error : E: Unable to locate package nvidia-container-toolkit
Ubuntu 18.04, upgraded all other packages.
root@uinference-115:/etc/apt/sources.list.d# apt-get install nvidia-container-toolkit
Reading package lists... Done
Building dependency tree
Reading state information... Done
E: Unable to locate package nvidia-container-toolkit
root@uinference-115:/etc/apt/sources.list.d# nvidia-smi
Wed Feb 19 11:40:55 2020
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 418.87.01 Driver Version: 418.87.01 CUDA Version: 10.1 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce RTX 2070 On | 00000001:00:01.0 Off | N/A |
| 0% 49C P8 13W / 175W | 0MiB / 7952MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 1 GeForce RTX 2070 On | 00000002:00:01.0 Off | N/A |
| 0% 53C P8 8W / 175W | 0MiB / 7952MiB | 0% Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| No running processes found |
+-----------------------------------------------------------------------------+

Same here. I think they are still working on that.

James-QiuHaoran on 19 Feb 2020

I am facing this too, any updates on this?

satyakrishnagorti on 3 Mar 2020

jeroenst on 28 Mar 2020

will-hill on 6 Apr 2020

Facing this too with Debian 18.04 , Docker 18.09.

RelinquishedAll on 21 Apr 2020

AntonAmes on 28 Apr 2020

Same problem Ubuntu 18.04, Docker 19.03.8
any updates?

luiztauffer on 5 May 2020

mtatum7 on 6 May 2020

Me too, is there any update?

Lance0218 on 13 May 2020

Hello!

In README document Ubuntu 16.04/18.04/20.04, Debian Jessie/Stretch/Buster section, there is commands are listing for installation.

When I run these commands respectively, they were running and no one is failing except the latest.
Because of this commands returning successly, I didn't consider warnings.

Then, I inspect commands. First command is:
~sh
distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
~

This command generates a variable, and it's value is as this:
~sh
$ echo distribution
debian
~

~sh
curl -s -L | sudo apt-key add -
~

This command is working properly, no problem. Next command is:

~sh
curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | sudo tee /etc/apt/sources.list.d/nvidia-docker.list
~

This command is not generating an error, instead generating an output:

~~~

Unsupported distribution!

Check https://nvidia.github.io/nvidia-docker

~~~

My distribution ( Debian 10 ) is listing as supported in Nvidia Docker documents but command outputs Unsupported distribution! .

So, when I run . /etc/os-release command, there is $ID variable exists but $VERSION_ID not exists.

I tried to check https://nvidia.github.io/nvidia-docker/debian/nvidia-docker.list address, and it's output was:

~~~

Unsupported distribution!

Check https://nvidia.github.io/nvidia-docker

~~~

Then , i add my VERSION_ID to my ID as: debian -> debian10 , then I tried to check https://nvidia.github.io/nvidia-docker/debian10/nvidia-docker.list address, and it's output was:

Gotcha!

Because of distribution variable didn't generated properly, I wasn't able to add package lists to my system.

TL;DR

Set distribution variable's value as

Ubuntu 18.04:
~sh
distribution=ubuntu18.04
~
Ubuntu 16.04:
~sh
distribution=ubuntu16.04
~
Ubuntu 14.04:
~sh
distribution=ubuntu14.04
~
Debian 10:
~sh
distribution=debian10
~
Debian 9:
~sh
distribution=debian9
~
Debian 8:
~
distribution=debian8
~

Then continue other commands.

~~~sh

Add the package repositories

sudo apt-get update && sudo apt-get install -y nvidia-container-toolkit
sudo systemctl restart docker
~~~

ileri on 13 May 2020

👍12 ❤4 🚀2 🎉2 😄2

root@localhost:/etc/apt/sources.list.d# sudo apt-get update && sudo apt-get install -y nvidia-container-toolkit
Hit:1 https://download.docker.com/linux/ubuntu bionic InRelease
Hit:2 https://nvidia.github.io/libnvidia-container/ubuntu18.04/ppc64el InRelease
Hit:3 https://nvidia.github.io/nvidia-container-runtime/ubuntu18.04/ppc64el InRelease
Hit:4 https://nvidia.github.io/nvidia-docker/ubuntu18.04/ppc64el InRelease
Reading package lists... Done
Reading package lists... Done
Building dependency tree
Reading state information... Done
E: Unable to locate package nvidia-container-toolkit

still happening.

adisubektie on 14 May 2020

I would also like to install nvidia-container-toolkit for Ubuntu 20.04 on Power. Is this still in the works? @RenaudWasTaken

eugeneswalker on 16 May 2020

We just released a new version of nvidia-docker on Tuesday 19-May. This is the support matrix for the new packages:

+----------------------+-------------+----------------+---------+-----------------+
|  OS Name / Version   |  Identifier | amd64 / x86_64 | ppc64le | arm64 / aarch64 |
+======================+=============+================+=========+=================+
| Amazon Linux 1       | amzn1       |       X        |         |                 |
| Amazon Linux 2       | amzn2       |       X        |         |                 |
| Amazon Linux 2017.09 | amzn2017.09 |       X        |         |                 |
| Amazon Linux 2018.03 | amzn2018.03 |       X        |         |                 |
| Open Suse Leap 15.0  | sles15.0    |       X        |         |                 |
| Open Suse Leap 15.1  | sles15.1    |       X        |         |                 |
| Debian Linux 9       | debian9     |       X        |         |                 |
| Debian Linux 10      | debian10    |       X        |         |                 |
| Centos 7             | centos7     |       X        |    X    |                 |
| Centos 8             | centos8     |       X        |    X    |        X        |
| RHEL 7.4             | rhel7.4     |       X        |    X    |                 |
| RHEL 7.5             | rhel7.5     |       X        |    X    |                 |
| RHEL 7.6             | rhel7.6     |       X        |    X    |                 |
| RHEL 7.7             | rhel7.7     |       X        |    X    |                 |
| RHEL 8.0             | rhel8.0     |       X        |    X    |        X        |
| RHEL 8.1             | rhel8.1     |       X        |    X    |        X        |
| RHEL 8.2             | rhel8.2     |       X        |    X    |        X        |
| Ubuntu 16.04         | ubuntu16.04 |       X        |    X    |                 |
| Ubuntu 18.04         | ubuntu18.04 |       X        |    X    |        X        |
| Ubuntu 19.04         | ubuntu19.04 |       X        |    X    |        X        |
| Ubuntu 19.10         | ubuntu19.10 |       X        |    X    |        X        |
| Ubuntu 20.04         | ubuntu20.04 |       X        |    X    |        X        |
+----------------------+-------------+----------------+---------+-----------------+

Please let us know if this resolves your issue.

klueska on 22 May 2020

Please let us know if this resolves your issue.

Seems to be working now :)
Tested on Ubuntu 16.04 and 18.06 on ppc64le

AntonAmes on 30 May 2020

THIS FIX worked for me.

Somehow the system is wrongly recognized as "debian" instead of "ubuntu"

view /etc/apt/sources.list content and locate docker entry. Update it to look something like this:
deb [arch=amd64] https://download.docker.com/linux/ubuntu bionic stable

Once saved, run the below commands
sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit