Nvidia-docker: CUDA / Docker & GPG error

[MattiasM80]

1. Issue or feature description:

Installation of cuda inside docker (ubuntu 16) fails. From the docker build log:

Hit:1 http://security.ubuntu.com/ubuntu xenial-security InRelease
Ign:2 http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64  InRelease
Hit:3 http://archive.ubuntu.com/ubuntu xenial InRelease
Ign:4 http://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1604/x86_64  InRelease
Hit:5 http://archive.ubuntu.com/ubuntu xenial-updates InRelease
Get:6 http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64  Release [564 B]
Hit:7 http://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1604/x86_64  Release
Hit:8 http://archive.ubuntu.com/ubuntu xenial-backports InRelease
Get:9 http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64  Release.gpg [801 B]
Ign:9 http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64  Release.gpg
Hit:11 http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64  Packages
Err:11 http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64  Packages
  Hash Sum mismatch
Fetched 1365 B in 0s (2658 B/s)
Reading package lists...
W: GPG error: http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64  Release: The following signatures were invalid: BADSIG F60F4B3D7FA2AF80 cudatools <[email protected]>
W: The repository 'http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64  Release' is not signed.
E: Failed to fetch store:/var/lib/apt/lists/partial/developer.download.nvidia.com_compute_cuda_repos_ubuntu1604_x86%5f64_Packages.gz  Hash Sum mismatch
E: Some index files failed to download. They have been ignored, or old ones used instead.

2. Steps to reproduce the issue

Dockerfile.txt

[flx42]

Seems to be the same than https://github.com/NVIDIA/nvidia-docker/issues/571 and https://github.com/NVIDIA/nvidia-docker/issues/613.
But the reporter in #613 said the issue is now fixed for him. Where are you located?

[MattiasM80]

In the US.
Is there anything on my side that I can do to resolve this? Is there another way to install the Nvidia dependencies?

[flx42]

Can you do the wget mentioned here? https://github.com/NVIDIA/nvidia-docker/issues/613#issuecomment-359136112

[MattiasM80]

RUN wget http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/Release
---> Running in 7e57c56dd794
--2018-01-22 19:29:52-- http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/Release
Resolving developer.download.nvidia.com (developer.download.nvidia.com)... 192.229.211.70, 2606:2800:21f:3aa:dcf:37b:1ed6:1fb
Connecting to developer.download.nvidia.com (developer.download.nvidia.com)|192.229.211.70|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 564 [application/octet-stream]
Saving to: 'Release'

0K 100% 72.2M=0s

2018-01-22 19:29:52 (72.2 MB/s) - 'Release' saved [564/564]

Removing intermediate container 7e57c56dd794

[MattiasM80]

@flx42 Anything I can do on my side to avoid this problem?

[flx42]

You can only workaround the problem. I suggest you use our prebuilt devel images on DockerHub, then remove the cuda.list file for now.

[MattiasM80]

When is a fix planned?
I need an ubuntu 16 image with cuda 8 (not 9). What docker FROM command should I use?
Could you be more specific on how to "remove the cuda.list file". Thanks!

[flx42]

I don't know when it will be fixed.

This is a potential workaround:

FROM nvidia/cuda:8.0-cudnn6-devel-ubuntu16.04

RUN rm /etc/apt/sources.list.d/cuda.list

[MattiasM80]

Thanks!
What would the non devel equivalent for that base image be?
FROM nvidia/cuda:8.0-cudnn6-ubuntu16.04 was not found.

[flx42]

nvidia/cuda:8.0-cudnn6-runtime-ubuntu16.04

[jedichien]

Here is my solution
in your Dockerfile add this line RUN apt-key adv --fetch-keys http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/7fa2af80.pub before apt-get update

[flx42]

That's not a good idea. Let us understand what it going on first. Thanks.

[3XX0]

@MattiasM80 Does it work for you now? If not, what's the output of apt-key export 7FA2AF80 in the container after you just pulled it?

And after the apt-get update can you try the following command:
apt-key adv --verify /var/lib/apt/lists/developer.download.nvidia.com_compute_cuda_repos_ubuntu1604_x86%5f64_Release.gpg /var/lib/apt/lists/developer.download.nvidia.com_compute_cuda_repos_ubuntu1604_x86%5f64_Release

[3XX0]

@MattiasM80 Is it resolved?

[MattiasM80]

@flx42 FROM nvidia/cuda:8.0-cudnn6-runtime-ubuntu16.04 worked excellent.
@3XX0 I didn't get to try `apt-key adv --verify /var/lib/apt/lists/developer.download.nvidia.com_compute_cuda_repos_ubuntu1604_x86%5f64_Release.gpg /var/lib/apt/lists/developer.download.nvidia.com_compute_cuda_repos_ubuntu1604_x86%5f64_Release````

Excuse me for the late reply.

[TaylorHere]

I got the same errs, the I just rm all the source about NVIDIA things, but it’s not a good way.