Nvidia-docker: Problem with repositories for Ubuntu 16.04

[sb5518]

When trying to create a docker image with nvidia-docker build the following issue arises:

W: GPG error: http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64 Release: The following signatures were invalid: BADSIG F60F4B3D7FA2AF80 cudatools [e-mail]
W: The repository 'http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64 Release' is not signed.
W: GPG error: http://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1604/x86_64 Release: The following signatures were invalid: BADSIG F60F4B3D7FA2AF80 cudatools [e-mail]
W: The repository 'http://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1604/x86_64 Release' is not signed.
E: Failed to fetch http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/Packages.gz Hash Sum mismatch
E: Failed to fetch http://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1604/x86_64/Packages.gz Hash Sum mismatch
E: Some index files failed to download. They have been ignored, or old ones used instead.

This occurred after compiling an image that was working perfectly until two days ago, and in different environments.

[flx42]

Can you try again @sb5518? We released CUDA 9.1 recently on this repository, and sometimes we hit issues with our CDN when a new release happens.

[sb5518]

Unfortunately I just tried again in two different environments and the issue persists. On one of the environments it shows the previous issue and in the other one:

E: Failed to fetch http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/Packages.gz Hash Sum mismatch
E: Some index files failed to download. They have been ignored, or old ones used instead.

[flx42]

Which country are you in? It might be a CDN issue since I'm not seeing that from the US.

[flx42]

Ping @sb5518

[flx42]

No update, closing. If it persists, try contacting a local NVIDIA representative, as it might be a CDN issue related to your region, I can't help you any further.

[YankeeMarco]

nvidia technical staff

[Pingmin]

I just got this too on Ubuntu 16.04 by performing "apt-get update" from Shanghai, China :

Error: http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64 Release.gpg
The following signatures were invalid： BADSIG F60F4B3D7FA2AF80 cudatools [e-mail]

=== Chinese ===

错误: http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64 Release.gpg
下列签名无效： BADSIG F60F4B3D7FA2AF80 cudatools [e-mail]
......
W: 校验数字签名时出错。此仓库未被更新，所以仍然使用此前的索引文件。GPG 错误：http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64 Release: 下列签名无效： BADSIG F60F4B3D7FA2AF80 cudatools [e-mail]
W: 无法下载 http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/Release.gpg 下列签名无效： BADSIG F60F4B3D7FA2AF80 cudatools [e-mail]
W: 部分索引文件下载失败。如果忽略它们，那将转而使用旧的索引文件。

[flx42]

@Pingmin can you try again in a few hours? If it persists, I will ask for a CDN cache flush.

[vincentbissonnette]

I also have the exact same issue here, on nvidia/cuda:8.0 and 9.0 docker images for Ubuntu 16.04.
I am located in France.

[flx42]

@vincentbissonnette Use nvidia/cuda:8.0-devel or nvidia/cuda:9.0-devel

[vincentbissonnette]

I am indeed using the devel images in both cases. Sorry that was unclear.

[flx42]

@vincentbissonnette can you post the log?

[vincentbissonnette]

Log from apt-get update :

Hit:1 http://archive.ubuntu.com/ubuntu xenial InRelease
Hit:2 http://ppa.launchpad.net/george-edison55/cmake-3.x/ubuntu xenial InRelease
Get:3 http://archive.ubuntu.com/ubuntu xenial-updates InRelease [102 kB]                      
Get:4 http://archive.ubuntu.com/ubuntu xenial-backports InRelease [102 kB]                               
Get:5 http://security.ubuntu.com/ubuntu xenial-security InRelease [102 kB]                               
Ign:6 http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64  InRelease
Get:7 http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64  Release [564 B]
Get:8 http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64  Release.gpg [819 B]
Get:9 http://archive.ubuntu.com/ubuntu xenial-updates/universe Sources [234 kB]
Get:10 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages [903 kB]
Err:8 http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64  Release.gpg      
  The following signatures were invalid: BADSIG F60F4B3D7FA2AF80 cudatools <[email protected]>
Get:11 http://archive.ubuntu.com/ubuntu xenial-updates/restricted amd64 Packages [13.1 kB]        
Get:12 http://archive.ubuntu.com/ubuntu xenial-updates/universe amd64 Packages [735 kB]
Get:13 http://security.ubuntu.com/ubuntu xenial-security/universe Sources [56.8 kB]
Get:14 http://archive.ubuntu.com/ubuntu xenial-updates/multiverse amd64 Packages [18.5 kB]
Get:15 http://security.ubuntu.com/ubuntu xenial-security/main amd64 Packages [539 kB]
Get:16 http://security.ubuntu.com/ubuntu xenial-security/restricted amd64 Packages [12.7 kB]
Get:17 http://security.ubuntu.com/ubuntu xenial-security/universe amd64 Packages [245 kB]
Get:18 http://security.ubuntu.com/ubuntu xenial-security/multiverse amd64 Packages [3492 B]
Fetched 3069 kB in 1s (1781 kB/s)   
Reading package lists... Done
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64  Release: The following signatures were invalid: BADSIG F60F4B3D7FA2AF80 cudatools <[email protected]>
W: Failed to fetch http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/Release.gpg  The following signatures were invalid: BADSIG F60F4B3D7FA2AF80 cudatools <[email protected]>
W: Some index files failed to download. They have been ignored, or old ones used instead.

I have also tried the usual steps of deleting the /var/lib/apt/lists folder and updating again, as well as apt-get clean, the issue still remains.

[flx42]

@Pingmin @vincentbissonnette what's the output of docker run --rm -ti nvidia/cuda:9.0-devel apt-key list?

[vincentbissonnette]

Output:

/etc/apt/trusted.gpg
--------------------
pub   1024D/437D05B5 2004-09-12
uid                  Ubuntu Archive Automatic Signing Key <[email protected]>
sub   2048g/79164387 2004-09-12

pub   4096R/C0B21F32 2012-05-11
uid                  Ubuntu Archive Automatic Signing Key (2012) <[email protected]>

pub   4096R/EFE21092 2012-05-11
uid                  Ubuntu CD Image Automatic Signing Key (2012) <[email protected]>

pub   1024D/FBB75451 2004-12-30
uid                  Ubuntu CD Image Automatic Signing Key <[email protected]>

pub   4096R/7FA2AF80 2016-06-24
uid                  cudatools <[email protected]>

[vincentbissonnette]

However, the issue appears to have been fixed, as I'm now unable to reproduce the error.
apt-get update now performs the update as usual:

Get:1 http://archive.ubuntu.com/ubuntu xenial InRelease [247 kB]
Get:2 http://security.ubuntu.com/ubuntu xenial-security InRelease [102 kB]     
Get:3 http://archive.ubuntu.com/ubuntu xenial-updates InRelease [102 kB]
Get:4 http://archive.ubuntu.com/ubuntu xenial-backports InRelease [102 kB]         
Ign:5 http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64  InRelease
Get:6 http://archive.ubuntu.com/ubuntu xenial/universe Sources [9802 kB]
Get:7 http://security.ubuntu.com/ubuntu xenial-security/universe Sources [56.8 kB]                    
Get:8 http://security.ubuntu.com/ubuntu xenial-security/main amd64 Packages [539 kB]                                         
Ign:9 http://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1604/x86_64  InRelease
Get:10 http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64  Release [564 B]  
Get:11 http://security.ubuntu.com/ubuntu xenial-security/restricted amd64 Packages [12.7 kB]
Get:12 http://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1604/x86_64  Release [564 B]
Get:13 http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64  Release.gpg [801 B]
Get:14 http://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1604/x86_64  Release.gpg [801 B]
Get:15 http://security.ubuntu.com/ubuntu xenial-security/universe amd64 Packages [245 kB]                            
Get:16 http://security.ubuntu.com/ubuntu xenial-security/multiverse amd64 Packages [3492 B]                
Get:17 http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64  Packages [109 kB]                  
Get:18 http://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1604/x86_64  Packages [14.4 kB]
Get:19 http://archive.ubuntu.com/ubuntu xenial/main amd64 Packages [1558 kB]
Get:20 http://archive.ubuntu.com/ubuntu xenial/restricted amd64 Packages [14.1 kB]
Get:21 http://archive.ubuntu.com/ubuntu xenial/universe amd64 Packages [9827 kB]
Get:22 http://archive.ubuntu.com/ubuntu xenial/multiverse amd64 Packages [176 kB]
Get:23 http://archive.ubuntu.com/ubuntu xenial-updates/universe Sources [234 kB]
Get:24 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages [903 kB]
Get:25 http://archive.ubuntu.com/ubuntu xenial-updates/restricted amd64 Packages [13.1 kB]
Get:26 http://archive.ubuntu.com/ubuntu xenial-updates/universe amd64 Packages [735 kB]
Get:27 http://archive.ubuntu.com/ubuntu xenial-updates/multiverse amd64 Packages [18.5 kB]
Get:28 http://archive.ubuntu.com/ubuntu xenial-backports/main amd64 Packages [5162 B]
Get:29 http://archive.ubuntu.com/ubuntu xenial-backports/universe amd64 Packages [7146 B]
Fetched 24.8 MB in 4s (5605 kB/s)                           
Reading package lists... Done

Any idea on what causes this issue ?

[flx42]

I'm not sure, our CDN can get into a bad state sometimes, it's difficult to debug since it seems to be region specific.
Please file a new issue if it happens again.

[Pingmin]

@flx42 Sorry, I just see these messages right now. It seems like it's OK now and I have updated CUDA to v9.1.85-1 for some days.

[riless]

I have the same Issue with nvidia/cuda:9.0-cudnn7-runtime-ubuntu16.04 (from France)

Get:18 http://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1604/x86_64  Packages [17.9 kB]
Err:18 http://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1604/x86_64  Packages
  Hash Sum mismatch

[flx42]

I've circulated the information internally, since I don't maintain this package repository.

[riless]

I've retried today, and now it works.

[classicsong]

I have the same problem these days:
E: GPG error: http://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1404/x86_64 Release: The following signatures were invalid: NODATA 1 NODATA 2

[flx42]

@classicsong try setting the repository as https and check if it changes something.
You will need the apt-transport-https package.

[kyeongsoo]

@flx42 Changing 'http' to 'https' works for me. Great thanks! In fact, I have been suffering from this issue since last month or so.

[jageshmaharjan]

I think it's a region-specific error, some region doesn't allow access on 'http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64'
The above link will redirect to this link "http://114.80.80.238:8089/ceshi/index.html". The content is in the Chinese language the translated version is:

Tips: The site is temporarily inaccessible

According to the relevant laws and regulations of the Ministry of Industry and Information, because your website has not yet been filed, the website cannot be accessed.

According to the Order No. 33 of the Ministry of Industry and Information Technology of the People's Republic of China: Websites that do not have record numbers cannot be solved
Analysis and opening, otherwise it is necessary to bear the relevant legal responsibility.

Please be sure that all enterprises and institutions must complete the record and open the website. Thank you for your support!

Century Connect Record Consultation Contact Information

Registration Advisory Telephone: [phone]

What we could do is to edit on "sudo nano /etc/apt/sources.list.d/cuda.list" file and replace
"http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64" with
"https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64"

[peterborkuti]

What helped in my case:

0) install apt-transport-https
1) Setting link from http: to https: in /etc/apt/sources.list.d/cuda.list
2) updating apt-key
check the https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/ page and find the .pub file and update apt-key with that .pub file's content, like this way:

wget -qO - https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/7fa2af80.pub | sudo apt-key add -

[goodloop]

Thanks @peterborkuti. Step 1 really helped me.

[reubenjohn]

@peterborkuti Needed to perform an additional Step 3: sudo apt clean before sudo apt update worked to fix "problems were found: Metadata files have errors".

[htgeis]

It happened again. I have try the way mentioned above but it all failed. The same image worked in yesterday.

[flx42]

@htgeis the CUDA repositories are being updated right now, check again in ~1h.

[shridharkini]

nvidia/cuda:9.0-cudnn7-devel-ubuntu16.04 has the same issues still.

[nuzhet]

I am getting this error today. It was working fine yesterday:

E: Failed to fetch https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1604/x86_64/Packages.gz Hash Sum mismatch

[khurramHazen]

same issue as @nuzhet

[flx42]

@nuzhet I can reproduce, I'm filing an issue internally.

[khurramHazen]

@flx42 ~Do you think using 9.1-cudnn7-devel-ubuntu16.04 will help?~

Edit: No it will not.

[gpascualg]

I can also reproduce with nvidia/cuda:9.2-cudnn7-devel-ubuntu18.04 and nvidia/cuda:9.2-cudnn7-devel-ubuntu16.04

Error on 18.04 is somewhat different and references 1604:

E: Failed to fetch https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1604/x86_64/Packages.gz  File has unexpected size (29068 != 30835). Mirror sync in progress? [IP: 192.229.221.58 443]
   Hashes of expected file:
    - Filesize:30835 [weak]
    - SHA256:5900f4f7f9ea42522a4d2ef07c58e8492bea610380c47e803d75b6a1a30cdebe
    - SHA1:96cb67d3d73ebfe35b732bb46cde0e3784adf168 [weak]
    - MD5Sum:292e5eb4f76dd981bf199fa6760146b0 [weak]
   Release file created at: Tue, 25 Sep 2018 18:11:32 +0000
E: Some index files failed to download. They have been ignored, or old ones used instead.

[marcoaleixo]

I'm also having this error using this Dockerfile - https://github.com/tensorflow/tensorflow/blob/master/tensorflow/tools/dockerfiles/dockerfiles/nvidia-jupyter.Dockerfile

I'm from Brazil.

[pzawistowski]

Same situation from Poland.

[mgolub2]

Same issue in USA NW

[rpengms]

+1.

[mfojtak]

Same problem when building docker image.

Failed to fetch https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1604/x86_64/Packages.gz  Hash Sum mismatch

[flx42]

Yeah, I agree there is an issue, but I don't control this repository. I'm pushing for a resolution.
In the meantime, if your driver is recent enough (driver 410), you can use the CUDA 10.0 image on ubuntu 18.04: nvidia/cuda:10.0-base-ubuntu18.04
The 18.04 repository is not affected.

[mirekphd]

@flx42 It's gone, seems to be resolved.

[khurramHazen]

It is resolved in nvidia/cuda:9.0-cudnn7-devel-ubuntu16.04 too.

[flx42]

Cool! Thanks for confirming (and reporting!).

[flx42]

Locking this issue, please file a new one next time there is a problem with the repo :)