Nvidia-docker: Invalid signature `BADSIG F60F4B3D7FA2AF80` on Ubuntu 16.04

Created on 20 Jan 2018 · 22Comments · Source: NVIDIA/nvidia-docker

My issue is very similar to #571; opening an new issue as the stuff suggested reply.
I am using nvidia/cuda:9.0-devel.

P.S: I am located in Toronto, Canada if that's the cause of the problem (as @flx42 mentioned might be a CDN issue).

Issue description

When I ran sudo apt-get update, I got following feedback from console.

W: GPG error: http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64  Release: The following signatures were invalid: BADSIG F60F4B3D7FA2AF80 cudatools <[email protected]>
W: The repository 'http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64  Release' is not signed.
N: Data from such a repository can't be authenticated and is therefore potentially dangerous to use.
N: See apt-secure(8) manpage for repository creation and user configuration details.

I have tried deleting & re-adding the key via different methods, while the issue still exists.

But when I ran docker run --runtime=nvidia --rm nvidia/cuda nvidia-smi, the docker seems okay. I got:

| NVIDIA-SMI 387.34                 Driver Version: 387.34                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 745     Off  | 00000000:01:00.0 Off |                  N/A |
| 20%   47C    P0    N/A /  N/A |    449MiB /  4042MiB |     25%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
+-----------------------------------------------------------------------------+

So right now I am not sure whether that BADSIG F60F4B3D7FA2AF80 key will be a potential bomb, or I can just ignore that one.

Any helps appreciated, thanks in advance.

Source

pwyq

Most helpful comment

Does it work if the repo is set as https?
Try the following inside the container:

$ apt-get update && apt-get install -y apt-transport-https
$ echo 'deb https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64 /' > /etc/apt/sources.list.d/cuda.list
$ apt-get update

flx42 on 20 Jan 2018

👍13

All 22 comments

Does it work if the repo is set as https?
Try the following inside the container:

$ apt-get update && apt-get install -y apt-transport-https
$ echo 'deb https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64 /' > /etc/apt/sources.list.d/cuda.list
$ apt-get update

flx42 on 20 Jan 2018

👍13

Hi, the error messages are still the same.

pwyq on 20 Jan 2018

Thanks @pwyq, inside the container, please share the output of:

$ wget http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/Release

This will tell me which server you are hitting.

flx42 on 20 Jan 2018

Hi, thanks for your patience :). I can't provide anything right now as I'm off work. Will get back to you on next Monday.

pwyq on 20 Jan 2018

Hi, @flx42, the problem was solved (I didn't change anything). I'm almost certain this is a CDN issue.

By the way, may I ask how can you tell which server I'm hitting from following info?

$ wget http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/Release
--2018-01-22 09:04:51--  http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/Release
Resolving localhost (localhost)... 127.0.0.1
Connecting to localhost (localhost)|127.0.0.1|:8080... connected.
Proxy request sent, awaiting response... 200 OK
Length: 564 [application/octet-stream]
Saving to: ‘Release’

Release             100%[===================>]     564  --.-KB/s    in 0s      

2018-01-22 09:04:51 (36.2 MB/s) - ‘Release’ saved [564/564]

pwyq on 22 Jan 2018

Well, here you can't since you have a proxy.

flx42 on 22 Jan 2018

@pwyq what's the output of apt-key export 7FA2AF80 in the container after you just pulled it?

After the apt-get update can you try the following command:
apt-key adv --verify /var/lib/apt/lists/developer.download.nvidia.com_compute_cuda_repos_ubuntu1604_x86%5f64_Release.gpg /var/lib/apt/lists/developer.download.nvidia.com_compute_cuda_repos_ubuntu1604_x86%5f64_Release

3XX0 on 23 Jan 2018

@3XX0

$ apt-key export 7FA2AF80
-----BEGIN PGP PUBLIC KEY BLOCK-----
Version: GnuPG v1

mQINBFdtt4UBEAC8FDSWMR07GJZ265giLn7kLF+EsJCWESUq6Cd13QN0JQ/tLibi
QlW4ZjeOnEH9VPlqh/mKqNMG4SwRt8S+GHpePMQrr0aOkiRGfCclnAWIZURSAP+t
PLelCt43fkw1BBTopd/0oOzO8kHu8j8WU4A8GHxqghfFWPv54FQs2iaZ2eWR7a6d
79IJrbDKaVCCiQrkhCM8m648pNKHhuoJ9cQXFV+uvwkpfmKWGQ4ultxlOyjLHJLF
vuML2RuAO9IxbdZjzeYNN+T+wjFIBVcPnwEO+WrYgvGkT4r9aqVqTeg3EPb7QclV
sKBVJdxk4jZl0y22HAWqScVi6SJ15uK9pXxywDZkbpuRBWx4ThWiGe/FiUa2igi9
/SIvqN2TBY0g18sRTrylVr1wE1UGa/y7nDx6PoGCP1frBt8YUYt3pkM8Xvb2CRxx
CyWwmuFEQHC6jCEWf7FnoBHBYQwTVGNrU0vkuIeDrm+ZAcv8wx+ie1hlFhqCCJnf
jqeQ0/zA9RPmCPOkLyTdSsNZtlxxk7bzCdTdFFKzBjGTR7Gz3SMSp23d11eIyRiF
HQsp2v0SvnPJ6OcgB95Hmo544vi3RuoVfovtDOdfSBCRxP+GhhxkKSrTleQjD0/r
CGkdG2Kox3m9YllAsvZchLXlS7bZV9mGRF61mVMjF3HJRUQfBBm89VPQ+QARAQAB
tCBjdWRhdG9vbHMgPGN1ZGF0b29sc0BudmlkaWEuY29tPokCNwQTAQgAIQUCV223
hQIbAwULCQgHAwUVCgkICwUWAgMBAAIeAQIXgAAKCRD2D0s9f6KvgNArEAChnfcW
rYItgt7xXXubT6E+KpJyJ0RPrXf51S2mhciFbjDl+3EXRMRjOutVmgWYPWUUZaKR
8Iez3Lz4BRmwYOWBLtdnOLbKoSsQUX95rnPFjfly/DFLfjKxz4NRBmh4r4/rCYWm
2hmnXmOAi8kV7fqx3g5XMpJ//N6+T8ctEol2iZ82GrXjadcRWE4rAe7UyuEzJ74y
6ZKIzk5ijdgEKtcaBhzEWvoV5Pr9nkn7ByGsdehKR/gNnjPMYXrklSHGfphJIsS2
S32lMk/kuRjihBcWcYBXIPEQ7CV+PNW2TlkZj/YqTg637sZHwkhcjcNzxeqKvRYG
8V7Ju5hTDxL1UQBmgDS3cRx1lw7tYRG5bS67tbC2dc/CpPkG5agiZ/WyoHQDnn4r
1fRuOFx694QR6+0rAP6171xEEoNAPaH7gdJdhWKiYiJD0T2EEbW7wBUi/EupeKRv
kR12R1jUa1mlpxNtWQxJ7qp98T9+DmkxI1XDmWx0/g4ryuicwLDSqoPgNcRNdSQb
b8YfTqrkqaDdYzwLr/n0YKW3cYIvIeisV0WxRjb6OP7oAlAtaAhImlIc//51qNO7
/WAud6qMtnhFoZayR/BzLKqnCioN5GYr9BAKskpPHe9cDKVS3fg+Qvc1sNJID+jf
k52PqyW24Qsr0A9+5zQyE4tH9dfv120gj9avmg==
=0nKc
-----END PGP PUBLIC KEY BLOCK-----

$ apt-key adv --verify /var/lib/apt/lists/developer.download.nvidia.com_compute_cuda_repos_ubuntu1604_x86%5f64_Release.gpg /var/lib/apt/lists/developer.download.nvidia.com_compute_cuda_repos_ubuntu1604_x86%5f64_Release
Executing: /tmp/tmp.xHiQFOmQoB/gpg.1.sh --verify
/var/lib/apt/lists/developer.download.nvidia.com_compute_cuda_repos_ubuntu1604_x86%5f64_Release.gpg
/var/lib/apt/lists/developer.download.nvidia.com_compute_cuda_repos_ubuntu1604_x86%5f64_Release
gpg: Signature made Thu 18 Jan 2018 04:02:34 PM EST using RSA key ID 7FA2AF80
gpg: Good signature from "cudatools <[email protected]>"
gpg: WARNING: Using untrusted key!

pwyq on 23 Jan 2018

👍2

@flx42 @3XX0 I meet the same problem:

W: GPG error: https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1604/x86_64  Release: The following signatures were invalid: BADSIG F60F4B3D7
FA2AF80 cudatools <[email protected]>W: The repository 'https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1604/x86_64  Release' is not signed.
E: Failed to fetch https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1604/x86_64/Packages.gz  Hash Sum mismatch
E: Some index files failed to download. They have been ignored, or old ones used instead.

The result of wget https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1604/x86_64/Release is:

# wget https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1604/x86_64/Release
--2018-06-19 09:12:50--  https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1604/x86_64/Release
Resolving developer.download.nvidia.com (developer.download.nvidia.com)... 119.188.74.163, 119.188.74.162, 119.188.74.165, ...
Connecting to developer.download.nvidia.com (developer.download.nvidia.com)|119.188.74.163|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 564 [application/octet-stream]
Saving to: 'Release'

Release                                   100%[===================================================================================>]     564  --.-KB/s    in 0s      

2018-06-19 09:12:52 (49.4 MB/s) - 'Release' saved [564/564]

The #cat Release :

Origin: NVIDIA
Label: NVIDIA CUDA
Architecture: x86_64
Date: Thu, 31 May 2018 02:05:48 +0000
MD5Sum:
 3fd9c7abb50441a77645a822c31632d0            92849 Packages
 930afb4079e9ccb03a6bc85be672abfd            24553 Packages.gz
SHA1:
 91cae9157974056be9fb011a0994a4ffb90fe834            92849 Packages
 5eb1185b7a97e9d262b6d1ffd0cb448424f9f867            24553 Packages.gz
SHA256:
 980a1ddfac37f27ff160228894c121c531d95077711495fd5a6dffff1f42400f            92849 Packages
 e54c31c63724ad023ded77dbf7a3b95e1cc3fc7dfa92198f02f6b08570762b19            24553 Packages.gz

The apt-key export 7FA2AF80 is :

The

apt-key adv --verify ./Release.gpg /var/lib/apt/lists/developer.download.nvidia.com_compute_machine-learning_repos_ubuntu1604_x86%5f64_Re
lease

is:

Executing: /tmp/tmp.mePybtekKm/gpg.1.sh --verify
./Release.gpg
/var/lib/apt/lists/developer.download.nvidia.com_compute_machine-learning_repos_ubuntu1604_x86%5f64_Release
gpg: Signature made Thu May 31 02:05:48 2018 UTC using RSA key ID 7FA2AF80
gpg: Good signature from "cudatools <[email protected]>"
gpg: WARNING: Using untrusted key!

The $docker run --rm -ti nvidia/cuda:9.1-devel apt-key list is:

Unable to find image 'nvidia/cuda:9.1-devel' locally
9.1-devel: Pulling from nvidia/cuda
b234f539f7a1: Pull complete 
55172d420b43: Pull complete 
5ba5bbeb6b91: Pull complete 
43ae2841ad7a: Pull complete 
f6c9c6de4190: Pull complete 
d8c68dd3efd7: Pull complete 
2a5a3d157169: Pull complete 
b5245bae531f: Pull complete 
4616f5613d58: Pull complete 
316e6a697dea: Pull complete 
Digest: sha256:32ed0cff1f856b66e244711af23601aa628de2e9de3b37748512299bf974cad6
Status: Downloaded newer image for nvidia/cuda:9.1-devel
/etc/apt/trusted.gpg
--------------------
pub   1024D/437D05B5 2004-09-12
uid                  Ubuntu Archive Automatic Signing Key <[email protected]>
sub   2048g/79164387 2004-09-12

pub   4096R/C0B21F32 2012-05-11
uid                  Ubuntu Archive Automatic Signing Key (2012) <[email protected]>

pub   4096R/EFE21092 2012-05-11
uid                  Ubuntu CD Image Automatic Signing Key (2012) <[email protected]>

pub   1024D/FBB75451 2004-12-30
uid                  Ubuntu CD Image Automatic Signing Key <[email protected]>

pub   4096R/7FA2AF80 2016-06-24
uid                  cudatools <[email protected]>

linrio on 19 Jun 2018

I've asked for a CDN flush. Thanks for the report.

flx42 on 19 Jun 2018

This is happening to our cloud as we speak. same exact errors on a bare metal server.

QthePirate on 13 Jul 2018

@QthePirate please share the logs and the output of the commands listed in https://github.com/NVIDIA/nvidia-docker/issues/613#issuecomment-359944005

flx42 on 13 Jul 2018

@flx42 I just solved this with the apt-transport-https fix.

QthePirate on 13 Jul 2018

I changed the download source. It's OK!

backup the source.list:
sudo cp /etc/apt/sources.list /etc/apt/sources.list.backup
modified the source.list and added mainland source(for example in Chinese Mainland):
vi /etc/apt/sources.list

3 Note the source in the metafile and add the following address:

#Ali Cloud
deb http://mirrors.aliyun.com/ubuntu/ trusty main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ trusty-security main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ trusty-updates main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ trusty-proposed main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ trusty-backports main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ trusty main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ trusty-security main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ trusty-updates main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ trusty-proposed main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ trusty-backports main restricted universe multiverse

update source
sudo apt-get update

5 upgrade software:

sudo apt-get dist-upgrade
sudo apt-get -f install

linrio on 7 Nov 2018

🎉6

@linrio you sure? trusty is for ubuntu 14

ChinoMars on 6 Dec 2018

@flx42
I am still hitting this issue:

W: GPG error: https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1604/x86_64 Release: The following signatures were invalid: BADSIG F60F4B3D7FA2AF80 cudatools cudatools@nvidia.com
W: The repository 'https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1604/x86_64 Release' is not signed.
E: Failed to fetch https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1604/x86_64/Packages.gz Hash Sum mismatch
E: Some index files failed to download. They have been ignored, or old ones used instead.

Any solution?

zionwu on 24 Dec 2018

i'm in mainland and i solved the problem with adding <ip of developer.download.nvidia.com> developer.download.nvidia.cn to the host, so you can also change your dns setting or set the proxy to do the same

lavender37 on 5 Oct 2020

🚀1

@lavender37 您好，请问一下您的这个是怎么具体解决的，感谢

vitahlin on 13 Oct 2020

@lavender37 您好，请问一下您的这个是怎么具体解决的，感谢

我的问题是在创建docker时在apt update过程中出现了Invalid signature BADSIG F60F4B3D7FA2AF80
但是遗憾的是现在我已经完全无法连接developer.download.nvidia.com除非使用proxy
简单来说我当时使用gpg verify了developer.download.nvidia.com和developer.download.nvidia.cn的下载内容
然后说明我的个人见解，当使用docker国内连接到developer.download.nvidia.com时似乎会自动转到developer.download.nvidia.cn，而验证gpg时时，developer.download.nvidia.com的内容是能验证通过而developer.download.nvidia.cn的不能（原因未知）
所以考虑到这种差异，我在host文件中将developer.download.nvidia.cn解析到developer.download.nvidia.com对应的ip地址（原理等同于proxy），最终验证通过

lavender37 on 17 Oct 2020

👍7 🚀4

@lavender37 您好，请问一下您的这个是怎么具体解决的，感谢

我的问题是在创建docker时在apt update过程中出现了Invalid signature BADSIG F60F4B3D7FA2AF80
但是遗憾的是现在我已经完全无法连接developer.download.nvidia.com除非使用proxy
简单来说我当时使用gpg verify了developer.download.nvidia.com和developer.download.nvidia.cn的下载内容
然后说明我的个人见解，当使用docker国内连接到developer.download.nvidia.com时似乎会自动转到developer.download.nvidia.cn，而验证gpg时时，developer.download.nvidia.com的内容是能验证通过而developer.download.nvidia.cn的不能（原因未知）
所以考虑到这种差异，我在host文件中将developer.download.nvidia.cn解析到developer.download.nvidia.com对应的ip地址（原理等同于proxy），最终验证通过

问题解决了，感谢🙏

vitahlin on 22 Oct 2020

英伟达仓库的 CDN 缓存背锅。签名 Release.gpg 缓存了个新的，Release 本体却没有更新，还是旧的。整个仓库的同步和缓存竟然不是原子化的，太糟糕了。
It's ALL Nvidia CDN's FAULT. The CDN cached a newer signature Release.gpg but did NOT refresh Release's cache at the same time. I could not imagine that the entire repository is not synchronized and cached atomically. This is too AWFUL.

OLD Release + NEW Release.gpg => BAD signature.

NVIDIA, ~~thank~~ you!

$ gpg --verify Release.gpg Release
gpg: Signature made Thu Jan 28 00:03:08 2021 UTC using RSA key ID 7FA2AF80
gpg: BAD signature from "cudatools <[email protected]>"

$ gpg --verify Release.gpg Release 2>&1 | grep -i made
gpg: Signature made Thu Jan 28 00:03:08 2021 UTC using RSA key ID 7FA2AF80
$ head Release | grep -i date
Date: Tue, 19 Jan 2021 20:03:37 +0000

As shown above, the Release's date is Tue, 19 Jan, but the signature made on Thu Jan 28.

至于说一直再复读的这句话：
As for

This is not an issue for nvidia-docker itself.

All issues related to images running on top of nvidia-docker should be directed here:
https://forums.developer.nvidia.com/c/accelerated-computing/cuda/cuda-setup-and-installation/8

我看到确实有人转去论坛发帖问了，只不过嘛，自始自终一个回复都没有XD。
I found some guys did go to the forum and post this issue there, but they just never recieve any response... XD

ynyyn on 31 Jan 2021

主要是两个地址冲突了,developer.download.nvidia.cn 和developer.download.nvidia.com的地址会被跳转到.cn的地址,这两个地址是不一样的pub,可以把原来的developer.download.nvidia.com都改成developer.download.nvidia.cn