Rustup: Cannot install nightly: checksum failed

Created on 2 Oct 2020  路  13Comments  路  Source: rust-lang/rustup

Problem


rustup install nightly fails with the error message:

error: component download failed for cargo-x86_64-unknown-linux-gnu
error: caused by: checksum failed

Steps

  1. Run rustup install nightly

Possible Solution(s)

Notes
Output of rustup install nightly --verbose:

verbose: read metadata version: '12'
verbose: Use of (currently) unmaintained command line interface.
verbose: The exact API of this command may change without warning
verbose: Eventually this command will be a true alias.  Until then:
verbose:   Please use `rustup toolchain install` instead
verbose: updating existing install for 'nightly-x86_64-unknown-linux-gnu'
verbose: toolchain directory: '/home/robin/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu'
info: syncing channel updates for 'nightly-x86_64-unknown-linux-gnu'
verbose: creating temp file: /home/robin/.rustup/tmp/ndkps923b_0vh7t1_file
verbose: downloading file from: 'https://static.rust-lang.org/dist/channel-rust-nightly.toml.sha256'
verbose: downloading with reqwest
verbose: deleted temp file: /home/robin/.rustup/tmp/ndkps923b_0vh7t1_file
verbose: creating temp file: /home/robin/.rustup/tmp/hrr_efz98x1511gx_file.toml
verbose: downloading file from: 'https://static.rust-lang.org/dist/channel-rust-nightly.toml'
verbose: downloading with reqwest
verbose: checksum passed
verbose: creating temp file: /home/robin/.rustup/tmp/uq9p4_nl3iub6a1__file
verbose: downloading file from: 'https://static.rust-lang.org/dist/channel-rust-nightly.toml.asc'
verbose: downloading with reqwest
verbose: deleted temp file: /home/robin/.rustup/tmp/uq9p4_nl3iub6a1__file
verbose: Good signature from on https://static.rust-lang.org/dist/channel-rust-nightly.toml from:
verbose: from builtin Rust release key
verbose:   RSA/85AB96E6-FA1BE5FE - Rust Language (Tag and Release Signing Key) <[email protected]>
verbose:   Fingerprint: 108F 6620 5EAE B0AA A8DD 5E1C 85AB 96E6 FA1B E5FE
verbose: deleted temp file: /home/robin/.rustup/tmp/hrr_efz98x1511gx_file.toml
info: latest update on 2020-10-02, rust version 1.48.0-nightly (154f1f544 2020-10-02)
info: downloading component 'cargo'
verbose: downloading file from: 'https://static.rust-lang.org/dist/2020-10-02/cargo-nightly-x86_64-unknown-linux-gnu.tar.xz'
verbose: downloading with reqwest
verbose: resuming partial download
info: Retrying download for 'https://static.rust-lang.org/dist/2020-10-02/cargo-nightly-x86_64-unknown-linux-gnu.tar.xz'
verbose: downloading file from: 'https://static.rust-lang.org/dist/2020-10-02/cargo-nightly-x86_64-unknown-linux-gnu.tar.xz'
verbose: downloading with reqwest
error: component download failed for cargo-x86_64-unknown-linux-gnu
error: caused by: checksum failed

Output of rustup --version: rustup 1.22.1 (b01adbbc3 2020-07-08)
Output of rustup show:

Default host: x86_64-unknown-linux-gnu
rustup home:  /home/robin/.rustup

installed toolchains
--------------------

stable-x86_64-unknown-linux-gnu (default)
nightly-x86_64-unknown-linux-gnu
stage1
1.36.0-x86_64-unknown-linux-gnu
1.40.0-x86_64-unknown-linux-gnu
1.45.2-x86_64-unknown-linux-gnu

installed targets for active toolchain
--------------------------------------

i686-unknown-linux-gnu
x86_64-unknown-linux-gnu

active toolchain
----------------

stable-x86_64-unknown-linux-gnu (default)
rustc 1.46.0 (04488afe3 2020-08-24)
bug
Source
picture robinkrahl
👍1

Most helpful comment

Hello all, and sorry for today's outage. I prepared a small writeup explaining what happened and what we're doing to prevent the problem from happening again.

All the files related to a release live at https://static.rust-lang.org/dist/YYYY-MM-DD, and since there is usually at most a nightly release each day no artifacts are overridden during normal operations. Today's nightly was released at 00:35 UTC, and shipped commit 8fe73e80d762. Its artifacts were stored at https://static.rust-lang.org/dist/2020-10-02/*.

One of the projects I'm currently working on is a revamp of our release process. The goal is to make it easier to maintain and extend the release tooling, while improving its security, speed and cost. Today I was starting test releases on our staging environment but, due to a bug in a script I was writing, a nightly release started on the production environment instead of staging.

The production release process I mistakenly started began at 11:05 UTC for commit 154f1f544dd, and started to override the files in https://static.rust-lang.org/dist/2020-10-02/*. The process ended at 11:41 UTC, and as usual the tooling issued a CDN invalidation for the manifest file used by Rustup.

Unfortunately the tooling assumed only one nightly was released every day, and it issued invalidations just for the manifest, not for the artifacts uploaded at https://static.rust-lang.org/dist/2020-10-02/*. This meant people downloaded the fresh manifest (with the checksums for commit 154f1f544dd in it) and the old cached artifacts (generated for commit 8fe73e80d762), causing the checksum mismatch errors.

Once we were alerted of the issue we began investigating what happened and, once we verified no infrastructure compromise happened and we understood the root cause, we issued a CDN invalidation for the whole static.rust-lang.org domain. That removed the old artifacts from the cache and allowed rustup downloads to resume as normal.

Following up on this outage, we'll change the release tooling to also invalidate all artifacts it uploads, and we'll add safeguards to prevent multiple nightlies from being released the same day.

picture pietroalbini on 2 Oct 2020
❤16 👍11

All 13 comments

Similar error on Travis CI: https://travis-ci.org/github/jugglerchris/rust-html2text/jobs/732229335

robinkrahl picture robinkrahl on 2 Oct 2020

Worked fine for me:

info: syncing channel updates for 'nightly-2020-10-02-x86_64-unknown-linux-gnu'
info: latest update on 2020-10-02, rust version 1.48.0-nightly (8fe73e80d 2020-10-01)
info: downloading component 'cargo'
info: downloading component 'rust-std'
 21.9 MiB /  21.9 MiB (100 %)   5.4 MiB/s in  4s ETA:  0s
info: downloading component 'rustc'
 54.8 MiB /  54.8 MiB (100 %)   3.7 MiB/s in 13s ETA:  0s
info: installing component 'cargo'
info: Defaulting to 500.0 MiB unpack ram
info: installing component 'rust-std'
 21.9 MiB /  21.9 MiB (100 %)   9.2 MiB/s in  2s ETA:  0s
info: installing component 'rustc'
 54.8 MiB /  54.8 MiB (100 %)   8.9 MiB/s in  6s ETA:  0s

  nightly-2020-10-02-x86_64-unknown-linux-gnu installed - rustc 1.48.0-nightly (8fe73e80d 2020-10-01)
bjorn3 picture bjorn3 on 2 Oct 2020

@bjorn3:

That鈥檚 a different version. From my output:

info: latest update on 2020-10-02, rust version 1.48.0-nightly (154f1f544 2020-10-02)

From your output:

info: latest update on 2020-10-02, rust version 1.48.0-nightly (8fe73e80d 2020-10-01)
robinkrahl picture robinkrahl on 2 Oct 2020
👍4

I have same issue

MikailBag picture MikailBag on 2 Oct 2020
👍3

I have the same issue as well

cjgratacos picture cjgratacos on 2 Oct 2020
👍2

Same here:

$ rustup toolchain install nightly
info: syncing channel updates for 'nightly-x86_64-unknown-linux-gnu'
info: latest update on 2020-10-02, rust version 1.48.0-nightly (154f1f544 2020-10-02)
info: downloading component 'cargo'
info: Retrying download for 'https://static.rust-lang.org/dist/2020-10-02/cargo-nightly-x86_64-unknown-linux-gnu.tar.xz'
  5.2 MiB /   5.2 MiB (100 %)   4.7 MiB/s in  1s ETA:  0s
error: component download failed for cargo-x86_64-unknown-linux-gnu

Also:

$ rustup --version
rustup 1.22.1 (b01adbbc3 2020-07-08)

And:

$ rustup update
info: syncing channel updates for 'stable-x86_64-unknown-linux-gnu'
info: checking for self-updates

  stable-x86_64-unknown-linux-gnu unchanged - rustc 1.46.0 (04488afe3 2020-08-24)

info: cleaning up downloads & tmp directories
homeworkprod picture homeworkprod on 2 Oct 2020

This is being looked into by the release team.

kinnison picture kinnison on 2 Oct 2020
❤5

I have same issue https://github.com/bandprotocol/bandchain/pull/2695/checks?check_run_id=1198672311

Benzbeeb picture Benzbeeb on 2 Oct 2020
👀1

We're aware of the issue, and we're looking into a solution.

pietroalbini picture pietroalbini on 2 Oct 2020
❤1

Same here. Github Action: https://github.com/Phala-Network/phala-blockchain/runs/1198792195

h4x3rotab picture h4x3rotab on 2 Oct 2020

The issue is now fixed.

pietroalbini picture pietroalbini on 2 Oct 2020
❤6

Hello all, and sorry for today's outage. I prepared a small writeup explaining what happened and what we're doing to prevent the problem from happening again.

All the files related to a release live at https://static.rust-lang.org/dist/YYYY-MM-DD, and since there is usually at most a nightly release each day no artifacts are overridden during normal operations. Today's nightly was released at 00:35 UTC, and shipped commit 8fe73e80d762. Its artifacts were stored at https://static.rust-lang.org/dist/2020-10-02/*.

One of the projects I'm currently working on is a revamp of our release process. The goal is to make it easier to maintain and extend the release tooling, while improving its security, speed and cost. Today I was starting test releases on our staging environment but, due to a bug in a script I was writing, a nightly release started on the production environment instead of staging.

The production release process I mistakenly started began at 11:05 UTC for commit 154f1f544dd, and started to override the files in https://static.rust-lang.org/dist/2020-10-02/*. The process ended at 11:41 UTC, and as usual the tooling issued a CDN invalidation for the manifest file used by Rustup.

Unfortunately the tooling assumed only one nightly was released every day, and it issued invalidations just for the manifest, not for the artifacts uploaded at https://static.rust-lang.org/dist/2020-10-02/*. This meant people downloaded the fresh manifest (with the checksums for commit 154f1f544dd in it) and the old cached artifacts (generated for commit 8fe73e80d762), causing the checksum mismatch errors.

Once we were alerted of the issue we began investigating what happened and, once we verified no infrastructure compromise happened and we understood the root cause, we issued a CDN invalidation for the whole static.rust-lang.org domain. That removed the old artifacts from the cache and allowed rustup downloads to resume as normal.

Following up on this outage, we'll change the release tooling to also invalidate all artifacts it uploads, and we'll add safeguards to prevent multiple nightlies from being released the same day.

pietroalbini picture pietroalbini on 2 Oct 2020
❤16 👍11

As a followup, I opened a PR to the new release tooling I'm working on implementing the changes I mentioned in the previous comment. Once the PR is merged I'll backport the changes to the currently used tooling.

pietroalbini picture pietroalbini on 5 Oct 2020
❤1
Was this page helpful?
0 / 5 - 0 ratings

Related issues

matthiaskrgr picture matthiaskrgr  路  3Comments
TomasHubelbauer picture TomasHubelbauer  路  3Comments
ghost picture ghost  路  3Comments
mcandre picture mcandre  路  4Comments
fenhl picture fenhl  路  4Comments