Teleport: v5.0.1 and v5.0.2 tarballs sha256 changed?

[carlocab]

Summary

The checksum of the v5.0.2 and v5.0.1 release tarballs changed.

Relevant information

We're attempting to update Homebrew's version of Go here: https://github.com/Homebrew/homebrew-core/pull/66355.

While testing the new version, CI produced the following error:

==> brew install --build-from-source teleport
==> FAILED
==> Downloading https://github.com/gravitational/webassets/archive/72412062d6d55ec7faa9707abf500d703e7d09da.tar.gz
==> Downloading from https://codeload.github.com/gravitational/webassets/tar.gz/72412062d6d55ec7faa9707abf500d703e7d09da
==> Downloading https://github.com/gravitational/teleport/archive/v5.0.1.tar.gz
==> Downloading from https://codeload.github.com/gravitational/teleport/tar.gz/v5.0.1
Error: SHA256 mismatch
Expected: 3145e0b93c7b38fdf94e8cd995699fca8f78c26228bd19b1aad66b4749fd7719
  Actual: 2b3bece3c38e8425c41045fd4e6c7dee285dfb8220d1d147a86c684375f69cf8

This shows a change in the sha256 of the v5.0.1 tarball. Similarly, when Homebrew updated teleport to v5.0.2, the sha256 was reported as

f90b7040f326611c33c8e63ef6d72745096e7b29fcf46474b4055cc8d86a3211

Now it is

430f02c4ec20ae7cd77ae3c263225fd6e4692a000eaeae7c81114c50595dec25

I can update the sha256 associated with the formula, but CI will come back to me with the following error:

teleport:
  * stable sha256 changed without the url/version also changing; please create an issue upstream to rule out malicious circumstances and to find out why the file changed.

Can I confirm that nothing is amiss here?

Environment (if relevant)

N/A

[carlocab]

Edit: Sorry, I was confused when I wrote this comment. Please ignore this; I've updated my original post to clarify.

---

When I opened this issue, I checked your release tarball's sha256. It matched the one listed in my original post:

2b3bece3c38e8425c41045fd4e6c7dee285dfb8220d1d147a86c684375f69cf8

Now, if I try to download it again, it reports the following sha256:

430f02c4ec20ae7cd77ae3c263225fd6e4692a000eaeae7c81114c50595dec25

[klizhentas]

@carlocab I gave it a try and the checksums matched for 5.0.1 and 5.0.2 respectively

~ $ shasum -a 256 teleport-5.0.1.tar.gz
2b3bece3c38e8425c41045fd4e6c7dee285dfb8220d1d147a86c684375f69cf8  teleport-5.0.1.tar.gz
~ $ shasum -a 256 teleport-5.0.2.tar.gz
430f02c4ec20ae7cd77ae3c263225fd6e4692a000eaeae7c81114c50595dec25  teleport-5.0.2.tar.gz

[carlocab]

Hmm, yes, I see. Sorry, my original post was about the v5.0.1 tarball. My second comment was mistaken, as I was looking at v5.0.2.

However, when Homebrew updated teleport to 5.0.1 (https://github.com/Homebrew/homebrew-core/commit/356ddcc941e0b17882a8f3bd8ecab9d8e8ae3c30#diff-5c6f19f4acfd46cdd135dc6abd85a833178756485b9c119bcc1d8bf49ba9ce05), the sha256 was recorded as

3145e0b93c7b38fdf94e8cd995699fca8f78c26228bd19b1aad66b4749fd7719

Now it is reporting something else.

[carlocab]

Actually, your v5.0.2 tarball also had its sha256 changed.

Yesterday, it was updated at Homebrew (https://github.com/Homebrew/homebrew-core/commit/2178f0fa3867bf97e717f0b293d0a020ed85f474#diff-5c6f19f4acfd46cdd135dc6abd85a833178756485b9c119bcc1d8bf49ba9ce05) and reported a sha256 of

f90b7040f326611c33c8e63ef6d72745096e7b29fcf46474b4055cc8d86a3211

Now it is something else.

[carlocab]

@klizhentas I updated my original post, as the way I had written it was confusing. (I was also confused myself then, so that might explain it.) Thanks again for looking into it.

[klizhentas]

I wonder if it's a case of this problem

[carlocab]

It may well be, but Homebrew downloads thousands of tarballs from GitHub and doesn't encounter that problem too often. For example, Homebrew CI will have downloaded over 500 tarballs, most of them from GitHub, in this run: https://github.com/Homebrew/homebrew-core/pull/66355/checks?check_run_id=1575924257

Only teleport's tarball reported a change in its sha256.

The most common explanation is that those releases were re-tagged, which changes their sha256. See, for example, https://github.com/alexei-led/pumba/issues/190, https://github.com/Aloxaf/silicon/issues/141, https://github.com/Findomain/Findomain/issues/121, https://github.com/cmyr/cargo-instruments/issues/27

[carlocab]

Looking into it some more, I don't think it's an instance of that problem. Whenever it happens, _lots of_ tarballs have their sha256s change. In this case it was only yours.

[klizhentas]

@carlocab I will check with the team and let you know

[klizhentas]

I have also created support request for Github to pull any audit data related to the hash change

[russjones]

@carlocab Any chance the old tarball is still available? We can compare the archives to see what changed.

[carlocab]

Homebrew doesn't store the tarballs, unfortunately. We only keep a record of the sha256.

[SMillerDev]

If @chenrui333 didn't run brew cleanup yet they might still have it.

[klizhentas]

We checked the logs of our CI/CD system and did not find any changes to the tags after the original release. We are waiting for additional information from Github.

[klizhentas]

While doing some digging, we have noticed the strange difference between source code in git vs Github generated tarball:

$ diff -r teleport-5.0.2 teleport

diff -r teleport-5.0.2/vendor/k8s.io/client-go/pkg/version/base.go teleport/vendor/k8s.io/client-go/pkg/version/base.go
58,59c58,59
< gitVersion   string = "v0.0.0-master+17679dcec2"
< gitCommit    string = "17679dcec24796955b96485d5922b6d695735a3b" // sha1 from git, output of $(git rev-parse HEAD)
---
> gitVersion   string = "v0.0.0-master+$Format:%h$"
> gitCommit    string = "$Format:%H$" // sha1 from git, output of $(git rev-parse HEAD)

Here is a guess from one of our engineers, @webvictim:

17679dcec24796955b96485d5922b6d695735a3b is the commit SHA of the v5.0.2 tag. https://git-scm.com/docs/pretty-formats states that %H is a Git format string which is expanded to the commit hash. My guess would be that this is a recently introduced bug in whatever code on Github’s end is handling the Git checkout -> ZIP file conversion. For some reason it’s parsing that format placeholder in our vendored client-go module and expanding it to the commit SHA, causing any previously-computed hashes to change.

[chenrui333]

If @chenrui333 didn't run brew cleanup yet they might still have it.

Those PRs are managed by action runner and I dont have cached artifacts in my local :(

[webvictim]

To expand on this a little: https://github.com/gravitational/teleport/blob/master/vendor/k8s.io/client-go/pkg/version/base.go#L55-L59 shows that the tag formats are processed and expanded when git archive is run. My guess is that this command is what Github uses to generate the archives.

[webvictim]

To expand on this a little: https://github.com/gravitational/teleport/blob/master/vendor/k8s.io/client-go/pkg/version/base.go#L55-L59 shows that the tag formats are processed and expanded when git archive is run with export-subst set in the .gitattributes file (as it is in https://github.com/gravitational/teleport/blob/master/vendor/k8s.io/client-go/pkg/version/.gitattributes)

My guess is that git archive is what Github uses to generate the archives. We could delete the .gitattributes file and that would prevent this from happening again, but it would push the module out of sync with upstream.