Pkg.jl: An incompatible version of MbedTLS is installed at wrong version_slug for Julia 1.0.5 on Travis

[tkf]

Following MWE shows some strange behavior:

• Pkg says Installed MbedTLS ──────── v0.6.8 but using MbedTLS tries to load MbedTLS 1.0.2.
• julia uses X4xar as the version_slug which is correct one for MbedTLS v0.6.8. However, the actual directory contains MbedTLS 1.0.2.

$ julia -e 'using Pkg; pkg"add MbedTLS"'
   Cloning default registries into /home/travis/.julia/registries
   Cloning registry General from "https://github.com/JuliaRegistries/General.git"
 Resolving package versions...
 Installed BinaryProvider ─ v0.5.10
 Installed MbedTLS ──────── v0.6.8
  Updating `~/build/tkf/MyPlayground.jl/Project.toml`
  [739be429] + MbedTLS v0.6.8
  Updating `~/build/tkf/MyPlayground.jl/Manifest.toml`
  [b99e7846] + BinaryProvider v0.5.10
  [739be429] + MbedTLS v0.6.8
  [2a0f44e3] + Base64 
  [ade2ca70] + Dates 
  [8ba89e20] + Distributed 
  [b77e0a4c] + InteractiveUtils 
  [8f399da3] + Libdl 
  [37e2e46d] + LinearAlgebra 
  [56ddb016] + Logging 
  [d6f4376e] + Markdown 
  [de0858da] + Printf 
  [9a3f8284] + Random 
  [ea8e919c] + SHA 
  [9e88b42a] + Serialization 
  [6462fe0b] + Sockets 
  [8dfed614] + Test 
  [4ec0a83e] + Unicode 
The command "julia -e 'using Pkg; pkg"add MbedTLS"'" exited with 0.

$ head /home/travis/.julia/packages/MbedTLS/X4xar/src/MbedTLS.jl
module MbedTLS
using Random, Sockets, MbedTLS_jll
import Sockets: TCPSocket
export
# Message digests
    MD_NONE,
    MD_MD2,
    MD_MD4,
The command "head /home/travis/.julia/packages/MbedTLS/X4xar/src/MbedTLS.jl" exited with 0.

$ cat /home/travis/.julia/packages/MbedTLS/X4xar/REQUIRE
cat: /home/travis/.julia/packages/MbedTLS/X4xar/REQUIRE: No such file or directory
The command "cat /home/travis/.julia/packages/MbedTLS/X4xar/REQUIRE" exited with 1.

$ cat /home/travis/.julia/packages/MbedTLS/X4xar/Project.toml
name = "MbedTLS"
uuid = "739be429-bea8-5141-9913-cc70e7f3736d"
authors = ["Jacob Quinn <[email protected]>"]
version = "1.0.2"
[deps]
Dates = "ade2ca70-3891-5945-98fb-dc099432e06a"
MbedTLS_jll = "c8ffd9c3-330d-5841-b78e-0817d7145fa1"
Random = "9a3f8284-a2c9-5f02-9a11-845980a1fd5c"
Sockets = "6462fe0b-24de-5631-8697-dd941f90decc"
[compat]
julia = "1.3"
MbedTLS_jll = "2"
[extras]
Distributed = "8ba89e20-285c-5b6f-9357-94700520ee1b"
Test = "8dfed614-e22c-5e08-85e1-65c5234f0b40"
[targets]
test = ["Test", "Distributed"]
The command "cat /home/travis/.julia/packages/MbedTLS/X4xar/Project.toml" exited with 0.

$ julia -e 'using MbedTLS'
ERROR: LoadError: ArgumentError: Package MbedTLS does not have MbedTLS_jll in its dependencies:
- If you have MbedTLS checked out for development and have
  added MbedTLS_jll as a dependency but haven't updated your primary
  environment's manifest file, try `Pkg.resolve()`.
- Otherwise you may need to report an issue with MbedTLS
Stacktrace:
 [1] require(::Module, ::Symbol) at ./loading.jl:836
 [2] include at ./boot.jl:317 [inlined]
 [3] include_relative(::Module, ::String) at ./loading.jl:1044
 [4] include(::Module, ::String) at ./sysimg.jl:29
 [5] top-level scope at none:2
 [6] eval at ./boot.jl:319 [inlined]
 [7] eval(::Expr) at ./client.jl:393
 [8] top-level scope at ./none:3
in expression starting at /home/travis/.julia/packages/MbedTLS/X4xar/src/MbedTLS.jl:3
ERROR: Failed to precompile MbedTLS [739be429-bea8-5141-9913-cc70e7f3736d] to /home/travis/.julia/compiled/v1.0/MbedTLS/bf9T0.ji.
Stacktrace:
 [1] error(::String) at ./error.jl:33
 [2] compilecache(::Base.PkgId, ::String) at ./loading.jl:1203
 [3] _require(::Base.PkgId) at ./loading.jl:960
 [4] require(::Base.PkgId) at ./loading.jl:858
 [5] require(::Module, ::Symbol) at ./loading.jl:853

https://travis-ci.org/github/tkf/MyPlayground.jl/builds/714907915#L214
https://github.com/tkf/MyPlayground.jl/blob/8483fda3a60371f897fbefec3f6ed473d863c089/.travis.yml

Note that X4xar is the correct version_slug of MbedTLS v0.6.8.

julia> Base.version_slug(Base.UUID("739be429-bea8-5141-9913-cc70e7f3736d"), Base.SHA1("2d94286a9c2f52c63a16146bb86fd6cdfbf677c6"))
"X4xar"

where 2d94286a9c2f52c63a16146bb86fd6cdfbf677c6 is the tree sha of b71d85de30afb1b4638d1169fa1f6432094a665a (v0.6.8) https://github.com/JuliaLang/MbedTLS.jl/commit/b71d85de30afb1b4638d1169fa1f6432094a665a.

(Originally reported in https://github.com/JuliaLang/MbedTLS.jl/issues/224)

[KristofferC]

Downloading https://api.github.com/repos/JuliaLang/MbedTLS.jl/tarball/2d94286a9c2f52c63a16146bb86fd6cdfbf677c6 which is the tree-sha for v0.6.8 seems to give the 1.0.2 tarball...

[KristofferC]

Wait, it seems to redirect to https://codeload.github.com/JuliaLang/MbedTLS.jl/legacy.tar.gz/master...

[KristofferC]

Eeh, is this the case for all our API tarball downloads? That's a bit worrying.

[staticfloat]

I can confirm that the https://api.github.com/repos/<org>/<repo>/tarball/<treehash>.tar.gz API endpoint seems to be redirecting to the latest master via the URL https://codeload.github.com/<org>/<repo>/legacy.tar.gz/master. Neither tree hashes nor commit hashes work; everything seems to redirect to the same tarball.

I guess we should have been verifying treehashes this whole time. ;)

[StefanKarpinski]

This used to work, so GitHub broke this and this is really screwing us over badly: the URL rewrite is baked into the client and cannot be fixed after the fact. Another check mark in "don't rely on some 3rd party service that can go away or break their own API" column. Thank goodness we're switching to our own protocol and servers and away from downloading from GitHub.

[KristofferC]

Is there any info about this API being deprecated?

[tkf]

It looks like it's still documented here? https://docs.github.com/en/rest/reference/repos#download-a-repository-archive

[StefanKarpinski]

I've tweeted about it: https://twitter.com/home. Does GitHub have a place to file issues these days?

[Nosferican]

So... Doing https://api.github.com/repos/JuliaLang/MbedTLS.jl/tarball/b71d85de30afb1b4638d1169fa1f6432094a665a
which is the sha1 of the commit for v0.6.8 works. Per the API docs, when the Git ref is not valid it defaults to giving the latest. In other words, it should be using the sha1 for the commit rather than the tree-sha apparently. Refs may be branch, SHA, tag. So our API call should probably be: https://api.github.com/repos/JuliaLang/MbedTLS.jl/tarball/v0.6.8

[tkf]

https://codeload.github.com/JuliaLang/MbedTLS.jl/legacy.tar.gz/2d94286a9c2f52c63a16146bb86fd6cdfbf677c6 seems to work. So it's only the redirection part that is not working...

[StefanKarpinski]

It's been worked around in a very hacky way.

[tkf]

Wow, that's a clever hack. Thanks a lot for the quick fix!

It's working now https://travis-ci.org/github/tkf/MyPlayground.jl/jobs/714938750#L191

[DilumAluthge]

So our API call should probably be: https://api.github.com/repos/JuliaLang/MbedTLS.jl/tarball/v0.6.8

We cannot rely on the tag existing in the repo.

[KristofferC]

Yes, and tags can be changed afterwards.

[Nosferican]

Yeah, it seems it really is an issue with the redirect when using Git Tree SHA1 as the ref... Well Pkg server is now serving v1.5 at least.

[DilumAluthge]

As many have pointed out above, using the Pkg server will prevent this issue. However, you will still get this issue if you opt-out of the Pkg server. So if e.g. I set export JULIA_PKG_SERVER="" in my .bashrc file, and then I open Julia 1.5 or Julia 1.6, then Pkg will fall back to downloading tarballs from the GitHub API, and I will still have this issue.

[DilumAluthge]

Even if GitHub fixes the issue now, there's nothing stopping them from changing the API again in the future.

The safest option may be to stop using the GitHub API completely, and always just clone the Git repository. We will need to make this change in Pkg master, but we will also need to backport it to Julia 1.0 and 1.5.

It would be nice to also backport it to Julia 1.1, 1.2, 1.3, and 1.4, but I'm not sure what our backport policy is for unmaintained releases of Julia.

[KristofferC]

Even if GitHub fixes the issue now, there's nothing stopping them from changing the API again in the future.

The safest option may be to stop using the GitHub API completely, and always just clone the Git repository.

No, Git cloning is an awful package manager experience. If we are talking about making changes to old Julia versions, we should backport the verification check.

[DilumAluthge]

Even if GitHub fixes the issue now, there's nothing stopping them from changing the API again in the future.
The safest option may be to stop using the GitHub API completely, and always just clone the Git repository.

No, Git cloning is an awful package manager experience. If we are talking about making changes to old Julia versions, we should backport the verification check.

That's a good idea.

[DilumAluthge]

Although, if I understand correctly, we don't do the verification check on Windows.

[DilumAluthge]

Is there anyway that we can also add the verification check on Windows?

[KristofferC]

Well, it fails in some cases (@staticfloat probably knows the exact details when) but then it falls back to the git clone version. So it could probably just be enabled.

[staticfloat]

However, you will still get this issue if you opt-out of the Pkg server.

Not with the recent work-around-hack that was merged into General.

Is there anyway that we can also add the verification check on Windows?

Not on older Julia versions; we don't get executable bits properly.

If we are talking about making changes to old Julia versions, we should backport the verification check.

I agree that we should do this where possible (e.g. on Linux and MacOS) but going forward, we will implement checks based on Tar.jl that make use of in-tarball-treehash verification, which can happen on any machine.

[KristofferC]

It seems to work again I think. Tried https://api.github.com/repos/JuliaLang/MbedTLS.jl/tarball/2d94286a9c2f52c63a16146bb86fd6cdfbf677c6 and it seems to give the correct files now.

[KristofferC]

Also tried adding MbedTLS on 1.0.5 with the old registry and I get the correct version now.

[tkf]

Looks like it's fixed now: https://discourse.julialang.org/t/pkg-downtime-incident/44288/19