Pkg.jl: Feature request: support version numbers with build metadata
Suppose I want to Pkg.add a specific version of a package. For example, suppose that I want to install version 0.5.3 of the Example package. I can do this as follows:
julia> import Pkg
julia> Pkg.add(Pkg.PackageSpec(name = "Example", version = "0.5.3"))
Cloning default registries into `~/.julia`
Cloning registry from "https://github.com/JuliaRegistries/General.git"
Added registry `General` to `~/.julia/registries/General`
Resolving package versions...
Installed Example ─ v0.5.3
Updating `~/.julia/environments/v1.4/Project.toml`
[7876af07] + Example v0.5.3
Updating `~/.julia/environments/v1.4/Manifest.toml`
[7876af07] + Example v0.5.3
I can also accomplish the same thing using the REPL mode:
(@v1.4) pkg> add [email protected]
Cloning default registries into `~/.julia`
Cloning registry from "https://github.com/JuliaRegistries/General.git"
Added registry `General` to `~/.julia/registries/General`
Resolving package versions...
Installed Example ─ v0.5.3
Updating `~/.julia/environments/v1.4/Project.toml`
[7876af07] + Example v0.5.3
Updating `~/.julia/environments/v1.4/Manifest.toml`
[7876af07] + Example v0.5.3
But this does not work with JLL packages. For example, suppose that I want to install version 2.23.0+1 of the Git_jll package. I tried this, but it gives me an error:
julia> import Pkg
julia> Pkg.add(Pkg.PackageSpec(name = "Git_jll", version = "2.23.0+1"))
ERROR: ArgumentError: invalid version range: "2.23.0+1"
Stacktrace:
[1] Pkg.Types.VersionRange(::String) at /Users/dilum/dev/forks-DilumAluthge/julia/usr/share/julia/stdlib/v1.4/Pkg/src/versions.jl:114
[2] VersionSpec at /Users/dilum/dev/forks-DilumAluthge/julia/usr/share/julia/stdlib/v1.4/Pkg/src/versions.jl:197 [inlined]
[3] #Package#129 at /Users/dilum/dev/forks-DilumAluthge/julia/usr/share/julia/stdlib/v1.4/Pkg/src/API.jl:996 [inlined]
[4] top-level scope at REPL[2]:1
I also tried the REPL mode, but it also gives an error:
(@v1.4) pkg> add [email protected]+1
ERROR: ArgumentError: invalid version range: "2.23.0+1"
Stacktrace:
[1] Pkg.Types.VersionRange(::String) at /Users/dilum/dev/forks-DilumAluthge/julia/usr/share/julia/stdlib/v1.4/Pkg/src/versions.jl:114
[2] Union{Pkg.Types.VersionRange, String, Pkg.REPLMode.Rev}(::String) at /Users/dilum/dev/forks-DilumAluthge/julia/usr/share/julia/stdlib/v1.4/Pkg/src/REPLMode/argument_parsers.jl:33
[3] iterate at ./generator.jl:47 [inlined]
[4] collect_to!(::Array{String,1}, ::Base.Generator{Array{String,1},Type{Union{Pkg.Types.VersionRange, String, Pkg.REPLMode.Rev}}}, ::Int64, ::Int64) at ./array.jl:710
[5] collect_to_with_first!(::Array{String,1}, ::String, ::Base.Generator{Array{String,1},Type{Union{Pkg.Types.VersionRange, String, Pkg.REPLMode.Rev}}}, ::Int64) at ./array.jl:689
[6] _collect(::Array{String,1}, ::Base.Generator{Array{String,1},Type{Union{Pkg.Types.VersionRange, String, Pkg.REPLMode.Rev}}}, ::Base.EltypeUnknown, ::Base.HasShape{1}) at ./array.jl:683
[7] collect_similar at ./array.jl:607 [inlined]
[8] map at ./abstractarray.jl:2072 [inlined]
[9] parse_package(::Array{Pkg.REPLMode.QString,1}, ::Dict{Symbol,Any}; add_or_dev::Bool) at /Users/dilum/dev/forks-DilumAluthge/julia/usr/share/julia/stdlib/v1.4/Pkg/src/REPLMode/argument_parsers.jl:10
[10] (::Pkg.REPLMode.var"#64#69")(::Array{Pkg.REPLMode.QString,1}, ::Dict{Symbol,Any}) at /Users/dilum/dev/forks-DilumAluthge/julia/usr/share/julia/stdlib/v1.4/Pkg/src/REPLMode/command_declarations.jl:89
[11] Pkg.REPLMode.Command(::Pkg.REPLMode.Statement) at /Users/dilum/dev/forks-DilumAluthge/julia/usr/share/julia/stdlib/v1.4/Pkg/src/REPLMode/REPLMode.jl:365
[12] iterate at ./generator.jl:47 [inlined]
[13] _collect(::Array{Pkg.REPLMode.Statement,1}, ::Base.Generator{Array{Pkg.REPLMode.Statement,1},Type{Pkg.REPLMode.Command}}, ::Base.EltypeUnknown, ::Base.HasShape{1}) at ./array.jl:678
[14] collect_similar at ./array.jl:607 [inlined]
[15] map at ./abstractarray.jl:2072 [inlined]
[16] do_cmd(::REPL.LineEditREPL, ::String; do_rethrow::Bool) at /Users/dilum/dev/forks-DilumAluthge/julia/usr/share/julia/stdlib/v1.4/Pkg/src/REPLMode/REPLMode.jl:378
[17] do_cmd at /Users/dilum/dev/forks-DilumAluthge/julia/usr/share/julia/stdlib/v1.4/Pkg/src/REPLMode/REPLMode.jl:376 [inlined]
[18] (::Pkg.REPLMode.var"#24#27"{REPL.LineEditREPL,REPL.LineEdit.Prompt})(::REPL.LineEdit.MIState, ::Base.GenericIOBuffer{Array{UInt8,1}}, ::Bool) at /Users/dilum/dev/forks-DilumAluthge/julia/usr/share/julia/stdlib/v1.4/Pkg/src/REPLMode/REPLMode.jl:542
[19] #invokelatest#1 at ./essentials.jl:712 [inlined]
[20] invokelatest at ./essentials.jl:711 [inlined]
[21] run_interface(::REPL.Terminals.TextTerminal, ::REPL.LineEdit.ModalInterface, ::REPL.LineEdit.MIState) at /Users/dilum/dev/forks-DilumAluthge/julia/usr/share/julia/stdlib/v1.4/REPL/src/LineEdit.jl:2354
[22] run_frontend(::REPL.LineEditREPL, ::REPL.REPLBackendRef) at /Users/dilum/dev/forks-DilumAluthge/julia/usr/share/julia/stdlib/v1.4/REPL/src/REPL.jl:1055
[23] run_repl(::REPL.AbstractREPL, ::Any) at /Users/dilum/dev/forks-DilumAluthge/julia/usr/share/julia/stdlib/v1.4/REPL/src/REPL.jl:206
[24] (::Base.var"#764#766"{Bool,Bool,Bool,Bool})(::Module) at ./client.jl:383
[25] #invokelatest#1 at ./essentials.jl:712 [inlined]
[26] invokelatest at ./essentials.jl:711 [inlined]
[27] run_main_repl(::Bool, ::Bool, ::Bool, ::Bool, ::Bool) at ./client.jl:367
[28] exec_options(::Base.JLOptions) at ./client.jl:305
[29] _start() at ./client.jl:484
So, as far as I can tell, there is currently no way to Pkg.add a specific version of a JLL package.
cc: @staticfloat @giordano
DilumAluthge
All 19 comments
I can also confirm that Project.toml is not happy if you specify the build number in the compat section
giordano
on 29 Dec 2019
I can also confirm that
Project.tomlis not happy if you specify the build number in the compat section
Correct. Suppose you have a Project.toml file with the following contents:
[deps]
Git_jll = "f8c6e375-362e-5223-8a59-34ff63f689eb"
[compat]
Git_jll = "2.23.0+1"
Here is the error you get if you start Julia with julia --project and then try to instantiate:
DMBP:foo dilum$ julia --project
_
_ _ _(_)_ | Documentation: https://docs.julialang.org
(_) | (_) (_) |
_ _ _| |_ __ _ | Type "?" for help, "]?" for Pkg help.
| | | | | | |/ _` | |
| | |_| | | | (_| | | Version 1.4.0-DEV.660 (2019-12-23)
_/ |\__'_|_|_|\__'_| | Commit 27eb582279 (5 days old master)
|__/ |
julia> import Pkg
julia> Pkg.instantiate()
ERROR: Could not parse compatibility version for dependency `Git_jll`
Stacktrace:
[1] pkgerror(::String) at /Users/dilum/dev/forks-DilumAluthge/julia/usr/share/julia/stdlib/v1.4/Pkg/src/Types.jl:54
[2] read_project_compat(::Dict{String,Any}, ::Pkg.Types.Project) at /Users/dilum/dev/forks-DilumAluthge/julia/usr/share/julia/stdlib/v1.4/Pkg/src/project.jl:66
[3] Pkg.Types.Project(::Dict{String,Any}) at /Users/dilum/dev/forks-DilumAluthge/julia/usr/share/julia/stdlib/v1.4/Pkg/src/project.jl:121
[4] read_project(::IOStream; path::String) at /Users/dilum/dev/forks-DilumAluthge/julia/usr/share/julia/stdlib/v1.4/Pkg/src/project.jl:140
[5] #32 at /Users/dilum/dev/forks-DilumAluthge/julia/usr/share/julia/stdlib/v1.4/Pkg/src/project.jl:144 [inlined]
[6] open(::Pkg.Types.var"#32#33"{String}, ::String; kwargs::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at ./io.jl:298
[7] open at ./io.jl:296 [inlined]
[8] read_project(::String) at /Users/dilum/dev/forks-DilumAluthge/julia/usr/share/julia/stdlib/v1.4/Pkg/src/project.jl:143
[9] Pkg.Types.EnvCache(::Nothing) at /Users/dilum/dev/forks-DilumAluthge/julia/usr/share/julia/stdlib/v1.4/Pkg/src/Types.jl:283
[10] EnvCache at /Users/dilum/dev/forks-DilumAluthge/julia/usr/share/julia/stdlib/v1.4/Pkg/src/Types.jl:279 [inlined]
[11] Pkg.Types.Context() at ./util.jl:736
[12] #instantiate#114 at /Users/dilum/dev/forks-DilumAluthge/julia/usr/share/julia/stdlib/v1.4/Pkg/src/API.jl:741 [inlined]
[13] instantiate() at /Users/dilum/dev/forks-DilumAluthge/julia/usr/share/julia/stdlib/v1.4/Pkg/src/API.jl:741
[14] top-level scope at REPL[2]:1
caused by [exception 1]
invalid version specifier: 2.23.0+1
Stacktrace:
[1] error(::String) at ./error.jl:33
[2] semver_spec(::String) at /Users/dilum/dev/forks-DilumAluthge/julia/usr/share/julia/stdlib/v1.4/Pkg/src/versions.jl:273
[3] read_project_compat(::Dict{String,Any}, ::Pkg.Types.Project) at /Users/dilum/dev/forks-DilumAluthge/julia/usr/share/julia/stdlib/v1.4/Pkg/src/project.jl:64
[4] Pkg.Types.Project(::Dict{String,Any}) at /Users/dilum/dev/forks-DilumAluthge/julia/usr/share/julia/stdlib/v1.4/Pkg/src/project.jl:121
[5] read_project(::IOStream; path::String) at /Users/dilum/dev/forks-DilumAluthge/julia/usr/share/julia/stdlib/v1.4/Pkg/src/project.jl:140
[6] #32 at /Users/dilum/dev/forks-DilumAluthge/julia/usr/share/julia/stdlib/v1.4/Pkg/src/project.jl:144 [inlined]
[7] open(::Pkg.Types.var"#32#33"{String}, ::String; kwargs::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at ./io.jl:298
[8] open at ./io.jl:296 [inlined]
[9] read_project(::String) at /Users/dilum/dev/forks-DilumAluthge/julia/usr/share/julia/stdlib/v1.4/Pkg/src/project.jl:143
[10] Pkg.Types.EnvCache(::Nothing) at /Users/dilum/dev/forks-DilumAluthge/julia/usr/share/julia/stdlib/v1.4/Pkg/src/Types.jl:283
[11] EnvCache at /Users/dilum/dev/forks-DilumAluthge/julia/usr/share/julia/stdlib/v1.4/Pkg/src/Types.jl:279 [inlined]
[12] Pkg.Types.Context() at ./util.jl:736
[13] #instantiate#114 at /Users/dilum/dev/forks-DilumAluthge/julia/usr/share/julia/stdlib/v1.4/Pkg/src/API.jl:741 [inlined]
[14] instantiate() at /Users/dilum/dev/forks-DilumAluthge/julia/usr/share/julia/stdlib/v1.4/Pkg/src/API.jl:741
[15] top-level scope at REPL[2]:1
Works if you pass a version number.
fredrikekre
on 16 Jan 2020
Works if you pass a version number.
Do you have a working example? I've tried passing a version number, but I get this error:
julia> import Pkg
julia> Pkg.add(Pkg.PackageSpec(name = "Git_jll", version = "2.23.0+1"))
ERROR: ArgumentError: invalid version range: "2.23.0+1"
Stacktrace:
[1] Pkg.Types.VersionRange(::String) at /Users/dilum/dev/forks-DilumAluthge/julia/usr/share/julia/stdlib/v1.5/Pkg/src/versions.jl:114
[2] VersionSpec at /Users/dilum/dev/forks-DilumAluthge/julia/usr/share/julia/stdlib/v1.5/Pkg/src/versions.jl:197 [inlined]
[3] #Package#129 at /Users/dilum/dev/forks-DilumAluthge/julia/usr/share/julia/stdlib/v1.5/Pkg/src/API.jl:996 [inlined]
[4] top-level scope at REPL[2]:1
I just figured out what you meant.
DilumAluthge
on 16 Jan 2020
You meant Pkg.add(Pkg.PackageSpec(name = "Git_jll", version = v"2.23.0+1")).
DilumAluthge
on 16 Jan 2020
I can confirm that it works:
julia> import Pkg
julia> Pkg.add(Pkg.PackageSpec(name = "Git_jll", version = v"2.23.0+1"))
Cloning default registries into `~/.julia`
Cloning registry from "https://github.com/JuliaRegistries/General.git"
Added registry `General` to `~/.julia/registries/General`
Resolving package versions...
Installed LibCURL_jll ── v7.66.0+1
Installed Git_jll ────── v2.23.0+1
Installed Gettext_jll ── v0.20.1+1
Installed XML2_jll ───── v2.9.9+1
Installed PCRE2_jll ──── v10.31.0+0
Installed Expat_jll ──── v2.2.7+0
Installed MbedTLS_jll ── v2.16.0+1
Installed Zlib_jll ───── v1.2.11+7
Installed OpenSSL_jll ── v1.1.1+0
Installed LibSSH2_jll ── v1.9.0+1
Installed Libiconv_jll ─ v1.16.0+0
Updating `~/.julia/environments/v1.5/Project.toml`
[f8c6e375] + Git_jll v2.23.0+1
Updating `~/.julia/environments/v1.5/Manifest.toml`
[2e619515] + Expat_jll v2.2.7+0
[78b55507] + Gettext_jll v0.20.1+1
[f8c6e375] + Git_jll v2.23.0+1
[deac9b47] + LibCURL_jll v7.66.0+1
[29816b5a] + LibSSH2_jll v1.9.0+1
[94ce4f54] + Libiconv_jll v1.16.0+0
[c8ffd9c3] + MbedTLS_jll v2.16.0+1
[458c3c95] + OpenSSL_jll v1.1.1+0
[efcefdf7] + PCRE2_jll v10.31.0+0
[02c8fc9c] + XML2_jll v2.9.9+1
[83775a58] + Zlib_jll v1.2.11+7
[2a0f44e3] + Base64
[ade2ca70] + Dates
[b77e0a4c] + InteractiveUtils
[76f85450] + LibGit2
[8f399da3] + Libdl
[56ddb016] + Logging
[d6f4376e] + Markdown
[44cfe95a] + Pkg
[de0858da] + Printf
[3fa0cd96] + REPL
[9a3f8284] + Random
[ea8e919c] + SHA
[9e88b42a] + Serialization
[6462fe0b] + Sockets
[cf7118a7] + UUIDs
[4ec0a83e] + Unicode
So, the original issue I reported is resolved. However, the related issue that @giordano mentioned in this comment (with additional details in this comment) is still open - cannot put the build number in the compat section of Project.toml.
Should we rename this issue to focus on the compat section? Or should we close this issue as resolved and open a new issue for the compat section?
DilumAluthge
on 16 Jan 2020
Even though passing as a version number works, it seems like the version selected is simply the latest version of the same patch release. Here I request 0.3.7+2 but 0.3.7+4 is added:
julia> Pkg.status()
Status `/tmp/Project.toml`
(empty environment)
julia> Pkg.add(PackageSpec(name="OpenBLAS_jll", version=v"0.3.7+2"))
Resolving package versions...
Updating `/tmp/Project.toml`
[4536629a] + OpenBLAS_jll v0.3.7+4
Updating `/tmp/Manifest.toml`
[4536629a] + OpenBLAS_jll v0.3.7+4
[2a0f44e3] + Base64
[ade2ca70] + Dates
[b77e0a4c] + InteractiveUtils
[76f85450] + LibGit2
[8f399da3] + Libdl
[56ddb016] + Logging
[d6f4376e] + Markdown
[44cfe95a] + Pkg
[de0858da] + Printf
[3fa0cd96] + REPL
[9a3f8284] + Random
[ea8e919c] + SHA
[9e88b42a] + Serialization
[6462fe0b] + Sockets
[cf7118a7] + UUIDs
[4ec0a83e] + Unicode
JackDunnNZ
on 16 Jan 2020
Even though passing as a version number works, it seems like the version selected is simply the latest version of the same patch release. Here I request
0.3.7+2but0.3.7+4is added:
That’s not good. Seems like a bug to me.
DilumAluthge
on 16 Jan 2020
That’s not good. Seems like a bug to me.
If it is a bug, it is in the bot that makes PR for jll packages to General and produces these version numbers. Pkg doesn't support this type of version numbers. This was known to the people doing _jll stuff but the advantage of being able to do multiple builds and have the major, minor, patch version correspond to the library version was likely considered more important than the drawback of being able to select different build versions to install (you should always want the latest?).
Pkg could be made to accept these version numbers but that is a feature request.
KristofferC
on 16 Jan 2020
the advantage of being able to do multiple builds and have the major, minor, patch version correspond to the library version was likely considered more important than the drawback of being able to select different build versions to install (you should always want the latest?).
That's exactly what we said, yes. We discussed the relative advantages of decoupling JLL versioning from the underlying source library, and we decided that being able to specify exactly which _source library_ to choose from was more important than which build; although it would of course be better to be able to choose both, it's much more important for us to be able to have LibFoo v3.3.3 and LibFoo v3.3.4 distinguishable, since the (vast) majority of our build-increment-only JLL package versions are build system improvements, turning on more features, adding more platforms, etc... Very rarely is there a behavioral difference in a build-number-only bump.
staticfloat
on 16 Jan 2020
Works if you pass a version number.
What am I missing?
julia> using Pkg
julia> Pkg.add(PackageSpec(; name = "SuiteSparse_jll", version = v"5.4.0+2"))
Updating registry at `~/.julia/registries/General`
Updating git-repo `https://github.com/JuliaRegistries/General.git`
Resolving package versions...
Downloading artifact: OpenBLAS
Updating `~/.julia/environments/v1.4/Project.toml`
[bea87d4a] + SuiteSparse_jll v5.4.0+4
Updating `~/.julia/environments/v1.4/Manifest.toml`
[d00139f3] + METIS_jll v5.1.0+1
[4536629a] + OpenBLAS_jll v0.3.7+5
[bea87d4a] + SuiteSparse_jll v5.4.0+4
I'm not sure how that would work, I don't see the build number in the PackageSpec:
julia> dump(PackageSpec(; name = "SuiteSparse_jll", version = v"5.4.0+2"))
Pkg.Types.PackageSpec
name: String "SuiteSparse_jll"
uuid: Nothing nothing
version: Pkg.Types.VersionSpec
ranges: Array{Pkg.Types.VersionRange}((1,))
1: Pkg.Types.VersionRange
lower: Pkg.Types.VersionBound
t: Tuple{UInt32,UInt32,UInt32}
1: UInt32 0x00000005
2: UInt32 0x00000004
3: UInt32 0x00000000
n: Int64 3
upper: Pkg.Types.VersionBound
t: Tuple{UInt32,UInt32,UInt32}
1: UInt32 0x00000005
2: UInt32 0x00000004
3: UInt32 0x00000000
n: Int64 3
tree_hash: Nothing nothing
repo: Pkg.Types.GitRepo
source: Nothing nothing
rev: Nothing nothing
path: Nothing nothing
pinned: Bool false
mode: Pkg.Types.PackageMode PKGMODE_PROJECT
@giordano Although it doesn't error, I think the build number is still not respected
00vareladavid
on 9 Feb 2020
@giordano Although it doesn't error, I think the build number is still not respected
Yeah, exactly. It will always install the latest build number. But at least it doesn’t error 🤷♂️
DilumAluthge
on 9 Feb 2020
You can probably add by commit sha as a workaround.
KristofferC
on 10 Feb 2020
Yep,
Pkg.add(PackageSpec(; name = "SuiteSparse_jll", rev = "202ebf9db961644ad225cb53549dedc8f1df9453"))
does the trick
giordano
on 10 Feb 2020
.... while it does the trick it feels like a workaround. Is there a plan to extend the dependency management to the binarybuildert tools (inlcuding the build numbers)?
j-fu
on 1 Apr 2020
It is indeed a workaround. This would only get solved by someone going through the codebase and add support for this, adding proper testing and performance checks.
KristofferC
on 1 Apr 2020
Related issues
cossio
·
3Comments
cossio
·
3Comments
DilumAluthge
·
3Comments
innerlee
·
4Comments
moustachio-belvedere
·
3Comments
Most helpful comment
That's exactly what we said, yes. We discussed the relative advantages of decoupling JLL versioning from the underlying source library, and we decided that being able to specify exactly which _source library_ to choose from was more important than which build; although it would of course be better to be able to choose both, it's much more important for us to be able to have
LibFoo v3.3.3andLibFoo v3.3.4distinguishable, since the (vast) majority of our build-increment-only JLL package versions are build system improvements, turning on more features, adding more platforms, etc... Very rarely is there a behavioral difference in a build-number-only bump.