Pkg.jl: Optionally warn if `update` cannot install latest version of a package

[fingolfin]

The following has bitten me and colleagues many times: we want to install the latest version of a package (e.g. CxxWrap), so I do ] update CxxWrap, and I see this:

(v1.3) pkg> update CxxWrap
  Updating registry at `~/.julia/registries/General`
  Updating git-repo `https://github.com/JuliaRegistries/General.git`
 Resolving package versions...
  Updating `~/.julia/environments/v1.3/Project.toml`
 [no changes]
  Updating `~/.julia/environments/v1.3/Manifest.toml`
 [no changes]

Odd, I wonder, nothing happened... Did I already have the latest version? Hm, let's check:

(v1.3) pkg> status CxxWrap
    Status `~/.julia/environments/v1.3/Project.toml`
  [b99e7846] BinaryProvider v0.5.8
  [1f15a43c] CxxWrap v0.8.2

Oh no, that's not the latest. Which leads me to my first question:

1. Is there any way to find out what the latest version of a package from the Julia REPL? Or get a list of all available package versions?

What's going on? Well, after some searching around, I may discover that some other package has a dependency on CxxWrap with an upper bound that prevents any newer version of CxxWrap installed. The only way I found to do that was to manually go through each installed/active package, and look at its dependencies. So my second question:

1. Is there any way to get a list of active packages that depend on a given package?

And finally, the feature request from the headline of this issue:

1. Would it be possible/acceptable/... to change the behavior of update to print a warning if a package is not updated to the latest due to a constraint?

I can simulate this by removing the package and then adding it back with a version constraint:

(v1.3) pkg> rm CxxWrap
  Updating `~/.julia/environments/v1.3/Project.toml`
  [1f15a43c] - CxxWrap v0.8.2
  Updating `~/.julia/environments/v1.3/Manifest.toml`
 [no changes]

(v1.3) pkg> add [email protected]
 Resolving package versions...
ERROR: Unsatisfiable requirements detected for package Singular [bcd08a7b]:
 Singular [bcd08a7b] log:
 ├─possible versions are: 0.1.0 or uninstalled
 ├─restricted to versions 0.1.0 by an explicit requirement, leaving only versions 0.1.0
 └─restricted by compatibility requirements with CxxWrap [1f15a43c] to versions: uninstalled — no versions left
   └─CxxWrap [1f15a43c] log:
     ├─possible versions are: [0.7.0-0.7.4, 0.8.0-0.8.2, 0.9.0] or uninstalled
     └─restricted to versions 0.9.0 by an explicit requirement, leaving only versions 0.9.0

But that requires me to know that a newer version is available (see question 1); I guess if there was a "pseudo version" to indicate the latest, that'd also work; something like add CxxWrape@latest ?

Also, it seems I have to remove the package first? At least I couldn't figure out a way to make a version specifier work with update; doing ]update [email protected] does the same as update CxxWrap.

CC @fieker

[KristofferC]

Yeah, providing some short info about this in the default Pkg.status as well as giving a hint to another command, say, Pkg.compat_status() that shows more extensively what is holding what back, makes sense to me.

[getzze]

Same problem here, It is very hard to know the last version of the packages and the dependency tree. So it is hard to know (1) that there is a problem, (2) how to solve it. I would like to tell what I did so it can help to identify the problems.

I was getting this error:

┌ Info: Precompiling DifferentialEquations [0c46a032-eb83-5123-abaf-570d42b7fbaa]
└ @ Base loading.jl:1273
WARNING: Method definition vec(Number) in module DiffEqDiffTools at ~/.julia/packages/DiffEqDiffTools/3mm8U/src/jacobians.jl:114 overwritten in module FiniteDiff at ~/.julia/packages/FiniteDiff/IPtYL/src/jacobians.jl:128.
  ** incremental compilation may be fatally broken for this module **

DiffEqDiffTools.jl has just been renamed to FiniteDiff.jl, so I ran ]up but everything seemed up to date...

So I identified the packages that still had a dependency to DiffEqDiffTools in the Manifest:

grep -B1 -A4 -i -n "DiffEqDiffTools" ~/.julia/environments/v1.3/Manifest.toml

That gave me DiffEqBase, OrdinaryDiffEq and StochasticDiffEq. I checked on their github page what was their last versions and they were not those I had in my Manifest file. So the next step was to run ]preview add [pkg]#master to force them to the last version and see which package was preventing them from updating. Here is the full sequence I ran on the REPL:

https://gist.github.com/getzze/4c76f701adb961d2ca99cc68297f84f7

I think it would be good to display if a package is not in its last version and also to have a function to list the dependency of a package, or even a dependency tree.

[KristofferC]

If your code requires a specific version you should generally put that in your Project file under [compat]. So if you require DifferentialEquations 0.8 you put

[compat]
DifferentialEquations = "0.8"

If that version cannot be installed, an error message with "the dependency of a package, or even a dependency tree." will be printed. If you have no compat entry you are saying that you are fine with whatever version the resolver can get you.

[getzze]

I am using the packages in rolling release :) Is there a way to specify that you always want the last version then?

[fieker]

On Fri, Feb 07, 2020 at 05:50:59AM -0800, Kristoffer Carlsson wrote:

If your code requires a specific version you should generally put that in your Project file under [compat]. So if you require DifferentialEquations 0.8 you put

[compat]
DifferentialEquations = "0.8"

If, that cannot be installed, an error message with "the dependency of
a package, or even a dependency tree." will be printed. If you have no
compat entry you are saying that you are fine with whatever version
the resolver can get you.

Yes, this is fine. However I cannot figure out WHY the resolver is doing
(or not doing) what it does, decisions are based on information that I
usually do not have, namely dependencies of packages that I did not
install, they are dependencies of s.th. else.
It's the information about availability of (newer) versions together

with the reason(s) why they are rejected that is missing.

You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
https://github.com/JuliaLang/Pkg.jl/issues/1655#issuecomment-583397157

[BioTurboNick]

Just going to +1 this idea. Spent a while wondering why the dependency wasn't updating before I remembered to check my package's compat listing.

In case the issue is slightly different: My [compat] restricted a package to 0.5, but ]update did not warn me that v0.6 was available but being ignored due to [compat].

[azev77]

+1 from me as well, as mentioned in closed issue (1774) & Discourse.
This is one of my biggest usability problems.

[CarloLucibello]

Is there any workaround? i.e. any way to know which of the installed packages is not on the latest version and why?

[azev77]

@CarloLucibello
in the Julia package manager type status & it will show which version of a pkg you currently have installed.
Then go to the packages github page (eg MLJ.jl) and see if the version you have installed is the latest version.

[drcxcruz]

Julia definitely needs to automatically provide warning when a package cannot be updated with the ]update command. I would guess 100s if not 1000s of people do not have the latest version of their packages without even know it.

[odow]

This is _the_ most common issue we face with JuMP users. It comes up multiple times a week.

[KristofferC]

If users need a specific version of a package the way to do that is to put it in the compat section. So if JuMP version 2.3 or higher is needed, you put

[compat]
JuMP = "2.3"

in the project file.

Encouraging users to use smaller project files instead of always using the global one also makes it less likely for an unrelated package to hold back some version.

[KristofferC]

I would guess 100s if not 1000s of people do not have the latest version of their packages without even know it.

The reason they run with these versions is then because either:

• another package would have to downgrade
• another package in the project is not compatible with the newer version.

[KristofferC]

Having a compat command or something along those lines that show what packages are not at the latest version and what packages hold them back would however indeed be a useful feature.

[joaquimg]

If users need a specific version of a package the way to do that is to put it in the compat section. So if JuMP version 2.3 or higher is needed, you put

[compact]
JuMP = "2.3"

in the project file.

Yes, sure thing. We do that.

Encouraging users to use smaller project files instead of always using the global one also makes it less likely for an unrelated package to hold back some version.

Agree 1000% ! Personally, I do that a lot, I'd say @odow also. We do recommend that to users.

A few comments though:

Many of these users we are talking about are very beginners, many might be coming to JuMP without any prior Julia knowledge because they want to move away from the commercial packages.
Therefore, it might be a little too much to expect they grasp the envs idea quickly. (We recommended a few times, so we might add a session on the JuMP manual).

Second, JuMP ecosystem is reasonably large, and many people can create/wrap solvers and make them useful through JuMP.
However, as JuMP evolves, some of the solvers (mainly the not "officially" maintained ones) become incompatible. To add more to the mix, many of these beginner users want to compare different solvers for their problems, thus, one solver package frequently locks another one (or even JuMP itself) to an earlier version.
Hence, depending on the order things are added they lead to diverse issues silently. Because Pkg does not alert that something was not added in the latest version.

The compat would be very helpful to shorten topics because people helping the ones in need would have a grasp of what is happening easier and quicker.

A possible new feature in Pkg, highlighting that added/update or even existing packages are not in the latest release might reduce the number of this kind of inquiry. Some users might not even go to discourse and give up on the ecosystem :(

I understand that this might not be easy to implement and might not be a current priority for Pkg, however, I hope it can be a point of interest soon :)

Also, we should add some details to JuMP docs to cover all fronts.

[BioTurboNick]

+1 to this issue again.

Was trying to use Pluto.jl. Initial add installed 0.12.4. update brought it up to 0.12.10, but the newest version is 0.12.11. Spent an hour trying to figure out why, couldn't find what the cause was.

On a hunch, removed Atom.jl, and suddenly it worked. Turns out the issue was that Atom.jl had a compat entry for FuzzyCompletions.jl 0.3. Pluto.jl added a compat entry for FuzzyCompletions.jl 0.2 in 0.12.11.

Would have been really helpful to know the two conflicting packages and the affected dependency.

EDIT to add: Yes, this was in the global environment. I was trying to follow along with the MIT Computational Thinking course for Julia. It might be a relatively rare problem, but having no hint of how to solve it is a big blocking issue for users.

[stillyslalom]

As a potential compromise between verbosity and ease of use, how about only printing compatibility constraints if package arguments are provided? It seems like the most frequent use case for this functionality would be querying why a particular package is failing to upgrade, and that's addressed by something like

(@v1.6) pkg> st Pluto
Status `C:\Users\alexa\.julia\environments\v1.6\Project.toml`
  [c3e4b0f8] Pluto v0.12.10
Warning: latest version is v0.12.11. Constrained by explicit requirement
  [c52e3926] Atom v0.12.28
due to shared transitive dependency
  [fb4132e2] FuzzyCompletions v0.4.0

[BioTurboNick]

@stillyslalom - That's a nice compromise for detail, but I'd advocate for some hint of the constraint in the typical workflow (color? asterisk? append with [constrained]?). Maybe a hint at the end of the output like "use status <pkgname> for information" when this occurs?

[azev77]

I'll repeat my note from closed issue #2526
Would it be possible for pkg.status() to reveal the current & latest version of the package:
(w different colors if current version ≠ latest version)

(@v1.6) pkg> status
      Status `C:\Users\azevelev\.julia\environments\v1.6\Project.toml`
              Package    Current   Latest
  [f6006082] `EvoTrees   v0.5.3    v0.7.0`
  [9bbee03b]  NaiveBayes v0.5.0    v0.5.0
  [add582a8] `MLJ        v0.15.2   v0.16.0`
  [d491faf4] `MLJModels  v0.13.3   v0.14.1` 

I strongly believe this is one of the biggest issues faced by new Julia users.

While we're here, it would be nice if update() told the user:
which version was updated, which version was latest
If updated version != latest, which packages are causing conflict...

[ptoche]

It would be very useful to have. So here's my experience, for illustration. On a clean install (having removed the .julia directory entirely), after installing the 25 or so packages I often use, I discovered that:

julia> Pkg.status("Plots")
      Status `~/Julia/workspace/Project.toml`
  [91a5bcdd] Plots v0.29.9

To put things in perspective, the current version of Plots is v.1.13.2. So naturally I tried to "force-add" the latest version. Following instructions from the Plots manual, I ran

julia> Pkg.pkg"add Plots#master"

Trying to decipher the output, suggests that a package EnKF is holding things back. But that's not easy to understand at first. And the approach is indirect, the sort of information below would be useful to have when simply running Pkg.add("Plots") and the incompatibility is detected by Pkg and friends. Google sent me here. I decided to leave a little footprint.

julia> Pkg.pkg"add Plots#master"
     Cloning git-repo `https://github.com/JuliaPlots/Plots.jl.git`
    Updating git-repo `https://github.com/JuliaPlots/Plots.jl.git`
    Updating registry at `~/.julia/registries/General`
   Resolving package versions...
ERROR: Unsatisfiable requirements detected for package PotentialFlow [73af2aaf]:
 PotentialFlow [73af2aaf] log:
 ├─possible versions are: 0.1.0-0.2.2 or uninstalled
 ├─restricted by compatibility requirements with RecipesBase [3cdcf5f2] to versions: 0.2.0-0.2.2 or uninstalled
 │ └─RecipesBase [3cdcf5f2] log:
 │   ├─possible versions are: 0.4.0-1.1.1 or uninstalled
 │   └─restricted to versions 1 by Plots [91a5bcdd], leaving only versions 1.0.0-1.1.1
 │     └─Plots [91a5bcdd] log:
 │       ├─possible versions are: 1.13.2 or uninstalled
 │       └─Plots [91a5bcdd] is fixed to version 1.13.2
 └─restricted by compatibility requirements with EnKF [685896a8] to versions: 0.1.0-0.1.7 — no versions left
   └─EnKF [685896a8] log:
     ├─possible versions are: 0.1.0 or uninstalled
     └─restricted to versions * by an explicit requirement, leaving only versions 0.1.0
Stacktrace:
  [1] propagate_constraints!(graph::Pkg.Resolve.Graph, sources::Set{Int64}; log_events::Bool)
    @ Pkg.Resolve /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Pkg/src/Resolve/graphtype.jl:1048
  [2] propagate_constraints! (repeats 2 times)
    @ /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Pkg/src/Resolve/graphtype.jl:989 [inlined]
  [3] simplify_graph!(graph::Pkg.Resolve.Graph, sources::Set{Int64}; clean_graph::Bool)
    @ Pkg.Resolve /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Pkg/src/Resolve/graphtype.jl:1503
  [4] simplify_graph! (repeats 2 times)
    @ /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Pkg/src/Resolve/graphtype.jl:1503 [inlined]
  [5] resolve_versions!(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec})
    @ Pkg.Operations /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Pkg/src/Operations.jl:405
  [6] targeted_resolve(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}, preserve::Pkg.Types.PreserveLevel)
    @ Pkg.Operations /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Pkg/src/Operations.jl:1210
  [7] tiered_resolve(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec})
    @ Pkg.Operations /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Pkg/src/Operations.jl:1196
  [8] _resolve
    @ /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Pkg/src/Operations.jl:1216 [inlined]
  [9] add(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}, new_git::Vector{Base.UUID}; preserve::Pkg.Types.PreserveLevel, platform::Base.BinaryPlatforms.Platform)
    @ Pkg.Operations /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Pkg/src/Operations.jl:1231
 [10] add(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}; preserve::Pkg.Types.PreserveLevel, platform::Base.BinaryPlatforms.Platform, kwargs::Base.Iterators.Pairs{Symbol, Base.TTY, Tuple{Symbol}, NamedTuple{(:io,), Tuple{Base.TTY}}})
    @ Pkg.API /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:203
 [11] add(pkgs::Vector{Pkg.Types.PackageSpec}; io::Base.TTY, kwargs::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ Pkg.API /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:79
 [12] add(pkgs::Vector{Pkg.Types.PackageSpec})
    @ Pkg.API /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:77
 [13] do_cmd!(command::Pkg.REPLMode.Command, repl::Pkg.REPLMode.MiniREPL)
    @ Pkg.REPLMode /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Pkg/src/REPLMode/REPLMode.jl:408
 [14] do_cmd(repl::Pkg.REPLMode.MiniREPL, input::String; do_rethrow::Bool)
    @ Pkg.REPLMode /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Pkg/src/REPLMode/REPLMode.jl:386
 [15] top-level scope
    @ REPL[7]:1

[jkrumbiegel]

Maybe a simple way to make the update-to-latest path easier is to introduce a --latest flag which would have the same effect as specifying the last version number manually. Only that one doesn't have to look that up, which makes the command easier to use.

Something like

]add Distributions --latest
]up Revise --latest