Pkg.jl: Packages downgraded after `Pkg.update()`?

[ptoche]

Unless I'm mistaken, packages were downgraded without a prior warning (and without an explanation):

I just called

import Pkg
Pkg.update()

The tail-end of the console shows:

    Updating `~/Julia/workspace/Project.toml`
  [b4f34e82] ↑ Distances v0.10.2 ⇒ v0.10.3
  [91a5bcdd] ~ Plots v1.13.2 `https://github.com/JuliaPlots/Plots.jl.git#master` ⇒ v1.13.2 `https://github.com/JuliaPlots/Plots.jl.git#master`
  [92933f4c] ↑ ProgressMeter v1.5.0 ⇒ v1.6.0
    Updating `~/Julia/workspace/Manifest.toml`
  [4fba245c] ↑ ArrayInterface v3.1.9 ⇒ v3.1.10
  [324d7699] ↑ CategoricalArrays v0.9.6 ⇒ v0.9.7
  [d360d2e6] ↑ ChainRulesCore v0.9.40 ⇒ v0.9.41
  [b630d9fa] - CheapThreads v0.2.3
  [34da2185] ↑ Compat v3.27.0 ⇒ v3.28.0
  [bcd4f6db] ↑ DelayDiffEq v5.29.3 ⇒ v5.30.0
  [2b5f629d] ↑ DiffEqBase v6.60.0 ⇒ v6.60.1
  [b4f34e82] ↑ Distances v0.10.2 ⇒ v0.10.3
  [7a1cc6ca] ↑ FFTW v1.4.0 ⇒ v1.4.1
  [5789e2e9] ↑ FileIO v1.8.1 ⇒ v1.8.2
  [82e4d734] ↑ ImageIO v0.5.3 ⇒ v0.5.4
  [5ab0869b] ↑ KernelDensity v0.6.2 ⇒ v0.6.3
  [bdcacae8] ↓ LoopVectorization v0.12.12 ⇒ v0.11.2
  [7269a6da] ↑ MeshIO v0.4.6 ⇒ v0.4.7
  [1dea7af3] ↑ OrdinaryDiffEq v5.52.7 ⇒ v5.53.0
  [a03496cd] ↑ PlotlyBase v0.5.2 ⇒ v0.5.3
  [91a5bcdd] ~ Plots v1.13.2 `https://github.com/JuliaPlots/Plots.jl.git#master` ⇒ v1.13.2 `https://github.com/JuliaPlots/Plots.jl.git#master`
  [92933f4c] ↑ ProgressMeter v1.5.0 ⇒ v1.6.0
  [476501e8] ↓ SLEEFPirates v0.6.15 ⇒ v0.6.10
  [2913bbd2] ↑ StatsBase v0.33.6 ⇒ v0.33.8
  [789caeaf] ↑ StochasticDiffEq v6.33.1 ⇒ v6.33.2
  [7792a7ef] - StrideArraysCore v0.1.5
  [c3572dad] ↑ Sundials v4.4.1 ⇒ v4.4.3
  [0c5d862f] ↑ Symbolics v0.1.24 ⇒ v0.1.25
  [8290d209] ↓ ThreadingUtilities v0.4.1 ⇒ v0.2.5
  [f269a46b] ↑ TimeZones v1.5.3 ⇒ v1.5.4
  [3d5dd08c] ↓ VectorizationBase v0.19.32 ⇒ v0.18.14

There is no warning and unless I missed it there wasn't any information about it in the docs.

Just a few days ago I noticed I was using a version of Plots from 3 or 4 years ago because some obscure package I had installed and forgotten about was holding it back. Now that I've carefully cleaned things up, I'm surprised to see several downgrades. Particularly as I do not know the reasons. For instance, ThreadingUtilities v0.4.1 ⇒ v0.2.5 suggests an important downgrade. Maybe it's not a big deal, the console message doesn't state anything about it. Should I worry at all?

[KristofferC]

Unless I'm mistaken, packages were downgraded without a prior warning (and without an explanation):

Pkg tries to give you the latest version of packages within the constraint that all those versions have to be compatible with each other. Sometimes upgrading one package makes it incompatible with other packages.

Should I worry at all?

It depends, do you need ThreadingUtilities to be on v0.4? In that case, you should add

[compact]
ThreadingUtilities = 0.4

to your Project file.

[ptoche]

Thanks Kristoffer. Is there a way to run Pkg.update() while forcing no-downgrades? I'm not too bothered if there are minor downgrades, but I was surprised to find Plots being stuck at a version of several years back because it was being held back by some package I'd forgotten about, despite regularly running Pkg.update(). Packages like ThreadingUtilities are installed by other packages and it's not easy to know if I should force a more recent version or not. Is there a better way of managing my packages? Thanks.

[StefanKarpinski]

Is there a way to run Pkg.update() while forcing no-downgrades?

Since the resolver prefers not to downgrade anything unless it's forced to, if there was a "no downgrades" option, you would instead get an error that there is no way to satisfy your version requirements, which is not any better. The only things that can really be improved here is making it clearer what the core conflict is that's causing a downgrade and potentially allowing you to choose which downgrades you'd prefer until those conflicts are resolved either by new releases of by fixing compat bounds. Currently the way you can make that choice is by putting your own version requirements into your project file.

[ptoche]

Thanks for your explanations Stefan.

making it clearer what the core conflict is that's causing a downgrade and potentially allowing you to choose which downgrades

Yes, that would be very useful indeed. Perhaps it would be too verbose for other users' taste and could be made available as an option. It's true that now that I'm aware of this issue, I won't be running Pkg.update() so often and ifever I do in the future, I'll be carefully reading the information printed to screen.

you can make that choice is by putting your own version requirements into your project file.

Thanks for the suggestion. Would that particular version requirement override all other requirements stated elsewhere? Let me give an example. Suppose I want to have as recent a version of the Plots package as is reasonable, but I also use another package for some small calculation somewhere in a part of my package that is not called all that often. The package in question has the following line (literally copied):

[[Plots]]
deps = ["Base64", "Contour", "Dates", "FFMPEG", "FixedPointNumbers", "GR", "GeometryTypes", "JSON", "LinearAlgebra", "Measures", "NaNMath", "Pkg", "PlotThemes", "PlotUtils", "Printf", "REPL", "Random", "RecipesBase", "Reexport", "Requires",     "Showoff", "SparseArrays", "Statistics", "StatsBase", "UUIDs"]
git-tree-sha1 = "0d7cf139b508defb467ac4fd655f4bd4d92f30cf"
uuid = "91a5bcdd-55d7-5caf-9e0b-520d859cae80"
version = "0.27.0"

Say I am not using the plotting capabilities of that package. Would putting the version number of Plots in my project file override the version number requested by that other package?

Of course there are workarounds when you know what you want. I could contact the package maintainer (but perhaps that package really needs version "0.27.0" for certain things). Or I could clone the package and change the relevant lines myself. Or I could go to the source code and copy-paste the relevant parts of that package that I need.

[StefanKarpinski]

There is no overriding of requirements and specific versions aren't requested: there are compatibility bounds, often lower but sometimes upper; all requirements must be satisfied, including yours. When there's some conflict, there tends to be more than one Pareto-optimal solution to the version selection problem. The resolver chooses one of these solutions, which uses a less recent version of some dependency, but it could have used a less recent version of some other dependency. If there was no conflict, it could just use the most recent version of all dependencies, but there is and it can't. You can guide it to choose a different solution by requiring that the dependency that it chose the older version of be installed at a higher version (if that's possible). The core issue here is that these version resolution problems are very complex (literally NP-hard) and it's an open problem how to figure out and clearly present what the "core conflict" is. If we knew how to do that, we would. Even computing what all the possible optimal solutions are is hard and not something we can currently do.

[ptoche]

I see! In this case, an option to approve/reject downgrades one by one as they present themselves to the console sounds like maybe the only reasonable solution. Pushing responsibility to the user.

[ptoche]

Long story short, this happened when I was adding the Flux package:

StaticArrays v1.1.2 ⇒ v0.12.5

I checked the Manifest.toml, but the version listed there was 1.1.1. The dependencies were also good. I then thought I could upgrade StaticArrays by listing a version in my Manifest.toml. So I changed the last line to: version = "1.1.2"

[[StaticArrays]]
deps = ["LinearAlgebra", "Random", "Statistics"]
git-tree-sha1 = "da4cf579416c81994afd6322365d00916c79b8ae"
uuid = "90137ffa-7385-5640-81b9-e52037218182"
version = "0.12.5"

But Julia then crashed. However, I was then able to find incompatibilities. The problem was with ImplicitPlots:

ERROR: Unsatisfiable requirements detected for package StaticPolynomials [62e018b1]:
 StaticPolynomials [62e018b1] log:
 ├─possible versions are: 0.2.2-1.3.3 or uninstalled
 ├─restricted by compatibility requirements with StaticArrays [90137ffa] to versions: uninstalled
 │ └─StaticArrays [90137ffa] log:
 │   ├─possible versions are: 0.8.0-1.2.0 or uninstalled
 │   └─restricted to versions 1.1.2 by an explicit requirement, leaving only versions 1.1.2
 └─restricted by compatibility requirements with ImplicitPlots [55ecb840] to versions: 1.3.3 — no versions left
   └─ImplicitPlots [55ecb840] log:
     ├─possible versions are: 0.1.0-0.2.1 or uninstalled
     ├─restricted to versions * by an explicit requirement, leaving only versions 0.1.0-0.2.1
     └─restricted by compatibility requirements with StaticArrays [90137ffa] to versions: 0.2.1 or uninstalled, leaving only versions: 0.2.1
       └─StaticArrays [90137ffa] log: see above

Removing ImplicitPlots did not fix the problem. StaticArrays did not update and could not be removed either.

I decided to destroy the environment an re-install all packages except ImplicitPlots. I keep a list of the packages I use regularly, about 30 of them, and re-installing was quite quick:

62 dependencies successfully precompiled in 124 seconds (227 already precompiled)

Now the latest version of StaticArrays is listed in the manifest file.

The strange thing was that the downgrade of StaticArrays was caused by adding Flux, while the problem really was with ImplicitPlots. I had contacted the package maintainer a few days ago and he fixed the compat list, so I'm not sure why updating ImplicitPlots did not fix the issue.

[ptoche]

I put the following in my Project.toml file and hopefully that will prevent StaticArrays from being downgraded in the future (as Kristoffer suggested above).

[compat]
StaticArrays = "1.2.0"

I'd like to test to see if installing ImplicitPlots would downgrade StaticArrays again. Is there something equivalent to Rs update.packages(ask=TRUE)? If you don't like what the package manager proposes to do (upgrades/downgrades), you can decline interactively.

More generally, how would you identify some package versions that lag very far behind? How to, say:

1. extract the version number for each package listed in `Manifest.toml`
2. check the package repository for the latest version and make a list of them

I can do 1 with

deps = [pair.second for pair in Pkg.dependencies()]
direct_deps = filter(p -> p.is_direct_dep, deps)
[(x.name, x.version) for x in direct_deps]
pkg_name_version = [(x.name, x.version) for x in direct_deps]

32-element Vector{Tuple{String, Any}}:
  ("IJulia", v"1.23.2")
  ("DataFrames", v"1.1.1")
  ## etc.