Pkg.jl: AssertionError in is_package_downloaded

[timholy]

(AbstractPlotting) pkg> up
    Updating registry at `~/.julia/registries/General`
    Updating git-repo `[email protected]:JuliaRegistries/General.git`
    Updating registry at `~/.julia/registries/HolyLabRegistry`
    Updating git-repo `[email protected]:HolyLab/HolyLabRegistry.git`
No Changes to `~/.julia/dev/AbstractPlotting/Project.toml`
No Changes to `~/.julia/dev/AbstractPlotting/Manifest.toml`
ERROR: AssertionError: sourcepath !== nothing
Stacktrace:
  [1] is_package_downloaded(ctx::Pkg.Types.Context, pkg::Pkg.Types.PackageSpec)
    @ Pkg.Operations ~/src/julia-master/usr/share/julia/stdlib/v1.6/Pkg/src/Operations.jl:1757
  [2] #12
    @ ~/src/julia-master/usr/share/julia/stdlib/v1.6/Pkg/src/Operations.jl:118 [inlined]
  [3] _all
    @ ./reduce.jl:923 [inlined]
  [4] #all#696
    @ ./reducedim.jl:886 [inlined]
  [5] all
    @ ./reducedim.jl:886 [inlined]
  [6] is_instantiated(ctx::Pkg.Types.Context)
    @ Pkg.Operations ~/src/julia-master/usr/share/julia/stdlib/v1.6/Pkg/src/Operations.jl:118
  [7] instantiate(ctx::Pkg.Types.Context; manifest::Nothing, update_registry::Bool, verbose::Bool, platform::Base.BinaryPlatforms.Platform, allow_autoprecomp::Bool, kwargs::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ Pkg.API ~/src/julia-master/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:1301
  [8] precompile(ctx::Pkg.Types.Context; internal_call::Bool, kwargs::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ Pkg.API ~/src/julia-master/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:917
  [9] _auto_precompile(ctx::Pkg.Types.Context)
    @ Pkg.API ~/src/julia-master/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:902
 [10] #up#38
    @ ~/src/julia-master/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:74 [inlined]
 [11] up(pkgs::Vector{Pkg.Types.PackageSpec})
    @ Pkg.API ~/src/julia-master/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:72
 [12] do_cmd!(command::Pkg.REPLMode.Command, repl::REPL.LineEditREPL)
    @ Pkg.REPLMode ~/src/julia-master/usr/share/julia/stdlib/v1.6/Pkg/src/REPLMode/REPLMode.jl:408
 [13] do_cmd(repl::REPL.LineEditREPL, input::String; do_rethrow::Bool)
    @ Pkg.REPLMode ~/src/julia-master/usr/share/julia/stdlib/v1.6/Pkg/src/REPLMode/REPLMode.jl:386
 [14] do_cmd
    @ ~/src/julia-master/usr/share/julia/stdlib/v1.6/Pkg/src/REPLMode/REPLMode.jl:377 [inlined]
 [15] (::Pkg.REPLMode.var"#24#27"{REPL.LineEditREPL, REPL.LineEdit.Prompt})(s::REPL.LineEdit.MIState, buf::IOBuffer, ok::Bool)
    @ Pkg.REPLMode ~/src/julia-master/usr/share/julia/stdlib/v1.6/Pkg/src/REPLMode/REPLMode.jl:550
 [16] #invokelatest#2
    @ ./essentials.jl:707 [inlined]
 [17] invokelatest
    @ ./essentials.jl:706 [inlined]
 [18] run_interface(terminal::REPL.Terminals.TextTerminal, m::REPL.LineEdit.ModalInterface, s::REPL.LineEdit.MIState)
    @ REPL.LineEdit ~/src/julia-master/usr/share/julia/stdlib/v1.6/REPL/src/LineEdit.jl:2441
 [19] run_frontend(repl::REPL.LineEditREPL, backend::REPL.REPLBackendRef)
    @ REPL ~/src/julia-master/usr/share/julia/stdlib/v1.6/REPL/src/REPL.jl:1126
 [20] (::REPL.var"#44#49"{REPL.LineEditREPL, REPL.REPLBackendRef})()
    @ REPL ./task.jl:406

Project and Manifest in this gist. (But I have lots of locally devved packages with changes, so this may be hard to replicate.)

[timholy]

https://github.com/JuliaArrays/MosaicViews.jl/pull/26/checks?check_run_id=1734570524 is a publicly-accessible example. I'm struggling to replicate it locally, though. Any tips?

[fredrikekre]

Thanks. Pkg fails to lookup the source of MosaicViews since it is not in the manifest. Compare e.g.

┌ Info: is_package_downloaded
│   project_file = "/home/fredrik/dev/pkg-dbg/MosaicViews/Project.toml"
│   pkg =
│    PackageSpec(
│      name = PaddedViews
│      uuid = 5432bcbf-9aad-5242-b902-cca2824c8663
│      tree_hash = 91d229e113e8975a399e40d7c0b1ddf4da6d3c59
│      version = v"0.5.7"
└    )
┌ Info: is_package_downloaded
└   sourcepath = "/home/fredrik/.julia/packages/PaddedViews/VweJw"
┌ Info: is_package_downloaded
│   project_file = "/home/fredrik/dev/pkg-dbg/MosaicViews/Project.toml"
│   pkg =
│    PackageSpec(
│      name = MosaicViews
│      uuid = e94cdb99-869f-56ef-bcf0-1ae2bcbe0389
│      version = v"0.2.4"
└    )
┌ Info: is_package_downloaded
└   sourcepath = nothing
ERROR: AssertionError: sourcepath !== nothing

where MosaicViews have neither a git-tree-sha1 nor a path in its PackageSpec.

[fredrikekre]

For now you can use another project and where you dev MosaicViews and then add ImageCore.

Btw, why are you testing ImageCorehere? It does not seem to depend on MosaicViews. Did you intend to test MosaicViews with ImageCore#master? Edit: Must have looked in the wrong place.

[DilumAluthge]

@timholy Are you trying to do reverse dependency testing?

I saw @oxinabox implement a nice reverse dependency testing setup recently. I think he's going to write it up as a blog post or a Discourse post soon.

If you are indeed doing reverse dependency testing, you can probably just copy and reuse his workflow file.

[DilumAluthge]

Here's the PR: https://github.com/JuliaDiff/ChainRulesCore.jl/pull/281

Here's the workflow file: https://github.com/JuliaDiff/ChainRulesCore.jl/blob/master/.github/workflows/IntegrationTest.yml

@oxinabox Is this the most recent/up-to-date version of your reverse dependency testing GitHub Actions workflow?

[fredrikekre]

I was wrong. On Pkg.add("ImageCore") this is added to the manifest:

[[MosaicViews]]
deps = ["ImageCore", "MappedArrays", "OffsetArrays", "PaddedViews"]
uuid = "e94cdb99-869f-56ef-bcf0-1ae2bcbe0389"
version = "0.2.4"

but Pkg should have inserted the path there, I guess? https://github.com/JuliaLang/Pkg.jl/pull/1958 is probably not enough.

[timholy]

Thanks for the help! I have a dual interest in getting the packages working and in debugging Pkg. I might hold off on implementing that workflow file for a few days just to see if we can use it to test Pkg fixes? Of course I could always back it out in a later commit, so it's not a big deal either way.

[fredrikekre]

It was trivial to reproduce locally.

[timholy]

How? I did this:

$ JULIA_DEPOT_PATH=/tmp/pkgs julia
pkg> dev MosaicViews
pkg> add ImageCore

and it succeeded...which is not what I wanted.

[fredrikekre]

But there you are not running with MosaicViews as the current project.

$ cd $(mktemp -d)
$ git clone https://github.com/JuliaArrays/MosaicViews.jl . 2> /dev/null
$ pkg --project=. add ImageCore 2> /dev/null
$ pkg --project=. instantiate 
ERROR: LoadError: AssertionError: sourcepath !== nothing

[oxinabox]

Here's the workflow file: https://github.com/JuliaDiff/ChainRulesCore.jl/blob/master/.github/workflows/IntegrationTest.yml

@oxinabox Is this the most recent/up-to-date version of your reverse dependency testing GitHub Actions workflow?

Yes, that is the latest file.
I posted a link and a brief discription of how it works on discourse the other day
https://discourse.julialang.org/t/github-actions-workflow-for-reverse-dependency-integration-testing/53536?u=oxinabox

It's fairly simple, now that it works.

[timholy]

As @fredrikekre kindly pointed out, this was fixed by #2338. I spent ~20 minutes trying to write a test:

diff --git a/test/new.jl b/test/new.jl
index 3c3bbc4c..3848857e 100644
--- a/test/new.jl
+++ b/test/new.jl
@@ -2411,6 +2411,19 @@ end
             Pkg.add(path="A")
         end
     end
+    isolate(loaded_depot=true) do
+        cd_tempdir() do dir
+            Pkg.generate("A")
+            Pkg.generate("B")
+            git_init_and_commit("A")
+            git_init_and_commit("B")
+            Pkg.activate("A")
+            Pkg.add(path="B")
+            # Pkg.activate("B")
+            # # This shouldn't error even though A has a dependency on B
+            # Pkg.add(path="A")
+        end
+    end
 end

 @testset "Offline mode" begin

but unfortunately it passes even without #2338.

EDIT: nvm, I figured something out.

[fonsp]

You can also get this error when using Julia 1.5 to instantiate a Manifest that was created by Julia 1.6. Either use Julia 1.6, or run Pkg.update() or Pkg.resolve() to fix it.

[KristofferC]

Yeah, Pkg was not well equipped for the transition of normal packages into stdlibs. It wasn't really anything that was considered plausible to happen.