Julia: Inconsistent visibility of Artifacts & TOML

[timholy]

I created a simple (code-free) package called ArtifactsTester using PkgTemplates, where the only thing in the repo is the following runtests.jl file:

using ArtifactsTester
using Test

@testset "ArtifactsTester.jl" begin
    @test Base.identify_package(Base.__toplevel__, "Artifacts") !== nothing
    @test Base.identify_package(Main, "Artifacts") !== nothing
end

It works when run from the REPL, but not when run under Pkg.test:

shell> cd test
/home/tim/.julia/dev/ArtifactsTester/test

julia> include("runtests.jl")
Test Summary:      | Pass  Total
ArtifactsTester.jl |    2      2
Test.DefaultTestSet("ArtifactsTester.jl", Any[], 2, false)

julia> using Pkg

julia> Pkg.test("ArtifactsTester")
    Testing ArtifactsTester
Status `/tmp/jl_U2sCWD/Project.toml`
  [8466d077] ArtifactsTester v0.1.0 `~/.julia/dev/ArtifactsTester`
  [8dfed614] Test
Status `/tmp/jl_U2sCWD/Manifest.toml`
  [8466d077] ArtifactsTester v0.1.0 `~/.julia/dev/ArtifactsTester`
  [2a0f44e3] Base64
  [8ba89e20] Distributed
  [b77e0a4c] InteractiveUtils
  [56ddb016] Logging
  [d6f4376e] Markdown
  [9a3f8284] Random
  [9e88b42a] Serialization
  [6462fe0b] Sockets
  [8dfed614] Test
    Testing Running tests...
ArtifactsTester.jl: Test Failed at /home/tim/.julia/dev/ArtifactsTester/test/runtests.jl:5
  Expression: Base.identify_package(Base.__toplevel__, "Artifacts") !== nothing
   Evaluated: nothing !== nothing
Stacktrace:
 [1] top-level scope
   @ ~/.julia/dev/ArtifactsTester/test/runtests.jl:5
 [2] top-level scope
   @ ~/src/julia-master/usr/share/julia/stdlib/v1.6/Test/src/Test.jl:1113
 [3] top-level scope
   @ ~/.julia/dev/ArtifactsTester/test/runtests.jl:5
ArtifactsTester.jl: Test Failed at /home/tim/.julia/dev/ArtifactsTester/test/runtests.jl:6
  Expression: Base.identify_package(Main, "Artifacts") !== nothing
   Evaluated: nothing !== nothing
Stacktrace:
 [1] top-level scope
   @ ~/.julia/dev/ArtifactsTester/test/runtests.jl:6
 [2] top-level scope
   @ ~/src/julia-master/usr/share/julia/stdlib/v1.6/Test/src/Test.jl:1113
 [3] top-level scope
   @ ~/.julia/dev/ArtifactsTester/test/runtests.jl:5
Test Summary:      | Fail  Total
ArtifactsTester.jl |    2      2
ERROR: LoadError: Some tests did not pass: 0 passed, 2 failed, 0 errored, 0 broken.
in expression starting at /home/tim/.julia/dev/ArtifactsTester/test/runtests.jl:4
ERROR: Package ArtifactsTester errored during testing
Stacktrace:
  [1] pkgerror(::String, ::Vararg{String, N} where N)
    @ Pkg.Types ~/src/julia-master/usr/share/julia/stdlib/v1.6/Pkg/src/Types.jl:52
  [2] test(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}; coverage::Bool, julia_args::Cmd, test_args::Cmd, test_fn::Nothing)
    @ Pkg.Operations ~/src/julia-master/usr/share/julia/stdlib/v1.6/Pkg/src/Operations.jl:1580
  [3] test(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}; coverage::Bool, test_fn::Nothing, julia_args::Cmd, test_args::Cmd, kwargs::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ Pkg.API ~/src/julia-master/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:328
  [4] test(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec})
    @ Pkg.API ~/src/julia-master/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:315
  [5] #test#62
    @ ~/src/julia-master/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:67 [inlined]
  [6] test
    @ ~/src/julia-master/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:67 [inlined]
  [7] #test#61
    @ ~/src/julia-master/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:66 [inlined]
  [8] test
    @ ~/src/julia-master/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:66 [inlined]
  [9] test(pkg::String; kwargs::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ Pkg.API ~/src/julia-master/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:65
 [10] test(pkg::String)
    @ Pkg.API ~/src/julia-master/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:65
 [11] top-level scope
    @ REPL[4]:1

[timholy]

The same proves to be true for TOML

[KristofferC]

Pkg.test modifies the LOAD_PATH (so you can't load stdlibs you don't depend on) so maybe that has something to do with it?

[timholy]

We don't have a way to have version-conditional test dependencies, do we?

[KristofferC]

No, but Artifacts already exist as a registered package that you can use pre 1.6 and soon things will be the same for TOML (https://github.com/JuliaRegistries/General/pull/21412)

[StefanKarpinski]

Also Downloads, which will be a new stdlib for downloading stuff.

[KristofferC]

Is there anything to do here? I think the observed behavior is intended.