Pkg.jl: Clarify current fragility of `test/Project.toml` approach

[cscherrer]

@fredrikekre 's comment here says,

Don't use the functionality of test/Project.tomls which is pretty beta (and documented as such)

But this is not clear at all from current documentation here, which only says

The exact interaction between Project.toml, test/Project.toml and their corresponding Manifest.tomls are not fully worked out, and may be subject to change in future versions. The old method of adding test-specific dependencies, described in the next section, will therefore be supported throughout all Julia 1.X releases.

The section title here is "Test-specific dependencies in Julia 1.2 and above", and the qualification seems to me to only indicate that the current approach may change in the future, not that it's "beta" in any way.

I have set up a few repositories using this approach, only to find out today that it's extremely brittle. Can the current documentation be updated to clarify this?

[phipsgabler]

There's something else that's not entirely clear here -- the docs say that "Pkg will add the tested package itself implictly". This works fine when running ] test, but I also expected the following to behave the same (that's on Julia 1.3.1):

(AutoGibbs) pkg> activate test
Activating environment at `~/.julia/dev/AutoGibbs/test/Project.toml`

julia> using AutoGibbs
ERROR: ArgumentError: Package AutoGibbs not found in current path:
- Run `import Pkg; Pkg.add("AutoGibbs")` to install the AutoGibbs package.

I guess this is related to the fact that the sub-project thing is still very much under discussion?

[KristofferC]

I guess this is related to the fact that the sub-project thing is still very much under discussion?

Yes, this would work with subprojects but with the current design, Pkg.test does "magic" to put these things into a new project that includes everything. My goal is to get subprojects into 1.6 (which will likely be the new LTS).

[FelixBenning]

I just tried this approach and I can not get it to work:

test/Project.toml

[deps]
Test = "8dfed614-e22c-5e08-85e1-65c5234f0b40

if I activate . :

(MyProject) pkg> test
....
ERROR: LoadError: ArgumentError: Package Tests not found in current path:
- Run `import Pkg; Pkg.add("Tests")` to install the Tests package.

if I activate ./test I can neither do

(test) pkg> test
ERROR: trying to test unnamed project

nor

(test) pkg>  test MyPackage
ERROR: The following package names could not be resolved:
 * MyPackage (not found in project or manifest)

I also tried adding the uuid of MyPackage to the test/Project.toml which complains about the package not being registered (of course).