Pkg.jl: Test a specific testset

[innerlee]

Would be great.

[KristofferC]

I think this first requires a rewrite of the Base test system to have some sort of TestRunner very similar to the BenchmarkGroup in BenchmarkTools.jl: https://github.com/JuliaCI/BenchmarkTools.jl/blob/master/doc/manual.md#the-benchmarkgroup-type.

We could then pass arguments to that runner and it would choose what tests to execute. What we currently do is just include a file (runtests.jl) and run it from top to bottom. This provides almost no options for customizing the execution behavior.

[innerlee]

That makes sense.
Just a random question:
Is it possible to make it like pytest? Specify a test filename, and then do that test?

[KristofferC]

The way test is documented now is "run test/runtests.jl. We could document test bleh to run test/bleh.jl but it seems like a fairly minor improvement (you could just do include("test/bleh.jl").... I think there needs to be a rewrite of the testing system that is customizable and extensible and until then there is nothing really that can be done from this side.

[KristofferC]

Until the testing system itself has support for this, nothing to do here.