Turing.jl: Re-organise `test` folder to improve clarity

[willtebbutt]

At the time of writing I'm trying to add some tests to Turing to enable some headway to be made on #674 , and I'm struggling a bit to navigate through our testing infrastructure. This is partly my own fault for having not worked a huge amount with it recently, but I do think that there's substantial room for improvement to make it more straightforward. Here are a few desiderata that I think are reasonable:

• For each file in src, there should be exactly one file in test that contains the corresponding unit / integration tests.
• The file structure in test should be identical to src.
• runtests.jl should contain a human-readable hierarchy of @testsets corresponding to the file structure. The names of each testset should correspond to the file / folder that the testset corresponds to.
• Each file in the test directory contains an outer testset with the name of the file, and may (generally should) contain nested test sets to group together related tests.

The first two points are, I feel, important because they make it trivial to figure out where all of the tests corresponding to a particular source file are. They also make it easy to run only the tests that you're interested in -- you just comment out the tests that you don't want. While this mechanism is arguably less sophisticated than our current mechanism, it lets you do everything that you want and is immediately obvious how to use. I really struggle to come up with a situation in which our current system provides a significant benefit over just commenting unwanted tests while developing, provided that the structuring of the test sets is done correctly.

Moreover, this set up makes makes it straightforward to see where test coverage is light -- you really just have to look a one file per source file.

The last two points have a similar rationale to the points before. An example runtests.jl file would be something like this:

@testset "Turing" begin
    @testset "core" begin
        include("core/ad.jl")
        include("core/compiler.jl")
        include("core/container.jl")
        include("core/Core.jl")
        include("core/VarReply.jl")
    end

    @testset "inference" begin
        @testset "adapt" begin
            include("core/inference/adapt/adapt.jl")
            # etc
        end
        @testset "support" begin
            include("core/inference/support/hmc_core.jl") 
        end
        include("core/inference/dynamichmc.jl")
        #etc
    end

    @testset "utilities" begin
        include("utilities/distributions.jl")
        #etc
    end
end

See Stheno.jl for an example of what I'm on about.

What are your thoughts @yebai ?

[cpfiffer]

I agree with this 100%. I have only worked a small amount with the Turing's testing suite, but I have found it very difficult to navigate through.

[yebai]

I agree that the current test folder is a bit overcomplicated. Some restructuring would save the team a lot of time for testing. I'll see what we can do about this. Any recommendations on better practice, related packages for organising tests, are appreciated.

@mohamed82008 @KDr2

[mohamed82008]

I second @willtebbutt 's proposal. It is also good to wrap the test prep code in functions. This is good to know which pieces are interacting together and which are not. I don't think there is any common testing package used in the community beside Base.Test.

[willtebbutt]

It is also good to wrap the test prep code in functions.

Could you expand a little on this @mohamed82008 ? I'm not quite sure what you mean.

[mohamed82008]

Like for example testing single argument AD can be done using:

function test_ad_single(f, x)
     @test ....
end
test_ad_single(x -> logpdf(Binomial(10, x), 5), 0.2)
....

So that we don't have a lot of repetition in the test files.

[willtebbutt]

Ah, yes, I'm a massive fan of this kind of thing. It's particularly salient in Turing's case, given how often we're likely to apply the same tests to lots of different models.

[willtebbutt]

When we do something here, the style guide should be updated.

[yebai]

Fixed by #731