Julia: Tests can break Test

Created on 26 Nov 2019 · 7Comments · Source: JuliaLang/julia

If you dev the brand-new https://github.com/JuliaMath/CheckedArithmetic.jl and then do pkg> test CheckedArithmetic, all seems well. But then insert error("oops") here, and you'll find to your dismay that the tests still pass.

If you comment out the "check" tests, then it fails as expected.

EDIT: package has been fixed. It was due to a habit of writing return val at the end of a block, and mistakenly doing that in the @check macro definition.

testsystem

Source

timholy

😄2

All 7 comments

This is also funny when putting it on e.g. L42:

@checked: Error During Test at /home/fredrik/dev/CheckedArithmetic/test/runtests.jl:12
  Got exception outside of a @test
  fsdfds
  Stacktrace:
   [1] error(::String) at ./error.jl:33
   [2] top-level scope at /home/fredrik/dev/CheckedArithmetic/test/runtests.jl:42
   [3] top-level scope at /home/fredrik/julia13/usr/share/julia/stdlib/v1.3/Test/src/Test.jl:1107
   [4] top-level scope at /home/fredrik/dev/CheckedArithmetic/test/runtests.jl:13
   [5] top-level scope at /home/fredrik/julia13/usr/share/julia/stdlib/v1.3/Test/src/Test.jl:1107
   [6] top-level scope at /home/fredrik/dev/CheckedArithmetic/test/runtests.jl:12
   [7] include at ./boot.jl:328 [inlined]
   [8] include_relative(::Module, ::String) at ./loading.jl:1105
   [9] include(::Module, ::String) at ./Base.jl:31
   [10] include(::String) at ./client.jl:424
   [11] top-level scope at none:6
   [12] eval(::Module, ::Any) at ./boot.jl:330
   [13] exec_options(::Base.JLOptions) at ./client.jl:263
   [14] _start() at ./client.jl:460

   Testing CheckedArithmetic tests passed

You get an error with a stacktrace, but the process exits with 0.

fredrikekre on 26 Nov 2019

The error seems to come from the @check macro, e.g. @test @check(3+5) == 12 will succeed. I think it comes from your @check macro, it shouldn't return in the returned quoted expression. I.e. we observe the same behavior here if you put a return 0 within a @testset.

Very cool package by the way!