Julia: 0.5 `@test_skip` causes CI to fail

[sbromberger]

per https://discourse.julialang.org/t/how-to-use-test-skip-properly/2628

The following code will cause CI to fail, even though @test_skip should not contribute to CI success/failure:

using LightGraphs
using LightGraphs.LinAlg
using Base.Test

@testset "foo" begin
@test_skip 1 == 2
end

yields

julia> Pkg.test("LightGraphs")
INFO: Testing LightGraphs
Test Summary: | Broken  Total
  foo         |      1      1
ERROR: LoadError: Some tests did not pass: 0 passed, 0 failed, 0 errored, 1 broken.

causing a CI failure.

This is apparently fixed in 0.6, but I haven't been able to confirm that yet.

[tkelman]

Bisect identifies 8982605c1998 as fixing this. I'm glad the version of that PR that we merged was bisect-clean.

That commit is a bit too large and disruptive to backport, but a small subset of it would probably be good enough to fix this bug. Likely just the counting of broken tests in determining whether a testset succeeds or fails.

[c42f]

I just came across this myself for @test_broken. Here's the workaround I'm using until 0.5.2:

using Base.Test

if VERSION < v"0.6-"
    # Override Test.@test_broken, which is broken on julia-0.5!
    # See https://github.com/JuliaLang/julia/issues/21008
    macro test_broken(exs...)
        esc(:(@test !($(exs...))))
    end
end

[tkelman]

fixed in 0.5.2 which will be tagged and uploaded shortly