Julia: Allow @test_nothrow to catch 'no error' for simple smoke tests

[djsegal]

I want to be able to do a quick @test_throws nothing Foo(42) to run a simple smoke test.

The current documentation makes it look like this is not an option

[oxinabox]

I've spent far too much time thinking about this, a few weeks back.

You can readily approximate that by doing something like @test Foo(42) != π
which will always have a test result of Pass, (assuming that it does indeed not return pi),
unless there is a exception -- then it will instead have a test result of Error
This macro @test_throws nothing, would turn that Error result into a Fail result.

After a fair bit of thought on that, getting into the semantics of what is an Error vs what is a Fail,
is a rabbit hole that does not go anywhere useful. As far as it goes it is right now: Fails are almost always a fault in src, Errors can be either a fault in src or a fault in test or a fault in the enviroment (eg Out of Memory).

@test_throws nothing, would allow a Fail to be causes by anything that can cause a Error, like OOM.
Now, that is a loss of information.
But it doesn't matter, because Fail or Error. when your test cases do not Pass you investigate it and fix it.

I think preferable would be the syntax @test_doesnotthrow, or @test_noerror

One thing to note, is that this is not present in most testing frameworks.

• xUnit removed it, in v2.0 (https://github.com/xunit/xunit/issues/188)
• jUnit does not have it
• Midge does not have it
• Python's testing does not have it

The only framework I've fount that has it is nUnit, which as DoesNotThrow in classic mode, and ThrowsNothing in constraint mode.

[djsegal]

@test_no_error ... would probably be good.

I know it's not the same, but rspec for ruby uses the syntax:

expect{Object.public_instance_methods}.to_not raise_error(NameError)

---

which came from:

• https://www.relishapp.com/rspec/rspec-expectations/v/2-0/docs/matchers/expect-error#expect-no-error

[StefanKarpinski]

I don't really get the application. Why do you need a testing construct for this at all? Every statement is implicitly @test_no_error; if an error is thrown, the test suite fails.

[djsegal]

I see where you're coming from.

• I just wanted it to be _more explicit_ in test files that something failed at construction time
• As opposed to just having a random line in your test file that some good samaritan might delete

---

Feel free to close issue at your own discretion

[kshyatt]

just having a random line in your test file that some good samaritan might delete

FWIW I feel like this problem can be addressed by commenting the tests more aggressively than we currently do. I often look at the base Julia tests and think "it sure would be nice to know what this is trying to show works!" Even a simple "test that X doesn't explode immediately" comment block would be good.

[djsegal]

• My only argument against comments is that they're out of date the second you write them
• If anything, I'd prefer something like:

@test_no_error Foo(42) "Foo can be constructed with integer value"

[ararslan]

I agree with the others that I don't think such a construct is that generally useful. Though for the sake of completeness, you _can_ obtain what you're looking for relatively easily using existing machinery, it's just a little hacky and ugly:

using Base.Test
f(x::Int) = 1
@test !isa(try f(1) catch ex ex end, Exception)    # Passes
@test !isa(try f(1.0) catch ex ex end, Exception)  # Fails

Assuming you know the value of Foo(42), wouldn't @test Foo(42) == something be a sufficient proxy to testing that something isn't thrown? You end up getting more information, since if it passes you know both that nothing was thrown and that you get the expected value.

[StefanKarpinski]

Ok, I see where this is coming from. You could do this: @test isa(ex, Any), although that's not very pretty. You could also write this macro:

macro no_error(ex)
    quote
        try
            $(esc(ex))
            true
        catch
            false
        end
    end
end

julia> @no_error 1 + 2
true

julia> @no_error error()
false

julia> @test @no_error 1 + 2
Test Passed
  Expression: @no_error 1 + 2

julia> @test @no_error error()
Test Failed
  Expression: @no_error error()
ERROR: There was an error during testing
 in record(::Base.Test.FallbackTestSet, ::Base.Test.Fail) at ./test.jl:397
 in do_test(::Base.Test.Returned, ::Expr) at ./test.jl:281

The main issue with using this is that no hint is given as to what error is thrown on failure. Hmm. I'm a bit ambivalent about whether this warrants a whole new @test_no_error macro.

[tpapp]

I would find this construct useful for testing functions which (mainly) have side effects. Eg when I implement a method for Base.show, testing that it just runs is useful (of course the right way would be capturing the output in an IOBuffer and comparing it to something predefined, but it is nice to have something intermediate which is more robust to code changes).

[cossio]

Currently we have @test_warn, @test_nowarn, @test_throws. I think a @test_nothrow completes the picture.

Here is a possible use case. Suppose you want to test a method that returns no results. But you know it throws an exception if some condition is not met. If you know in a certain context the condition should hold, the only way to test it would be to make sure that the method doesn't throw the exception.

I don't agree with a generic @test_nothrow to test that the method does not throw _any_ exception. Instead we should have a @test_nothrow except where except is a specific (or abstract) exception. A possible semantic would be:

1. The test results in Error if an exception other than except is thrown.
2. The test Fails if except is thrown.
3. The test Passes if no exceptions are thrown.

[StefanKarpinski]

Sure, I guess if someone wants to put together a @test_nothrow PR we could add this. Those semantics seem sensible enough.