Nunit-console: NUnit 3 Console runner executes Explicit tests when filtering by namespace

[soodputter]

NUnit 3 Console runner executes Explicit tests when filtering by namespace.

We're using NUnit3, the test dll is compiled against NUnit Framwork 3.9.0.0 and using NUnit Console Runner version 3.7.0.0.
We're using Following filter expression:

--where namespace==Some.Namespace.Having.ExplicitTests

but we're finding all the Explicit tests + tests from Explicit Fixture are executed.

The filter works fine when filter by Class or Category.

So the Issue is something similar to
https://github.com/nunit/nunit/issues/1141
https://github.com/nunit/nunit/issues/1039

Any help on this will be highly appreciated!

[CharliePoole]

If we treat this as a bug, we will be introducing a certain inconsistency, as I'll explain below. For sure, this should be better documented, however.

NUnit's principle for "explicit selection" is that the test is being examined and selected because of characteristics on that class, like its name or categories. Non-explicit selection, OTOH, means that some parent test was explicitly selected, and this test is being run as a consequence.

Consider the following two selection expressions:

  --where test==Some.Namespace.Having.Explicit.Tests
  --where namespace==Some.Namespace.Having.Explicit.Tests

They do different things, which is probably surprising to many users. Initially, I resisted introducing the namespace filter because of this confusion. But in the end we accepted it and I believe we need to live with that choice and just document it better.

In the first example, using test, the actual test suite represented by the namespace is the __only__ test selected. It is run and all the non-explicit tests under it are also run. That includes nested namespace suites as well. Explicit tests, of course, are not run.

In the second example, using namespace, the same namespace suite is directly selected but so are __all__ the tests in that same namespace. That is, the namespace field of the method representing the test is checked and it is selected because __it's namespace__ is equal to the stated value. Consequently, two things happen: first, the nested namespaces are not included and secondly, the tests are explicitly selected.

Workarounds:

1. If you want the behavior of test, then simply use it.
2. If you want to include a namespace but exclude nested namespaces, you can use something like

  --where test==My.Namespace && test!=My.Namespace.Nested

I'll leave it up to the rest of the @nunit/framework-team to decide whether to make namespace work differently, but I strongly recommend against it. We should document this, however.

[jnm2]

I was hoping to suggest the exact same change for Category, since people think of it (like namespace) as only a container rather than a single test which is being explicitly chosen. I can't say definitively that we should, but I thought it would be nice to whitelist the types of comparison which are capable of explicitly selecting tests.

Also, with Category, there's no workaround like there is with --where test=My.Namespace.

[CharliePoole]

I'd strongly suggest that we shouldn't. If I understand what you are saying, you want explicit tests under a suite with a given category to be run if that category is selected. But ExplicitAttribute was designed to prevent exactly that from happening. Changing the meaning is not minor, because it requires folks to re-design how they use categories. IMO... this is another one of those things that could once (NUnit 2.3 IIRC) been done in several different perfectly valid ways. We chose a particular way and we end up having to stick with it as a result.

Just for interest, Explicit was originally a named property of the CategoryAttribute, so it has always been tied with categories. In the original design, an Explicit category had the same effect that putting both the category and ExplicitAttribute on the same method has today. The drawback of this was that you could use the same string as a category with and without explicit in different places, leading to confusion. In many ways, it would have been better if categories were objects.

It is a long-time problem that users assume either category inheritance or that categories are containers. If I get around to writing a book, that will be a whole chapter! For now, the docs could explain it better. Reviewing the docs for Explicit, I see that everything I wrote above is in there, but would be missed by most people. It's only when closely read by someone who knows how NUnit works that you can see it's there, which is not a good thing as far as documentation goes.

We should at least document (well) what it now does. Then people could tell us if they want something else. Even though all of us are also users of NUnit, I'm always suspicious of "improvements" that one of us initiates or argues for.

[jnm2]

If I understand what you are saying, you want explicit tests under a suite with a given category to be run if that category is selected.

I'm trying to say the opposite: maybe we should whitelist which comparisons are considered 'explicitly-selecting' comparisons so that Category and Namespace filters never cause explicit tests to run.

[CharliePoole]

I'm confused by your term "whitelist." There is no code anyhwhere that looks at a list of ways to select things and decides whether explicit affects that particular thing. There are filters, which are able to select things and which can indicate whether the selection was explicit or not.

It sounds like you want to change category filters and namespace filters work so they work differently.

IMO, that's premature if we are saying that many people don't understand how they already work. Maybe we should try explaining it better. Of course, it's also a pretty big breaking change to change how they work. Namespace filters are new, so not so bad to change. Category filters have worked as they do for a long time, however.

[jnm2]

Yes, that's what I'm saying. There is no such code which requires certain kinds of comparison (like TestName) in order to explicitly select a test, and I sort of think making filters work this way could be nice. But perhaps I only want that because I haven't figured out how to leverage them properly.

[soodputter]

Thank you for the replies.

Our scenarios is to test all the TestFixtures under a namespace but we want to skip all the Explicit marked tests. Some TestFixtures have explicit tests too and those shall not be executed. Hence we are using filter expression for namespace and providing the exact namespace as input parameter like:
--where namespace==NUnit.ExplicitAttributeTests, this execute all tests including explicit tests and nothing is skipped.
But when we use something like
--where class==NUnit.ExplicitAttributeTests.ExplicitExecutionReferenceTests, all explicit tests are skipped. So we though the explicit tests can also be skipped by using namespace filter.

Example code: In this example filter by namespace will run all test cases.

namespace NUnit.ExplicitAttributeTests
{
    using NUnit.Framework;

    [TestFixture]
    public class ExplicitExecutionReferenceTests
    {
        [Test, Explicit("Test shall be skipped")]
        public void TestUsingExplicitAttribute()
        {
            Assert.Pass("Test executed explicitly");
        }

        [Test]
        public void TestWithoutExplicitAttribute()
        {
            Assert.Pass("Test shall be executed and shall not be skipped");
        }
    }

    [TestFixture, Explicit]
    public class ExplicitFixtureTests
    {
        [Test]
        public void TestInvokedExplicitly()
        {
            Assert.Pass("Test executed and invoked explicitly");
        }
    }
}

^{[jnm2—added code fences for readability]}

[CharliePoole]

Odd and interesting that class is different. In fact, you can think about the class either as a container of the method or an attribute of the method (since it's part of the MethodInfo).

Even though class and namespace can both be seen in those two ways, we chose differently in each case. The key aspect of namespace that caused us to treat it as we did was that the user requesting the feature wanted to include only classes with the __exact__ namespace, excluding any sub-namespaces. That led us away from treating it as a container. Since we only recently implemented namespace a short while back, we could probably get away with changing it's operation, but we would still have to deal with contained namespaces in the correct way.

All that said, I wonder if we can't simply leave things as they are and document better how each selection element actually works. It seems to me that the original request is taken care of here by changing to use test or class rather than namespace. If we should agree that this is a doc issue, I'll take it on.

[jnm2]

For now, I'm content leaving it a doc issue.

[CharliePoole]

As usual, I'll edit then ask for a review.

[CharliePoole]

Looking at the documentation, it actually seemed correct to me. However, I added some examples to make it clearer. Please review before closing this.

[CharliePoole]

@nunit/framework-and-engine-team Can somebody review this?

[mikkelbu]

@CharliePoole I assume that you only edited the explicit attribute page? Overall it looks good to me. I was going to say that I missed the information about the difference between test, class, namespace filtering on the page, but I found the information on the test selection language page.

[CharliePoole]

Yes, exactly. It seemed like that what was needed on that page was some reference to using the command-line filter commands with Explicit tests.

[GraueEminenz79]

@CharliePoole had such a lovely explanation on 27th of Feb which didn't make into the documentation, imho. To be honest I am still confused when it comes to test and namespace filtering whilst having Explicit attributed tests (and/or fixtures).
Just my .02

[mikkelbu]

@GraueEminenz79 I've created nunit/docs#290 to do this