Nunit-console: Add namespace keyword to Test Selection Language

[tom-dudley]

Suppose I currently have some tests sitting under My.Namespace and some other tests sitting under My.Namespace.SubNamespace, I can't identify an easy way to select just the former as using --where "class =~ My.Namespace" will result in running all of the tests.

What I suggest is that the keywords namespace is added to the Test Selection Language allowing such filtering, such as --where "namespace == My.Namespace"

[CharliePoole]

Your example can be run a number of ways. The two simplest would be:

--where "class =~ My.Namespace and class !~ My.Namespace.SubNamespace"
--where "test == My.Namespace and test != My.Namespace.SubNamespace"

Tests are treated as nesting, which is why it's possible to use the equality and inequality operators in the second example as opposed to a regular expression.

Adding namespace is indeed a possible extension and could allow a slightly shorter expression. However, I'm afraid that some users would expect My.Namespace to include My.Namespace.SubNamespace, leading to more confusion. I'm inclined to make this a "won't fix" item. What do other @nunit/core-team and @nunit/framework-team members think?

[jnm2]

If I have five subnamespaces and want to execute only the tests in the outer namespace, I'd like to be able to match an exact namespace.

I generally mimic the namespace structure of the assembly under test, so I can see this happening easily.

[CharliePoole]

@jnm2 OK... that's a point to consider.
.
Of course, there are workarounds: (1) stop organizing tests that way or (2) use categories. The latter is the classic NUnit answer of course.

Still it would be easy to add this. Again, my main concern is user confusion. I don't think it's obvious that
namespace == My.Namespace doesn't include My.Namespace.SubNamespace. The docs will explain it of course but even so I wonder if we would be confusing users.

[jnm2]

or (2) use categories. The latter is the classic NUnit answer of course.

Which is what I do, but is less ad-hoc.

The docs will explain it of course but even so I wonder if we would be confusing users.

The users should think of it as the entire namespace, the same as if they wrote typeof(x).Namespace ==. I guess it'll be a learning thing. Unless you like EntireNamespace ==, but at some level that seems redundant and it begs the question of what Namespace would then mean.

[CharliePoole]

I see what you're saying. When I'm looking at a single class, I understand "namespace" as you do. When I'm running tests from the command-line or gui, I think of it as being inclusive. The namespace code element stands for a test suite in the logical NUnit view of tests. We'll have to explain it carefully if we do it.

Let's wait to see if some other opinions pop up here.

[tom-dudley]

I should mention that the scenario jnm2 initially outlined with multiple subnamespaces was my motivator for creating this issue, as currently I'm working on a large codebase containing a vast number of subnamespaces making using the excludes somewhat cumbersome.

I definitely appreciate the conundrum regarding namespaces being inclusive and unfortunately I don't currently have a good answer for how both cases might be handled nicely.

[ChrisMaddock]

I agree I'd expect a namespace filter to be inclusive - but I don't have a solution to offer for the problem at hand!

[jnm2]

How about NamespaceExact?

[ChrisMaddock]

I think something such as that would be the best option.

I was considering suggesting a === operator for exact match, but just the thought of it made me feel a little ill. :-p

[JustinRChou]

Categories seem to already do a fine job for test selection.

And I never noticed that C# didn't allow for Using System.*; so I don't know sure if using .* notation for loading subnamespaces would make sense.

[jnm2]

Categories seem to already do a fine job for test selection.

Categories aren't as great for this IMO. For one thing they aren't ad-hoc; I can't filter on something clearly obvious without taking half an hour and annotating all my fixtures in the namespace with a new category. For another thing it's maintenance overhead with opportunity for error, and it's all redundant considering the namespace.

[CharliePoole]

OK, I added it to the backlog as an enhancement. We should support both equal tests and regular expressions, just as we do for other things.

[CharliePoole]

This will require a corresponding framework change, by the way.

[JustinRChou]

Why would there be a framework?

Isn't this just an update to the test selection langauge?

[CharliePoole]

The console parses the TSL using a service of the engine, available to all clients. The parser produces filters in the form of XML that can be passed back to the engine when running the tests. The engine sends the filter XML directly to the framework, which acts on it. The filter implementation for this feature would probably contain something like

<namespace>My.Selected.Namespace</namespace>

Without modification, the framework won't know what to do with that.

Full implementation will require

1. Designing the filter in XML
2. Implementing the TSL to create the filter in the engine.
3. Implementing the filter action itself in the framework.

2 and 3 can actually be done in any order.

[JustinRChou]

I see now.
Namespace just gets translated into a property filter which is wrong.

[CharliePoole]

Right. The parser assumes that any left-hand side operand it doesn't recognize is the name of a property.

[tom-dudley]

So is the consensus that the keyword should be namespace and that == would be used to match the exact namespace (and not subnamespaces), and =~ can be used for subnamespaces?

[jnm2]

So is the consensus that the keyword should be namespace and that == would be used to match the exact namespace (and not subnamespaces), and =~ can be used for subnamespaces?

Well, I had another thought. =~ is not good for subnamespaces because =~Foo would match both Foo.Bar and FoolishNamespace, while =~Foo. would not match Foo.

That seems to indicate that we really do need two separate keywords. Since so many people think namespace is intuitively inclusive, what about Namespace and NamespaceExact?

[JustinRChou]

Is that how console regex works?
Shouldn't =~Foo just match Foo?
And you could use =~^Foo$|^Foo\. to get any Foo. namespace and Foo itself. I hope.

[jnm2]

Oops, I did not realize =~ was regex. From examples in this thread, assumed it had to be StartsWith. I failed at finding a reference for the test selection language. Anyone know where it is?

[JustinRChou]

Hidden within the depths of the nunit docs.
https://github.com/nunit/docs/wiki/Test-Selection-Language

[jnm2]

So is the consensus that the keyword should be namespace and that == would be used to match the exact namespace (and not subnamespaces), and =~ can be used for subnamespaces?

This is now sounding the best.
I don't like that I'd have to think carefully about writing a regex if I wanted the inclusive behavior, but I can't think of a better alternative. == is hardly the ideal operator to use to match a partial namespace no matter what the keyword is.

[jnm2]

I think the regex could be ^Foo($|\.) - I've used that before.
You just can't forget to escape namespace dots 😆 ^Foo.Bar($|\.) will match FoodBar

So more or less the idea is to calculate
c# $@"^{Regex.Escape(type.Namespace)}($|\.)"

[JustinRChou]

I was barking up that tree for a good 20 minutes till I gave up and went for the simple version.

[jnm2]

I was barking up that tree for a good 20 minutes till I gave up and went for the simple version.

Just checking. Does that mean there's something wrong with ^Foo\.Bar($|\.) that I should know about?

[JustinRChou]

I thought you were referring to the case with Foo, that is fine for all Foo.Bar namespaces.

[jnm2]

As long as the pattern ^some\.thing($|\.) works in every possible scenario, that's all I care about. It might be good to put this in the documentation as a hint to people hoping to match subnamespaces.

[CharliePoole]

Users who don't know how to write them won't be able to use regular expressions. Sounds fair to me. :smile:

Seriously, we can put in a few examples but it's not our job to teach programming basics.

[CharliePoole]

OTOH, what we should be sure we have documented is how we support regular expressions within the --where clause because that's not part of C#. For example that we allow various characters to quote the entire expression.

[jnm2]

@CharliePoole 😆 True. I was thinking of it as a time-saver though.

[JustinRChou]

I am guessing it's a no on adding subnamespaces == Foo which would be a shortcut that that regex.

[CharliePoole]

@JustinRChou If you want the inclusive behavior, just use test or class rather than namespace. The justification of namespace is that the OP wanted the non-inclusive behavior.

All... we just want to implement keyword namespace with the four standard operators... which will fall into place immediately as we recognize the keyword. Anyone who wants to do it an volunteer. Even though it requires work in both two different repos we will still implement each one separately, but hopefully close together in time. It seems to make most sense to do the framework first, otherwise the console will produce errors. I'll create an issue for the framework and mark this one blocked for now.

[CharliePoole]

Once the TSL implementation in nunitlite is updated it will be trivial to apply the same changes here.

[CharliePoole]

Here's a further bit of research somebody could do...

What happens right now if the console creates a filter like <filter><namespace></filter> ? Does the result contain information that allows us to display a message like

The namespace filter is not supported by this version of the framework.

We will have to be able to do that if we try to run tests that use an older version of the framework.

In fact, if we don't have a good solution for this, we probably shouldn't go ahead with the change. 😢

[JustinRChou]

It becomes a property element with name="namespace" in the console and gets processed as a property by the framework.
That seems to be the case for anything that isn't already a filter class.

Oh you meant if someone builds the namespace filter on the console first, then the framework throws a ArgumentException.

So far the exception in the TestResult.xml was <message><![CDATA[Exception has been thrown by the target of an invocation.]]></message>

[tom-dudley]

I'm happy to carry out the work for this (and the framework related change) if the issue @CharliePoole raised can be addressed.

@JustinRChou is right that an ArgumentException gets thrown in framework/Internal/TestFilter.cs which happens as the default case for a switch statement which switches on the filter element.

It seems to me that this exception should get handled and that The {0} filter is not supported by this version of the framework gets reported back to the user but this obviously necessitates a change in the framework. I can't think of a how this can be handled without making some change to the framework.

[CharliePoole]

@tom-dudley We'd be happy to get PRs from you. I was able to assign this issue to you, since you created it. However, the framework issue should be first, so I sent you an invitation as a contributor. That will allow me to assign the other issue, which I created, to you.

The exception from TestFilter should be reported by the engine, it would help to know exactly how it is currently reported, which might require making a temporary change to the engine filter generation to insert an element that the framework doesn't understand.

[tom-dudley]

Thanks @CharliePoole . Is it OK that version 3.x.y of the console-runner depends on 3.x.y of the framework for this then? Here what you currently get when making the required changes in console runner but not in the framework:

System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.ArgumentException: Invalid filter element: namespace
Parameter name: xmlNode
   at NUnit.Framework.Internal.TestFilter.FromXml(TNode node) in C:\dev\nunit\src\NUnitFramework\framework\Internal\TestFilter.cs:line 197
   at NUnit.Framework.Internal.TestFilter.FromXml(String xmlText) in C:\dev\nunit\src\NUnitFramework\framework\Internal\TestFilter.cs:line 135
   at NUnit.Framework.Api.FrameworkController.RunTests(ICallbackEventHandler handler, String filter) in C:\dev\nunit\src\NUnitFramework\framework\Api\FrameworkController.cs:line 371
   at NUnit.Framework.Api.FrameworkController.RunTestsAction..ctor(FrameworkController controller, String filter, Object handler) in C:\dev\nunit\src\NUnitFramework\framework\Api\FrameworkController.cs:line 582
   --- End of inner exception stack trace ---

[CharliePoole]

If version 3.7 of the console runner is used to execute tests that reference version 3.6 of the framework, it should work - at least in the sense that the user gets an intelligent message. I take it that the message you are seeing is the result of crashing the runner rather than simply reporting an error. We could live with that if we had to but I'd rather see it improved.

Because you are new at this - or at least new to us - I suggest we get the basic framework fix in first, then the console/engine fix and finally, as a separate commit, deal with backward compatibility. That's going to be a bit complicated:

1. One approach is for NUnit3FrameworkDriver to have knowledge of the version of the framework where this was introduced and to analyze the passed in filter.

2. Another approach is for the console runner to have the same knowldege and not allow use of namespace if dealing with an older framework version.

   [Both of the above require a lot of information sharing across components that are supposed to be independent, so I really don't like them.]

3. A better approach, I think, is for the NUnit3FrameworkDriver to catch the exception, recognize it and return XML for a non-runnable assembly, with the appropriate error message. That will allow us to run tests against multiple assemblies referencing __different__ versions of the framework, an uncommon event but one we support.

4. Ideally, the action of alternative 3 would take place in the framework itself. We may want to do that for the future. But for the current issue of backward compatibility, we can't change the older frameworks.

Bottom line, I think the viable choices are 3, 4 or 3 plus 4. Personally, I'm leaning toward 4 and letting the existing crash take place if you use namespace with a pre-3.7 version of the framework. What do others think?

Additional work after all this: the NUnit2FrameworkDriver also has to give an error when you try to use namespace filter with it. That can be done in advance, without calling the framework, since no versions of NUnit V2 support this.

[JustinRChou]

How would it affect backward compatibility?

But I would go with 3.

[CharliePoole]

Issue #162 is labeled as blocked, by which we mean "cannot yet be worked on." Once that issue us fixed and merged, we can look at this PR, which will have to be updated to reflect the merged reference.

[CharliePoole]

Last comment was meant to go on the PR.

[CharliePoole]

@tom-dudley I'm closing this manually as with the earlier issue.

General rules on PRs for future reference.

• DO use either "Fixes" or "Closes" the issue number, so it autocloses the issue if that's your intent. Usually it is, but sometimes it isn't, in which case I do something like "Partial fix for #"
• DON'T put the issue number in the title of the PR.

[CharliePoole]

@tom-dudley You did nice work here. 😄

[algot]

Is namespace filter is working in the 3.6.1?

I have tests under namespace "Namespace1.Namespace2.Namespace3"
When I run following command filter returns 0 tests
c:\dev\nunit-console\nunit3-console.exe --labels=All Autotests.dll --where "namespace == Namespace1.Namespace2.Namespace3"

NUnit Console Runner 3.6.1
Runtime Environment
OS Version: Microsoft Windows NT 10.0.15063.0
CLR Version: 4.0.30319.42000

Test Files
Autotests.dll

Test Filters
Where: namespace == Namespace1.Namespace2.Namespace3

Test Run Summary
Overall result: Passed
Test Count: 0, Passed: 0, Failed: 0, Warnings: 0, Inconclusive: 0, Skipped: 0

[CharliePoole]

You need at least version 3.7 of both the console and framework to use the feature.

On 3.6.1, you can use "test == namespace1.namespace2.namespace3" instead. However, it will include tests in any subnamespace under namespace3 in addition. The namespace keyword was added to allow you to specify the exact namespace, excluding any subnamespaces.