Nunit-console: Run (or fail on) uncategorised tests

[bacar]

We use categories to run different types of tests at different points of our build pipeline, but I would like a way to ensure that we do not accidentally miss any tests simply because a developer forgot to assign a category.

We're currently doing that by using category exclusion lists, but since we have many categories, these are getting a bit unwieldy.

Is there a way to:

• include all uncategorized tests, or
• immediately fail if there are any uncategorized tests?

Either mechanism would allow me to run uncategorized tests in a particular invocation by default.

[CharliePoole]

There is no specific feature for selecting all uncategorized tests - the question has never come up before.

If you have a specific set of categories thar are always used, you could do something like:

--where "cat!=CAT1 && cat != CAT2 && cat != CAT3"

This would pick up uncategorized tests as well as those that used a mis-spelled category.

[bacar]

Thanks. Yeah that's basically what we're doing at the moment - but it's a bit cumbersome when the list of categories gets long.

[CharliePoole]

You can put the expression in a file named (for example) UNCATEGORIZED. Put the whole thing on a single long line. Then use @UNCATEGORIZED in your command line.

Do you have a suggestion for a syntax we might support in our Test Selection Language for this feature?

[bacar]

Hm. Using a file would help actually. It's also remembering to maintain the file when a new category is added that's partly the issue.

How about e.g.:

cat is empty (feed the poor cat!)
cat is uncategorised
test is uncategorised

[HollyDOL]

I would like this condition as well..

cat is null
cat not null

'empty' sounds way too much like Category("") which is not equivalent to no Category.

[CharliePoole]

If we are accepting Category("") that's a bug!

I see two ways to go:

1. Introduce a new comparison operator like is, which is followed by one or more key words to give the precise meaning. For this issue with cat's, I would like cat is null.

2. Introduce a new keyword that doesn't require a comparison operator, like cat missing or cat null

Negation does not require anything new. If, for example, we accept cat is null then !(cat is null) will work without any further change.

What do people think is best between the two approaches?

[HollyDOL]

I would suggest either 'is' operator or simply
!cat
as it would be easier to define in the grammar.

[ChrisMaddock]

I think 'is' gives us more room for future expansion.

[CharliePoole]

It also fits into the existing grammar as an operator.

[jnm2]

I like is undefined or is unspecified the best. null is a catch-all term and it's okay here, but undefined or unspecified seem more communicative to me.

[CharliePoole]

Which do we like best... cat is null or cat is missing ?

[jnm2]

is missing almost sounds like something went wrong.

My order of preference:

• unspecified
• undefined
• missing
• null

[CharliePoole]

@jnm2 You beat me to it by 14 seconds!

I like is unspecified, which even allows for is specified. Other option is missing versus present. We should pick one, of course.

[tom-dudley]

If we're going the route if having is specified would is not specified make more sense? I could imagine this would get parsed as <not><specified><category/></specified></not> for free.

[CharliePoole]

It's a little outside of the language we already parse, since we don't recognize not coming after any other operators. So it would not be picked up automatically, but it could be handled specially after "is".

[richard-fine]

Did cat is unspecified, or something similar to it, ever get implemented? I don't see it mentioned in the documentation.

[CharliePoole]

It has been available from the beginning of NUnit 3. See https://docs.nunit.org/articles/nunit/running-tests/Test-Selection-Language.html

[CharliePoole]

Sorry, I didn't read closely enough. Nothing like cat is unspecified is implemented... otherwise this issue would be closed. :smile:

@ChrisMaddock This has been categorized as design since 2017. From reading old comments by myself and others, I guess the design required is to pick a syntax.

BTW, I notice somebody posted about a particular design being basically "for free." We know there's no such thing. Adding is as an operator is one thing. Adding not would be an additional operator after is, requiring a bit of additional effort. I think is plus some word specifying a quality is probably the easiest approach if we want to do this.

Of course, the simplest of all is to turn it down. :smiley:

Going slightly meta here... my understanding is that we normally let design and programming proceed together and review both when the PR is done. We add the design label if we want to see some sort of design before the work starts. Right?

[ChrisMaddock]

I think we should close this. This issue's been open for over four years now, with nobody yet willing to take it on.

If anyone is interested to do a pull request for this (@richard-fine?), I think it's a decent idea, and more than happy to talk through the design.