Nunit-console: Invalid assemblies no longer give an error message

[CharliePoole]

In the NUnit3 console runner 3.7, the argument x.dll gives an error message indicating the file was not found. Typing x.junk gives a message saying that the file type is not known.

In the current master build, the run fails without specifying a message in both cases.

[CharliePoole]

As discussed in PR #177, assembly-level errors like file not found or invalid file type stopped displaying in the console runner with the merging of #326. This is due to the fact that the assembly level normally does not have a site associated with it.

An easy fix was tried but has the drawback that it causes all suites containing a test with a warning, either directly or indirectly, to display as well. That problem is due to the fact that warnings never have a site associated with them either.

This will be a two part fix. The framework has to be updated so that a warning result caused by a child test uses FailureSite.Child In fact, it's essential that any failure or warning of a suite caused by a child test use that value explicitly.

However, even when the framework is updated, the console has to be able to deal with older framework versions that do not set FailureSite correctly. That's what the current issue has to do. Specifically, this issue needs to...

1. Ensure that assembly errors, failures and warnings are shown in the errors, failures and warnings report

2. Ensure that warnings are also shown in the report __only once__, that is, not repeated for the suites that contain them

I'm replicating this same issue to the framework repository for the work that is needed there.

[rprouse]

@CharliePoole this sounds important enough to get in the 3.10 release. You've assigned to yourself, do you have a timeline?

I am putting it in the milestone so we can track it. Please remove if you feel the priority is lower.

[CharliePoole]

PR shortly.

[CharliePoole]

@rprouse 3.10 release?

[CharliePoole]

@rprouse I marked this "critical" but with a twist from the normal definition. I don't think we need a hotfix, because the buggy code has never been released. I just don't think you should release with the bug in place since an invalid file, missing file or unsupported project type will all give a failure without any explanatory message. That seems like a reasonable extension of the definition of "critical."

[rprouse]

Sorry, mixing up framework and console. 3.8 release which was planned for next week, but I will hold off if necessary.

[CharliePoole]

I was waiting for travis to pass before doing the PR on this one, but I guess I'll stop doing that. 😄

In practice, the framework fix should be looked at before the console fix, so we don't have to fix the console twice. I'll move to that next.

[CharliePoole]

UPDATE: This turns out to not be an issue, since we only display ignored test cases.
Hmmm... looking at the framework, I am suspicious that we may have a similar problem with ignored tests as we have with warnings... the suite containing the ignored test gets ResultState.Ignored and FailureSite.Child is not specified. I'll run a test to see and update this PR if necessary.