Powershell: Nightly Builds failing after #5759

[markekraus]

5759 changes were merged without any [feature] tags for full testing. Now nightly and PR builds are failing.

Again, I'm asking that we start running feature tests by default in PRs. Trying to determine what is and is not a feature affecting change is proving to be pointless. If you are concerned about quick README updates, add a README tag to bypass feature tests. Otherwise, they should be on by default, IMO.

https://ci.appveyor.com/project/PowerShell/powershell-f975h

TEST FAILURES
Description: Validate Update-Help for module 'PSReadLine'
Name:        Validate Update-Help from the Web for all PowerShell Core modules...Validate Update-Help for module 'PSReadLine'
message:
Expected: {1}
But was:  {0}
stack-trace:
at line: 157 in C:\projects\powershell-f975h\test\powershell\engine\Help\UpdatableHelpSystem.Tests.ps1
157:     $helpFilesInstalled.Count | Should Be $expectedHelpFiles.Count
Description: Validate Update-Help for module 'PSReadLine'
Name:        Validate Update-Help -SourcePath for all PowerShell Core modules...Validate Update-Help for module 'PSReadLine'
message:
Expected: {1}
But was:  {0}
stack-trace:
at line: 157 in C:\projects\powershell-f975h\test\powershell\engine\Help\UpdatableHelpSystem.Tests.ps1
157:     $helpFilesInstalled.Count | Should Be $expectedHelpFiles.Count

https://travis-ci.org/PowerShell/PowerShell/jobs/331386328

TEST FAILURES
Description: Validate Update-Help from the Web for all PowerShell Core modules./
Name:        Validate Update-Help for module 'PSReadLine'
message:
Expected: {1}
But was:  {0}
stack-trace:
at line: 157 in /home/travis/build/PowerShell/PowerShell/test/powershell/engine/Help/UpdatableHelpSystem.Tests.ps1
157:     $helpFilesInstalled.Count | Should Be $expectedHelpFiles.Count
Description: Validate Update-Help -SourcePath for all PowerShell Core modules./
Name:        Validate Update-Help for module 'PSReadLine'
message:
Expected: {1}
But was:  {0}
stack-trace:
at line: 157 in /home/travis/build/PowerShell/PowerShell/test/powershell/engine/Help/UpdatableHelpSystem.Tests.ps1
157:     $helpFilesInstalled.Count | Should Be $expectedHelpFiles.Count
Description: ConsoleHost unit tests/Redirected standard input for 'interactive' use
Name:        Interactive redirected input: 
message:
Expected string length 7 but was 125. Strings differ at index 0.
Expected: {PS> 1+1}
But was:  {An error has occurred that was not properly handled. Additional information is shown below. The PowerShell process will exit.}
-----------^
stack-trace:
at line: 352 in /home/travis/build/PowerShell/PowerShell/test/powershell/Host/ConsoleHost.Tests.ps1
352:             $process.StandardOutput.ReadLine() | Should Be "PS> 1+1"
Description: ConsoleHost unit tests/Redirected standard input for 'interactive' use
Name:        Interactive redirected input:  -IntERactive
message:
Expected string length 7 but was 125. Strings differ at index 0.
Expected: {PS> 1+1}
But was:  {An error has occurred that was not properly handled. Additional information is shown below. The PowerShell process will exit.}
-----------^
stack-trace:
at line: 352 in /home/travis/build/PowerShell/PowerShell/test/powershell/Host/ConsoleHost.Tests.ps1
352:             $process.StandardOutput.ReadLine() | Should Be "PS> 1+1"
Description: ConsoleHost unit tests/Redirected standard input for 'interactive' use
Name:        Interactive redirected input:  -i
message:
Expected string length 7 but was 125. Strings differ at index 0.
Expected: {PS> 1+1}
But was:  {An error has occurred that was not properly handled. Additional information is shown below. The PowerShell process will exit.}
-----------^
stack-trace:
at line: 352 in /home/travis/build/PowerShell/PowerShell/test/powershell/Host/ConsoleHost.Tests.ps1
352:             $process.StandardOutput.ReadLine() | Should Be "PS> 1+1"
Description: ConsoleHost unit tests/Redirected standard input for 'interactive' use
Name:        Interactive redirected input w/ initial command
message:
Expected string length 7 but was 4. Strings differ at index 4.
Expected: {PS> 1+1}
But was:  {PS> }
---------------^
stack-trace:
at line: 372 in /home/travis/build/PowerShell/PowerShell/test/powershell/Host/ConsoleHost.Tests.ps1
372:             $process.StandardOutput.ReadLine() | Should Be "PS> 1+1"
Description: ConsoleHost unit tests/Redirected standard input for 'interactive' use
Name:        Redirected input explicit prompting (-File -)
message:
Expected string length 7 but was 125. Strings differ at index 0.
Expected: {PS> 1+1}
But was:  {An error has occurred that was not properly handled. Additional information is shown below. The PowerShell process will exit.}
-----------^
stack-trace:
at line: 388 in /home/travis/build/PowerShell/PowerShell/test/powershell/Host/ConsoleHost.Tests.ps1
388:             $process.StandardOutput.ReadLine() | Should Be "PS> 1+1"
Description: ConsoleHost unit tests/Redirected standard input for 'interactive' use
Name:        Redirected input no prompting (-Command -)
message:
Expected string length 1 but was 0. Strings differ at index 0.
Expected: {2}
But was:  {}
-----------^
stack-trace:
at line: 399 in /home/travis/build/PowerShell/PowerShell/test/powershell/Host/ConsoleHost.Tests.ps1
399:             $process.StandardOutput.ReadLine() | Should Be "2"
Description: ConsoleHost unit tests/Redirected standard input for 'interactive' use
Name:        Redirected input w/ nested prompt
message:
Expected string length 29 but was 4. Strings differ at index 4.
Expected: {PS> $host.EnterNestedPrompt()}
But was:  {PS> }
---------------^
stack-trace:
at line: 432 in /home/travis/build/PowerShell/PowerShell/test/powershell/Host/ConsoleHost.Tests.ps1
432:             $process.StandardOutput.ReadLine() | Should Be "PS> `$host.EnterNestedPrompt()"

https://travis-ci.org/PowerShell/PowerShell/jobs/331386331

TEST FAILURES
Description: Validate Update-Help from the Web for all PowerShell Core modules./
Name:        Validate Update-Help for module 'PSReadLine'
message:
Expected: {1}
But was:  {0}
stack-trace:
at line: 157 in /Users/travis/build/PowerShell/PowerShell/test/powershell/engine/Help/UpdatableHelpSystem.Tests.ps1
157:     $helpFilesInstalled.Count | Should Be $expectedHelpFiles.Count
Description: Validate Update-Help -SourcePath for all PowerShell Core modules./
Name:        Validate Update-Help for module 'PSReadLine'
message:
Expected: {1}
But was:  {0}
stack-trace:
at line: 157 in /Users/travis/build/PowerShell/PowerShell/test/powershell/engine/Help/UpdatableHelpSystem.Tests.ps1
157:     $helpFilesInstalled.Count | Should Be $expectedHelpFiles.Count
Description: ConsoleHost unit tests/Redirected standard input for 'interactive' use
Name:        Interactive redirected input: 
message:
Expected string length 7 but was 125. Strings differ at index 0.
Expected: {PS> 1+1}
But was:  {An error has occurred that was not properly handled. Additional information is shown below. The PowerShell process will exit.}
-----------^
stack-trace:
at line: 352 in /Users/travis/build/PowerShell/PowerShell/test/powershell/Host/ConsoleHost.Tests.ps1
352:             $process.StandardOutput.ReadLine() | Should Be "PS> 1+1"
Description: ConsoleHost unit tests/Redirected standard input for 'interactive' use
Name:        Interactive redirected input:  -IntERactive
message:
Expected string length 7 but was 125. Strings differ at index 0.
Expected: {PS> 1+1}
But was:  {An error has occurred that was not properly handled. Additional information is shown below. The PowerShell process will exit.}
-----------^
stack-trace:
at line: 352 in /Users/travis/build/PowerShell/PowerShell/test/powershell/Host/ConsoleHost.Tests.ps1
352:             $process.StandardOutput.ReadLine() | Should Be "PS> 1+1"
Description: ConsoleHost unit tests/Redirected standard input for 'interactive' use
Name:        Interactive redirected input:  -i
message:
Expected string length 7 but was 125. Strings differ at index 0.
Expected: {PS> 1+1}
But was:  {An error has occurred that was not properly handled. Additional information is shown below. The PowerShell process will exit.}
-----------^
stack-trace:
at line: 352 in /Users/travis/build/PowerShell/PowerShell/test/powershell/Host/ConsoleHost.Tests.ps1
352:             $process.StandardOutput.ReadLine() | Should Be "PS> 1+1"
Description: ConsoleHost unit tests/Redirected standard input for 'interactive' use
Name:        Interactive redirected input w/ initial command
message:
Expected string length 7 but was 4. Strings differ at index 4.
Expected: {PS> 1+1}
But was:  {PS> }
---------------^
stack-trace:
at line: 372 in /Users/travis/build/PowerShell/PowerShell/test/powershell/Host/ConsoleHost.Tests.ps1
372:             $process.StandardOutput.ReadLine() | Should Be "PS> 1+1"
Description: ConsoleHost unit tests/Redirected standard input for 'interactive' use
Name:        Redirected input explicit prompting (-File -)
message:
Expected string length 7 but was 125. Strings differ at index 0.
Expected: {PS> 1+1}
But was:  {An error has occurred that was not properly handled. Additional information is shown below. The PowerShell process will exit.}
-----------^
stack-trace:
at line: 388 in /Users/travis/build/PowerShell/PowerShell/test/powershell/Host/ConsoleHost.Tests.ps1
388:             $process.StandardOutput.ReadLine() | Should Be "PS> 1+1"
Description: ConsoleHost unit tests/Redirected standard input for 'interactive' use
Name:        Redirected input no prompting (-Command -)
message:
Expected string length 1 but was 0. Strings differ at index 0.
Expected: {2}
But was:  {}
-----------^
stack-trace:
at line: 399 in /Users/travis/build/PowerShell/PowerShell/test/powershell/Host/ConsoleHost.Tests.ps1
399:             $process.StandardOutput.ReadLine() | Should Be "2"
Description: ConsoleHost unit tests/Redirected standard input for 'interactive' use
Name:        Redirected input w/ nested prompt
message:
Expected string length 29 but was 4. Strings differ at index 4.
Expected: {PS> $host.EnterNestedPrompt()}
But was:  {PS> }
---------------^
stack-trace:
at line: 432 in /Users/travis/build/PowerShell/PowerShell/test/powershell/Host/ConsoleHost.Tests.ps1

PR build failures:

• https://travis-ci.org/PowerShell/PowerShell/jobs/331469515#L4276
• https://travis-ci.org/PowerShell/PowerShell/jobs/331469516#L3427

[iSazonov]

I don't see it as a problem. The idea of using Feature test is to make the build go fast.
I would prefer our daily tests to have better coverage.

[markekraus]

You don't see it as a problem that so much time is wasted when a feature affecting change is merged without feature tests passing?

It is a tremendous waste of time. I don't want fast builds. I want things merged without other things breaking and the way we do that is with thorough testing. We should absolutely switch around this project to do quick builds as the exception, not the rule.

The alternative is to have the maintainers be more diligent about this and triple check that a PR is feature affecting and has passed feature tests with the most recent commit. But that has proven a bad strategy as we continually have stuff merged that breaks feature tests (whether it's actually a regression or a test issue requires time an resources to figure out).

If our average turnaround for PRs was in the realm of hours, I might agree that fast tests make sense... but they aren't. Most PRs span days if not weeks. We introduced CI changes to stop existing tests when a new push is made (rolling builds) so multiple commits in a short period have no affect.

I'm pretty annoyed that I get work delayed and pushed back because someone merged a test breaking change. It's not the odd occurrence.. it happens way too often. Most of my changes require feature tests so I'm stuck with PRs that can't merge due to failures that shouldn't be there in the first place.

If the maintainers can't stay on top of this, then we absolutely should switch to feature tests as the rule and bypass as an option.

[iSazonov]

I believe that the main problem is that MSFT allocates too little resources to this project. Therefore quality suffers, we're too slow.

[Jaykul]

This seems to fall into the old agile maxim: "slow down to go fast"

It doesn't matter how fast your CI build is, if it's not testing the right things.

I mean, in this case, skipping tests in the non-feature build makes that slightly faster -- but dramatically slows down feature changes, because on the occasions where those builds fail due to unrelated changes, the authors have to struggle to track down the real source of the problem...