Pester: PSCore compatibility - Windows, Linux, macOS

[vors]

Hi team,

Are there any plans on Linux compatibility that could work with https://github.com/PowerShell/PowerShell ?
There is a hacky psl-pester fork https://github.com/PowerShell/psl-pester#note and we would like to remove it and start use the official pester.

[nohwnd]

We would love to have Pester compatible with Linux and OS X. Are there any breaking changes that we need to make to incorporate your changes? I did not go through your changes yet, could you just quickly summarize the main changes that needed to be done? (if you have that in your head, otherwise I will investigate myself).

[dlwyatt]

I went through the work of comparing the forks at one point, but most of the changes were things that were no longer needed due to work progressing on PowerShell itself. (Backslashes were changed to forward slashes, Add-Type was removed, etc.)

At this point, I'd recommend just running the PowerShell tests with the official version of Pester to see what breaks, rather than trying to sort out which changes in psl-pester are still needed.

[nohwnd]

Ok thanks. I run Pester on OS X the other day and roughly 50 of the tests failed, but I might have been running the V4 version, not really sure. I suppose we should, wrap up the V4 version somehow, and finally release it. And then start on Linux / OSX compatibility. At the moment all the v4 tests seem to pass, the only thing I am not sure about is Describes in Describes. That feature works totally different from what I would expect, and I am sure it would take a lot of effort to get working as a real cascade with proper mock scopes and so on.

That is not to say the work you done on it is not valued. The total opposite. The developement you've done on version 4 is truly amazing, and that's why I would like to see it released, before we get caught up in adding new features.

[vors]

I totally think we should not cherry-pick changes from psl-pester fork. As I said, it was created at the early times and quite hacky.
I could not adequately evaluate current compatibilities, because of #641

As part of enabling Linux compatibility it would be useful to get Linux builds in CI. Do we have this option in TeamCity?

[vors]

Polite ping :)
I did a quick run of pester tests on MacOS and it seems that thing became much better!
It's mostly tests that need to be fixed to avoid using C: drive and such.

[it-praktyk]

@nohwnd, @dlwyatt is something that can be added for Pester v4.1?

I've recently added Linux compatibility to https://github.com/it-praktyk/new-outputobject so I can try play with Pester in this area too.

[it-praktyk]

@Jaykul, did you check an option to add support for Gherkin on PSCore?

[it-praktyk]

I've checked if Gherkin tests can be used on Linux. Unfortunately currently not due to lack of System.Web.Extensions (errors attached below).

I assume that the PowerShell Core 6.0.0 beta 5 uses .Net Core 1.1 Missed System.Web.* maybe will be available in .Net Core 2.0 - documentation is incomplete yet.

PS <FOLDER_PATH>/Pester> Invoke-Pester -Script ./Functions/Gherkin.Tests.ps1
Executing all tests in './Functions/Gherkin.Tests.ps1'

Executing script ./Functions/Gherkin.Tests.ps1
WARNING: OS Information retrieval is not possible, reports will contain only partial system data
Exception calling "Parse" with "1" argument(s): "Could not load file or assembly 'System.Web.Extensions, Version=3.5.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35'. The system cannot find the file
specified.
"
    + CategoryInfo          : NotSpecified: (:) [Import-GherkinFeature], MethodInvocationException
    + FullyQualifiedErrorId : FileNotFoundException,Import-GherkinFeature
    + PSComputerName        : localhost

Exception calling "Parse" with "1" argument(s): "Could not load file or assembly 'System.Web.Extensions, Version=3.5.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35'. The system cannot find the file
specified.
"
    + CategoryInfo          : NotSpecified: (:) [Import-GherkinFeature], MethodInvocationException
    + FullyQualifiedErrorId : FileNotFoundException,Import-GherkinFeature
    + PSComputerName        : localhost

Cannot bind argument to parameter 'ReferenceObject' because it is null.
    + CategoryInfo          : InvalidData: (:) [Compare-Object], ParameterBindingValidationException
    + FullyQualifiedErrorId : ParameterArgumentValidationErrorNullNotAllowed,Microsoft.PowerShell.Commands.CompareObjectCommand
    + PSComputerName        : localhost

<Output partially ommited>

[it-praktyk]

Draft of documentation

Known limitation of Pester on PowerShell Core

• Pester unit tests for test a code coverage of DSC resources can't be executed - DSC is not supported on MacOSX, support on Linux requires additional software to be installed

• Gherkin tests don't work on Linux - even when newer libraries are used #866

[Jaykul]

I had to backport gherkin.dll to target .NET 3.5 in order to support PowerShell 2 because that's what Pester was prioritizing -- so no, it doesn't work in .NET Core.

However, The Official Gherkin.dll is .NetCoreApp 1.1

It should work fine in core -- but we would have to ship separate assemblies to maintain .Net 3.5 compatibility.

[vors]

Since 2.0 is going to be deprecated in the fall, should .net-core compatibility be a higher priority than 3.5 compatibility?

[nohwnd]

@vors I think we can just ship two assemblies and check which .NET powershell is using and select the correct assembly to load.

[Jaykul]

And complain, YET AGAIN to the PowerShell team that we have to write a bunch of really dumb code that nobody should have to write. Not to mention shipping multiple versions of a product in one package, because their packaging system can't help either.

[nohwnd]

@Jaykul You can write an angry email and have it ignored, or spend the time to add the decision and both assemblies in the repo, and post a PR :D (Or both!)

[it-praktyk]

Status update

Pester v4.0.6-RC (+ PR #831) updated in the way that all Pester tests pass on

• Linux - PSCore 6.0.0-beta.5
• Windows - PowerShell 5.1.15063.502

On Linux some tests need to be reverified - they are on the TO DO list below.

On Windows - PSCore 6.0.0-beta.6 - some tests fail - they are on TO DO list below.

Gherkin-style tests work on Windows (classic PowerShell and PSCore), fail on Linux, macOS required testing.

The updated code available: https://github.com/it-praktyk/Pester/tree/DevV4-1_Linux_compatibility_2.

Update 2017-09-10
Work will be continued in the branch https://github.com/pester/Pester/tree/Dev4-1_PSCore_compatibility .

TO DO

General

• [x] replace \nand/nwith[System.Environment]::NewLine` - the related issue #838 - mostly done

on Linux/MacOS

• [x] testing required

C:\ drive cases

• [x] verify/correct the test Mock.Tests.ps1\When mocking a command that has an ArgumentList parameter with validation

• [x] verify/correct the test Ouput.Tests.ps1\Format-PesterPath

• [x] verify/correct the test Output.Tests.ps1\Write-PesterStart

• [x] verify/correct the test TestResults.Tests.ps1\should write a failed test result - doesn't need to be changed

• [x] verify/correct the test [TestResults.Tests.ps1\Exporting Parameterized Tests (Newer format)] (https://github.com/it-praktyk/Pester/blob/fc4f760505b5520f7d3059ad4572e02c3c723ea5/Functions/TestResults.Tests.ps1#L168-L229)

HKLM PSDrive case

• [x] verify/correct the test Mock.Tests.ps1\Mocking Get-ItemProperty

on PSCore on Windows

• [x] verify/correct the test Output.Tests.ps1\ConvertTo-FailureLines - the related issue #838

• [x] handle the case loading of DSC module on PSCore (more than one module exists) Coverage.Tests.ps1
  It resolved "magically" (at least for me) in PowerShell Core 6.0.0-beta.7

  Describing Analyzing coverage of a DSC configuration
    [+] Has the proper number of breakpoints defined 292ms
    [-] Error occurred in Describe block 254ms
      ParseException: At C:\temp\6f07d56f-3035-4ecd-9be6-a8d2135a4eb6\TestScriptWithConfiguration.ps1:6 char:17
      + ...             Import-DscResource -ModuleName PSDesiredStateConfiguratio ...
      +                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      Multiple versions of the module 'PSDesiredStateConfiguration' were found. You can run 'Get-Module -ListAvailable -FullyQualifiedName PSDesiredStateConfiguration' to see available versions on the system, and then use the fully qualified name '@{ModuleName="PSDesiredStateConfiguration"; RequiredVersion="Version"}'.
      at <ScriptBlock>, <FILE_PATH>\Pester\Functions\Coverage.Tests.ps1: line 325
      at DescribeImpl, <FILE_PATH>\Pester\Functions\Describe.ps1: line 161

[it-praktyk]

@vors, can you test on MacOS, please?

[vors]

Not to mention shipping multiple versions of a product in one package, because their packaging system can't help either.

@Jaykul well, have you seen this?

    Directory: /usr/local/microsoft/powershell/6.0.0-beta.5/Modules/PackageManagement/1.1.4.0/coreclr


Mode                LastWriteTime         Length Name                                                                   
----                -------------         ------ ----                                                                   
--r---          5/24/17   9:03 PM          86184 Microsoft.PackageManagement.ArchiverProviders.dll                      
--r---          5/24/17   9:03 PM          73384 Microsoft.PackageManagement.CoreProviders.dll                          
--r---          5/24/17   9:03 PM         271528 Microsoft.PackageManagement.dll                                        
--r---          5/24/17   9:03 PM          84648 Microsoft.PackageManagement.MetaProvider.PowerShell.dll                
--r---          5/24/17   9:03 PM         174760 Microsoft.PackageManagement.NuGetProvider.dll                          
--r---          5/24/17   9:03 PM         182440 Microsoft.PowerShell.PackageManagement.dll                             


    Directory: /usr/local/microsoft/powershell/6.0.0-beta.5/Modules/PackageManagement/1.1.4.0/fullclr


Mode                LastWriteTime         Length Name                                                                   
----                -------------         ------ ----                                                                   
--r---          5/24/17   9:03 PM          86696 Microsoft.PackageManagement.ArchiverProviders.dll                      
--r---          5/24/17   9:03 PM          75432 Microsoft.PackageManagement.CoreProviders.dll                          
--r---          5/24/17   9:03 PM         298664 Microsoft.PackageManagement.dll                                        
--r---          5/24/17   9:03 PM          85160 Microsoft.PackageManagement.MetaProvider.PowerShell.dll                
--r---          5/24/17   9:03 PM         252584 Microsoft.PackageManagement.MsiProvider.dll                            
--r---          5/24/17   9:03 PM          30376 Microsoft.PackageManagement.MsuProvider.dll                            
--r---          5/24/17   9:03 PM         173736 Microsoft.PackageManagement.NuGetProvider.dll                          
--r---          5/24/17   9:03 PM         181928 Microsoft.PowerShell.PackageManagement.dll  

[nohwnd]

@it-praktyk I updated my powershell on MacOS to the latest version, and pulled your changes over the latest master. There is one test that fails. The test is pretty new, I merged it in last week imho, check if you are in sync with the latest master :)

The test should reach recursion limit of 100 and fail, related issue #785, and pr #824. Should I have a look on this or do you want to dig into it yourself?

Name                           Value
----                           -----
PSVersion                      6.0.0-beta
PSEdition                      Core
GitCommitId                    v6.0.0-beta.5

  [-] throws exception when self-imposed recursion limit is reached 343ms
      Expected: the expression to throw an exception with message {recursion depth limit}, an exception was raised, message was {The script failed due to call depth overflow.}
          from /Users/nohwnd/projects/pester_main/Functions/Assertions/Be.Tests.ps1:99 char:21
          +             { $a1 | Should be $a2 } | Should throw 'recursion depth l ...
          +                     ~~~~~~~~~~~~~
      99:             { $a1 | Should be $a2 } | Should throw 'recursion depth limit'
      at Invoke-LegacyAssertion, /Users/nohwnd/projects/pester_main/Functions/Assertions/Should.ps1: line 190
      at <ScriptBlock>, /Users/nohwnd/projects/pester_main/Functions/Assertions/Be.Tests.ps1: line 99

[vors]

Corresponding PR in PowerShell repo https://github.com/PowerShell/PowerShell/pull/4618

PowerShell test suite is a pretty big one and I noticed few new problems, compare to the psl-pester 3.x.x fork.

1. AppVeyor (windows) always (3 out of 3) times out. It likely indicates a hang. I.e. https://ci.appveyor.com/project/PowerShell/powershell/build/6.0.0-beta.5-4775
   To be fair, there is https://github.com/PowerShell/PowerShell/issues/3670 but these hangs seems different.

2. Travis complains about exceeding 4 Mb output limit https://travis-ci.org/PowerShell/PowerShell/jobs/266226694

3. This could be an explanation for the previous one. Tests output a bunch of new lines and maybe padding of spaces (when tried to do copy paste of pester output on macos it seems to be the case).
   

[it-praktyk]

@jpsnover, @TravisEz13, @dantraMSFT

We are almost done with porting Pester to PowerShell Core but we need help with handling DSC. If someone from Microsoft can help us?

Thank you in advance.

CC: @vors, @KarolKaczmarek

[vors]

@it-praktyk jfyi @vors @KarolKaczmarek are no longer with MSFT.
cc @SteveL-MSFT

[SteveL-MSFT]

@TravisEz13 can help after he's freed up from his current priorities

[Jaykul]

Can I just ask why @it-praktyk is having to do all the cross-platform stuff in his own repo and ask people to go help him do the work there?

[it-praktyk]

@Jaykul good questions.

My answers

why @it-praktyk is having to do all the cross-platform stuff

Because

• I can
• I have fun with it

in his own repo

Now, when almost (as I hope) it's done, I created a separate branch in Pester repo https://github.com/pester/Pester/tree/Dev4-1_PSCore_compatibility

and ask people to go help him do the work there?

IMHO it's power of open source projects that I can ask for help :-) DSC stuff it's too difficult for me or I'm just too tired (I'm going to go on holidays in October?).

[it-praktyk]

@vors

jfyi @vors @KarolKaczmarek are no longer with MSFT.

We are open for your contributions too :-P

[it-praktyk]

@SteveL-MSFT

@TravisEz13 can help

Thank you in advance.

after he's freed up from his current priorities

What?! Pester is not his priority?! ;-)

[it-praktyk]

Please be aware that currently some DSC-related changes are not merged to the branch Dev4-1_PSCore_compatibility - the awaiting pull request #869, due to bug - the issue #870.

I'll try to resolve it soon.

[vors]

@it-praktyk sure :D I was just replying to

If someone from Microsoft can help us?

[it-praktyk]

The new - at least for me - information about Desired State Configuration Core (DSC Core) - the blog post at the PowerShell Team Blog, DSC Future Direction Update.

[TravisEz13]

@it-praktyk There are several known issues with DSC and PowerShell Core. What kind of help are you needing with DSC?

[it-praktyk]

@TravisEz13, thank you for contact.

Generally, after when I read the blog post DSC Future Direction Update I assumed that it's too early to try fully support DSC on PSCore by Pester.

I hope that in the future - when DSC on PSCore will be more mature - you will try to improve support for it in Pester too.

For now - I'll just add a condition to not perform DSC-related tests for a code coverage and I'll add this to the list of limitations.

Can you recommend me any place where I can follow the progress of work related to the DSC?

[it-praktyk]

@JamesWTruher, your name was mentioned here #888 so I've checked your GitHub profile. I think that you can be interested in the current thread too.

[TravisEz13]

@it-praktyk The blog is the best channel I can think of right now. Message me on Gitter or Twitter (@TravisPlunk) and I'll see if I can get your connected with the DSC team.

[it-praktyk]

Today, for the first time, the Pester code from the branch Dev4-1_PSCore_compatibility passed standard Pester tests on

• Windows PowerShell 2.0 - 5.1
• Linux PowerShell 6.0.0-beta.8

On macOS (PowerShell 6.0.0-beta.8) still one tests (reported previously by @nohwnd) fails, but IMHO it's more PowerShell itself issue than Pester.

Gherkin is not supported on PSCore yet - is an explicitly excluded.

[vors]

@it-praktyk super cool! Imho it's ok to exclude this test on macOS with a comment about the issue bug number, so it could be merged into master and officially released.

[nohwnd]

@it-praktyk Super cool indeed. :) Please exclude the test, and then we can merge it. Next week I hopefully will have some time to get back to Pester, so I can start solving other issues and so on.

[it-praktyk]

Today I've prepared the new branch https://github.com/it-praktyk/Pester/tree/PSCore_compatibility_clean what will be a source of a pull request.

For the test that fails on macOS I'm not sure if should we skip it or set as inconclusive.

[it-praktyk]

I just created the pull request #925 - Update Pester to work on PowerShell Core at Windows, Linux, macOS.

The failing on macOS test is excluded.

[it-praktyk]

I just updated Pester wiki

• Pester on PSCore - limitations
• Development rules - technical

[Jaykul]

Why have you disabled Gherkin?

[it-praktyk]

@Jaykul, I don't understand why you are surprised.

I informed about issues with Gherkin related code on PSCore, mentioned you directly, and even sent you an email with a help request.

Please read

• #639
• #870
• https://github.com/Jaykul/Pester/pull/1
• email

Subject: Gherkin fails when Set-StrictMode is set - can you help?
Date: Fri, 29 Sep 2017 14:16:09 +0200
From: Wojciech Ściesiński
To: Jaykul

[nohwnd]

Achievement unlocked, thanks to everyone involved, mainly @it-praktyk !