Pester: v5: Discussion

[nohwnd]

🙋‍♀️

I am raising a few questions in v5 readme and this is the place to start discussion so you don't have to go through all the respective issues and ask there.

[rdbartram]

I like the idea of Add-Dependency to simplify importing scripts, but what about modules? I had some test classes I created for pester testing vmware PowerCLi but I had to mark them as .ps1 and use add-dependency in the beforeall in order that they worked. Previously, i had them as using module statements at the top of the .test.ps1 but that doesn't work anymore.

Could we do something with Add-Dependency so it could import modules/classes?

Where is the code for Add-Dependency? If I use get-command, I see its in the pester.runtime. Where is that?

Finally, I'm abit confused if we should be using Add-Dependency or not because here #1283, you say using beforeall and not add-dependency, but in the readme you say using Add-Dependency. Thanks for the clarification

[nohwnd]

@rdbartram when I wrote Add-Dependency I made it do the same thing as ., but then I realized that I am not using it as such. Instead I am using the Add-Dependency in the same manner as BeforeAll, so the goal of the issue is to use the same syntax so there is one less cmdlet to think about.

I don't know if I am able to do anything about using module, so that is still something I need to research.

All in all the readme is going a bit outdated, as I go and form how it will all work. :)

[rdbartram]

ok. so i'll stick to dot sourcing in the beforeall as in v4. Is a cool idea though, to simplify the dependencies. Could even make it something like frontmatter at the start of the tests.ps1

[Stephanevg]

After writing this, I am not 100% sur if this should be in this thread.
Although the -Passthru object format is not mentionned in the readme document, I think it would make sense to discuss it a bit, as the change is pretty heavy compared to the v4 one. If you prefere having me open a issue of it own to track this just let me know.

Hi,
I looked at the output of the -Passthru object a bit today, and there is really a very big difference with the v4 one.

Describe "simple test" {
    it 'Should be false' {
        $true | Should be $false
    }

    it 'Should be true' {
        $true | should be $true
    }
}

Currently, executing the following test in v4 will result in this output:

pester v4.X output

TagFilter         : 
ExcludeTagFilter  : 
TestNameFilter    : 
ScriptBlockFilter : 
TotalCount        : 2
PassedCount       : 1
FailedCount       : 1
SkippedCount      : 0
PendingCount      : 0
InconclusiveCount : 0
Time              : 00:00:00.2968309
TestResult        : {@{ErrorRecord=Expected $false, but got $true.; ParameterizedSuiteName=; Describe=simple test; Parameters=System.Collections.Specialized.OrderedDictionary; Passed=False; Show=All; 
                    FailureMessage=Expected $false, but got $true.; Time=00:00:00.0120719; Name=Should be false; Result=Failed; Context=; StackTrace=at <ScriptBlock>, 
                    C:\Users\taavast3\OneDrive\Repo\Projects\OpenSource\pesterPR\Pester\simpletest.ps1: line 3
                    3:         $true | Should be $false}, @{ErrorRecord=; ParameterizedSuiteName=; Describe=simple test; Parameters=System.Collections.Specialized.OrderedDictionary; Passed=True; Show=All; 
                    FailureMessage=; Time=00:00:00.0058684; Name=Should be true; Result=Passed; Context=; StackTrace=}}

pester v5 output

Content              : C:\Users\taavast3\OneDrive\Repo\Projects\OpenSource\pesterPR\Pester\simpletest.ps1
Type                 : File
PSTypename           : ExecutedBlockContainer
OneTimeTestSetup     : 
Focus                : False
DiscoveryDuration    : 00:00:00.2508997
ExecutedAt           : 9/28/2019 9:19:08 AM
EachTestTeardown     : 
Id                   : 
ShouldRun            : True
PluginData           : {}
ScriptBlock          : 
AggregatedPassed     : False
Tests                : {}
FrameworkData        : {PreviouslyGeneratedTests, PreviouslyGeneratedBlocks}
Passed               : True
IsRoot               : True
OwnDuration          : 00:00:00
BlockContainer       : @{Content=C:\Users\taavast3\OneDrive\Repo\Projects\OpenSource\pesterPR\Pester\simpletest.ps1; Type=File}
FrameworkDuration    : 00:00:00.5209437
Root                 : @{OneTimeTestSetup=; First=True; Focus=False; DiscoveryDuration=00:00:00.2508997; ExecutedAt=9/28/2019 9:19:08 AM; EachTestTeardown=; Id=; ShouldRun=True; 
                       PluginData=System.Collections.Hashtable; ScriptBlock=; AggregatedPassed=False; Tests=System.Collections.Generic.List`1[System.Object]; Name=Root; 
                       FrameworkData=System.Collections.Hashtable; Passed=True; IsRoot=True; OwnDuration=00:00:00; BlockContainer=; FrameworkDuration=00:00:00; Root=; OneTimeBlockTeardown=; 
                       Blocks=System.Collections.Generic.List`1[System.Object]; Exclude=False; OneTimeTestTeardown=; Executed=True; OneTimeBlockSetup=; EachBlockSetup=; Path=; EachTestSetup=; 
                       EachBlockTeardown=; Last=True; StandardOutput=; Tag=; Parent=; Duration=00:00:00; ErrorRecord=System.Collections.Generic.List`1[System.Object]}
OneTimeBlockTeardown : 
Blocks               : {@{OneTimeTestSetup=; First=True; Focus=False; DiscoveryDuration=00:00:00.0230244; ExecutedAt=9/28/2019 9:19:09 AM; EachTestTeardown=; Id=0; ShouldRun=True; 
                       PluginData=System.Collections.Hashtable; ScriptBlock=
                           it 'Should be false' {
                               $true | Should be $false
                           }

                           it 'Should be true' {
                               $true | should be $true
                           }
                       ; AggregatedPassed=True; Tests=System.Collections.Generic.List`1[System.Object]; Name=simple test; FrameworkData=System.Collections.Hashtable; Passed=True; IsRoot=False; 
                       OwnDuration=00:00:00.0137001; BlockContainer=; FrameworkDuration=00:00:00.3347260; Root=; OneTimeBlockTeardown=; Blocks=System.Collections.Generic.List`1[System.Object]; Exclude=False; 
                       OneTimeTestTeardown=; Executed=True; OneTimeBlockSetup=; EachBlockSetup=; Path=System.String[]; EachTestSetup=; EachBlockTeardown=; Last=True; StandardOutput=; Tag=System.String[]; 
                       Parent=; Duration=00:00:00.0137001; ErrorRecord=System.Collections.Generic.List`1[System.Management.Automation.ErrorRecord]}}
Exclude              : False
OneTimeTestTeardown  : 
Executed             : True
OneTimeBlockSetup    : 
EachBlockSetup       : 
EachTestSetup        : 
EachBlockTeardown    : 
Duration             : 00:00:00.0137001
ErrorRecord          : {}

I have written quite some automation around the object that -Passthru outputs in v3 / v4 and I was wondering if the object currently returned in v5 would be definitive one?

as much as I love all that extra information we get on the details of every run, I miss some of the information I have been used to through the different versions of pester.
Although that going up a major version we allow our selfs to introduce some breaking changes, I am a bit afraid of how much of my automation and CI checks I will need to change.

Also, it seems like that the information contained in the v5 object contains the details of the internal PesterRun.

Discussion

The part that would be really missing I think is the header of the v4 object.

TagFilter         : 
ExcludeTagFilter  : 
TestNameFilter    : 
ScriptBlockFilter : 
TotalCount        : 2
PassedCount       : 1
FailedCount       : 1
SkippedCount      : 0
PendingCount      : 0
InconclusiveCount : 0
Time              : 00:00:00.2968309
TestResult: 

I know that my self (and quite a lot of people I work with) use something like this to see if all tests passed.

$PesterRun = Invoke-Pester .\SimpleTest.ps1 -PassThru

If($PesterRun.FailedCount -ge 1){
  # Query failed pestertests from $PesterRun.TestsResults
}

This might be a naïve thought, but would it make sense to simply add a property to the existing v4 object, and put the new object of v5 in that one?
Something like RunDetails perhaps ?

Implementing the -Passthru as described above, could result in a -Passthruobject that would look like this

TagFilter         : 
ExcludeTagFilter  : 
TestNameFilter    : 
ScriptBlockFilter : 
TotalCount        : 2
PassedCount       : 1
FailedCount       : 1
SkippedCount      : 0
PendingCount      : 0
InconclusiveCount : 0
Time              : 00:00:00.2642866
TestResult        : {@{ErrorRecord=Expected $false, but got $true.; ParameterizedSuiteName=; Describe=simple test; Parameters=System.Collections.Specialized.OrderedDictionary; Passed=False; Show=All; 
                    FailureMessage=Expected $false, but got $true.; Time=00:00:00.0105793; Name=Should be false; Result=Failed; Context=; StackTrace=at <ScriptBlock>, 
                    C:\Users\taavast3\OneDrive\Repo\Projects\OpenSource\pesterPR\Pester\simpletest.ps1: line 3
                    3:         $true | Should be $false}, @{ErrorRecord=; ParameterizedSuiteName=; Describe=simple test; Parameters=System.Collections.Specialized.OrderedDictionary; Passed=True; Show=All; 
                    FailureMessage=; Time=00:00:00.0055245; Name=Should be true; Result=Passed; Context=; StackTrace=}}
RunDetails        : 

                    Content              : C:\Users\taavast3\OneDrive\Repo\Projects\OpenSource\pesterPR\Pester\simpletest.ps1
                    Type                 : File
                    PSTypename           : ExecutedBlockContainer
                    OneTimeTestSetup     : 
                    Focus                : False
                    DiscoveryDuration    : 00:00:00.2698095
                    ExecutedAt           : 9/28/2019 9:25:02 AM
                    EachTestTeardown     : 
                    Id                   : 
                    ShouldRun            : True
                    PluginData           : {}
                    ScriptBlock          : 
                    AggregatedPassed     : False
                    Tests                : {}
                    FrameworkData        : {PreviouslyGeneratedTests, PreviouslyGeneratedBlocks}
                    Passed               : True
                    IsRoot               : True
                    OwnDuration          : 00:00:00
                    BlockContainer       : @{Content=C:\Users\taavast3\OneDrive\Repo\Projects\OpenSource\pesterPR\Pester\simpletest.ps1; Type=File}
                    FrameworkDuration    : 00:00:00.4905946
                    Root                 : @{OneTimeTestSetup=; First=True; Focus=False; DiscoveryDuration=00:00:00.2698095; ExecutedAt=9/28/2019 9:25:02 AM; EachTestTeardown=; Id=; ShouldRun=True; 
                                           PluginData=System.Collections.Hashtable; ScriptBlock=; AggregatedPassed=False; Tests=System.Collections.Generic.List1[System.Object]; Name=Root; 
                                           FrameworkData=System.Collections.Hashtable; Passed=True; IsRoot=True; OwnDuration=00:00:00; BlockContainer=; FrameworkDuration=00:00:00; Root=; OneTimeBlockTeardown=; 
                                           Blocks=System.Collections.Generic.List1[System.Object]; Exclude=False; OneTimeTestTeardown=; Executed=True; OneTimeBlockSetup=; EachBlockSetup=; Path=; 
                    EachTestSetup=; 
                                           EachBlockTeardown=; Last=True; StandardOutput=; Tag=; Parent=; Duration=00:00:00; ErrorRecord=System.Collections.Generic.List1[System.Object]}
                    OneTimeBlockTeardown : 
                    Blocks               : {@{OneTimeTestSetup=; First=True; Focus=False; DiscoveryDuration=00:00:00.0299721; ExecutedAt=9/28/2019 9:25:03 AM; EachTestTeardown=; Id=0; ShouldRun=True; 
                                           PluginData=System.Collections.Hashtable; ScriptBlock=
                                               it 'Should be false' {
                                                   True | Should be False
                                               }

                                               it 'Should be true' {
                                                   True | should be True
                                               }
                                           ; AggregatedPassed=True; Tests=System.Collections.Generic.List1[System.Object]; Name=simple test; FrameworkData=System.Collections.Hashtable; Passed=True; 
                    IsRoot=False; 
                                           OwnDuration=00:00:00.0153029; BlockContainer=; FrameworkDuration=00:00:00.3050340; Root=; OneTimeBlockTeardown=; 
                    Blocks=System.Collections.Generic.List1[System.Object]; Exclude=False; 
                                           OneTimeTestTeardown=; Executed=True; OneTimeBlockSetup=; EachBlockSetup=; Path=System.String[]; EachTestSetup=; EachBlockTeardown=; Last=True; StandardOutput=; 
                    Tag=System.String[]; 
                                           Parent=; Duration=00:00:00.0153029; ErrorRecord=System.Collections.Generic.List1[System.Management.Automation.ErrorRecord]}}
                    Exclude              : False
                    OneTimeTestTeardown  : 
                    Executed             : True
                    OneTimeBlockSetup    : 
                    EachBlockSetup       : 
                    EachTestSetup        : 
                    EachBlockTeardown    : 
                    Duration             : 00:00:00.0153029
                    ErrorRecord          : {}

Like that we won't break code that is strongly based the structure of the v4 object, and still offer new functionality to the Pester end users.

[nohwnd]

There is a function that summarizes the new object to the old one. It does not do everything exactly yet but it should in the future. It’s probably not published right now but it is used internally to get the summary at the end of the run. That is imho the way to go. Just put the adapter function in the pipeline after Invoke-Pester and before your existing build infrastructure and you are good to go.

[BoSorensen]

Could you also mention the breaking change to Should -Throw, it was introduced in #1003 and means that the substring match has been replaced with the -like operator: link to the v4 docs https://pester.dev/docs/usage/assertions for reference.

I guess it means that it is not possible to use Should -Throw that works both for v4 and v5, if you want to do substring matching.

Perhaps #675 could be reconsidered ?

[nohwnd]

@BoSorensen Okay gotcha. I think that #675 or something similar would be useful for migration where you can run your test suite in a mixed mode. It would probably be difficult to merge the test results, but I have a Legacy result converter, so maybe not. 🙂

[nohwnd]

@Stephanevg Coverter to legacy object will be merged shortly. https://github.com/pester/Pester/pull/1472

[sdecker]

Are all the things listed in "Breaking changes" expected to be that way in the final release? It feels like many are "known issues". But some are actual breaking changes. If so could you please separate this section into two?

For example, not an exhaustive list
True breaking changes: these seems like they are expected not to work going forward

• Legacy syntax Should Be (without -) is removed, see Migrating from Pester v3 to v4
• Assert-VerifiableMocks was removed, Assert-MockCalled was renamed (but aliases exist), see Should -Invoke
• Assert-VerifiableMocks was removed, Assert-MockCalled was renamed (but aliases exist), see Should -Invoke

Really should be known issues: And these seem like known issues expected to be resolved or improved before final release

• Set-ItResult is published but does not work
• Code coverage report is not available.
• Mocks and Discovery are slow (whole execution is about 40% slower than on v4)

As it stands it's hard to make a business case of the value of moving to Pester 5 without knowing which are permanent.

[nohwnd]

@sdecker good point! I expect some of the known issues to be solved before 5.0 and some before 5.1, will, take that into consideration as well :)

[nohwnd]

@sdecker done 🙂

[dbafromthecold]

@nohwnd this may be me missing something, but has the -Script parameter for Invoke-Pester been removed?

Just tried updating the module to RC 5 and am getting...

A parameter cannot be found that matches parameter name 'Script'

When running something like...

Invoke-Pester -Script @{Path = "C:\pestertest\pestertest1.ps1";
Parameters = @{Param1 = $Value1;}}
-TestName 'MyPesterTest'
-PassThru -Show None

I checked the breaking features of the docs and didn't see it mentioned there. The function itself has -Script in the examples but it's not declared.

[nohwnd]

@dbafromthecold you are right the -Script does not exist anymore. It did too much stuff. It is called -Path now and accepts paths. The parametrized script running is not implemented and is documented here. https://github.com/pester/Pester/issues/1485

I added -Script to breaking changes list.

[dbafromthecold]

@nohwnd thanks! will get updating my scripts and testing.

[bergmeister]

@nohwnd I find the guidance to put everything that used to not be controlled into a BeforeAll block confusing because of the following scenario:
At the start a module with test helper functions gets imported (not within Pester controlled blocks initially). At first I just moved all the existing test initialization into a BeforeAll block but then inside the Describe block, the imported function is not available any more, therefore I had to move the bit with the module import outside of the BeforeAll block for it to work again. What do you think?

[nohwnd]

@bergmeister yeah that one is confusing. I saw you do that in PSScriptAnalyzer (worked recently a bit more on moving it to v5, not sure if I pushed into my branch). The original readme for alpha2 had guidance around this. And I did not come up with a good pattern for this yet. I experimented with BeforeDiscovery { }, and had Anywhere { } and Add-Dependency {} blocks before. All with different timings when they will run. But the way I ran the tests also changed two times.

Anyway, describing it in an article is probably the best thing to do now, because there are more situations where you might want to put your code outside of pester blocks.

[Stephanevg]

Ok, sorry in advance for the HUGE spam:

Hi,

I wanted to share some feedback regarding the experience of migrating to Pester v5.
I took a simple test of PSTHML https://github.com/Stephanevg/PSHTML/blob/master/Tests/link.tests.ps1

And tried to migrate it.

I had basically two things I needed to do:

1) Adding all the 'Arrange' code into the a beforeAll
2) I had to add the parameter names to the IT block

Old version

Describe "Testing link" {


        $Class = "MyClass"
        $Id = "MyID"
        $Style = "Background:green"
        $href = "woop"
        $rel = "author"
        $CustomAtt = @{"MyAttribute1"='MyValue1';"MyAttribute2"="MyValue2"}
        $string = link -href $href -rel $rel  -Attributes $CustomAtt -Style $Style -Class $class -id $id

        if($string -is [array]){
            $string = $String -join ""
        }

        it "Should contain opening and closing tags" {
            $string -match '^<link.*>' | should be $true
            #$string -match '.*</link>$' | should be $true

        }

# More it blocks...
} # End Describe block

Output

PS /Users/stephanevg/Code/PSHTML/Tests> $e = invoke-pester -Path ./link.tests.ps1

Starting test discovery in 1 files.
Discovering tests in /Users/stephanevg/Code/PSHTML/Tests/link.tests.ps1.
Found 5 tests. 64ms
Test discovery finished. 79ms


Running tests from '/Users/stephanevg/Code/PSHTML/Tests/link.tests.ps1'
Context Testing PSHTML
 Describing Testing link
   [-] Should contain opening and closing tags 10ms (2ms|7ms)
    ParameterBindingException: Parameter set cannot be resolved using the specified named parameters. One or more parameters issued cannot be used together or an insufficient number of parameters were provided.
    at <ScriptBlock>, /Users/stephanevg/Code/PSHTML/Tests/link.tests.ps1:35
   [-] Testing common parameters: Class 6ms (2ms|4ms)
    ParameterBindingException: Parameter set cannot be resolved using the specified named parameters. One or more parameters issued cannot be used together or an insufficient number of parameters were provided.
    at <ScriptBlock>, /Users/stephanevg/Code/PSHTML/Tests/link.tests.ps1:42
   [-] Testing common parameters: ID 6ms (2ms|4ms)
    ParameterBindingException: Parameter set cannot be resolved using the specified named parameters. One or more parameters issued cannot be used together or an insufficient number of parameters were provided.
    at <ScriptBlock>, /Users/stephanevg/Code/PSHTML/Tests/link.tests.ps1:45
   [-] Testing common parameters: Style 6ms (3ms|4ms)
    ParameterBindingException: Parameter set cannot be resolved using the specified named parameters. One or more parameters issued cannot be used together or an insufficient number of parameters were provided.
    at <ScriptBlock>, /Users/stephanevg/Code/PSHTML/Tests/link.tests.ps1:48
   [-] Testing Attributes parameters 8ms (4ms|5ms)
    ParameterBindingException: Parameter set cannot be resolved using the specified named parameters. One or more parameters issued cannot be used together or an insufficient number of parameters were provided.
    at <ScriptBlock>, /Users/stephanevg/Code/PSHTML/Tests/link.tests.ps1:56
Tests completed in 176ms
Tests Passed: 0, Failed: 5, Skipped: 0 NotRun: 0

New version Result

Describe "Testing link" {
BeforeAll -Scriptblock {

            $Class = "MyClass"
            $Id = "MyID"
            $Style = "Background:green"
            $href = "woop"
            $rel = "author"
            $CustomAtt = @{"MyAttribute1"='MyValue1';"MyAttribute2"="MyValue2"}
            $string = link -href $href -rel $rel  -Attributes $CustomAtt -Style $Style -Class $class -id $id

            if($string -is [array]){
                $string = $String -join ""
            }
        }



        it -Name "Should contain opening and closing tags" -test {
            $string -match '^<link.*>' | should -be $true
            #$string -match '.*</link>$' | should be $true

        }
# more IT blocks

} # End describe block

It fixed my test.

I wrote a lot (but like, really A LOT!) of tests without specifying the parameter names.

Pester Configuration Object

I LOVE IT!

Up until I have tested, that is really something that could make us think of migrating to v5.

One small suggestion, is maybe to expose the constructors to the end users via Functions (using a 'hybrid' approach). so that the creation of the Configuration is abstracted from the user.

example:

Function New-PesterConfiguration {
[CmdletBinding()]
Param(
  [System.Io.FileInfo]$Path,
  [String[]]$Tag,
  [String[]]$ExcludeTag
  [Switch]$CodeCoverage
)

If($CodeCoverage){
      $ccstate = $true
   }else{
       $ccstate = $False
    }
$configuration = [PesterConfiguration]@{
    Run = @{
        Path = $Path
    }
    Filter = @{
        Tag = $Tag
        ExcludeTag = $ExcludeTag
    }
    Should = @{
        ErrorAction = $PsBoundParameter.ErrorAction
    }

    CodeCoverage = @{
        Enable = $ccstate
    }
}

Return $Configuration

}

Like this the users will be able to benefit from all the guidance a Powershell function can offer when creation a new pester configuration (Comment based help, validate sets etc...).

Logging

I couldn't find how to enable the logging in the readme file.
I think it would be great to have the possibility to save the logs to file.

ConvertTo-Pester4Object

Thanks for adding that @nohwnd 👍
I tested it, and it works like a charm.

ConvertTo-NunitReport

it works really well. I like the new -AsString parameter.
ExportTo-NunitReport works as I would expect as well.

Note: I couldn't find a ConvertTo-JunitReport. It is probably on it's way, but we do rely on the JunitXML document as we use Gitlab, and it only supports that format.
IF there would a new cmdlet ConvertTo-JunitReportto come, I guess a Export-JunitReport should also be implemented. Wouldn't it make sense to Have a generic 'Export-pesterReport' function (or something around these lines) which could persist the following to file:

• NunitXML
• JunitXML
• PesterObject deserialized (CLIXML? / JSon ? )
  So, with a -Format parameter.

Internally, that function could either have it's own logic, or simply call the other ConverTo-J/NUnitReport functions.

I am just raising the thought here, simply to have the discussion now, and avoid having to rename the cmdlets once v5 is officially out.

Overall impression of RC1

I really really really like the Configuration Object. That will definitely be something very usefull. Especially for framework authors I think.

I have to admit that it confuses me a little bit that we have the that small pester message 'Amount of failed / passed tests etc...' AND the pester object that is returned at each run.

Saw this discussion -> https://github.com/pester/Pester/issues/1480
I think adding -Passthru back would make it more clear. It is a pattern that people already know, and less scripts will need to be updated.

I have just tested for regular Unit tests. Moving things to BeforeAllblocks seems to be the right migration path. (Also, It would be great if the migration can be smoothed a little bit with some Migration help scripts.)

I do have another test case I would like to takle, and that is for Framekworks that are written around Pester itself. At work we have a framwork that works 'kind of' like DBAChecks.

I do have the impression that these frameworks will benefit a lot of the refactoring of Pester.

When reading through the v5 readme with my "boss's vision" I don't see that much that can appeal us to migration from v4 to v5.
Of course, there is logging, the discovery, the new structure of the object, the decoupling of commands etc... But in the end, If I just want to use Pester's basic functionality to test my code (written with v4 syntax) v5 doesn't seem to offer something that will want us to really really migration to it, and go through the hassle of migrating ALL our tests (belive, we have a LOT of tests) to v5.

For now, the Pester v5 migration sounds like a super lot of work for me, without any direct benefits (outside of use the latest version of pester).

I can image that we would end up having a double approach:
1) leave our existing Tests written in v4 as is, and call Invoke-Pester from the v4 module .
2) when writing new tests, we would embrace Pester v5 recommendations (beforeAll etc...) , and use Invoke-Pester from the v5 module.

We would need to know in advance for which pester version a Tests.ps1 file is written. Not sure how to get that one to work.

[nohwnd]

@Stephanevg

I had to add the parameter names to the IT block
Why was that necessary? That should not be necessary. Were there errors?

One small suggestion, is maybe to expose the constructors to the end users via Functions (using a 'hybrid' approach). so that the creation of the Configuration is abstracted from the user.

That will come in the future on way or another. Right now each section can be created via a hashtable now, or by creating the whole object using $config = [PesterConfiguration]::Default and then assigning to the different properties. I understand that this is not ideal, but my immediate goal was to expose a simple api that can be used interactively, and then have a complete "api" into which everything else will be translated. So internally I can represent all the configuration via a single object, not matter in which way it is created.

I am planning to add a more intermediate api, which will have more parameters on Invoke-Pester, or possible functions to create the sections of the config. The thing is that I need to think about it a lot to make it usable. And I don't have the bandwidth to do that before 5.0.

That is why I am exposing the "raw" config object because it allows to config everything that is there to configure.

Like this the users will be able to benefit from all the guidance a Powershell function can offer when creation a new pester configuration (Comment based help, validate sets etc...).

Did you see the descriptions on each of the properties? Or that is not discoverable?

ConvertTo-JunitReport.

Is that in v4?

Export-pesterReport

That is what I wanted to avoid. Generic functions that do many things. You are also IMHO unlikely to have both NUnit and JUnit reports in one CI pipeline.

I couldn't find how to enable the logging in the readme file. I think it would be great to have the possibility to save the logs to file.

That would be in the debug section of config. https://github.com/pester/Pester/tree/v5.0#advanced-interface WriteDebugMessages = $true and WriteDebugMessagesFrom = "Filter"

It currently uses Write-Host to write to the screen, so it does not save into a file. And there is a rather huge perf penalty for enabling it. Imho adding -Log param with the basics like "Skip", "Mock", "Discovery", "Filter" would make it way more usable. Again, not enough time to polish all this out :)

I have to admit that it confuses me a little bit that we have the that small pester message 'Amount of failed / passed tests etc...' AND the pester object that is returned at each run.

That is already reverted and in rc2 there is -PassThru again. 🙂

migration can be smoothed a little bit with some Migration help scripts.

There is a script linked in the readme that puts all file setups to BeforeAll, I will try to update it to do the same for the code after Describe but before It

When reading through the v5 readme with my "boss's vision" I don't see that much that can appeal us to migration from v4 to v5.

Give it a try on a new project, and take advantage of stuff like skipping whole blocks based on the platform variables that PowerShell 6 has, or tagging on everything, even child Describes and tests, and mocks that almost never require you to specify the scope.

And then when you go back to v4 you will probably feel annoyed by all of this missing. As I was yesterday 😁

And there is more stuff coming, I have an idea how to make the mocks debuggable, will probably make parallel test running more native and so on. v5 is where new stuff will be happening.

We would need to know in advance for which pester version a Tests.ps1 file is written. Not sure how to get that one to work.

In the config you could define that all your pester5 tests have .Tests5.ps1 extension, there is an option for that in the Run section. That way Pester5 will pick up it's tests, and v4 will pickup the legacy tests.

[nohwnd]

@Stephanevg tagging you one more time, I posted the previous answer too soon. It was not a spam at all, thanks for sharing your experience 👍

[bergmeister]

Around -TestCases, which is now also evaluated at discovery time:
I have a test suite here with multiple, long test cases for different Context blocks. The first solution was to declare variables for the test case arrays but for long and multiple test case objects meaning that test and test case are far apart. I then though of inlining the test case array, but that would meaning that the test name and test scriptblock are far apart.
The final solution that I came up with is using the named parameters of It:

It -TestCases @(
    @{
        foo = 'bar'
    }
    @{
        foo = 'baz'
    }
) -Name 'ItName' -Test {
    $foo | Should -Match 'ba'
}

or

It 'ItName' -Test {
    $foo | Should -Match 'ba'
} -TestCases @(
    @{
        foo = 'bar'
    }
    @{
        foo = 'baz'
    }
)

I think I slightly prefer the last version. Just posting as seeing this pattern might help others

[nohwnd]

Around -TestCases, which is now also evaluated at discovery time:

Does this have any impact on this? I don't see anything that would actually run in your case, it's just a bunch of strings.

The final solution that I came up with is using the named parameters of It:

That looks okay to me, especially if the test is just a couple of lines long, then you can easily see that testcases are specified below.

---

If the problem actually was that the evaluation takes long time, then I think -LazySkip and -LazyTestCases which would take scriptblocks could be introduced.

[bergmeister]

Does this have any impact on this?

It was more around finding the most readable version, especially when touching a lot of code now, not performance

On a related note: Is there an easier/lazier way of using TestCases with just an array of strings instead of an array of hashtables if I have a simple test that only takes an array of test values All I could think of is doing something like this

$testValues= 'foo', 'bar', 'baz'
$testCases = $Instances.ForEach{ @{ _ = $_ } }

It would be nice if Pester shipped with something like an array to hashtable converter with the hashtable key being _ so that one can use the intuitive $_ in the test itself (not sure what to do about the angle syntax in the test name itself)

[nohwnd]

It would be nice if Pester shipped with something like an array to hashtableConverter with the hashtable key being _ so that one can use the intuitive $_ in the test itself (not sure what to do about the angle syntax in the test name itself)

And that is why it does not yet ship with such function, because people want to use it in different ways 🙂

[nohwnd]

Anyways I will start cutting Gherkin from Pester 5, and start renaming to __ where necessary.

[bergmeister]

And that is why it does not yet ship with such function, because people want to use it in different ways 🙂

I only partially agree: I agree that people need the freedom to do it in their way but at the moment, if one supplies e.g. an array of strings to the -TestCases parameter, then Pester fails with a parameter binding error that it cannot convert it to an IDictionary. Therefore having at least one default value of using an object array would be useful. Anyone who want to do some custom logic can still use their own array to hashtable converter. This way I don't see how adding support for arrays in -TestCases would break any existing usage or constrain usage.

How about having a -Whatif on Invoke-Pester to run only the test discovery? This would be useful as a simple build validation that there are no syntax errors, etc. without having to run the entire test suite?

[nohwnd]

having at least one default value of using an object array would be useful

You are right. The problem is that consuming arrays of any type is hard to make practical without inventing a lot of arbitrary rules that are hard to come up with, and then hard to define in code. So sticking with the least amount of ways to provide the test cases which still allow you to do everything, even though you sometimes need to adapt array to hashtable is what I do now. We can continue discussing it here https://github.com/pester/Pester/issues/832 there are some nice ideas in that thread. 🙂

How about having a -Whatif on Invoke-Pester to run only the test discovery?

Internally there is Find-Test. And publishing it as a separate cmdlet or as additional parameter on Invoke-Pester is a decision that prevents me from publishing it. I also want to first get the functionality that current Pester has, without adding a lot of new stuff, because the smaller the api is at the start the better.

[nohwnd]

Anyways I will start cutting Gherkin from Pester 5, and start renaming to __ where necessary.

Wrong thread 😁

[Stephanevg]

Hi @nohwnd ,
I do have a last issue / suggestion / comment regarding Pesterv5, and since it is not in official release yet, I hope we can find a way to inegrate that.

At work I use pester to do Infrastructure validation tests. In (very short) What we do, is we 'dehydrate' (Export-CliXML) the PesterObject, which we rehydrate to do some global parsing afterwards and some other stuff. This is done each time using the -PassThru parameter.

The object in Pester5 is very different than the one from v4. It is a good thing we have ConvertTo-Pester4Result (And btw, was that object introduced in v4 only? Was is not present before v3 ? If so, the name might be missleading then.).

In a scenario where I have already a lot of dehydrated pester 4 objects, and than I export new ones using pester 5, I realize that I need to somehow make a difference between the types of objects as the parsing logic I need to use is different according to the version of pester in which it got generated.

Wouldn't it make sense to add a property at the root of the pesterobject which can help to identify the major version of pester that was used to generate the object?

Adding the same property (with a different value of course) on the v4 object in a version 4.11 for example could help solve this issue, and open the door to a standardisation towards the pester object as such.

Something like PesterObjectVersion or perhaps PesterVersion or anything in that direction would help to make a difference easily and quickly.
(I'd be happy to implement it also)

[nohwnd]

Wouldn't it make sense to add a property at the root of the pesterobject which can help to identify the major version of pester that was used to generate the object?

Totally, that's a great idea :) I also need to work on actually make the object serializable. It is serializable now I think, but there are a lot of recursive properties on the unfiltered object. And that will be problematic with clixml (or any other serializer for that matter).

[Stephanevg]

I'll switch to the #1501 to continue the discussion to continue the discussion

[inammathe]

Sorry if this is a dumb question.. but what happened to the -OutputFile parameter of Invoke-Pester?

PS> Get-Help Invoke-pester -Parameter OutputFile
Get-Help: No parameter matches criteria OutputFile.

[bergmeister]

@inammathe When using the -CI, it will set the TestResult.Enabled property of the configuration object to $true and the file name will default to testResults.xml (but can be overriden via the TestResult.OutputPath property on the configuration object).
See the v5 readme for more details around the interfaces and the configuration object: https://github.com/pester/Pester/tree/v5.0#simple-and-advanced-interface

[nohwnd]

@inammathe https://github.com/pester/Pester/pull/1578 this PR will add them back together, and will be soon released in 5.0.1 to make the compatibility better for now. They are deprecated, and will be removed at some point later when the intermediate api has a comfortable replacement.

[vojtech-kasny]

Hello all. Has anybody problems with variables not imported to session in 1st run?
Pester v5.0.1

Code

BeforeAll {
    # Call module loader
    $LoaderPath = (Resolve-Path -Path (Join-Path $PSScriptRoot -ChildPath "..\..\..\common\ModuleLoader.ps1")).Path
    . $LoaderPath -PSModuleNames @('Vsts.Functions', 'Utility.Functions') -AdditionalFolderNamesWithModules '\Tipsweb\DownloadBuildArtifact\Modules'
    # Module to test
    $ModuleName = 'VstsBuild.Functions'
    # Module commands to test
    $ExportedCommands = Get-Command -Module $ModuleName
    # PSScriptAnalyzer rules setup
    $Severity = 'error' # Error|Warning
    $ScritpAnalyzerRules = Get-ScriptAnalyzerRule -Severity $Severity
    # PSScriptAnalyzer call
    $ScriptAnalyzerResult = Invoke-ScriptAnalyzer -Path (Get-Module $ModuleName).Path -Severity $Severity
}

Describe "Module '$ModuleName' PSScriptAnalyzer Tests" {
    It "Has at least 1 function" {
        $ExportedCommands.Count | Should -BeGreaterThan 0
    }
    foreach ($Rule in $ScritpAnalyzerRules) {
        It "Should pass rule '$Rule' on severity '$Severity'" {
            if ($ScriptAnalyzerResult.RuleName -contains $Rule) {
                ($ScriptAnalyzerResult | Where-Object {$_.RuleName -eq $Rule} -OutVariable Failures).Count | Should -Be 0
            }
        }
    }
    foreach ($cmd in $ExportedCommands) {
        $ErrorCount = $null
        [System.Management.Automation.PSParser]::Tokenize((Get-Command $cmd).Definition, [ref]$ErrorCount) | Out-Null
        It "'$cmd' has valid PowerShell code syntax" {
            $ErrorCount.Count | Should -Be 0
        }
    }
}

Run 1

Invoke-Pester C:\Dev\Repos\toolbox\Build\01_scripts\tipsweb\DownloadBuildArtifact\Tests -Output Detailed

Starting discovery in 1 files.
Discovering in VstsBuild.Functions.Tests.ps1.
Found 1 tests. 475ms
Discovery finished in 927ms.

Running tests from 'VstsBuild.Functions.Tests.ps1'
Describing Module '' PSScriptAnalyzer Tests
  [+] Has at least 1 function 472ms (350ms|122ms)
Tests completed in 3.33s
Tests Passed: 1, Failed: 0, Skipped: 0 NotRun: 0

Run 2

Invoke-Pester C:\Dev\Repos\toolbox\Build\01_scripts\tipsweb\DownloadBuildArtifact\Tests -Output Detailed

Starting discovery in 1 files.
Discovering in VstsBuild.Functions.Tests.ps1.
Found 13 tests. 38ms
Discovery finished in 51ms.

Running tests from 'VstsBuild.Functions.Tests.ps1'
Describing Module 'VstsBuild.Functions' PSScriptAnalyzer Tests
  [+] Has at least 1 function 4ms (2ms|2ms)
  [+] Should pass rule 'PSAvoidUsingUserNameAndPassWordParams' on severity 'error' 190ms (188ms|2ms)
  [+] Should pass rule 'PSAvoidUsingComputerNameHardcoded' on severity 'error' 7ms (5ms|3ms)
  [+] Should pass rule 'PSAvoidUsingConvertToSecureStringWithPlainText' on severity 'error' 11ms (7ms|4ms)
  [+] Should pass rule 'PSDSCUseIdenticalMandatoryParametersForDSC' on severity 'error' 10ms (8ms|2ms)
  [+] Should pass rule 'PSDSCUseIdenticalParametersForDSC' on severity 'error' 5ms (3ms|2ms)
  [+] Should pass rule 'PSDSCStandardDSCFunctionsInResource' on severity 'error' 6ms (4ms|2ms)
  [+] 'Get-VstsBranchBuild' has valid PowerShell code syntax 7ms (6ms|2ms)
  [+] 'Get-VstsSprintBuild' has valid PowerShell code syntax 11ms (8ms|3ms)
  [+] 'Invoke-GetBuildArtifact' has valid PowerShell code syntax 7ms (4ms|2ms)
  [+] 'Save-VstsBuildArtifact' has valid PowerShell code syntax 7ms (5ms|2ms)
  [+] 'Test-VstsBuildArtifact' has valid PowerShell code syntax 6ms (4ms|2ms)
  [+] 'Test-VstsBuildIsRunning' has valid PowerShell code syntax 6ms (4ms|2ms)
Tests completed in 720ms
Tests Passed: 13, Failed: 0, Skipped: 0 NotRun: 0

[nohwnd]

@vojtech-kasny in your example the code in BeforeAll which defines $ScritpAnalyzerRules will run after your foreach loop. BUT because there is a bug https://github.com/pester/Pester/issues/1579 in scoping when running via Invoke-Pester the variable will stay in scope even after the file is executed, and you will use it on the second run.

This will get fixed and your example will stop working.

To make this work every time, you would have to move the code that you need to generate the tests outside of BeforeAll and attach the variables that you need in a test using -TestCases. In your case that would be the iterator variable $Rule, like this It "asdfsafd" -TestCases { Rule = $Rule }` , this will attach the current value of $Rule to the It block, and will later define it automatically, because all Keys from TestCases are defined as variables to avoid manual writing of param block.

[vojtech-kasny]

@nohwnd Thanks Jakub. Adjusted code based on your recommendations and it works

[plastikfan]

I am new to Pester. I have created some plasters (derived from Adentures In Plaster and invoking pester with parameters -Path, -OutputFile and -OutputFormat. The blog article is quite old so I suppose it is probably out of date (finding it really hard to find something that is up to date), but this invocation is causing the following warning to appear:

WARNING: You are using Legacy parameter set that adapts Pester 5 syntax to Pester 4 syntax. This parameter set is deprecated, and does not work 100%. The -Strict and -PesterOption parameters are ignored, and providing advanced configuration to -Path (-Script), and -CodeCoverage via a hash table does not work. Please refer to https://github.com/pester/Pester/releases/tag/5.0.1#legacy-parameter-set for more information.

I am trying to not use a "Legacy" parameter set and the ... Please refer to ... and not sure what parameters should be used for version 5 of Pester.

It looks like I should either be using Simple or Advanced parameter set. I am trying to work my way through this description, but there isn't mapping between v4 parameters and v5.

As I said before I'm trying to figure out what parameters I should replace to be v5 compliant

• Path: currently specifies the test files (eg: '.../Tests/*'
• Output: currently specifies the xml file name with the test result eg '/testResult.xml'
• OutputFormat currently = NUnitxml

It's not clear to me how to adapt this call to invoke-pester for v5 compliancy:

Invoke-Pester -Path -ExcludePath -Tag -ExcludeTag -Output
-CI

The only one that is clear is -Path. I thought that the old -Output was the same as the new, but they are not (how do I now specify the output file?) and how do I specify the -OutputFormat, or do I not need to?

Thanks for any help

EDIT: I'm using a configuration object and this runs without warnings. However, I no longer have a test results xml file, what used to be specified by -OutputFile.

EDIT2: I have discovered that a lot of the old parameters have moved to configuration.TestResult. So, where we used to specify -OutputFormat NUnitxml and -Output 'testResults.xml', we can now populate these on configuration.TestResult. Hope this helps somebody else that might struggle with this as its not yet documented.

[nohwnd]

@plastikfan thanks for the question. The Legacy parameter set was added in hurry, and you are right there is no table. But you can go to the code here, and see the parameter set:
https://github.com/pester/Pester/blob/816c964ea85bb39898ca5180c405cdc8beaa41c9/src/Pester.ps1#L259

All of the parameters are in the Legacy parameter set, and most of them define extra aliases. Except for ExcludeTag, that relies on the feature of PowerShell that allows you to specify just enough of the name to not be ambiguous.

The mapping from the parameters to the values on the configuration object is done below in:

https://github.com/pester/Pester/blob/816c964ea85bb39898ca5180c405cdc8beaa41c9/src/Pester.ps1#L420

It would be nice if you could make PR that adds Legacy section into the readme with a table like the one in the Simple and Advanced parameter set 🙂

[plastikfan]

Hi, are there some strange scoping rules that apply to variables declared (or just set as we can't explicitly declare a variable with say a 'var' statement) inside pester Describe/Context/It blocks, which are not particularly intuitive?

Consider this test:

[System.Collections.Hashtable]$DefaultKrayolaTheme = @{
  'FORMAT'             = 'def:"<%KEY%>"="<%VALUE%>"';
  'KEY-PLACE-HOLDER'   = '<%KEY%>';
  'VALUE-PLACE-HOLDER' = '<%VALUE%>';
  'KEY-COLOURS'        = @('DarkCyan');
  'VALUE-COLOURS'      = @('DarkBlue');
  'OPEN'               = '••• (';
  'CLOSE'              = ') •••';
  'SEPARATOR'          = ' @@ ';
  'META-COLOURS'       = @('Yellow');
  'MESSAGE-COLOURS'    = @('Cyan');
  'MESSAGE-SUFFIX'     = ' ~~ '
}

Describe 'Get-KrayolaTheme' {
  Context 'Dark Theme, "KRAYOLA-THEME-NAME" not defined in Environment' {
    Context 'Given: Theme not specified' {
      it 'Should: return default valid theme' {
        Mock -ModuleName Elizium.Krayola Get-IsKrayolaLightTerminal { return $false }
        Mock -ModuleName Elizium.Krayola Get-EnvironmentVariable { return $null }

        $result = Get-KrayolaTheme -DefaultTheme $DefaultKrayolaTheme
        $result | Should -Not -BeNullOrEmpty
        $result | Should -BeOfType System.Collections.Hashtable
      }
    }
...

This is failing for a strange reason because, on entry into the function under test Get-KrayolaTheme, the argument DefaultTheme is $null even though it has been set to the file scoped DefaultKrayolaTheme.

However, when I change the test, so that DefaultKrayolaTheme is declared inside the It block, then it passes as expected because DefaultTheme is no longer $null and is correctly set to the local variable value.

So what am I missing here? DefaultKrayolaTheme is a global and should be applied to multiple tests, hence that is the reason why I want to set it once at the top of the file for it to be shared.

And setting the value inside of a BeforeAll block does not help, that block's scope doesn't apply to the It scope, so what-ever is set there isn't persistent.

What's happening here isn't intuitive and I can't find anything relevant in the Describe/Context & It documents referenced above.

Thanks for anyone's insight.

[plastikfan]

I have discovered that if I use the Global scope specifier, this works

[System.Collections.Hashtable]$Global:DefaultKrayolaTheme = @{ ...

I have also found that another technique (issue #330) is to use hash tables and populate them inside BeforeEach. However I would prefer to use Global, unless there is a good reason not to and it is much more concise.

This is not a Pester issue as it turns out, it's just one of the many frustrating unintuitive gotchyas of PowerShell.

[nohwnd]

$DefaultKrayolaTheme is defined during discovery and those variables are not guaranteed to be available during run. If you'd put that into a BeforeAll block it would work correctly because Pester would run the BeforeAll block before running It block. BeforeAll is a setup for multiple tests.

[plastikfan]

Thanks @nohwnd, I just tested your theory and can confirm that you're right. I didn't think this would have worked because I previously moved that code into the BeforeAll block, but the script analyser flags this as an unreferenced variable, so I just assumed it wouldn't work, without running it!

Your point about the discovery phase is noted, cheers.

[BoSorensen]

Was there a resolution to the pattern where you move setup code to a BeforeAll block and end up with warnings from PsScriptAnalyzer ?

See https://github.com/PowerShell/PSScriptAnalyzer/issues/711#issuecomment-476532100

[plastikfan]

Hi @BoSorensen, actually, the script analyzer is ok with the variable if you mark it as Global, so in my BeforeAll it's declared as follows:

[System.Collections.Hashtable]$Global:DefaultKrayolaTheme = @{ ...

Previously, I wasnt using $Global, which resulted in the script analyzer warning.

[BoSorensen]

Good tip, but has the drawback of polluting the $Global namespace with these variables.

[plastikfan]

Yeah you're right, but I'm tackling one problem at a time. I will resolve this properly in due course. And actually, I'm not dogmatic about these things, sometimes a global variable is ok, depends how you do it

[plastikfan]

Actually, I just realised, this is only a test, so there is no consequence to using a global here. This is an example I would say, the use of the global is ok.

[BoSorensen]

If you run Invoke-Pester from the PowerShell console you will have a lot of new variables in the $Global namespace, so it has consequences.

[plastikfan]

Ok, you're probably right, but I'm not going to waste time on it. I've got far bigger issues to be dealing with.

[BoSorensen]

Fair enough, the issue is also with PSScriptAnalyzer where the real bugfix should be applied.

[bergmeister]

ScriptAnalyzer cannot know how a variable is being used by Pester due to it being a DSL. ScriptAnalyzer is a linter and the nature of linters is that there can be false positives that the tools cannot make a decision on and need human input. Therefore in that case, that warning should be suppressed specifically for that specific variable in that file. Suppressing is a way of stating in the source code that 'I have reviewed the warning and consider it to be not an issue'.

[nohwnd]

@bergmeister that sounds very annoying to do manually. Does PSSA consider all scriptblocks to form their own scope?

Could PSSA consider all Before* and After* blocks to be dot-sourced, and therefore accessible in the current and all child scopes?

[bergmeister]

@nohwnd At the moment, PSSA's variable and parameter analysis is limited to a per-scriptblock basis, even Begin/Process blocks do not see each other. This is due to PowerShell's dynamic nature, where it cannot even assume that a child scriptblock is invoked in the current scope due to PowerShell's dynamic nature, see https://github.com/PowerShell/PowerShell/issues/12287 (allow-listing certain known cmdlets that do like e.g. Where-Object is considered though.). Therefore making PSSA aware of Pester's DSL has a long way to go but we'd for sure consider PRs. Sure, it can be done, but there's loads of things that could be done (like e.g. the Begin/Process issue) and therefore I want to quote the comment of @rjmholt here again as it's also a case of adapting the user's mindset (as there is simply not enough time from the PowerShell team or the community to fulfil all those asks):

Ultimately this rule is supposed to be a helpful heuristic rather than an absolute.

[nohwnd]

I totally understand all the arguments, here and in the linked comments. I am not looking for a perfect solution that will catch every case. Rather I am looking for a solution that will be mostly correct, requiring minimal manual intervention in the most used scenarios.

[rjmholt]

It would be possible to consider them dot-sourced, but if the Before* and After* blocks are provided out of order, the underlying assumption wouldn't play out.

I actually played around with ingesting Pester ASTs and handling them specially. It's quite possible (if you or I can read a Pester test and understand it without executing it, so can PSSA), but would be hardcoded to Pester's current layout and be tricky to change.

It's still something I'm interested in doing, just because it would be nice for the linter to encourage people to use good tools like Pester, but I put off the work to invest in PSSA2, and possibly come up with more general solutions, at least for the dot-sourcing. Essentially a good solution would consist of:

• Detecting any Before*, After* or Describe keywords in any scriptblock, or ideally just identifying that a file ends with .Tests.ps1
• At that point, collecting the whole test structure into a shape we know Pester uses
• Then have the rule traverse and build context in the same order Pester does
  
  
  
  • BeforeAll
  
  
  • Describe
  
  
  • BeforeAll
  
  
  • Context
  
  
  • BeforeAll
  
  
  • BeforeEach
  
  
  • It
  
  
  • AfterEach
  
  
  • AfterAll
  
  
  • etc
  
  

To be perfectly clear, I don't personally consider many of the outstanding issues in the declared vars rule to be bugs, but rather feature requests to improve the heuristics. In Pester's case, the heuristic depends on an exact understanding of Pester's semantics and whether any of that is likely to change.

[nohwnd]

To be perfectly clear, I don't personally consider many of the outstanding issues in the declared vars rule to be bugs, but rather feature requests to improve the heuristics.

👍👍👍

[BoSorensen]

Although being told to ignore certain PSScriptAnalyzer warnings is a slippery slope to ignoring warnings in general. But any minor improvement would be very welcome.

[AlexRedkin]

Good day!
I've got a question regarding mocking module functions. I import module and try to mock it's function, but it does not work. Could somebody please tell me what am I doing wrong?
Here is the .psm1 file:

using module psPAS

class Sample {
    Sample() {
    }

    [void] login([PSCredential] $credential) {
        New-PASSession -Credential $credential -BaseURI "https://none"
    }
}

function Get-Sample(){
    $sample = [Sample]::new()
    return $sample
}

Export-ModuleMember -Function Get-Sample

And here is the script I am trying to execute.

Import-Module "C:\Users\Admin\Downloads\class.psm1"

Import-Module -Name Pester -RequiredVersion 5.0.2


Describe "Sometest"{
    Context "testContext"{

        Mock -ModuleName class -CommandName Get-Sample -MockWith {return "bla-bla"}


        It "test"
            get-sample | should -Be "bla-bla"
        }

    }

I've already read documentation and tried to reproduse solutions from here
https://github.com/pester/Pester/issues/1351

[nohwnd]

@AlexRedkin That Mock should be inside of BeforeAll block.

[AlexRedkin]

@AlexRedkin That Mock should be inside of BeforeAll block.

Thank you for the clarification and awesome product! It works now!

[deadlydog]

For anybody else that upgraded their machine to Pester 5 (on purpose, or accidentally with a sweeping Update-Module command like me 😛 ) and doesn't have time to change their test code to account for the breaking changes, you can still use Pester v4 by sticking code similar to this at the top of your test scripts:

Install-Module -Name Pester -RequiredVersion 4.10.1 -Repository PSGallery
Import-Module -Name Pester -RequiredVersion 4.10.1

@nohwnd You could maybe make a note of this in the Pester ReadMe (perhaps in the Questions section) for others to be able to get back up and running quickly.

[petercharleston]

Hi everyone,
I am getting the same error when using pester runner in a pipeline task
I tried installing/importing version 4 module but get same error

The code i am using:

`[CmdletBinding()]
param(
[Parameter(Mandatory)]
[ValidateNotNullOrEmpty()]
[string]$ResourceGroupName
)

Install-Module -Name Pester -RequiredVersion 4.6.0 -Repository PSGallery
Import-Module -Name Pester -RequiredVersion 4.6.0

requires -Module Az.Accounts,Az.Resources

describe 'Template validation' {
it 'template passes validation check' {
$parameters = @{
TemplateFile = 'server.json'
ResourceGroupName = $ResourceGroupName
adminUsername = 'adam'
adminPassword = (ConvertTo-SecureString -String 'testing' -AsPlainText -Force)
vmName = 'TESTING'
}
(Test-AzResourceGroupDeployment @parameters).Details | should -Benullorempty
}
}`

[nohwnd]

What is the error?

[petercharleston]

WARNING: You are using Legacy parameter set that adapts Pester 5 syntax to Pester 4 syntax. This parameter set is deprecated, and does not work 100%. The -Strict and -PesterOption parameters are ignored, and providing advanced configuration to -Path (-Script), and -CodeCoverage via a hash table does not work. Please refer to https://github.com/pester/Pester/releases/tag/5.0.1#legacy-parameter-set for more information.

System.Management.Automation.RuntimeException: No test files were found and no scriptblocks were provided.
at Invoke-Pester, /home/vsts/.local/share/powershell/Modules/Pester/5.0.2/Pester.psm1: line 4197
at , /home/vsts/work/_tasks/Pester_cca5462b-887d-4617-bf3f-dcf0d3c622e9/0.1.3/Pester.ps1: line 160

[error]Write-Error: Pester returned errors

[error]Write-Error: Pester returned errors

Pester Script finished
Finishing: Pester

[bergmeister]

@petercharleston Are you sure you haven't already imported v5 accidentally before running your install & import script because your log indicates that you are invoking pester v5? Generally I suggest to rather use -MaximumVersion 4.99 on the Install-Module and Import-Module cmdlets.

[deadlydog]

Generally I suggest to rather use -MaximumVersion 4.99 on the Install-Module and Import-Module cmdlets.

I'd typically agree with this, but I've found there's issues with -MaximumVersion in PSGet v2 and it is often ignored or errs out. They're supposed to be fixed PSGet v3, but that's still in beta.

[petercharleston]

This is at the top of the pester task:
Pester Test Runner
Description : Run Pester tests by either installing the latest version of Pester at run time (if possible) or using the version shipped with the task (4.6.0)
Version : 0.1.3
Author : Pester

Could you advise on where or how i could have imported v5 accidentally please?

This pipeline was working on March 19th but did not work from Jun 16th

how could i force the task to use

Here is my pester script server.template.tests.ps1:
`[CmdletBinding()]
param(
[Parameter(Mandatory)]
[ValidateNotNullOrEmpty()]
[string]$ResourceGroupName
)

Install-Module -Name Pester -MaximumVersion 4.99 -Repository PSGallery
Import-Module -Name Pester -MaximumVersion 4.99

requires -Module Az.Accounts,Az.Resources

describe 'Template validation' {
it 'template passes validation check' {
$parameters = @{
TemplateFile = 'server.json'
ResourceGroupName = $ResourceGroupName
adminUsername = 'adam'
adminPassword = (ConvertTo-SecureString -String 'testing' -AsPlainText -Force)
vmName = 'TESTING'
}
(Test-AzResourceGroupDeployment @parameters).Details | should -Benullorempty
}
}`

I tried updating the pester script to include -MaximumVersion 4.99 but get the same error

here is my task in steps:

• task: Pester@0
  inputs:
  scriptFolder: "@{Path='$(System.DefaultWorkingDirectory)/server.template.tests.ps1'; Parameters=@{ResourceGroupName='$(azure_resource_group_name)'}}"
  resultsFile: "$(System.DefaultWorkingDirectory)/server.template.tests.XML"
  usePSCore: true
  run32Bit: False

[johndog]

In 5.0.2, $TestDrive seems to return $nullall the time now. Not listed as a breaking change.

(I upgraded from 4 to 5 unintentionally, looks like I need to go back)

[plastikfan]

Is it possible to mock out a scriptblock? I asked because scriptblocks are un-named functions and as far as I can see, Pester mocking depends on seeing the function invocation by it's name. I have a function that takes a scriptblock as a variable, but for the life of me, I've no idea how to mock out the script block and then test that that block has been invoked in a test.

Consider this function invoke-ForeachFile (invokes a custom script block for every file passed in via pipeline):

function invoke-ForeachFile {

  [Diagnostics.CodeAnalysis.SuppressMessageAttribute('PSShouldProcess', '')]
  [CmdletBinding(SupportsShouldProcess)]
  param(
    [Parameter(Mandatory, ValueFromPipeline = $true)]
    [System.IO.FileSystemInfo]$pipelineItem,

    [Parameter()]
    [scriptblock]$Condition = ( { return $true; }),

    [Parameter(Mandatory)]
    [scriptblock]$Body,

   # Some details omitted 
  )

  process {
    if (-not($broken)) {
      if (-not($pipelineItem.Attributes -band [System.IO.FileAttributes]::Directory)) {
        $pipelineFileInfo = [System.IO.FileInfo]$pipelineItem;

        if ($Condition.Invoke($pipelineFileInfo)) {
          $collection += $pipelineFileInfo;

          $shouldProcess = $PSCmdlet.ShouldProcess($pipelineFileInfo.FullName, $Description);
          $result = Invoke-Command -ScriptBlock $Body -ArgumentList @(
            $pipelineFileInfo, $index, $PassThru, $trigger, -not($shouldProcess)
          );

          # some details ommitted
        }
      }
    }
  }
}

... and the test:

  Context 'WhatIf not specified in custom scriptblock' {
    It 'given: files piped from same directory, WhatIf flagged' -Tag 'Current' {
      [int]$count = 0;
      [scriptblock]$block = { # THIS IS THE ScriptBlock, how to mock this? 
        [CmdletBinding(SupportsShouldProcess)]
        param(
          [System.IO.FileInfo]$FileInfo,
          [int]$Index,
          [System.Collections.Hashtable]$PassThru,
          [boolean]$Trigger
        )
        [ref]$count++; # THIS DOES NOT WORK (I'm not sure if Powershell forms a Closure here)
        Write-Host ">>> $($FileInfo.Name), WhatIf: $($WhatIf), Count: $([ref]$count)";
      }

      [string]$directoryPath = './Tests/Data/csv';
      Get-ChildItem $directoryPath | invoke-ForeachFile -body $block -Description "Operation blade" -WhatIf;
      $count | Should -Be 2;
    }
  }

As a workaround, I tried to use a $count variable inside the block, so that the test can test if the block was called, but even this doesn't work, even with the [ref] attribute!

So is this possible, if not is there a workaround?

[nohwnd]

@plastikfan this won't work because $count++ will create a new local variable called count and will assign the result of the parent count + 1 to it. So you will never see the result outside of that scope.

Same as if you'd do:

$count = 0
& {
    $count++

    Get-Variable -Scope Local -Name Count
    Get-Variable -Scope 1 -Name Count
}
"count is: $count"

Name                           Value
----                           -----
count                          1    <- the local var
count                          0    <- the parent var
count is: 0

But you can have are reference to an object and add to a property on it, that will work correctly, becuase you modify the value on the object instead of creating a local variable:

$c = @{ 
    Count = 0
}
& {
    $c.Count++
}

"count is: $($c.Count)"

It 'given: files piped from same directory, WhatIf flagged' -Tag 'Current' {
      $container = @{
          Count = 0
      }

      [scriptblock]$block = { # THIS IS THE ScriptBlock, how to mock this? 
        [CmdletBinding(SupportsShouldProcess)]
        param(
          [System.IO.FileInfo]$FileInfo,
          [int]$Index,
          [System.Collections.Hashtable]$PassThru,
          [boolean]$Trigger
        )
        $container.Count++; # THIS DOES NOT WORK (I'm not sure if Powershell forms a Closure here)
        Write-Host ">>> $($FileInfo.Name), WhatIf: $($WhatIf), Count: $($container.Count)";
      }

      [string]$directoryPath = 'c:\temp\';
      Get-ChildItem $directoryPath | invoke-ForeachFile -body $block -Description "Operation blade" -WhatIf;
      $container.Count | Should -Be 2;
    }

[plastikfan]

Thanks @nohwnd, I tried to use this technique (the [ref] attribute), because this is the way it has worked effectively for me in the past. The big difference that counts is the fact in my previous cases, the variable I was referencing was an object so it worked, in contrast with the way I was using it this time round on a scalar value, so thanks for clearing that up (I realise now that I can probably remove the [ref] attribute from my old code, it's probably redundant, but something tells me, it can't have worked, otherwise I wouldn't have put it in!). I assume, then that there is no resolution to being able to pass in a mock as a scriptblock. Just need to know so that I don't waste time trying to figure this out, I'll find another way like defining my own mock class/instance with embedded asserts, if Pester doesnt currently offer this capability.

[thor]

In 5.0.2, $TestDrive seems to return $nullall the time now. Not listed as a breaking change.

(I upgraded from 4 to 5 unintentionally, looks like I need to go back)

I do not experience the same here in Core, but I do experience that $TestDrive returns null in BeforeAll outside of a Describe block.

[GitHed]

I had this working in 4, but in 5, I can't see to use a variable in the describe block text:

This is basically a generic module checking script for module development that uses individual function files during dev, but gets packaged and signed as a single file during a build process.

Wrapping in BeforeAll doesn't allow the var to be used in the describe: Describe "Module Tests: $ModuleName" {.

BeforeAll {
    function Find-InParentPath {
        [CmdletBinding()]
        param(
            [String]
            $Path,

            [String]
            $Filter
        )

        Process {
            # Write-Verbose "$Path"
            $Items = Get-ChildItem -Path $Path -Filter $Filter
            if ($Items.Count -eq 1) {
                Write-Output (Get-Item -Path $Items.FullName)
            } else {
                Find-InParentPath -Path ((Get-Item -Path (Resolve-Path -Path $Path)).Parent.FullName) -Filter $Filter
            }
        }
    }

    $ManifestPath   = Find-InParentPath -Path $PSScriptRoot -Filter '*.psd1'
    $ModulePath     = $ManifestPath.Directory
    $ModuleName     = $ManifestPath.BaseName

    # Force reload the module from the file.
    Remove-Module -Force -Name $ModuleName
    $Manifest = Test-ModuleManifest -Path $ManifestPath
    Import-Module -Force -Name $ManifestPath

    # Did we load the right module?
    $Module = Get-Module -Name $ModuleName
    if (("$($Manifest.Name)" -ne "$($Module.Name)") -or ("$($Manifest.Name)" -ne "$($ModuleName)") -or ("$($Module.Name)" -ne "$($ModuleName)")) {
        Write-Warning "$($Manifest.Name) -ne $($Module.Name) -ne $($ModuleName)"
        throw "$($Manifest.Name) -ne $($Module.Name) -ne $ModuleName"
    }
}

Describe "Module Tests: $ModuleName" {
    Context 'Module Setup' {
        It "has the root module $ModuleName.psm1" {
            "$ModulePath\$ModuleName.psm1" | Should -Exist
        }

        It "has the a manifest file of $ModuleName.psd1" {
            "$ModulePath\$ModuleName.psd1" | Should -Exist
            "$ModulePath\$ModuleName.psd1" | Should -FileContentMatch "$ModuleName.psm1"
        }

        It "$ModuleName is valid PowerShell code" {
            $psFile = Get-Content -Path "$ModulePath\$ModuleName.psm1" -ErrorAction Stop
            $errors = $null
            $null = [System.Management.Automation.PSParser]::Tokenize($psFile, [ref]$errors)
            $errors.Count | Should -Be 0
        }

        It "Module's filelist files exist" {
            ForEach ($File in $Module.FileList) {
                $File | Should -Exist
            }
        }

        It "Module's filelist missing files" {
            $Files = Get-ChildItem -Recurse $ModulePath -File
            ForEach ($File in $Module.FileList) {
                $File | Should -BeIn $Files.FullName
            }
        }
    }
}

[nohwnd]

@GitHed What does not work in that example?

[GitHed]

@nohwnd

So, I was trying to simplify the issue to a specific case, but really, my idea of dynamic testing is broken.

I also probably should have just posted the full file instead of cutting parts to try to narrow the issue, I also had mistaken my first issue to just be a fault of my own.

I do seem to need to include a function in two locations now, in order to get the variable available for the describe context.

The current issue is I create context blocks inside of a function, and those don't seem to run because they aren't discovered.

Here's the full test file:

BeforeAll {
    function Find-InParentPath {
        [CmdletBinding()]
        param(
            [String]
            $Path,

            [String]
            $Filter
        )

        Process {
            # Write-Verbose "$Path"
            $Items = Get-ChildItem -Path $Path -Filter $Filter
            if ($Items.Count -eq 1) {
                Write-Output (Get-Item -Path $Items.FullName)
            } else {
                Find-InParentPath -Path ((Get-Item -Path (Resolve-Path -Path $Path)).Parent.FullName) -Filter $Filter
            }
        }
    }

    $ManifestPath   = Find-InParentPath -Path $PSScriptRoot -Filter '*.psd1'
    $ModulePath     = $ManifestPath.Directory
    $ModuleName     = $ManifestPath.BaseName

    Write-Verbose "ManifestPath: $($ManifestPath)"
    Write-Verbose "ModulePath:   $($ModulePath)"
    Write-Verbose "ModuleName:   $($ModuleName)"

    # Force reload the module from the file.
    Remove-Module -Force -Name $ModuleName
    $Manifest = Test-ModuleManifest -Path $ManifestPath
    Import-Module -Force -Name $ManifestPath


    # Did we load the right module?
    $Module = Get-Module -Name $ModuleName
    if (("$($Manifest.Name)" -ne "$($Module.Name)") -or ("$($Manifest.Name)" -ne "$($ModuleName)") -or ("$($Module.Name)" -ne "$($ModuleName)")) {
        Write-Warning "$($Manifest.Name) -ne $($Module.Name) -ne $($ModuleName)"
        throw "$($Manifest.Name) -ne $($Module.Name) -ne $ModuleName"
    }
}

Describe "Module Tests: $ModuleName" {
    BeforeAll {
        function Find-InParentPath {
            [CmdletBinding()]
            param(
                [String]
                $Path,

                [String]
                $Filter
            )

            Process {
                # Write-Verbose "$Path"
                $Items = Get-ChildItem -Path $Path -Filter $Filter
                if ($Items.Count -eq 1) {
                    Write-Output (Get-Item -Path $Items.FullName)
                } else {
                    Find-InParentPath -Path ((Get-Item -Path (Resolve-Path -Path $Path)).Parent.FullName) -Filter $Filter
                }
            }
        }

        function Test-Function {
            [CmdletBinding()]
            Param (
                [String]
                $Function,

                [String]
                $ModuleName,

                [ValidateScript({Test-Path -PathType Container -Path $_})]
                [String]
                $ModulePath,

                [ValidateScript({Test-Path -PathType Container -Path $_})]
                [String]
                $TestsPath,

                [ValidateSet('Private', 'Public')]
                [String]
                $Context,

                [Switch]
                $CheckFile
            )

            Context "Function: $ModuleName\$Function" {
                if ($CheckFile) {
                    It "$Function.ps1 should exist" {
                        "$ModulePath\$Context\$Function.ps1" | Should -Exist
                    }

                    if (Test-Path -Path "$ModulePath\$Context\$Function.ps1") {
                        It "$Function.ps1 is valid PowerShell code" {
                            $psFile = Get-Content -Path "$ModulePath\$Context\$Function.ps1" -ErrorAction Stop
                            $errors = $null
                            $null = [System.Management.Automation.PSParser]::Tokenize($psFile, [ref]$errors)
                            $errors.Count | Should -Be 0
                        }

                        It "$Function.ps1 should be an advanced Function" {
                            "$ModulePath\$Context\$Function.ps1" | Should -FileContentMatch 'Function'
                            "$ModulePath\$Context\$Function.ps1" | Should -FileContentMatch 'cmdletbinding'
                            "$ModulePath\$Context\$Function.ps1" | Should -FileContentMatch 'param'
                        }

                        It "$Function.ps1 should contain Write-Verbose blocks" {
                            "$ModulePath\$Context\$Function.ps1" | Should -FileContentMatch 'Write-Verbose'
                        }
                    }
                }

                Context "Get-Help $ModuleName\$Function" -Tag 'Help' {
                    $Help = Get-Help -Name $ModuleName\$Function -Category Function -ErrorAction SilentlyContinue

                    It "$Function should have a Synopsis" {
                        $Help.SYNOPSIS | Should -BeOfType String
                        $Help.SYNOPSIS | Should -Not -BeNullOrEmpty
                    }

                    It "$Function should have a Description" {
                        $Help.Description.Text | Should -BeOfType String
                        $Help.Description.Text | Should -Not -BeNullOrEmpty
                    }

                    It "$Function should have at least one Example" {
                        ForEach ($Example in $Help.Examples) {
                            $Example.example.code | Should -Not -BeNullOrEmpty
                        }
                    }

                    Context "Get-Help $Function -Parameter *" {
                        ForEach ($HelpParam in $Help.Parameters.Parameter) {
                            if (('WhatIf' -ne $HelpParam.Name) -and ('Confirm' -ne $HelpParam.Name)) {
                                It "Parameter $($HelpParam.Name) should have a Description" {
                                    $HelpParam.Description.Text | Should -BeOfType String
                                    $HelpParam.Description.Text | Should -Not -BeNullOrEmpty
                                }

                                if (('false' -eq $HelpParam.Required) -and ('switch' -ne $HelpParam.Type.Name)) {
                                    It "Parameter $($HelpParam.Name) should have a Default value if not Required" {
                                        $HelpParam.defaultValue | Should -Not -BeNullOrEmpty
                                    }
                                }
                            }
                        }
                    }
                }

                if ($CheckFile) {
                    Context "Pester Tests $Function" {
                        It ".\Tests\$Function.Tests.ps1 should exist" {
                            "$TestsPath\$Function.Tests.ps1" | Should -Exist
                        }

                        It ".\Tests\$Function.Tests.ps1 is valid PowerShell code" {
                            $psFile = Get-Content -Path "$TestsPath\$Function.Tests.ps1" -ErrorAction Stop
                            $errors = $null
                            $null = [System.Management.Automation.PSParser]::Tokenize($psFile, [ref]$errors)
                            $errors.Count | Should -Be 0
                        }
                    }
                }
            }
        }

        $ManifestPath   = Find-InParentPath -Path $PSScriptRoot -Filter '*.psd1'
        $ModulePath     = $ManifestPath.Directory
        $ModuleName     = $ManifestPath.BaseName
        $Module         = Get-Module -Name $ModuleName
        $TestsPath      = Find-InParentPath -Path $PSScriptRoot -Filter 'Tests'
        $PublicFiles    = Test-Path -Path "$($ModulePath)\Public" -PathType Container
        $PrivateFiles   = Test-Path -Path "$($ModulePath)\Private" -PathType Container

        Write-Verbose "ManifestPath: $($ManifestPath)"
        Write-Verbose "ModulePath:   $($ModulePath)"
        Write-Verbose "ModuleName:   $($ModuleName)"
        Write-Verbose "TestsPath:    $($TestsPath)"

        # Get the module functions
        $ModuleFunctions_Public  = Get-Command -Module $ModuleName -CommandType Function
        $ModuleFunctions_All     = $Module.Invoke({Get-Command -CommandType Function -Module $ModuleName})
        $ModuleFunctions_All     = $ModuleFunctions_All | Where-Object {$_.ModuleName -eq $ModuleName}
        $ModuleFunctions_Private = Compare-Object -ReferenceObject $ModuleFunctions_All -DifferenceObject $ModuleFunctions_Public | Select-Object -ExpandProperty InputObject

        Write-Verbose "ModuleFunctions_All.Count: $($ModuleFunctions_All.Count)"
        Write-Verbose "ModuleFunctions_Public.Count: $($ModuleFunctions_Public.Count)"
        Write-Verbose "ModuleFunctions_Private.Count: $($ModuleFunctions_Private.Count)"

        Write-Verbose "ModuleFunctions_Private: $($ModuleFunctions_Private)"
        Write-Verbose "ModuleFunctions_Public:  $($ModuleFunctions_Public)"
    }

    Context 'Module Setup' {
        It "has the root module $ModuleName.psm1" {
            "$ModulePath\$ModuleName.psm1" | Should -Exist
        }

        It "has the a manifest file of $ModuleName.psd1" {
            "$ModulePath\$ModuleName.psd1" | Should -Exist
            "$ModulePath\$ModuleName.psd1" | Should -FileContentMatch "$ModuleName.psm1"
        }

        It "$ModuleName is valid PowerShell code" {
            $psFile = Get-Content -Path "$ModulePath\$ModuleName.psm1" -ErrorAction Stop
            $errors = $null
            $null = [System.Management.Automation.PSParser]::Tokenize($psFile, [ref]$errors)
            $errors.Count | Should -Be 0
        }

        It "Module's filelist files exist" {
            ForEach ($File in $Module.FileList) {
                $File | Should -Exist
            }
        }

        It "Module's filelist missing files" {
            $Files = Get-ChildItem -Recurse $ModulePath -File
            ForEach ($File in $Module.FileList) {
                $File | Should -BeIn $Files.FullName
            }
        }
    }

    Context 'Functions' {
        if ($null -ne $ModuleFunctions_Private) {
            if (Get-Member -Name 'Count' -InputObject $ModuleFunctions_Private) {
                if ($ModuleFunctions_Private.Count -gt 0) {
                    Context 'Private Functions' -Tag 'Private' {
                        ForEach ($Function in $ModuleFunctions_Private) {
                            Test-Function -Function $Function -ModuleName $ModuleName -ModulePath $ModulePath -TestsPath $TestsPath -Context 'Private' -CheckFile:$PrivateFiles
                        }
                    }
                }
            }
        }

        if ($null -ne $ModuleFunctions_Public) {
            if (Get-Member -Name 'Count' -InputObject $ModuleFunctions_Public) {
                Context 'Public Functions' -Tag 'Public' {
                    ForEach ($Function in $ModuleFunctions_Public) {
                        Test-Function -Function $Function -ModuleName $ModuleName -ModulePath $ModulePath -TestsPath $TestsPath -Context 'Public' -CheckFile:$PublicFiles
                    }
                }
            }
        }
    }
}

And it doesn't matter if it's combined or the single file, the Functions context will not have any discoverable tests, and thus just won't run.

And for reference, here's the difference in test numbers between 4 and 5.

5.0.2

Starting discovery in 1 files.
Discovery finished in 48ms.
Tests completed in 1.12s
Tests Passed: 5, Failed: 0, Skipped: 0 NotRun: 0

4.9.0

    ____            __
   / __ \___  _____/ /____  _____
  / /_/ / _ \/ ___/ __/ _ \/ ___/
 / ____/  __(__  ) /_/  __/ /
/_/    \___/____/\__/\___/_/
Pester v4.9.0
Executing all tests in 'F:\Projects\Modules\ModuleName\Tests\ModuleName.Tests.ps1'
Tests completed in 24.43s
Tests Passed: 225, Failed: 52, Skipped: 0, Pending: 0, Inconclusive: 0

[nohwnd]

@GitHed Your Test-Function is defined in BeforeAll which will make it run after Discovery, so when you reach the Functions Context it won't generate anything because the Test-Function is not defined yet. Also some of your variables won't be defined in your tests like $Help won't be defined in your tests, because you define them in the body of Describe / Context.

[GitHed]

@nohwnd Thanks for the response!

So I re-arranged that code because reading suggested it needed to be in a beforeall block to work. But I've basically gone back to what I had before...

Side tracking here a little:
It does seem that there's an empty context variable in the it blocks:
Get-Content : Cannot find path 'F:\Projects\Modules\ModuleName\@{Name=; Path=}\.ps1' because it does not exist.

I pass a string (which granted is not there), but not an object with properties.

$PesterResults.Passed[25]


Name              : Context leaks?
Path              : {Module Tests: ModuleName, Functions, Public Functions, Function: ModuleName\FunctionName...}
Data              : {}
ExpandedName      : Context leaks?
Result            : Passed
ErrorRecord       : {}
StandardOutput    :
Duration          : 00:00:00.0203784
ItemType          : Test
Id                :
ScriptBlock       :
                                $Context.name | Should -BeNullOrEmpty
                                $Context.path | Should -BeNullOrEmpty

Tag               :
Focus             : False
Skip              : False
Block             : [-] Function: ModuleName\FunctionName
First             : True
Last              : False
Include           : False
Exclude           : False
Explicit          : False
ShouldRun         : True
Executed          : True
ExecutedAt        : 7/29/2020 3:35:54 AM
Passed            : True
Skipped           : False
UserDuration      : 00:00:00.0083888
FrameworkDuration : 00:00:00.0119896
PluginData        :
FrameworkData     :

Changing the parameter name of the test-function function stops outputting the empty context object in the error.

[-] Module Tests: ModuleName.Functions.Public Functions.Function: ModuleName\FunctionName.FunctionName.ps1 should exist 12ms (7ms|5ms)
 Expected path 'F:\Projects\Modules\ModuleName\\.ps1' to exist, but it did not exist.

You can check this with this simple test file:

Describe "describe" {
    Context 'Context' {
        it 'it' {
            $Context | Should -Not -BeNullOrEmpty
        }
    }
}

Now, how do I actually get a variable into an it block? I noticed none of the examples really show that, but they do suggest this should work:

Context "Function: $ModuleName\$Function" {
        BeforeAll {
            function Get-FunctionFilePath {
                Write-Output "$($ModulePath)\$($Type)\$($Function).ps1"
            }
        }

        It "$Function.ps1 should exist" {
            $FilePath = Get-FunctionFilePath
            $FilePath | Should -Exist
        }

Defining Get-FunctionFilePath in a Begin section or before the Context block of the Test-Function function just results in not defined in the it blocks, placing it in BeforeAll makes it available, but still null.

[plastikfan]

Using the & operator to invoke a function in a Pester test is proving to be difficult. The following code DOES work:

  BeforeAll {
    function Show-Greeting {
      param(
        [string]$Person
      )
      Write-Host "Hello $Person";  
    }

  Context 'given: a test function' {
    It 'should: be invokable via the & operator' {
      $parameters = @{
        'Person' = 'The Nephilim'
      }
      & Show-Greeting @parameters;
    }
  }

The invoke is directly inside the test case. However, a module function that uses & operator to invoke a function indirectly does not work. The only way that this works, is when the function being invoked is moduule source code, not a test function defined in the BeforeAll or It block. So this does not work:

      It 'Should: traverse creating directories only' -Tag 'Current' {

        function Show-InternalMirror {
          param(
            [Parameter(Mandatory)]
            [System.IO.DirectoryInfo]$Underscore,

            [Parameter(Mandatory)]
            [int]$Index,

            [Parameter(Mandatory)]
            [System.Collections.Hashtable]$PassThru,

            [Parameter(Mandatory)]
            [boolean]$Trigger,

            [Parameter(Mandatory)]
            [string]$Format
          )

          [string]$result = $Format -f ($Underscore.Name);
          Write-Host "Custom function; Show-InternalMirror: '$result'";
          @{ Product = $Underscore }
        }
        [System.Collections.Hashtable]$parameters = @{
          'Format' = '---- {0} ----'
        }
        Invoke-MirrorDirectoryTree -Path $sourcePath `
          -DestinationPath $destinationPath -CreateDirs `
          -Functee 'Show-InternalMirror' -FuncteeParams $parameters -WhatIf:$whatIf;
      }

So Invoke-MirrorDirectoryTree is a module function that invokes Functee with the & operator. The only (very un-desirable) workaround is to include the test function Show-InternalMirror within the module, but I don't like doing this because it is a function that is of no benefit to the end-user and is not required in a production environment. I have a lot of testcases that are afflicted with this issue, which means I have a lot of duff test functions in the production module.

The error shown is something like this

     | Invoke-TraverseDirectory(top-level) Error: (Exception calling "Invoke" with "4" argument(s): "The term
     | 'Show-InternalMirror' is not recognized as the name of a cmdlet, function, script file, or operable program. Check
     | the spelling of the name, or if a path was included, verify that the path is correct and try again."), for item:
     | 'traverse'

Is there an issue with using & operator for internally defined Pester test functions and how can I define a test function within a Pester test case that can be invoked by a module function under test?

EDIT: I tried InModuleScope and that makes no difference.

[plastikfan]

Are there any guidelines for writing Pester tests that run on local machine but also work on Azure? At the moment, you can write tests thast work locally, but they don't run on Azure, there are quite a few differences. I'm finding that if I tweek my build process (I'm using Invoke-Build) to try and get tests running on Azure, they will break the regular local run. Problems are mainly to do with importing the built module and getting access to the exported functions that don't seem to be available when running the tests on Azure.

[hugh-martin]

I realize this is probably more of a PowerShell issue than a Pester issue, but are there any workarounds for testing functions that return PowerShell v5 class objects? When I debug a Pester test, I can see the object, but the test itself sees it as null.

[nohwnd]

@plastikfan Show-InternalMirror question: Did that work in Pester 4? To me it seems like it should never work, or work for a very specific case, but not for many others.

@plastikfan Azure question: Do you have an example of pipeline that does not work, while it works locally (or the other way around)? Is this just Pester v5 problem?

[nohwnd]

@hugh-martin could you show an example please? The class support in Pester is quite limited, though.

[plastikfan]

@plastikfan Show-InternalMirror question: Did that work in Pester 4? To me it seems like it should never work, or work for a very specific case, but not for many others.

Hi @nownd, as I'm relatively new to Pester, I am not sure if this would work in v4. I've just come to accept this is a current limitation of Pester. Using the & op on a named function simply does not work unfortunately, so at the moment as a workaround I have to define dummy test functions that get delivered with the module (which is less than desirable).

@plastikfan Azure question: Do you have an example of pipeline that does not work, while it works locally (or the other way around)? Is this just Pester v5 problem?

And this is not an issue with Pester, it is purely an Azure issue and what makes this worse is that there is next to no information about testing Powershell modules in an Azure pipeline. There is documentation on using powershell scripting, but modules have not been adequately catered for. The main issue is the scope of a powershell session within a task/job/stage etc, because what you'll find is you can define a Job to bootstrap the session (install dependencies), but the next job will fail, becasue there is a new session and what what bootstrapped no longer exits. I was hoping somebody else here might have had better luck testing PowerrShell modules on Azure. I'm attending a virtual tech meeting with MS talking about Azure pipelines. I'm going to bring the issue of PS modules on Azure and ask why there's no documetation on this.

[nohwnd]

@plastikfan you can pass a function reference which will work, and probably will also still work nicely with the name. Because & will invoke both name and well as the commnd info object. -Functee (Get-Command Show-InternalMirror), and [object] $Functee

[nohwnd]

@plastikfan I run all Pester tests in v5.0 in AzDO, but my way might be specific.

[plastikfan]

@plastikfan I run all Pester tests in v5.0 in AzDO, but my way might be specific.

Ok, is the pipeline public? I'd like to take a look to see if I can resolve my issues.

[plastikfan]

@plastikfan you can pass a function reference which will work, and probably will also still work nicely with the name. Because & will invoke both name and well as the commnd info object. -Functee (Get-Command Show-InternalMirror), and [object] $Functee

I didnt know there was such a thing as a function reference (via Get-Command). I can see that will resolve another issue that I have developed a work-around for, which having seen this is now redundant. I'll try this technique out in my test and hopefully that will work.

[nohwnd]

@plastikfan yes it is public: https://nohwnd.visualstudio.com/Pester/_build?definitionId=6&_a=summary

[plastikfan]

Great thanks @nohwnd I'll study that in great detail and see where I went wrong. Cheers

[hugh-martin]

@nohwnd It looks like I had the same problem as @AlexRedkin. I had to move my mock into a BeforeAll or it block. Sorry for the false alarm.

[hugh-martin]

Am I correct regarding BeforeAll and AfterAll blocks:

• A BeforeAll block can reside above the first Describe block.
• An AfterAll block cannot reside above the first Describe block.
• BeforeAll / AfterAll blocks can be the first / last block within a Describe block, respectively.
• BeforeAll / AfterAll blocks can be the first / last block within a Context block, respectively.
• BeforeAll / AfterAll blocks cannot reside within It blocks.
• BeforeAll blocks cannot contain tests, but can contain mocks.
• AfterAll blocks cannot contain tests or mocks (or at least mocks would not do anything).

[nohwnd]

@hugh-martin

• A BeforeAll block can reside above the first Describe block. - yes
• An AfterAll block cannot reside above the first Describe block. - yes, but that should be temporary limitation
• BeforeAll / AfterAll blocks can be the first / last block within a Describe block, respectively. - yes, in fact they can be anywhere in the respective block
• BeforeAll / AfterAll blocks can be the first / last block within a Context block, respectively. - yes, same as above. Describe and Context are for the most part the same thing, where there are not the same is Mock -Scope, there we distinguish Context from Describe, but in all other places they are interchangeable
• BeforeAll / AfterAll blocks cannot reside within It blocks. - yes
• BeforeAll blocks cannot contain tests, but can contain mocks. - yes, mocks can be placed in BeforeAll, but tests should be placed outside of it
• AfterAll blocks cannot contain tests or mocks (or at least mocks would not do anything). - yes, same as above, but as you say mocking in teardown is pointless

[hugh-martin]

Thanks @nohwnd.

[hugh-martin]

I'm trying to make sure I understand how scoping works, but my test script is getting some unexpected results.

BeforeAll {
    function GoForIt {
        Write-Output "bob"
    }
}

Describe "Function test" {

    Context "Test 1" {

        it "'bob' (hard-coded) should be bob" {
            $myvar = GoForIt
            $myvar | Should -Be "bob"
        }

        it "There should be a myvar variable (1)" {
            (Get-Variable myvar -ErrorAction SilentlyContinue | Measure-Object).Count | Should -Be 1 -Because "the variable was just defined and used"
        }
    }

    Context "Test 2" {

        it "There should NOT be a myvar variable (1)" {
            (Get-Variable myvar -ErrorAction SilentlyContinue | Measure-Object).Count | Should -Be 0 -Because "this is a new Context block"
        }

    }

    Context "Test 3" {

        BeforeAll {
            $myvar = GoForIt
        }

        it "There should be a myvar variable (1)" {
            (Get-Variable myvar -ErrorAction SilentlyContinue | Measure-Object).Count | Should -Be 1 -Because "The variable was defined in the BeforeAll block"
        }

        it "'$myvar' (variable) should be bob" {
            $myvar | Should -Be "bob"
        }

        it "There should be a myvar variable (2)" {
            (Get-Variable myvar -ErrorAction SilentlyContinue | Measure-Object).Count | Should -Be 1
        }
    }
}

Here are the results:

Describing Function test
 Context Test 1
   [+] 'bob' (hard-coded) should be bob 10ms (7ms|2ms)
   [-] There should be a myvar variable (1) 14ms (12ms|2ms)
    Expected 1, because the variable was just defined and used, but got 0.
    at (Get-Variable myvar -ErrorAction SilentlyContinue | Measure-Object).Count | Should -Be 1 -Because "the variable was just defined and used", C:\MyTemp\Test1.Tests.ps1:17
    at <ScriptBlock>, C:\MyTemp\Test1.Tests.ps1:17
 Context Test 2
   [+] There should NOT be a myvar variable (1) 8ms (6ms|2ms)
 Context Test 3
   [+] There should be a myvar variable (1) 8ms (6ms|2ms)
   [+] '' (variable) should be bob 9ms (7ms|2ms)
   [+] There should be a myvar variable (2) 7ms (5ms|2ms)
Tests completed in 765ms
Tests Passed: 5, Failed: 1, Skipped: 0 NotRun: 0

• The failure in Context Test 1 is unexpected.

• The result in Context Test 2 is good.

• In Context Test 3, all tests are successful, but the $myvar value does not appear in the output of the second test, yet the test is successful.

[nohwnd]

@hugh-martin

• The failure in Context Test 1 is unexpected.
  
  
  
  • There should be a myvar variable (1) - this fails because you defined the variable in the first It not the second one. Variables don't transfer between It blocks because you would not want the data of all tests to be muddled together.
  
  

• The result in Context Test 2 is good.

  • True for the same reason, the variable was defined in It and will not leave that It, so you don't see it defined in another context.

• In Context Test 3, all tests are successful, but the $myvar value does not appear in the output of the second test, yet the test is successful.

  • this is a regressions in v5 I capture the name of the test, during discovery, so at that point $myVar is not defined, because the BeforeAll that defines it did not run yet, and so the value does not appear in the name. The test itself will pass because that scriptblock runs durin the Run phase, and BeforeAll will run first, then the test will run, and the variable will be available. I don't have a workaround for the name not being defined.

[hugh-martin]

So when you say

BeforeEach, It and AfterEach now run in the same scope, but are still isolated from their parent to avoid leaking variables and test cross-pollution,

it doesn't mean that all It blocks in a single parent (Context or Describe) block share the same scope as I thought. Does this mean that v5 is much more tightly scoped than v4? I apologize for the newbie questions; just trying to get the basics down. Pester has been great for operational validation, but I'm just starting to use it for code testing. Understanding scope means knowing how to structure the tests (or at least knowing how NOT to!). Thanks for the help.

[hugh-martin]

I rewatched your Pester 5 video and I think I understand how to restructure my code and code blocks to get the benefits of Pester 5.

[nohwnd]

@hugh-martin No problem. Yes in some ways the scoping is more tight and logical.

This is how it looks like in v5:

# & { } is a new scope
# . { } is dot-sourcing into the current scope
# Each variable represents the scriptblock provided, to the 
# Pester block of that name, e.g. BeforeAll { $a = 10 }, 
# then in this diagram $BeforeAll = { $a = 10 }
# or It "is 10" { $a | Should -Be 10 }, 
# then $It = { $a | Should -Be 10 }

& {
    . $BeforeAll
    & {
        . $BeforeEach
        . $It
        . $AfterEach
    }
    . $AfterAll
}

And this is how it looked in v4.

. $BeforeAll
& {
    . $BeforeEach
    &{
        . $It
    }
    . $AfterEach
}
. $AfterAll

Notice that v5 has one extra scope that isolates each group (that would be a top-level Describe block) from the script itself. So variables don't leak from first Describe in the file to the next one.

And that It is in the same scope as BeforeEach and AfterEach, this is to ensure that you can move a piece of your test into common setup and get the same behavior, and that you can use values that you set in It in your teardown (e.g. when making sure a file is deleted.

For completeness here is how it looks if we include ensuring that After* blocks will run, try catch or try finally does not create a new scope so it does not have any effect on our discussion, but I already wrote it so maybe you might find it useful:

& {
    try{
        . $BeforeAll
        & {
            try {
                . $BeforeEach
                . $It
            }
            finally {
                . $AfterEach
            }
        }
    }
    finally {
        . $AfterAll
    }
}

[hugh-martin]

This is super helpful. Thank you.

[nohwnd]

Comment 100 💯

[plastikfan]

I was wondering what other people think of the current way that Should assertion failures are reported. In my view, I think it is slightly misleading and could/should be reported in a much more succint and direct manner. Eg when I see as Should assertion failure (I'm not going to consider the "Because" parameter in this):

RuntimeException: Expected 19, but got 18.
MethodInvocationException: Exception calling "Invoke" with "4" argument(s): "Expected 19, but got 18."

I am not immediatetly drawn to a Should assertion failure. The "invoke" with "4" argument(s)" is a red herring as its got nothing to do with why the exception occurred. Instead this tends to lead me into a blind alley of perhaps ive invoked with either too few or too many arguments.

Instead, all I need to see is that its a Should assertion failure, that immediately shuts down any other possible cause of the error. I know that "Exception: Expected 19, but got 18" is the issue, but incluing the term Should assertion failure would vastly improve this error reporting. I accept I my thought process may be different to others, but this has always bugged me and has led me donw the wrong path too many times, so I thought I would like to hear others' opinions. And actually, MethodInvocationException is a bit of a misnoma too, for the same reason just stated. Couldn't we have a new Pester execption along the lines of ShouldAssertionException?

[nohwnd]

@plastikfan do you have an example of code that produces this? And on which version of PowerShell and Pester?

Describe "a" {
    It "b" {
        18 | Should -Be 19
    }
}

Describing a
  [-] b 7ms (6ms|1ms)
   Expected 19, but got 18.
   at 18 | Should -Be 19, C:\Users\jajares\Desktop\should.tests.ps1:3
   at <ScriptBlock>, C:\Users\jajares\Desktop\should.tests.ps1:3
Tests completed in 322ms
Tests Passed: 0, Failed: 1, Skipped: 0 NotRun: 0

[plastikfan]

Hi @nownd, looks like I need to look at this aspect of my code again. I thought the excpetion was occuring as a result of the assertion failure, but it's probably something else.

[majkinetor]

I switched to Pester 5 on latest project, here are my observations so far:

1. Code lenses in VSCode is a killer feature. Entire team is more effective now as they always had to previously do mumbo-jumbo to make current test easier to debug (moving it to a top and putting exit bellow it, skipping contexts before etc.)
2. Tags/Skip on everything was must have - now I can run a selection of tests from the shell without any problem, I can also mark any pester entity with predefined tags and devs can quickly execute them.
3. VSCode integrated console crashes every 10-20 minutes or so, after executing tests. Not a big deal.
4. I miss integrated HTML report such as Format-Pester that can be easily and without any dependencies produced on CI builds. This is particularly problematic now that JUnit format is removed, the one that most CI servers know how to view (although I still prefer just offline report and not depend on 3rd party).
5. I really think tests should have time shown (#1611) in the output (so I don't have to look into XML just for it)

All in all, its totally great and much better then 5- versions. I use it INSTEAD dotnet test for CORE REST services. This had some challenges like code coverage and metrics that I had to overcome (not pester related but I will see to publish my InfluxDb metric sender script and Grafana dashboard that tracks behaviors across multiple runs; code coverage turned out to be doable with dotCover CLI).

The only thing that I really miss now are parallel builds.

[nohwnd]

@majkinetor
1 & 2 That is great to hear 🥳
3 Same here, but it seems to be more stable in the Preview version of the plugin, but still pretty flaky.
4 there is proposal for html output I think, but I don't want to integrate yet another tool. Fixing JUnit output is long overdue, but I will hopefully get to it.
5 Maybe I would consider it for the Detailed output. will have to look how it looks like.

[majkinetor]

4 there is proposal for html output I think, but I don't want to integrate yet another tool. Fixing JUnit output is long overdue, but I will hopefully get to it.

Sounds like basic stuff IMO. Its standalone too (doesn't introduce more complexity). I am sure that there are 0 people who would object on having it, plus it has potential to popularize the tool (as nice dashboards always do). I hope to find some time to make one.

[LethiferousMoose]

@nohwnd Based on your comment above:
"An AfterAll block cannot reside above the first Describe block. - yes, but that should be temporary limitation".

Is this something that is coming the future?
What is the best way to migrate tests that currently use a try/finally block for setup/teardown if AfterAll is not yet supported?

[nohwnd]

@LethiferousMoose implemented in https://github.com/pester/Pester/pull/1707 shipped in 5.1.0-beta2

[rcdailey]

I'm struggling with how to share variables between test cases in Pester 5. I do something like this:

$InstallDir = 'C:\Program Files\MyProduct'

Describe "Product Installation" {
    Context "Bin Directory" {
        It "Program exists" {
            Join-Path $InstallDir 'Bin\MyProgram.exe' | Should -Exist
        }
    }
}

Inside the It block, it can't access the value for $InstallDir. If I put the definition of this variable inside Describe.BeforeAll or Describe.BeforeEach, it still doesn't work. How am I supposed to share variables between multiple test cases like this?

[LethiferousMoose]

BeforeAll normally works for me, I do that in a lot of tests. Here's one of mine that works:

Describe 'Write-Logger' {
    BeforeAll {
        [string] $projectRoot = $PSScriptRoot + '/../../../'
        [string] $logPath = $projectRoot + 'log'
        [string] $logFolder = "${PID}_" + $Host.InstanceId.ToString()
        [string] $logFile = '000_Pester_Invoke-Pester.log'
        [string] $logFilePath = Join-Path -Path $logPath -ChildPath ($logFolder + '/' + $logFile)
    }

    Context 'Logging enabled' {
        It 'Write with default settings' {
            Write-Logger -Message 'TestDefault1', 'TestDefault2'
            Start-Sleep -Seconds 1

            Test-Path $logFilePath | Should -Be $true
            [string[]] $fileContent = Get-Content $logFilePath
            $fileContent.Count | Should -Be 2
            $fileContent[0] | Should -Match '\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3} TestDefault1'
            $fileContent[1] | Should -Match '\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3} TestDefault2'
        }
    }
}

[nohwnd]

@rcdailey put $InstallDir = 'C:\Program Files\MyProduct' into BeforeAll as @LethiferousMoose suggests:

BeforeAll { 
    $InstallDir = 'C:\Program Files\MyProduct'
}

[chscott]

I maintain a module, and I like to be able to both test individual functions as I'm modifying them and then test those same functions once they are bundled into a module. Prior to v5, I could have both test modes in the same file, but I've been unable to make that work due to the new scoping in v5. Now I have two test files like these:

# Load the module
Import-Module 'MyModule.psm1'

InModuleScope 'MyModule' {
  Describe 'MyFunction' {
    # Tests
  }
}

Describe 'MyFunction' {
  BeforeAll {
    # Load the script
    . 'MyFunction.ps1'
  }

  # Tests
}

This arrangement works for me, but I have to maintain the tests in two different files, which is a pain. Is it possible to eliminate this type of duplication?

[rjmholt]

Finally piling into this issue with a question myself...

I've got a set of tests where I want to generate the testcases based on files that are checked in to the repo, i.e. I can't declare them statically:

    # Test cases come from the examples folder
    $exampleDir = (Resolve-Path "$PSScriptRoot/../../examples").Path
    $examples = Get-ChildItem -LiteralPath $exampleDir

    $testCases = $examples | ForEach-Object {
        $basePath = $_.FullName
        $scriptPath = Join-Path -Path $basePath -ChildPath 'test.ps1'
        $templatePath = Join-Path -Path $basePath -ChildPath 'template.json'

        if ((Test-Path -LiteralPath $scriptPath) -and (Test-Path -LiteralPath $templatePath))
        {
            @{ Name = $_.Name; ScriptPath = $scriptPath; TemplatePath = $templatePath }
        }
    }

(This is because I want my examples to be continuously validated against my module).

But trying to declare $testCases anywhere (BeforeAll at the top level in the file, BeforeAll in Describe, directly in Describe) will evaluate the code, but not pass it through to my test. I assume this is a consequence of the discovery phase concept.

Is there a way for me to dynamically generate test cases, or otherwise dynamically produce tests, in Pester 5?

[nohwnd]

@chscott please show me how you did it before. That sounds like something you should definitely be able to do in v5 :)

@rjmholt you are looking for the mythical Generating tests section of my usage docs PR, that section I am yet to write.

Here are some common gotchas and the discovery process explained

But there are few examples of generating tests already in our tests. This one being most complete. But there are more in that file. https://github.com/pester/Pester/blob/d689c3cf8834ff64e0dda735dcb35db933c12125/tst/Pester.RSpec.ts.ps1#L1189-L1240

In that example I am combining all three things together: passing external data, and generating describes & contexts, and generating Its. And also the new ability to expand dotted code, and passing any array to -Foreach / -TestCases.

More examples are also in these release notes: https://github.com/pester/Pester/releases/tag/5.1.0-beta1 Just know that New-TestContainer was renamed to New-PesterContainer in 5.1.0-rc1

[chscott]

@chscott please show me how you did it before. That sounds like something you should definitely be able to do in v5 :)

I no longer have an example of it that I could readily find, so let me show what doesn't work in v5. I'm not married to this approach; if there's a better way to accomplish what I'm trying to do, I'm all ears.

$test = @{
  Name        = 'returns True if the arrays are equal'
  ScriptBlock = {
    $test = @(
      'element1',
      'element2'
    )
    $expected = @(
      'element1',
      'element2'
    )
    Assert-ArrayEquality $test $expected | Should -Be $true
  }
}

$describe = @{
  Name        = 'Assert-ArrayEquality'
  ScriptBlock = {
    Write-Host "Inside Describe: test.Name is '$($test.Name)'"

    It -Name $test.Name -Test $test.ScriptBlock
  }
}

if ($global:MODULE) {
  Write-Host "Before InModuleScope: test.Name is '$($test.Name)'"
  InModuleScope $global:MODULE {
    Describe -Name $describe.Name -Fixture $describe.ScriptBlock
  }
} else {
  Write-Host "Before Describe: test.Name is '$($test.Name)'"
  Describe -Name $describe.Name -Fixture $describe.ScriptBlock
}

The central challenge is trying to reuse the same Describe script block in and out of a module context. When $global:MODULE is not set, things work fine:

Starting discovery in 1 files.
Discovering in C:\src\module\common\test\unit\Assert-ArrayEquality.Tests.ps1.
Before Describe: test.Name is 'returns True if the arrays are equal'
Inside Describe: test.Name is 'returns True if the arrays are equal'
Found 1 tests. 8ms
Discovery finished in 12ms.

Running tests from 'C:\src\module\common\test\unit\Assert-ArrayEquality.Tests.ps1'
Describing Assert-ArrayEquality
  [+] returns True if the arrays are equal 1ms (1ms|1ms)
Tests completed in 78ms
Tests Passed: 1, Failed: 0, Skipped: 0 NotRun: 0

But when I set $global:MODULE, I lose access to the variables.

Starting discovery in 1 files.
Discovering in C:\src\module\common\test\unit\Assert-ArrayEquality.Tests.ps1.
Before InModuleScope: test.Name is 'returns True if the arrays are equal'
System.Management.Automation.ParameterBindingValidationException: Cannot bind argument to parameter 'Name' because it is an empty string.

I have the same issue if I try to move the InModuleScope portion to just the It level:

$test = @{
  Name        = 'returns True if the arrays are equal'
  ScriptBlock = {
    $test = @(
      'element1',
      'element2'
    )
    $expected = @(
      'element1',
      'element2'
    )
    Assert-ArrayEquality $test $expected | Should -Be $true
  }
}

$describe = @{
  Name        = 'Assert-ArrayEquality'
  ScriptBlock = {
    Write-Host "Inside Describe: test.Name is '$($test.Name)'"

    if ($global:MODULE) {
      InModuleScope $global:MODULE {
        It -Name $test.Name -Test $test.ScriptBlock
      }
    } else {
      It -Name $test.Name -Test $test.ScriptBlock
    }
  }
}

Describe -Name $describe.Name -Fixture $describe.ScriptBlock

[chscott]

I got some clues from https://github.com/pester/Pester/issues/1603 that I was able to use to get this working.

$test = @{
  Name        = 'returns True if the arrays are equal'
  ScriptBlock = {
    $test = @(
      'element1',
      'element2'
    )
    $expected = @(
      'element1',
      'element2'
    )
    Assert-ArrayEquality $test $expected | Should -Be $true
  }
}

$describe = @{
  Name        = 'Assert-ArrayEquality'
  ScriptBlock = {
    Write-Host "Inside Describe: test.Name is '$($test.Name)'"
    It -Name $test.Name -Test $test.ScriptBlock
  }
}

if ($global:MODULE) {
  Write-Host "Before InModuleScope: test.Name is '$($test.Name)'"
  InModuleScope $global:MODULE -Parameters @{ Name = $describe.Name; Fixture = $describe.ScriptBlock } {
    param($Name, $Fixture)
    Describe -Name $Name -Fixture $Fixture
  }
} else {
  Write-Host "Before Describe: test.Name is '$($test.Name)'"
  Describe Describe -Name $describe.Name -Fixture $describe.ScriptBlock
}

[nohwnd]

@chscott Seems a bit complicated. Seems to me that you just need to wrap the whole file in InModuleScope when the $globalModule is defined, or don't wrap it when executing normally. So I would just put all the test code in the scriptblock and then either invoke it in module scope or not. This way you can use the normal Pester syntax and avoid juggling with variables.

$script:aaa = "script"
Get-Module m | Remove-Module
New-Module -Name m -ScriptBlock {
    $script:aaa = "module"
} -PassThru | Import-Module

$scriptBlock = { 
    Describe "Assert-ArrayEquality" {
        It "I run outside of module because the script variable is not shadowed by the module variable" {
            $script:aaa | Should -Be "script"
        }

        It "I run in module because the script variable is shadowed by the module variable" {
            $script:aaa | Should -Be "module"
        }
    }
}

# outside module
$global:MODULE = $null
if ($global:Module) { 
    InModuleScope -ModuleName $global:Module -ScriptBlock $scriptBlock
}
else { 
    & $scriptBlock
}

# in module
$global:MODULE = "m"
if ($global:Module) { 
    InModuleScope -ModuleName $global:Module -ScriptBlock $scriptBlock
}
else { 
    & $scriptBlock
}

# output: 

Found 4 tests. 14ms
Discovery finished in 24ms.

Running tests from 'C:\Users\jajares\Desktop\in-out.tests.ps1'
Describing Assert-ArrayEquality
  [+] I run outside of module because the script variable is not shadowed by the module variable 7ms (3ms|4ms)
  [-] I run in module because the script variable is shadowed by the module variable 9ms (8ms|1ms)
   Expected strings to be the same, but they were different.
   String lengths are both 6.
   Strings differ at index 0.
   Expected: 'module'
   But was:  'script'
   at $script:aaa | Should -Be "module", C:\Users\jajares\Desktop\in-out.tests.ps1:14
   at <ScriptBlock>, C:\Users\jajares\Desktop\in-out.tests.ps1:14

Describing Assert-ArrayEquality
  [-] I run outside of module because the script variable is not shadowed by the module variable 8ms (5ms|3ms)
   Expected strings to be the same, but they were different.
   String lengths are both 6.
   Strings differ at index 0.
   Expected: 'script'
   But was:  'module'
   at $script:aaa | Should -Be "script", C:\Users\jajares\Desktop\in-out.tests.ps1:10
   at <ScriptBlock>, C:\Users\jajares\Desktop\in-out.tests.ps1:10
  [+] I run in module because the script variable is shadowed by the module variable 4ms (2ms|1ms)
Tests completed in 409ms
Tests Passed: 2, Failed: 2, Skipped: 0 NotRun: 0

[plastikfan]

I'm having problems with InModuleScope, trying to test an internal module function, so I'm making use of InModuleScope as follows, for function under test format-ColouredLine:

  BeforeAll {
    Get-Module Elizium.Loopz | Remove-Module
    Import-Module .\Output\Elizium.Loopz\Elizium.Loopz.psm1 `
      -ErrorAction 'stop' -DisableNameChecking;

    InModuleScope Elizium.Loopz {
      [string]$script:LineKey = 'LOOPZ.HEADER-BLOCK.LINE';
      [string]$script:CrumbKey = 'LOOPZ.HEADER-BLOCK.CRUMB-SIGNAL';
      [string]$script:MessageKey = 'LOOPZ.HEADER-BLOCK.MESSAGE';

      function show-result {
        param(
          [string]$Ruler,
          [object[]]$Snippets
        )
        Write-Host "$Ruler";
        Write-InColour -TextSnippets $Snippets;
      }
    }
  }

  BeforeEach {
    InModuleScope Elizium.Loopz {
      [string]$script:_ruler = '........................................................................................................................................';
    }
  }
...

  Context 'given: Plain Line' {
    It 'should: create coloured line without crumb or message' -Tag 'Current' {
      InModuleScope Elizium.Loopz {
        [System.Collections.Hashtable]$passThru = @{
          'LOOPZ.HEADER-BLOCK.LINE' = $LoopzUI.EqualsLine;
        }

        $line = format-ColouredLine -PassThru $passThru -LineKey $LineKey -CrumbKey $CrumbKey;
        $line[0][0] | Should -BeExactly $LoopzUI.EqualsLine;
        show-result -Ruler $_ruler -Snippets $line;
      }
    }
  } # given: Plain line

and running this test results in

CommandNotFoundException: The term 'show-result' is not recognized as a name of a cmdlet, function, script file, or executable program.

Note how my other variables are correctly bound in LineKey, CrumbKey and MessageKey.

Also, just to note, the only thig failing in this test is accessing the function _show-result_, which has been defined inside the module scope and being invoked also from inside module scope. The rules pertaining to variables and functions do not appear to be the same with regards to InModuleScope. So how can I define test function show-result, so that it is accessible to test code?

I suppose to simplify further:

  Context 'ask: question' {
    It 'should: just work' -Tag 'Current' {
      InModuleScope Elizium.Loopz {
        show-result -Ruler '.........................................' -Snippets @(@('First Snippet', 'red'), @('Second Snippet', 'blue'));
      }
    }
  }

... which fails for the exact same reason.

[plastikfan]

I've finally resolved this problem of mine. If you prefix the function with script:, ie

      function script:show-result {
        param(
          [string]$Ruler,
          [object[]]$Snippets
        )
        Write-Host "$Ruler";
        Write-InColour -TextSnippets $Snippets;
      }

then this fixes the issue. This was a pure guess on my part and could do with an explanation and being documented.

Actually, using the scope specifier 'script:' works when you're running InModuleScope. I discovered that when testing a public Module function (and hence not using InModuleScope), any test function declared inside BeforeAll/After or inside It, which needs to be invoked, should be defined in the global scope; script scope does not work in this scenario.

[hugh-martin]

First, thanks very much for the inclusion of -ForEach in v5.1.1!! I'm starting to migrate my operational validation tests to v5.1x and it's going well. One thing I'm finding with v5 is that I'm using hashtables more and more to move data around to make things easier. One issue I'm running into is validating the hashtable data coming into a function via a parameter. Right now, I'm looping through the keys to make sure they are what's expected and then looping through the values and validating the data for each. In short, it's a pain.

Is it safe to assume that I cannot pass a PowerShell class to Invoke-Pester's -TestCases param? If so, what's the best way to validate hashtable data that's coming into a function? Thanks.

[hugh-martin]

Here's a situation I'm running into that I'm not sure how best to convert to v5. In one of our operational validation tests, we're looping through VMware datastores. In that process, we're collecting various properties of each datastore. Based on those properties, we currently have If / Else statements that determine which It statements to run. This doesn't work as is in v5. I realize I could move the If / Else statements into the BeforeAll scriptblock and generate new hashtables as needed and use TestCases to go back through the datastores, but that doesn't seem very efficient. Suggestions? Thanks.

Here's a bit of the code from each section:

BeforeDiscovery {

    #...
    #Create an array of all datastores
    $script:arrDatastores = Get-Datastore | Sort-Object Name
    #...

}

Describe "Storage checks" -ForEach $arrDatastores {

    BeforeAll {
        $objDatastore = $_

        [string]$dsName = $objDatastore.Name
        [int]$dsFreeGB = $objDatastore.FreeSpaceGB

        #Check availability and accessibility
        if ($objDatastore.State -eq "Available" -and $objDatastore.Accessible -eq $true) {
            $bolDatastoreIsHealthy = $true
        }
        else {
            $bolDatastoreIsHealthy = $false
        }

        #Check free space (skip datastores that don't have VMs on them)

        #Get all the VMs on this datastore
        $objVMsOnThisDatastore = $objDatastore | Get-VM


        #Determine how many of the VMs on this datastore are connected only via the virtual CD drive
        [boolean]$VMDisksOnThisDatastore = $false
        foreach ($objVMOnThisDatastore in $objVMsOnThisDatastore) {
            $objHardDisks = $objVMOnThisDatastore | Get-HardDisk
            foreach ($objHardDisk in $objHardDisks) {
                $DiskDatastoreName = ($objHardDisk | Get-Datastore).Name
                if ($DiskDatastoreName -eq $dsName) {
                    $VMDisksOnThisDatastore = $true
                }
            }
            #Break out of the loop if any disks are found; no need to continue searching
            if ($VMDisksOnThisDatastore) {
                Break
            }
        }

    } #End BeforeAll


    Context "Datastore availability and free space checks (vCenter level)" {

        it "Datastore <dsName> should be both available and accessible." {
            $bolDatastoreIsHealthy | Should -Be $true
        }

        #Here's where the trouble begins...

        if ($VMDisksOnThisDatastore) {
            #This datastore has VM disks on it
            It "Datastore <dsName> (containing one or more VMs with <dsFreeGB> GB free space) should have more than <MinVMDatastoreFreeSpaceGB> GB free" {
                $dsFreeGB | Should -BeGreaterThan $script:MinVMDatastoreFreeSpaceGB
            }
        }
        else {
            #This datastore does NOT have VM disks on it
            It "Datastore <dsName> (containing no VMs with <dsFreeGB> GB free space) should have more than <MinUtilDatastoreFreeSpaceGB> GB free" {
                $dsFreeGB | Should -BeGreaterThan $MinUtilDatastoreFreeSpaceGB
            }
        }
    }

    Context "Next group of tests for this datastore..." {
        #...
    }

}

[fflaten]

@hugh-martin The problem is that your if-test is running during Discovery, while you variable is set during Run (when BeforeAll is executed). To minimize changes, you could move the if-test inside the tests and skip the test when it's not needed, e.g. if ($VMDiskOnThisDatastore) { Set-ItResult -Skipped -Because "Not relevant" } and vice versa.

However the "proper" way would be to move data-collection/logic related to test-generation into the discovery phase. Ex.

BeforeDiscovery {
    $numbersToTest= 1..5 | % { [pscustomobject]@{Number = $_ } }
}

Describe 'Test <_.Number>' -ForEach $numbersToTest {
    BeforeAll {
        #$MyNumber from BeforeAll doesn't exist here
        $number = $_.Number
    }

    BeforeDiscovery {
        #$number from BeforeAll doesn't exist here
        $MyNumber = $_.Number
        $isEvenNumber = $MyNumber % 2 -eq 0
    }

    Context 'Using If' {
        if($isEvenNumber) {
            It 'Test even number <number>' {
                $number % 2 | Should -Be 0
            }
        } else {
            It 'Test odd number <number>' {
                $number % 2 | Should -Be 1
            }
        }
    }

    Context 'Using TestCases (ugly)' {
        It 'Test even number <number>' -TestCases (@($isEvenNumber) -eq $true) {
            $number % 2 | Should -Be 0
        }
        It 'Test odd number <number>' -TestCases (@($isEvenNumber) -eq $false) {
            $number % 2 | Should -Be 1
        }
    }

    Context 'Using Skip switch (skipped tests are shown in output/report)' {
        It 'Test even number <number>' -Skip:($isEvenNumber -eq $false) {
            $number % 2 | Should -Be 0
        }
        It 'Test odd number <number>' -Skip:($isEvenNumber -eq $true) {
            $number % 2 | Should -Be 1
        }
    }
}

[hugh-martin]

I'm guessing this pattern won't work. I'm trying to loop through each VMware cluster, then loop through each host in that cluster, so I'm using a -ForEach on the Describe (for each cluster) and then attempting to use a -TestCases on the It (for each host in that cluster). It's not working, presumably since both will use $_. Can someone suggest another way to do this in Pester 5? In Pester 4, I just used nested foreach loops.

Describe "Datastore count checks (host level)" -ForEach (Get-Cluster | Sort-Object) {

    BeforeEach {
        $strClusterName = $_.Name
        [int]$intClusterDatastoreCount = (Get-Cluster $strClusterName | Get-Datastore | Measure-Object).Count
        $objClusterHosts = Get-Cluster $strClusterName | Get-VMHost | Sort-Object Name
    }

    It "Host <_.Name> should have the same number of datastores as its cluster" {
        [int]$intHostDatastoreCount = ($_ | Get-Datastore | Measure-Object).Count
        $intHostDatastoreCount | Should -BeExactly $intClusterDatastoreCount
    } -TestCases $objClusterHosts
}

[hugh-martin]

With respect to the location of the BeforeEach, this pattern makes sense to me, since the BeforeEach is inside the -ForEach loop:

BeforeDiscovery {
    $collection = @("one","two","three")
}

Describe "describe" -ForEach ($collection) {

    BeforeEach {
        $strNum = $_
    }

    It "<strNum> should be three" {
        $strNum | Should -Be "three"
    }
}

However, this does not make sense to me. In this case, I would expect the BeforeEach to run just one time, since the Describe does not have a -ForEach and the BeforeEach is outside the -TestCases loop on the It statement. However, it works, and putting the BeforeEach inside the It statement definitely does not. Is this the intended behavior?

BeforeDiscovery {
    $collection = @("one","two","three")
}

Describe "describe" {

    BeforeEach {
        $strNum = $_
    }

    It "<strNum> should be three" {
        $strNum | Should -Be "three"
    } -TestCases $collection
}

[fflaten]

I agree it can feel a bit confusing. Foreach/testcases was probably designed to be used with an array of hashtables in which case you'd get variables per dictionary key in the current object (hashtable) and not rely on $_. When using hashtables it would be very clear when you would overwrite a variable since you reused a key-name. Now that you can use an array of objects directly, then $_ will be the current object (ex. the whole hashtable-object or a string in you sample).

That being said, it's easy enough to explain. When you use It -TestCases ... you actually generate x duplicate tests with different data-input (each testcase-object). BeforeEach runs before each test so it's natural to expect $_ to be the current testcase-object just like it will be inside the test. The docs should probably have a note about this.

BeforeAll however is executed once before the first test in a describe/context. At this point the latest "current object" is the current object provided in the describe/context foreach. Having that in mind, if you need to use the context/describe object in BeforeEach, you would capture it to a variable in BeforeAll, ex. BeforeAll { $myDescribeObject = $_ }; BeforeEach { Write-Host "BeforeEach does something to $myDescribeObject" }.

This is untested, but my immediate suggestion to the scenario in your previous problem would be:

BeforeDiscovery {
    $clusters = Get-Cluster | Sort-Object
}

Describe "Datastore count checks (host level)" -ForEach $clusters {

    BeforeDiscovery {
        # Setup required for TestCases / data-driven tests
        $objClusterHosts = Get-Cluster $_.Name | Get-VMHost | Sort-Object Name
    }

    BeforeAll {
        # Prepare cluster-specific values
        $strClusterName = $_.Name
        [int]$intClusterDatastoreCount = (Get-Cluster $strClusterName | Get-Datastore | Measure-Object).Count
    }

    It "Host <_.Name> should have the same number of datastores as its cluster" {
        [int]$intHostDatastoreCount = ($_ | Get-Datastore | Measure-Object).Count
        $intHostDatastoreCount | Should -BeExactly $intClusterDatastoreCount
    } -TestCases $objClusterHosts
}

[hugh-martin]

@fflaten That worked like a charm. Thank you very much! I have a couple other use cases to apply this to.

[Glober777]

Hi, I've been looking for the details regarding the duration values that are being reported by Pester v5 in its console output. For example, when I run the test, I see the output as follows:

...
[+] Some test <duration value 1> (<duration value 2> | <duration value 3>)
...

Can someone please shed some light on what these duration values actually mean?

Thanks!

[fflaten]

@Glober777 It's duration (user|framework). Duration is total duration, user is test code duration and the last is time used by the framework for mock setup etc.

See https://github.com/pester/Pester/issues/1402

[nohwnd]

I enabled discussions for the repo. This one is not needed anymore. Closing.