Navcontainerhelper: Tests report a failure but generate no output

[RafaelKoch]

Describe the issue
As of a couple of hours ago we see failing tests (cannot be sure, but they might have actually succeeded before...) with no output on any callstack or failing test function or the like. See the below screenshot: Some test codeunits report a failure but no actual functions run.

NavContainerHelper is version 0.6.5.9

Screenshots

Tests run, but some report failures that are likely no failures

The output does not include the failed tests

Additional context

• does it happen all the time? -> yes, most builds failing now
• did it use to work? -> yes, probably related to 0.6.5.9

[RafaelKoch]

Just realized.... reports failure AND success for the same test codeunit. Very strange...

[freddydk]

I need the script and full output of the script
And if possible - the output using -debugMode on Run-Tests.
Then I can either advice a script change or fix the issue.

[MJECloud]

Which script exactly?
For container creation or for running the test?

[freddydk]

Both if possible - getting/running test as a minimum.

[MJECloud]

Creation of Container:

$User = "UserName"
$Password = "MySecretPassword"
$Credentials = New-Object -TypeName System.Management.Automation.PSCredential -ArgumentList $User,(ConvertTo-SecureString -String $Password -AsPlainText -Force)

$RepoPath = $env:BUILD_REPOSITORY_LOCALPATH
$myscripts = @(@{'MainLoop.ps1' = 'while ($true) { start-sleep -seconds 10 }'})
$additionalParameters = @(
    '--storage-opt "size=50GB"',
    '-e CustomNavSettings=EnableTaskScheduler=False',
    "--volume ""$($RepoPath):C:\app""",
    '-e CustomNavSettings=ServicesUseNTLMAuthentication=true'
)

$LicenseFile = "URLToLicenseFile"

$User = "UserName"
$Password = "MySecretPassword"
$Credential = New-Object -TypeName System.Management.Automation.PSCredential -ArgumentList $User,(ConvertTo-SecureString -String $Password -AsPlainText -Force)

New-NavContainer -accept_eula `
                -accept_outdated `
                -containerName "BC" `
                -imageName "mcr.microsoft.com/businesscentral/sandbox:de-ltsc2019" `
                -licenseFile $LicenseFile `
                -Credential $Credential `
                -doNotExportObjectsToText `
                -alwaysPull `
                -shortcuts "Desktop" `
                -auth NavUserPassword `
                -additionalParameters $additionalParameters `
                -memoryLimit 10GB `
                -assignPremiumPlan `
                -updateHosts `
                -useBestContainerOS `
                -myScripts $myscripts

Getting and running Tests

$ResultXMLPath = (-join($env:BUILD_REPOSITORY_LOCALPATH,"\",$Config.ContainerName,".results.xml"))
$User = "UserName"
$Password = "MySecretPassword"
$Credential = New-Object -TypeName System.Management.Automation.PSCredential -ArgumentList $User,(ConvertTo-SecureString -String $Password -AsPlainText -Force)


$FailedTests = @()
$first = $true


$Tests = Get-TestsFromBCContainer -containerName "BC" -credential $Credential -ignoreGroups


$Tests | ForEach-Object {
    Invoke-ScriptInBCContainer -containerName "BC" -scriptblock {
        if ((Get-Process Microsoft.Dynamics.Nav.Server).WS -gt 6Gb) {
            Write-Host "Memory usage $((Get-Process Microsoft.Dynamics.Nav.Server).WS) - restarting Service Tier"
            Set-NavServerInstance $serverInstance -restart
        }
    }

    if(-not (Run-TestsInBCContainer -containerName "BC" `
                                    -credential $Credential `
                                    -XUnitResultFileName $ResultXMLPath `
                                    -AppendToXUnitResultFile:(!$first) `
                                    -testCodeunit $_.Id `
                                    -returnTrueIfAllPassed `
                                    -restartContainerAndRetry `
                                    -AzureDevOps error `
                                    -detailed)){ $FailedTests += $_.Id }
    $first = $false
}


if($FailedTests.Count -gt 0) {
    Write-Host $FailedTests
    if(!(Test-Path C:\FailedTests)) {
        New-Item -Path C:\FailedTests -Force -ItemType Directory
    }
    $Date = Get-Date -Format yyyyMMdd
    $FailedTests | ConvertTo-Json | Out-File (-join("C:\FailedTests\FailedTests-",$Date,"-",$($env:BUILD_BUILDID),".json"))
    Write-Error "There are $($FailedTests.Count) failing tests!"
} else {
    "Tests passed"
}    

[freddydk]

Do you have the full output (with -debugMode as well)
-debugmode on both Get-Test and Run-Test, thanks

[MJECloud]

Sorry no DebugMode
This is the only Output

2020-05-12T08:59:25.5902359Z Codeunit 60001 GOB Reg Perm. Test Success (25.597 seconds) 2020-05-12T08:59:25.6418078Z Testfunction TestUnitopIsInstalled Success (0.216 seconds) 2020-05-12T08:59:25.6493410Z Testfunction TestMandatoryPermissionSetsAreFound Success (0.03 seconds) 2020-05-12T08:59:25.6556159Z Testfunction TestPermissionSetContainsNoDeadObjectReferences Success (1.676 seconds) 2020-05-12T08:59:25.6619793Z Testfunction TestPermissionSetContainsAllRequiredObjectReferences Success (22.856 seconds) 2020-05-12T08:59:53.0269564Z Codeunit 5021832 GOB Test Event Mgt. Setup Failure (22.116 seconds) 2020-05-12T08:59:55.8797154Z Codeunit 5021832 GOB Test Event Mgt. Setup Success (2.852 seconds) 2020-05-12T08:59:55.8865587Z Testfunction TestMakeEventMgtSetup Success (0.173 seconds) 2020-05-12T08:59:55.8937474Z Testfunction TestCreateEventtype Success (0.017 seconds) 2020-05-12T08:59:55.9005152Z Testfunction TestCreateEventQualification Success (0.013 seconds) 2020-05-12T08:59:55.9083889Z Testfunction TestCreateSubjectArea Success (0.017 seconds) 2020-05-12T08:59:55.9143236Z Testfunction TestCreateCertificateType Success (0.016 seconds) 2020-05-12T08:59:55.9310964Z Testfunction TestCreateItemTypeServiceForParticipant Success (0.204 seconds) 2020-05-12T08:59:55.9381186Z Testfunction TestCreateEventTemplateWithServiceLines Success (0.203 seconds) 2020-05-12T08:59:55.9452568Z Testfunction TestCreateEventHeader Success (0.597 seconds) 2020-05-12T08:59:55.9518359Z Testfunction TestCreateCalculationTemplate Success (0.093 seconds) 2020-05-12T08:59:56.0663294Z Testfunction TestLecturerEnabledUI Success (0.69 seconds) 2020-05-12T08:59:56.0733158Z Testfunction MarkContactAsLecturer Success (0.05 seconds) 2020-05-12T08:59:56.0803526Z Testfunction TestDocumentTemplateTypeSelectionForEventTemplate Success (0.033 seconds) 2020-05-12T08:59:56.0871085Z Testfunction TestReportSelection_Listofattendance Success (0.014 seconds) 2020-05-12T08:59:56.0941348Z Testfunction TestReportSelection_SuccessorsInformation Success (0 seconds) 2020-05-12T08:59:56.1001691Z Testfunction TestReportSelection_EnrollingConf Success (0 seconds) 2020-05-12T08:59:56.1086639Z Testfunction TestReportSelection_ParticipantList Success (0.017 seconds) 2020-05-12T08:59:56.1156620Z Testfunction TestReportSelection_BookingConfirmation Success (0.013 seconds) 2020-05-12T08:59:56.1227682Z Testfunction TestReportSelection_CancellationConfirmation Success (0 seconds) 2020-05-12T09:00:30.5352526Z Codeunit 5021867 GOB Test Event Part. Services Failure (27.09 seconds) 2020-05-12T09:00:45.0851065Z Codeunit 5021867 GOB Test Event Part. Services Success (14.363 seconds) 2020-05-12T09:00:45.1239396Z Testfunction TestCopyEventTemplateServices Success (0.22 seconds) 2020-05-12T09:00:45.1313044Z Testfunction TestEmptyCustomerGroupEventTemplateServices Success (0.033 seconds) 2020-05-12T09:00:45.1389464Z Testfunction TestItemNotServiceInEventTemplateServices Success (0.03 seconds) 2020-05-12T09:00:45.1455971Z Testfunction TestCreateNewTemplateServiceWhenEventTemplateClosed Success (0.017 seconds) 2020-05-12T09:00:45.1526429Z Testfunction TestInsertNewServiceWhenEventClosed Success (0.25 seconds) 2020-05-12T09:00:45.2614270Z Testfunction TestChangeofEventTemplateInEventHeader Success (0.296 seconds) 2020-05-12T09:00:45.2676061Z Testfunction TestChangeEventToInhouseEvent Success (0.25 seconds) 2020-05-12T09:00:45.2741111Z Testfunction TestEventTemplateServiceMaxFieldLengths Success (0.034 seconds) 2020-05-12T09:00:45.2828980Z Testfunction TestEventHeaderMaxFieldLengths Success (0.17 seconds) 2020-05-12T09:00:45.2880028Z Testfunction TestEventWithParticipants_NoOfParticipants Success (1.583 seconds) 2020-05-12T09:00:45.2945219Z Testfunction TestEventWithParticipants_BillToCust Success (1.877 seconds) 2020-05-12T09:00:45.3008706Z Testfunction TestEventWithParticipants_CreateServiceLines Success (1.63 seconds) 2020-05-12T09:00:45.3074732Z Testfunction TestEventWithParticipants_TestPriceInServiceLines Success (1.66 seconds) 2020-05-12T09:00:45.3139660Z Testfunction TestEventWithParticipants_ComplementEventLineWithBillToValuesFromContact Success (1.58 seconds) 2020-05-12T09:00:45.3215755Z Testfunction TestEventWithParticipants_ComplementEventLineWithBillToValuesFromCompany Success (1.787 seconds) 2020-05-12T09:00:45.3290454Z Testfunction TestEventWithParticipants_DontComplementEventLineWithBillToValues Success (1.646 seconds) 2020-05-12T09:00:45.3368892Z Testfunction TestEventContactServicesWithEarlyBird Success (0.393 seconds) 2020-05-12T09:00:56.1750554Z Codeunit 5021889 GOB Test Calc. Event Mgt. Failure (3.398 seconds) 2020-05-12T09:00:58.9396951Z Codeunit 5021889 GOB Test Calc. Event Mgt. Success (2.436 seconds) 2020-05-12T09:00:58.9852330Z Testfunction TestCreateEventHeaderFromTemplateWithCalcLines Success (0.407 seconds) 2020-05-12T09:00:58.9947022Z Testfunction TestTransferCalcLinesFromNewTemplate Success (0.236 seconds) 2020-05-12T09:00:59.1169160Z Testfunction TestCompletePreCalculation Success (0.424 seconds) 2020-05-12T09:00:59.1240804Z Testfunction TestBreakEvenNotPossible Success (0.186 seconds) 2020-05-12T09:00:59.1318668Z Testfunction TestCompletePreCalculationWhenEventIsClosed Success (0.174 seconds) 2020-05-12T09:00:59.1395320Z Testfunction TestCalcLineEventGLAccountMaxLengths Success (0.173 seconds) 2020-05-12T09:01:25.6610253Z Codeunit 5021914 GOB Test Events Failure (19.459 seconds) 2020-05-12T09:01:41.1535289Z Codeunit 5021914 GOB Test Events Success (15.04 seconds) 2020-05-12T09:01:41.1928228Z Testfunction TestEventHeaderSalespersonMaxLength Success (0.344 seconds) 2020-05-12T09:01:41.2007773Z Testfunction TestEventDatesOnNewEvent Success (0.126 seconds) 2020-05-12T09:01:41.2071762Z Testfunction TestEventHeaderValidateSalespersonWithPivacyBlocked Success (0.123 seconds) 2020-05-12T09:01:41.2140135Z Testfunction TestEventHeaderActivityCodeMaxLength Success (0.114 seconds) 2020-05-12T09:01:41.2206659Z Testfunction TestEventHeaderTeamCodeMaxLength Success (0.11 seconds) 2020-05-12T09:01:41.2266692Z Testfunction TestEventHeaderValidateStartDateOnSelectedWeekday Success (0.123 seconds) 2020-05-12T09:01:41.2328551Z Testfunction TestEventHeaderValidateStartDateWithNotSelectedWeekday Success (0.11 seconds) 2020-05-12T09:01:41.2400708Z Testfunction TestEventHeaderValidateStartTime Success (0.11 seconds) 2020-05-12T09:01:41.2463585Z Testfunction TestEventLineRegisterParticipantWhileEventStatusIsPlanning Success (0.236 seconds) 2020-05-12T09:01:41.2858154Z Testfunction TestEventLineRegisterParticipantTwice Success (0.22 seconds) 2020-05-12T09:01:41.3922607Z Testfunction TestEventTestReportWithAllCriteria Success (1.324 seconds) 2020-05-12T09:01:41.3988366Z Testfunction TestEventTestReportWithMissingCriteria Success (1.284 seconds) 2020-05-12T09:01:41.4082394Z Testfunction TestAssignLecturersToEvent Success (0.627 seconds) 2020-05-12T09:01:41.4162232Z Testfunction TestPreAssignLecturersToEventFromTemplate Success (0.253 seconds) 2020-05-12T09:01:41.4262646Z Testfunction TestCopyHTMLTextFromTemplate Success (0.05 seconds) 2020-05-12T09:01:41.4324802Z Testfunction TestCreateEventFromEventTemplate Success (0.13 seconds) 2020-05-12T09:01:41.4408579Z Testfunction TestCreateEventFromWrongEventTemplate Success (0.09 seconds) 2020-05-12T09:01:41.4507584Z Testfunction TestSetDirectionWhenLecturerStatusUndetermined Success (0.177 seconds) 2020-05-12T09:01:41.4587887Z Testfunction TestCalculationOfAvailableSeatsByInsert Success (1.333 seconds) 2020-05-12T09:01:41.4664501Z Testfunction TestCalculationOfAvailableSeatsByModify Success (1.107 seconds) 2020-05-12T09:01:41.4745708Z Testfunction TestCalculationOfAvailableSeatsByDelete Success (1.207 seconds) 2020-05-12T09:01:41.4832365Z Testfunction TestCreateSegmentFromEventFromEvent_Basic Success (1.24 seconds) 2020-05-12T09:01:41.4913433Z Testfunction TestCreateSegmentFromEventFromEvent_NoOfSegLines Success (1.4 seconds) 2020-05-12T09:01:41.5021256Z Testfunction TestCreateNewEventTask Success (1.283 seconds) 2020-05-12T09:01:41.5084563Z Testfunction TestDefaultPriceGroup Success (0.467 seconds) 2020-05-12T09:01:41.6124065Z Testfunction TestExpandCompaniesFromSegment Success (0.153 seconds) 2020-05-12T09:02:12.3566760Z Codeunit 5022056 GOB Test Event Part. Account. Failure (23.734 seconds) 2020-05-12T09:02:21.7647117Z Codeunit 5022056 GOB Test Event Part. Account. Success (8.682 seconds) 2020-05-12T09:02:21.8077156Z Testfunction TestEventLinePostParticipantServicesforOneParticipant Success (1.374 seconds) 2020-05-12T09:02:21.8149846Z Testfunction TestEventLinePostParticipantServicesforMoreParticipants Success (1.67 seconds) 2020-05-12T09:02:21.8614526Z Testfunction TestEventLinePostParticipantServiceswithDiffrentInvoiceRecipient Success (0.957 seconds) 2020-05-12T09:02:21.8681493Z Testfunction TestEventLinePostedParticipantServicesforOneParticipant Success (1.217 seconds) 2020-05-12T09:02:21.8743637Z Testfunction TestPostEventLineWithMultipleServicesDifferentBillToCode Success (0.996 seconds) 2020-05-12T09:02:21.8801550Z Testfunction TestEventDimensionSetInSalesHeaderAndLine Success (1.113 seconds) 2020-05-12T09:02:31.8320913Z Codeunit 5022092 GOB Test Event Factboxes Failure (2.278 seconds) 2020-05-12T09:02:35.3807551Z Codeunit 5022092 GOB Test Event Factboxes Success (2.797 seconds) 2020-05-12T09:02:35.4210937Z Testfunction TestEventTemplateRevenue Success (0.526 seconds) 2020-05-12T09:02:35.4296309Z Testfunction TestPartDetailsInvoicedAmount Success (1.027 seconds) 2020-05-12T09:03:00.5830608Z Codeunit 5229390 GOB Test Event Lect. Account. Failure (18.147 seconds) 2020-05-12T09:03:09.9003645Z Codeunit 5229390 GOB Test Event Lect. Account. Success (8.561 seconds) 2020-05-12T09:03:09.9704428Z Testfunction TestCalculateTotalLecturerFeesForAnEvent Success (0.94 seconds) 2020-05-12T09:03:09.9778038Z Testfunction TestPostLecturerFees Success (0.9 seconds) 2020-05-12T09:03:09.9846897Z Testfunction TestPostLecturerFees_DirectPosting Success (1.103 seconds) 2020-05-12T09:03:10.0929358Z Testfunction TestPostLecturerFees_NoFee Success (0.82 seconds) 2020-05-12T09:03:10.1002648Z Testfunction TestPostLecturerFees_MissingItemNo Success (0.647 seconds) 2020-05-12T09:03:10.1062637Z Testfunction TestPostLecturerFees_MissingQuantity Success (0.663 seconds) 2020-05-12T09:03:10.1131333Z Testfunction TestPreAssignLecturerFeesToEventFromTemplate Success (0.233 seconds) 2020-05-12T09:03:10.1195410Z Testfunction TestCancelLecturerWithInvoicedFee Success (0.85 seconds) 2020-05-12T09:03:10.1259056Z Testfunction TestEventDimensionSetInPurchaseHeaderAndLine Success (0.98 seconds) 2020-05-12T09:03:20.5218516Z Codeunit 5229397 GOB TestContactFactbox Failure (3.603 seconds) 2020-05-12T09:03:25.4828057Z Codeunit 5229397 GOB TestContactFactbox Success (4.047 seconds) 2020-05-12T09:03:25.5219215Z Testfunction TestContactFactboxAssignLecturer_undetermined Success (0.453 seconds) 2020-05-12T09:03:25.5295226Z Testfunction TestContactFactboxAssignLecturer_confirmed Success (0.173 seconds) 2020-05-12T09:03:25.5370331Z Testfunction TestContactFactboxAssignLecturer_total Success (0.167 seconds) 2020-05-12T09:03:25.5426175Z Testfunction TestContactFactBoxEventRevenue Success (0.96 seconds) 2020-05-12T09:03:25.5486610Z Testfunction TestContactFactBoxCompanyParticipations Success (0.957 seconds) 2020-05-12T09:04:04.9826064Z Codeunit 5229654 GOB Test Event Cancelation Failure (32.436 seconds) 2020-05-12T09:04:29.0795812Z Codeunit 5229654 GOB Test Event Cancelation Success (23.227 seconds) 2020-05-12T09:04:29.1240120Z Testfunction CancelPostedParticipantServices Success (2.56 seconds) 2020-05-12T09:04:29.1314033Z Testfunction CancelPostedParticipantServicesAndPostCreditMemo Success (3.246 seconds) 2020-05-12T09:04:29.2401549Z Testfunction CancelPostedParticipantServicesAndCreateCreditMemo Success (2.814 seconds) 2020-05-12T09:04:29.2463448Z Testfunction CancelParticipantWithNotPostedDocuments Success (0.737 seconds) 2020-05-12T09:04:29.2531869Z Testfunction CancelParticipantWithWrongEventLineStatus Success (0.373 seconds) 2020-05-12T09:04:29.2594402Z Testfunction AbortParticipantWithOpenServices Success (0.36 seconds) 2020-05-12T09:04:29.2661375Z Testfunction AbortParticipantOutsideOfEventPeriod Success (0.44 seconds) 2020-05-12T09:04:29.2727504Z Testfunction TestCancelEvent_LecturerAssignments Success (0.73 seconds) 2020-05-12T09:04:29.2790206Z Testfunction CancelPostedParticipantServicesAndPostDocuments Success (4.177 seconds) 2020-05-12T09:04:29.2857353Z Testfunction CancelPostedParticipantServicesAndPostCancelationFee Success (1.954 seconds) 2020-05-12T09:04:29.2966411Z Testfunction AbortPostedParticipantServices Success (1.743 seconds) 2020-05-12T09:04:29.3035552Z Testfunction CancelOpenParticipantServices Success (0.907 seconds) 2020-05-12T09:04:29.3100927Z Testfunction CancelParticipantWithoutCancelationServiceItem Success (0.437 seconds) 2020-05-12T09:04:29.3168601Z Testfunction CancelParticipantWithoutCancelationCode Success (0.353 seconds) 2020-05-12T09:04:30.0235976Z 5021832 5021867 5021889 5021914 5022056 5022092 5229390 5229397 5229654 2020-05-12T09:04:30.2899270Z ##[error]There are 9 failing tests! 2020-05-12T09:04:30.3360548Z ##[section]Finishing: Run Tests

[MJECloud]

I'll try to reproduce and get a build with -debugMode

[MJECloud]

In the attachement you find the output with -debugMode
It's to long for a comment.

OutputTestsDebugMode.txt

[freddydk]

I can repro the problem.
Do you want an updated pstestfunctions.ps1, which fixes the problem to try out?

[MJECloud]

Sure, would like to test it :)

[freddydk]

You can grab the updated pstestfunctions from the PR.
Running CI pipeline now - release a new containerhelper afterwards.

[freddydk]

0.6.5.10 has shipped with this fix, thanks for reporting it.

[MJECloud]

Sorry for the late response, took a while to run the build.

It works!
Thank you for your quick fix! :)

[freddydk]

Thanks. There are really so many different ways to run tests and combinations of things, so i literally had my test machines running all combinations of codeunits and testfunctions in different extensions on 14.x, 15.x and 16.x - I still missed one...