Navcontainerhelper: Run-TestsInBCContainer fails with error UnhandledException : Object reference not set to an instance of an object.

[jeesus450]

Describe the issue
Running standard test codeunit in Cronus database with Run-TestsInBCContainer fails with following error:
Codeunit 138696 Page Troubleshooting Test UnhandledException : Object reference not set to an instance of an object.
at , C:\ProgramData\BcContainerHelper\Extensions\bc17-de-temp\PsTestTool\ClientContext.ps1: line 117
at , : line 76
Current Interaction: Microsoft.Dynamics.Framework.UI.Client.Interactions.InvokeActionInteraction
Time spend: 3 seconds
at AwaitState, C:\ProgramData\BcContainerHelper\Extensions\bc17-de-temp\PsTestTool\ClientContext.ps1: line 217
at InvokeInteraction, C:\ProgramData\BcContainerHelper\Extensions\bc17-de-temp\PsTestTool\ClientContext.ps1: line 236
at InvokeAction, C:\ProgramData\BcContainerHelper\Extensions\bc17-de-temp\PsTestTool\ClientContext.ps1: line 460
at Run-Tests, C:\ProgramData\BcContainerHelper\Extensions\bc17-de-temp\PsTestTool\PsTestFunctions.ps1: line 809
at , : line 57

It works when executing the tests from the web client.

After the run-testsinbccontainer restarts the instance. There is Tenant Profile table locked for some time and web client cannot be used.

I also experienced this error with Codeunit 138697 Profile Import/Export Test. Same error: UnhandledException : Object reference not set to an instance of an object.

Scripts used to create container and cause the issue

import-module bccontainerhelper

New-BCContainer -containerName bc17-de-temp `
                -accept_eula -accept_outdated `
                -artifacturl "https://bcartifacts.azureedge.net/onprem/17.1.18256.18792/de" `
                -licenseFile "<path to license file>" `
                -updateHosts `
                -useBestContainerOS `
                -useGenericImage "mcr.microsoft.com/dynamicsnav:10.0.14393.3630-generic" `
                -includeTestToolkit `
                -auth UserPassword

Add the codeunit into the test tool manually from the web client

import-module bccontainerhelper

Run-TestsInBcContainer -containerName bc17-de-temp `
-companyName "Cronus AG" `
-credential (Get-Credential) `
-testSuite "DEFAULT" `
-testCodeunit 138696

Full output of scripts

{
    "container.logs":  [
                           "Using artifactUrl https://bcartifacts.azureedge.net/onprem/17.1.18256.18792/de",
                           "Using installer from C:\\Run\\150-new",
                           "Installing Business Central",
                           "Installing from artifacts",
                           "Starting Local SQL Server",
                           "Starting Internet Information Server",
                           "Copying Service Tier Files",
                           "Copying PowerShell Scripts",
                           "Copying dependencies",
                           "Copying ReportBuilder",
                           "Importing PowerShell Modules",
                           "Determining Database Collation from c:\\dl\\onprem\\17.1.18256.18792\\de\\database\\Demo Database NAV (17-0).bak",
                           "Changing Database Server Collation to Latin1_General_100_CI_AS",
                           "Restoring CRONUS Demo Database",
                           "Modifying Business Central Service Tier Config File for Docker",
                           "Creating Business Central Service Tier",
                           "Installing SIP crypto provider: \u0027C:\\Windows\\System32\\NavSip.dll\u0027",
                           "Copying Web Client Files",
                           "Copying Client Files",
                           "Copying ModernDev Files",
                           "Copying additional files",
                           "Copying ConfigurationPackages",
                           "Copying Test Assemblies",
                           "Copying Applications",
                           "Starting Business Central Service Tier",
                           "Importing license file",
                           "Stopping Business Central Service Tier",
                           "Installation took 151 seconds",
                           "Installation complete",
                           "Initializing...",
                           "Setting host.containerhelper.internal to 192.168.240.1 in container hosts file",
                           "Starting Container",
                           "Hostname is bc17-de-temp",
                           "PublicDnsName is bc17-de-temp",
                           "Using NavUserPassword Authentication",
                           "Creating Self Signed Certificate",
                           "Self Signed Certificate Thumbprint 2319BE8F2483026200D87FA5884E51CAEA20A493",
                           "Modifying Service Tier Config File with Instance Specific Settings",
                           "Starting Service Tier",
                           "Registering event sources",
                           "Creating DotNetCore Web Server Instance",
                           "Using license file \u0027c:\\run\\my\\license.flf\u0027",
                           "Import License",
                           "Creating http download site",
                           "Setting SA Password and enabling SA",
                           "Creating tasane as SQL User and add to sysadmin",
                           "Creating SUPER user",
                           "Container IP Address: 192.168.251.208",
                           "Container Hostname  : bc17-de-temp",
                           "Container Dns Name  : bc17-de-temp",
                           "Web Client          : http://bc17-de-temp/BC/",
                           "Dev. Server         : http://bc17-de-temp",
                           "Dev. ServerInstance : BC",
                           "Setting bc17-de-temp to 192.168.251.208 in host hosts file",
                           "",
                           "Files:",
                           "http://bc17-de-temp:8080/ALLanguage.vsix",
                           "",
                           "WARNING: You are running a container which is 213 days old.",
                           "Microsoft recommends that you always run the latest version of our containers.",
                           "",
                           "Initialization took 39 seconds",
                           "Ready for connections!"
                       ],
    "docker.info":  [
                        "Client:",
                        " Debug Mode: false",
                        " Plugins:",
                        "  cluster: Manage Docker clusters (Docker Inc., v1.2.0)",
                        "",
                        "Server:",
                        " Containers: 7",
                        "  Running: 1",
                        "  Paused: 0",
                        "  Stopped: 6",
                        " Images: 5",
                        " Server Version: 19.03.5",
                        " Storage Driver: windowsfilter",
                        "  Windows: ",
                        " Logging Driver: json-file",
                        " Plugins:",
                        "  Volume: local",
                        "  Network: ics internal l2bridge l2tunnel nat null overlay private transparent",
                        "  Log: awslogs etwlogs fluentd gcplogs gelf json-file local logentries splunk syslog",
                        " Swarm: inactive",
                        " Default Isolation: process",
                        " Kernel Version: 10.0 14393 (14393.3564.amd64fre.rs1_release.200303-1942)",
                        " Operating System: Windows Server 2016 Datacenter Version 1607 (OS Build 14393.3564)",
                        " OSType: windows",
                        " Architecture: x86_64",
                        " CPUs: 4",
                        " Total Memory: 64GiB",
                        " Name: RDES-FIN-DEV02",
                        " ID: MIQJ:YOEV:FAXO:UHUF:FQHQ:7YQA:N6DY:GOLD:2MBU:3IVU:AJ6V:5YX3",
                        " Docker Root Dir: D:\\ProgramData\\Docker",
                        " Debug Mode: false",
                        " Registry: https://index.docker.io/v1/",
                        " Labels:",
                        " Experimental: false",
                        "  127.0.0.0/8",
                        " Live Restore Enabled: false",
                        ""
                    ],
    "container.labels":  {
                             "country":  "de",
                             "created":  "202008281422",
                             "eula":  "https://go.microsoft.com/fwlink/?linkid=861843",
                             "maintainer":  "Dynamics SMB",
                             "nav":  "",
                             "osversion":  "10.0.14393.3630",
                             "platform":  "17.0.18204.18738",
                             "tag":  "0.1.0.20",
                             "version":  "17.1.18256.18792"
                         },
    "container.ping":  [
                           "",
                           "Pinging bc17-de-temp [192.168.251.208] with 32 bytes of data:",
                           "Reply from 192.168.251.208: bytes=32 time\u003c1ms TTL=128",
                           "Reply from 192.168.251.208: bytes=32 time\u003c1ms TTL=128",
                           "Reply from 192.168.251.208: bytes=32 time\u003c1ms TTL=128",
                           "Reply from 192.168.251.208: bytes=32 time\u003c1ms TTL=128",
                           "",
                           "Ping statistics for 192.168.251.208:",
                           "    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),",
                           "Approximate round trip times in milli-seconds:",
                           "    Minimum = 0ms, Maximum = 0ms, Average = 0ms"
                       ],
    "container.env":  [
                          "auth=NavUserPassword",
                          "ExitOnError=N",
                          "accept_eula=Y",
                          "accept_outdated=Y",
                          "passwordKeyFile=c:\\run\\my\\aes.key",
                          "artifactUrl=https://bcartifacts.azureedge.net/onprem/17.1.18256.18792/de",
                          "username=tasane",
                          "locale=de-DE",
                          "securePassword=\u003cspecified\u003e",
                          "databaseServer=",
                          "enableApiServices=Y",
                          "licenseFile=\u003cspecified\u003e",
                          "databaseInstance=",
                          "useSSL=N",
                          "removePasswordKeyFile=Y",
                          "COMPLUS_NGenProtectedProcess_FeatureEnabled=0"
                      ]
}

Additional context

• It happens all the time
• It did work once in the middle of december 2020 - i had a full test run, where this error did not pop up

[jeesus450]

I just tested with the latest version of onprem de using following artifacts url:
https://bcartifacts.azureedge.net/onprem/17.3.20469.20605/de

Same issue.

[jeesus450]

bccontainerhelper version is 1.0.19

[freddydk]

The best way of running tests in 17 is to use run-tests with -extensionid - once for every app.
That should work.

Get-BcContainerAppInfo | % { run-tests -extensionid $_.appid }

or something like that.

[jeesus450]

What is the main difference between executing tests in this way compared to the old way? What is handled differently?

Do we need to consider something special when developing our own test apps to make it possible to execute the tests with the new way?

[freddydk]

No change to the tests.
you don't have to populate the test suite anymore
it is more stable - we know there are issues with the old way when running very large amounts of tests - it is just very hard to find and fix (especially when a working alternative exists)

[jeesus450]

Is there a way to filter on test codeunits with the new way? To pick some specific tests codeunits or exclude some.

[freddydk]

You run tests one app at a time.
Disabledtests excludes tests you don't want to run.
You can find samples on howto on my blog.

[jeesus450]

i will test the new way and report back.

[jeesus450]

I ran following command against the same container:

Get-BcContainerAppInfo -containerName bc17-de-temp | % {Run-TestsInBcContainer -containerName bc17-de-temp -extensionId $_.appid -credential $credentials } 

and now it failed with following message:

Codeunit 138500 Common Demodata Failure (0.151 seconds)
    Testfunction ConfigTemplateCodeShouldStartWithTablePrefix Failure (0.016 seconds)
    Testfunction PaymentMethodsWithUseForInvoicing Failure (0.01 seconds)
UnhandledException : Object reference not set to an instance of an object.
at <ScriptBlock>, C:\ProgramData\BcContainerHelper\Extensions\bc17-de-temp\PsTestTool\ClientContext.ps1: line 117
at <ScriptBlock>, <No file>: line 76
Current Interaction: Microsoft.Dynamics.Framework.UI.Client.Interactions.InvokeActionInteraction
Time spend: 11 seconds
at AwaitState, C:\ProgramData\BcContainerHelper\Extensions\bc17-de-temp\PsTestTool\ClientContext.ps1: line 217
at InvokeInteraction, C:\ProgramData\BcContainerHelper\Extensions\bc17-de-temp\PsTestTool\ClientContext.ps1: line 236
at InvokeAction, C:\ProgramData\BcContainerHelper\Extensions\bc17-de-temp\PsTestTool\ClientContext.ps1: line 460
at Run-Tests, C:\ProgramData\BcContainerHelper\Extensions\bc17-de-temp\PsTestTool\PsTestFunctions.ps1: line 447
at <ScriptBlock>, <No file>: line 57
ClientSession State is InError (Wait time 11 seconds)****

[freddydk]

Could you paste in the full script end2end causing this issue?
thanks

[jeesus450]

This time I used 17.3 onprem de container. Artifact url is in the following script I used to create the container:

import-module bccontainerhelper

New-BCContainer -containerName bc17-de-temp `
                -accept_eula -accept_outdated `
                -artifacturl "https://bcartifacts.azureedge.net/onprem/17.3.20469.20605/de" `
                -licenseFile "path to the license file" `
                -updateHosts `
                -useBestContainerOS `
                -useGenericImage "mcr.microsoft.com/dynamicsnav:10.0.14393.3630-generic" `
                -includeTestToolkit `
                -auth UserPassword

Then I just ran the following commands:

$credentials=(Get-Credential)
Get-BcContainerAppInfo -containerName bc17-de-temp | % {Run-TestsInBcContainer -containerName bc17-de-temp -extensionId $_.appid -credential $credentials }

[freddydk]

You should remove these parameters:

            -useBestContainerOS `
            -useGenericImage "mcr.microsoft.com/dynamicsnav:10.0.14393.3630-generic" `

Artifacts are always running best container os.
The generic image you point out is very old.

I don't know whether this is the solution, but it could be.

I will try to run a repro here.

[jeesus450]

we need sql 2017. Can I enforce it somehow?

[freddydk]

-useGenericImage "$(GetBestGenericImageName)-sql2017"

[freddydk]

I tried this with de and get an error like you.
When running w1 it works.
Not sure whether this can be related to some specific tests in the german version but I will have to investigate that.

[freddydk]

You can use:

$disabledTests = @'
[
    {
        "codeunitName": "Common Demodata",
        "method" : "*"
    }
]
'@

and add-disabledTests ($disabledTests | ConvertFrom-Json) to your run-tests, then it will disable all tests from that codeunit and the German tests can be run.
Maybe you can even find the method that is failing and disable that.
I do not know what the problem is, but I will not have the time to dive into the German tests and why they are failing here.

[freddydk]

If anybody can figure out why this specific test codeunit isn't running in docker, I would be happy to take a PR or add a fix.
thanks

[ralf-escher]

We have to deal with this (or somekind of similar) issue for quite a long time.
We figured out that our tests fail (with the same error as above) if we are posting an Item Journal (to get stock for our integration tests), but never could isolate the problem (gave up after one year...).

Never thought about our German environment.

We solved the problem by running the tests outside the container:
building up a ClientContext and running the Run-Tests routine.
At least the Get-TestsFromBCContainer function is working pretty well.