Navcontainerhelper: New-BCContainer fails to create container if using artifact (instead of image) with Transparent network

[phamhainguyen68]

Issue: New-BCContainer fails midway, when creating container with Transparent network using artifact

Troubleshooting step:

1. Running New-BCContainer with default NAT network: no issue
2. Running New-BCContainer using image, instead of artifact with both Transparent and NAT networks: works in both scenarios
3. The output of the scripts suggests it fails at restoring database (New-NAVDatabase at line 182 of C:\run\navinstall.ps1 script)

Determining Database Collation from c:\dl\sandbox\16.3.14085.15298\us\BusinessCentral-US.bak
Restoring CRONUS Demo Database
The database restore operation failed due to the following error returned by SQL Server:
A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (
provider: SQL Network Interfaces, error: 26 - Error Locating Server/Instance Specified)
at <ScriptBlock>, C:\Run\navinstall.ps1: line 182
at <ScriptBlock>, C:\Run\start.ps1: line 202
at <ScriptBlock>, <No file>: line 1Error

1. With that information, I logged into the container (using docker exec) and manually run the New-NAVDatabase command:
2. Exactly same error:

New-NAVDatabase -DatabaseServer "localhost" -DatabaseInstance "SQLEXPRESS" -DatabaseName "CRONUS" -FilePath "c:\dl\sandbox\16.3.14085.15298\us\BusinessCentral-US.bak" -DestinationPath "c:\databases" -Timeout 300

New-NAVDatabase : The database restore operation failed due to the following error returned by SQL Server:
A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL
Server is configured to allow remote connections. (provider: SQL Network Interfaces, error: 26 - Error Locating Server/Instance Specified)
At line:1 char:1
+ New-NAVDatabase -DatabaseServer "localhost" -DatabaseInstance "SQLEXP ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     + CategoryInfo          : NotSpecified: (0:Int32) [New-NAVDatabase], NavCommandException
     + FullyQualifiedErrorId : gMSAnavnst,Microsoft.Dynamics.Nav.Management.Cmdlets.NewNavDatabase

• HOWEVER, The command works when I manually specify the DatabaseServer parameter as localhost\SQLEXPRESS

New-NAVDatabase -DatabaseServer "localhost\SQLEXPRESS" -DatabaseName "CRONUS" -FilePath "c:\dl\sandbox\16.3.14085.15298\us\BusinessCentral-US.bak" -DestinationPath "c:\databases" -Timeout 300

DatabaseServer       DatabaseInstance DatabaseName
--------------       ---------------- ------------
localhost\SQLEXPRESS                  CRONUS

Conclusion In Transparent Network, New-NAVDatabase command can only connect to SQL server when specifying server name and instance in DatabaseServer parameter as localhost\SQLEXPRESS.

Additional information with a bit further testing, I found out that this issue also applies to image method. We never caught it because image method does not have database restore process through New-NAVdatabase command.

Scripts used to create container and cause the issue

$artifactUrl=Get-BCArtifactUrl -type sandbox -country us -select Latest
Write-Host -ForegroundColor Yellow "Using this artifact $artifactUrl to create container"

New-BCContainer -accept_eula -accept_outdated -containerName TEST -memoryLimit 8G `
-artifactUrl $artifactUrl `
-alwaysPull -doNotCheckHealth -auth Windows  -restart unless-stopped `
-shortcuts Desktop -doNotExportObjectsToText `
-additionalParameters @('--network=TransparentNet --security-opt "credentialspec=file://gMSAnavnst.json"')

Full output of scripts (please look at the last 15 lines)

Using this artifact https://bcartifacts.azureedge.net/sandbox/16.3.14085.15298/us to create container
NavContainerHelper is version 0.7.0.23
NavContainerHelper is running as administrator
Host is Microsoft Windows Server 2019 Standard - ltsc2019
Docker Client Version is 19.03.5
Docker Server Version is 19.03.5
Fetching all docker images
Pulling image mcr.microsoft.com/dynamicsnav:10.0.17763.1339-generic
10.0.17763.1339-generic: Pulling from dynamicsnav
Using image mcr.microsoft.com/dynamicsnav:10.0.17763.1339-generic
Disabling Health Check (always report healthy)
Creating Container TEST
Version: 16.3.14085.15298-US
Style: sandbox
Platform: 16.0.14073.15236
Generic Tag: 0.1.0.13
Container OS Version: 10.0.17763.1339 (ltsc2019)
Host OS Version: 10.0.17763.1339 (ltsc2019)
Using locale en-US
Using process isolation
Disabling the standard eventlog dump to container log every 2 seconds (use -dumpEventLog to enable)
Additional Parameters:
--network=TransparentNet --security-opt "credentialspec=file://gMSAnavnst.json"
Files in C:\ProgramData\NavContainerHelper\Extensions\TEST\my:
- AdditionalOutput.ps1
- MainLoop.ps1
- SetupVariables.ps1
- updatehosts.ps1
Creating container TEST from image mcr.microsoft.com/dynamicsnav:10.0.17763.1339-generic
28f2330544d7b0f17a0d2346bb8c06fa4472e578bb152dac25fe4d25d5b46389
Waiting for container TEST to be ready
Using artifactUrl https://bcartifacts.azureedge.net/sandbox/16.3.14085.15298/us
Using installer from C:\Run\150-new
Installing Business Central
Installing from artifacts
Starting Local SQL Server
Starting Internet Information Server
Copying Service Tier Files
Copying PowerShell Scripts
Copying dependencies
Copying ReportBuilder
Importing PowerShell Modules
Determining Database Collation from c:\dl\sandbox\16.3.14085.15298\us\BusinessCentral-US.bak
Restoring CRONUS Demo Database
The database restore operation failed due to the following error returned by SQL Server:
A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (
provider: SQL Network Interfaces, error: 26 - Error Locating Server/Instance Specified)
at <ScriptBlock>, C:\Run\navinstall.ps1: line 182
at <ScriptBlock>, C:\Run\start.ps1: line 202
at <ScriptBlock>, <No file>: line 1Error
Using artifactUrl https://bcartifacts.azureedge.net/sandbox/16.3.14085.15298/us
Using installer from C:\Run\150-new
Installing Business Central
Installing from artifacts
Starting Local SQL Server
Starting Internet Information Server
Copying Service Tier Files
Copying PowerShell Scripts
Copying dependencies
Copying ReportBuilder
Importing PowerShell Modules
Determining Database Collation from c:\dl\sandbox\16.3.14085.15298\us\BusinessCentral-US.bak
Restoring CRONUS Demo Database
The database restore operation failed due to the following error returned by SQL Server:
A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (
provider: SQL Network Interfaces, error: 26 - Error Locating Server/Instance Specified)
at <ScriptBlock>, C:\Run\navinstall.ps1: line 182
at <ScriptBlock>, C:\Run\start.ps1: line 202
at <ScriptBlock>, <No file>: line 1
Initialization of container TEST failed
At C:\Program Files\WindowsPowerShell\Modules\navcontainerhelper\0.7.0.23\ContainerHandling\Wait-NavContainerReady.ps1:44 char:17
+ ...             throw "Initialization of container $containerName failed" ...
+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : OperationStopped: (Initialization of container TEST failed:String) [], RuntimeException
    + FullyQualifiedErrorId : Initialization of container TEST failed

[freddydk]

Could you try to add

-imageName myimg

to the command to make containerhelper create an image on the fly and run that.

[phamhainguyen68]

Thank you for the fast response, Fred. Using imageName fixes the issue.

I guess this is because during the creation of the image, the temp container is spun up using NAT network and is able to restore the database. After the image has been created, the command builds the real container using the normal image method, hence no issue.

This does not fix the root cause (New-NAVDatabase) though. Could you escalate this to your internal team to see if they have any idea?

[freddydk]

Could it be that the issue is not the network but the gMSA setting?
This is kind of the same problem as is reported here: https://github.com/microsoft/nav-docker/issues/478

[phamhainguyen68]

Hi Freddy, It is indeed related to the use of gMSA. If I do not use that switch, the database restore step runs fine.

However, my situation is a bit different from the one in microsoft/nav-docker#478. The SQL service in my container does not stop after the collation change. It runs all the time.

In my case, the root issue is in the New-NAVDatabase command. The command is only able to connect to the SQL server using -DatabaseServer localhost\SQLEXPRESS (default setting in navinstall.ps1 script is -DatabaseServer localhost)

[phamhainguyen68]

Update: New-NAVDatabase command also works with -DatabaseServer "localhost,1433" -DatabaseInstance "SQLEXPRESS" (i.e. specifying the default SQL port, 1433).

[phamhainguyen68]

Update: using _-Verbose_ switch for New-NAVDatabase command, I found out that, if we use localhost for _-DatabaseServer_ switch, somehow, the command converts it to the gMSA account name when executing:

New-NAVDatabase -DatabaseServer "localhost" -DatabaseInstance "SQLEXPRESS" -Verbose
VERBOSE: Performing the operation "New-NAVDatabase" on target "DatabaseServer = gMSAnavnst, DatabaseInstance = SQLEXPRESS, DatabaseName = CRONUS".

That is not the case for localhost,1433 or localhost\SQLEXPRESS

New-NAVDatabase -DatabaseServer "localhost,1433" -DatabaseInstance "SQLEXPRESS" -Verbose
VERBOSE: Performing the operation "New-NAVDatabase" on target "DatabaseServer = localhost,1433, DatabaseInstance = SQLEXPRESS, DatabaseName = CRONUS".
VERBOSE: Restoring database 'CRONUS' from backup file: 'c:\dl\sandbox\16.3.14085.15298\us\BusinessCentral-US.bak'...
VERBOSE: Restore of database 'CRONUS' from file 'c:\dl\sandbox\16.3.14085.15298\us\BusinessCentral-US.bak' completed 
successfully.

New-NAVDatabase -DatabaseServer "localhost\SQLEXPRESS" -Verbose
VERBOSE: Performing the operation "New-NAVDatabase" on target "DatabaseServer = localhost\SQLEXPRESS, DatabaseInstance = , DatabaseName = CRONUS".
VERBOSE: Restoring database 'CRONUS' from backup file: 'c:\dl\sandbox\16.3.14085.15298\us\BusinessCentral-US.bak'...
VERBOSE: Restore of database 'CRONUS' from file 'c:\dl\sandbox\16.3.14085.15298\us\BusinessCentral-US.bak' completed successfully.

Edit: I think New-NAVDatabase command is hardcoded to convert "localhost" to the DNS/hostname of the server. When I use gMSA, the container will use gMSA's account name for its network name. With this finding, the quickest ways to overcome this is either modify the New-NAVDatabase command to use localhost\SQLEXPRESS, or just use imagename switch like you suggested

C:\Windows\System32\drivers\etc> ipconfig /all

Windows IP Configuration

   Host Name . . . . . . . . . . . . : gMSAnavnst
   Primary Dns Suffix  . . . . . . . : ourdomain.com
   Node Type . . . . . . . . . . . . : Hybrid
   IP Routing Enabled. . . . . . . . : No
   WINS Proxy Enabled. . . . . . . . : No

[freddydk]

That is really interresting.
If you do not use gMSA, then I assume that New-NavDatabase won't use the hostname right?
My assumption is that this is a problem with gMSA and not with New-NavDatabase, right?

[phamhainguyen68]

Hi Freddy,

Regardless of using gMSA or not, the New-NAVDatabase always uses hostname if specify localhost in _DatabaseServer_ switch.

The reason why the command works when not using gMSA is because in this situation, the hostname is the same as the container's name, not the gMSA's account name.

For example below, I created a container named TEST without gMSA. In this case, the hostname of the container is also TEST. When I ping this hostname from within the container, it is ping-able since it is equal to localhost (127.0.0.1) internally.

New-NAVDatabase -DatabaseServer "localhost" -DatabaseInstance "SQLEXPRESS" -DatabaseName "CRONUS" -Verbose
VERBOSE: Performing the operation "New-NAVDatabase" on target "DatabaseServer = TEST, DatabaseInstance = SQLEXPRESS, DatabaseName = CRONUS".

C:\Windows\system32> docker exec -it TEST powershell
Windows PowerShell
Copyright (C) Microsoft Corporation. All rights reserved.
   Host Name . . . . . . . . . . . . : TEST

PS C:\Program Files\Microsoft Dynamics NAV\160\Service> ping TEST

Pinging TEST [fe80::e4cd:d17f:aa3e:f4d7%46] with 32 bytes of data:
Reply from fe80::e4cd:d17f:aa3e:f4d7%46: time<1ms
Reply from fe80::e4cd:d17f:aa3e:f4d7%46: time<1ms
Reply from fe80::e4cd:d17f:aa3e:f4d7%46: time<1ms
Reply from fe80::e4cd:d17f:aa3e:f4d7%46: time<1ms



md5-6b31a8f3008bfe72737e35d12a783236



New-NAVDatabase -DatabaseServer "localhost" -DatabaseInstance "SQLEXPRESS" -DatabaseName "CRONUS" -Verbose
VERBOSE: Performing the operation "New-NAVDatabase" on target "DatabaseServer = gMSAnavnst, DatabaseInstance = SQLEXPRESS, DatabaseName = CRONUS".



md5-478df5d5050f23a3c0fb64019cf5d6ac



PS C:\Program Files\Microsoft Dynamics NAV\160\Service> ipconfig /all

Windows IP Configuration

   Host Name . . . . . . . . . . . . : gMSAnavnst



md5-478df5d5050f23a3c0fb64019cf5d6ac



PS C:\Program Files\Microsoft Dynamics NAV\160\Service> ping gMSAnavnst
Ping request could not find host gMSAnavnst. Please check the name and try again.

[freddydk]

Looked in the source code for New-NavDatabase and you are right - the switch happens there.
But - it happens deep down in helper files used by lots of other things and New-NavDatabase behaves differently if it determines that the database is on a local server.
This means that there might be other places, where this switch happens and/or it might have other consequences to make the switch.

Could you try this:

I assume, that if you run

[System.Net.Dns]::GetHostEntry('localhost')

Inside the container, it returns your gMSAnavst.

Could you insert this (before the call to new-navdatabase):

set-content -Path 'c:\windows\system32\drivers\etc\hosts' -Value "127.0.0.1 $([System.Net.Dns]::GetHostEntry('localhost').HostName)"

Just as a test to see whether this fixes the issue.

This should cause all commands to know that this is localhost - and it would also cause other commands using the same low level function to resolve.

[phamhainguyen68]

Hi Freddy,

Thank you for your suggestion. Modifying hosts file was my first thought as well, but I do not know how to do that during the creation of container, specifically, before the database restore step. I can do that manually by running PowerShell inside the container and it fixes the issue with New-NAVDatabase. However I have to wait for New-BCContainer command to fail and exit out.

I even tried to create a DNS record to point gMSAnavnst to 127.0.0.1 in our internal DNS server, but the container did not pick it up (it prioritizes its own internal record I guess).

Also, New-NAVDatabase gets the "localhost" data from the same source as _ipconfig /all_ command, not from _hostname_ or _[System.Net.Dns]::GetHostEntry('localhost')_. In normal scenario (i.e. no gMSA), all of these return the same result, TEST. However, in gMSA setup, _ipconfig/all_ returns gMSAnavnst, while _hostname_ or _[System.Net.Dns]::GetHostEntry('localhost')_ returns TEST

PS C:\> ipconfig /all

Windows IP Configuration

   Host Name . . . . . . . . . . . . : gMSAnavnst
   Primary Dns Suffix  . . . . . . . : ourdomain.com


PS C:\> hostname
TEST2

PS C:\> [System.Net.Dns]::GetHostEntry('localhost')

HostName Aliases AddressList
-------- ------- -----------
TEST2    {}      {::1, 127.0.0.1}

Update: _$env:computername_ also returns same result like _ipconfig /all_ in gMSA scenario. So I guess, your command can be modified as
set-content -Path 'c:\windows\system32\drivers\etc\hosts' -Value "127.0.0.1 $env:computername"

[freddydk]

Download https://raw.githubusercontent.com/microsoft/nav-docker/master/generic/Run/150-new/navinstall.ps1 and place it on your machine (ex. c:tempnavinstall.ps1)
Insert the line above in the beginning.
run new-bcContainer with -myscripts @("c:tempnavinstall.ps1")
You could insert a Write-Host to see that your file gets executed.

[phamhainguyen68]

Awesome! I will try this.

Thank you for the suggestion.

[freddydk]

Let me know if this worked, then I will add this line in the generic image if the host name starts with gmsa.

[phamhainguyen68]

Hi Freddy,

It works! I put the command (to modify the host file) right after line 10 and 11 of navinstall.ps1 file.
This is the command I used:

set-content -Path 'c:\windows\system32\drivers\etc\hosts' -Value "127.0.0.1 $env:computername"

You should not rely on the _host name starts with gMSA_ condition, simply because people can name their gMSA account in whatever way they want. It does not have to start with _gMSA_. Example: _BCgMSAaccount_, which makes the host name starts with _BC_, not _gMSA_

I think you can just hardcode this command without any condition (i.e. it always runs), because it gets the host name from $env:computername, which:

• returns the gMSA account name if you use gMSA

PS C:\> $env:computername
gMSAnavnst

PS C:\> hostname
TEST
PS C:\>

• returns the normal host name if you do not use gMSA.

PS C:\> $env:computername
TEST

PS C:\> hostname
TEST

In other word, it works with both situations and do not have any negative consequence.

[freddydk]

I don't want to just always add the line - it will probably break something we don't think about right here.

But I could test whether computername is different from hostname - and if that is the case - then add computername to hosts like you do here.

Thanks for the investigations.

[phamhainguyen68]

That is a good point!

Thank you very much for all the prompt and informative responses. Please let me know when you have implemented the change.

Have a good weekend!

[freddydk]

Generic image 0.1.0.14 has been released and the code is added to start.ps1 (not navinstall.ps1)

[phamhainguyen68]

Hi Freddy,

Could you also seed the change to onprem artifacts? It only works with sandbox at the moment.

Thank you in advance!

[phamhainguyen68]

@freddydk

Hi Freddy,

Could you also seed the change to onprem artifacts? It only works with sandbox at the moment.

Thank you in advance!

[freddydk]

This bug fix was not in the artifacts - it was in the generic image.
If something is wrong in onprem containers - either you have a cached image or something else is wrong.
copy/paste the full output of the onprem container generation and script.

[phamhainguyen68]

@freddydk I just realized that as well. Turns out, I am now having exact same issue with this one: https://github.com/microsoft/nav-docker/issues/478. The SQL service just crashed after changing Collation. This only occurs for onprem image. I will follow up with you through that issue 478 from now on.

Determining Database Collation from c:\dl\onprem\16.4.14693.15445\na\database\Demo Database NAV (16-0).bak
Changing Database Server Collation to Latin1_General_100_CI_AS
Restoring CRONUS Demo Database
The database restore operation failed due to the following error returned by SQL Server:
A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allo
w remote connections. (provider: TCP Provider, error: 0 - The remote computer refused the network connection.)
at <ScriptBlock>, C:\Run\navinstall.ps1: line 182
at <ScriptBlock>, C:\Run\start.ps1: line 213
at <ScriptBlock>, <No file>: line 1

PS C:\> Get-Service *SQL*

Status   Name               DisplayName
------   ----               -----------
Stopped  MSSQL$SQLEXPRESS   SQL Server (SQLEXPRESS)

[freddydk]

It will probably happen if you use a sandbox image, which uses a different collation than the original.