Azure-pipelines-agent: container jobs fails to connect to docker daemon on Windows Server 2019

[dtzar]

Having issue with YAML?

No

Having issue with Tasks?

No

Having issue with software on Hosted Agent?

No

Having generic issue with Azure-Pipelines/VSTS/TFS?

No, but did check the developer community if a similar issue existed.

Have you tried troubleshooting?

Yes, none of these steps apply.

Agent Version and Platform

Version of your agent? 2.144.2

OS of the machine running the agent?
Windows Server 2019 Datacenter with Containers (Azure marketplace VM)

Installed docker via:

Install-Module -Name DockerMsftProvider -Repository PSGallery -Force
Install-Package -Name docker -ProviderName DockerMsftProvider -Force -RequiredVersion 18.09
Start-Service docker

Azure DevOps Type and Version

dev.azure.com

What's not working?

It seems the agent does not properly initialize the docker daemon and crashes every time any job is run with this error:

2019-02-02T00:37:51.7189718Z ##[section]Starting: Initialize container
2019-02-02T00:37:51.7810243Z ##[command]C:\Program Files\Docker\docker.EXE version --format '{{.Server.Version}}'
2019-02-02T00:37:52.5526451Z error during connect: Get http://%2F%2F.%2Fpipe%2Fdocker_engine/v1.39/version: open //./pipe/docker_engine: Access is denied. In the default daemon configuration on Windows, the docker client must be run elevated to connect. This error may also indicate that the docker daemon is not running.
2019-02-02T00:37:52.5532460Z '
2019-02-02T00:37:52.7600346Z ##[error]Exit code 1 returned from process: file name 'C:\Program Files\Docker\docker.EXE', arguments 'version --format '{{.Server.Version}}''.
2019-02-02T00:37:52.7631665Z ##[section]Finishing: Initialize container

Docker works fine on the host server as I can pull down and run images down without any issues.

I then tried installing Docker for Desktop on the Windows Server to see if this would change the behavior and it did when I ran Linux containers it gave this output:

2019-01-31T19:34:53.4489829Z ##[section]Starting: Initialize container
2019-01-31T19:34:53.4894022Z ##[command]C:\Program Files\Docker\Docker\Resources\bin\docker.EXE version --format '{{.Server.Version}}'
2019-01-31T19:34:53.9039369Z '18.09.1'
2019-01-31T19:34:53.9181062Z Docker daemon version: '18.09.1'
2019-01-31T19:34:53.9181274Z ##[command]C:\Program Files\Docker\Docker\Resources\bin\docker.EXE version --format '{{.Client.Version}}'
2019-01-31T19:34:54.2232180Z '18.09.1'
2019-01-31T19:34:54.2333506Z Docker client version: '18.09.1'
2019-01-31T19:34:54.2438312Z ##[command]C:\Program Files\Docker\Docker\Resources\bin\docker.EXE ps --all --quiet --no-trunc --filter "label=67dc44"
2019-01-31T19:34:54.4390361Z ##[command]C:\Program Files\Docker\Docker\Resources\bin\docker.EXE pull buhler/buildagent-dotnet-tools:latest
2019-01-31T19:34:56.1161406Z latest: Pulling from buhler/buildagent-dotnet-tools
2019-01-31T19:34:56.1274705Z 3889bb8d808b: Pulling fs layer
2019-01-31T19:34:56.1336236Z 900b07d43172: Pulling fs layer
…
2019-01-31T19:34:56.1437942Z e5a401be5270: Waiting
2019-01-31T19:34:56.7146487Z image operating system "windows" cannot be used on this platform
2019-01-31T19:34:56.7364693Z ##[warning]Docker pull failed with exit code 1, back off 8.2 seconds before retry.
2019-01-31T19:35:04.9469722Z ##[command]C:\Program Files\Docker\Docker\Resources\bin\docker.EXE pull buhler/buildagent-dotnet-tools:latest
2019-01-31T19:35:06.2913977Z latest: Pulling from buhler/buildagent-dotnet-tools
2019-01-31T19:35:06.2974490Z 3889bb8d808b: Pulling fs layer
…
2019-01-31T19:35:06.3055069Z e5a401be5270: Waiting
2019-01-31T19:35:06.8159314Z image operating system "windows" cannot be used on this platform
2019-01-31T19:35:06.8253636Z ##[warning]Docker pull failed with exit code 1, back off 4.774 seconds before retry.
2019-01-31T19:35:11.6054183Z ##[command]C:\Program Files\Docker\Docker\Resources\bin\docker.EXE pull buhler/buildagent-dotnet-tools:latest
2019-01-31T19:35:12.9581187Z latest: Pulling from buhler/buildagent-dotnet-tools
2019-01-31T19:35:12.9583507Z 3889bb8d808b: Pulling fs layer
2019-01-31T19:35:12.9583993Z 900b07d43172: Pulling fs layer
2019-01-31T19:35:12.9585059Z d5f1862af3b5: Pulling fs layer
…
2019-01-31T19:35:12.9650637Z e5a401be5270: Waiting
2019-01-31T19:35:12.9650690Z cce31346ada7: Waiting
2019-01-31T19:35:12.9650723Z 445d98c6f2ed: Waiting
2019-01-31T19:35:13.4790691Z image operating system "windows" cannot be used on this platform
2019-01-31T19:35:13.5528467Z ##[error]Docker pull failed with exit code 1
2019-01-31T19:35:13.5538955Z ##[section]Finishing: Initialize container

Problem is I desire to use Windows based containers and switching Docker for Desktop to use Windows containers produces this error in Docker for Desktop:

Failed to register Docker daemon Windows service
   at Docker.Backend.Processes.WindowsDockerDaemon.TryToStartService(Settings settings, String args, Dictionary`2 env) in C:\workspaces\stable-18.09.x\src\github.com\docker\pinata\win\src\Docker.Backend\Processes\WindowsDockerDaemon.cs:line 241
   at Docker.Backend.Processes.WindowsDockerDaemon.Start(Settings settings, String daemonOptions) in C:\workspaces\stable-18.09.x\src\github.com\docker\pinata\win\src\Docker.Backend\Processes\WindowsDockerDaemon.cs:line 165
   at Docker.Backend.ContainerEngine.Windows.DoStart(Settings settings, String daemonOptions) in C:\workspaces\stable-18.09.x\src\github.com\docker\pinata\win\src\Docker.Backend\ContainerEngine\Windows.cs:line 225
   at Docker.Backend.ContainerEngine.Windows.Start(Settings settings, String daemonOptions) in C:\workspaces\stable-18.09.x\src\github.com\docker\pinata\win\src\Docker.Backend\ContainerEngine\Windows.cs:line 98
   at Docker.Core.Pipe.NamedPipeServer.<>c__DisplayClass9_0.<Register>b__0(Object[] parameters) in C:\workspaces\stable-18.09.x\src\github.com\docker\pinata\win\src\Docker.Core\pipe\NamedPipeServer.cs:line 48
   at Docker.Core.Pipe.NamedPipeServer.RunAction(String action, Object[] parameters) in C:\workspaces\stable-18.09.x\src\github.com\docker\pinata\win\src\Docker.Core\pipe\NamedPipeServer.cs:line 180

Agent and Worker's Diagnostic Logs

I can privately provide the full log to the engineer troubleshooting, but this seems to be the most relevant part of the worker diagnostic log:

[2019-02-01 23:57:45Z INFO JobRunner] Total job steps: 5.
[2019-02-01 23:57:45Z INFO JobRunner] Total accessible running process: 140.
[2019-02-01 23:57:45Z INFO JobRunner] Run all job steps.
[2019-02-01 23:57:45Z INFO StepsRunner] Processing step: DisplayName='Initialize container', ContinueOnError=False, Enabled=True
[2019-02-01 23:57:45Z INFO ExpressionManager] Evaluating: SucceededNode()
[2019-02-01 23:57:45Z INFO ExpressionManager] Result: True
[2019-02-01 23:57:45Z INFO StepsRunner] Starting the step.
[2019-02-01 23:57:45Z INFO ContainerOperationProvider] Container name: 0434e615fc544b10ab4119b4c5aa60a5
[2019-02-01 23:57:45Z INFO ContainerOperationProvider] Container image: buhler/buildagent-dotnet-tools:latest
[2019-02-01 23:57:45Z INFO ContainerOperationProvider] Container registry: 00000000-0000-0000-0000-000000000000
[2019-02-01 23:57:45Z INFO ContainerOperationProvider] Container options: 
[2019-02-01 23:57:45Z INFO ContainerOperationProvider] Skip container image pull: False
[2019-02-01 23:57:45Z INFO HostContext] Well known directory 'Bin': 'C:\agents\build-devops-01-01\bin'
[2019-02-01 23:57:45Z INFO HostContext] Well known directory 'Root': 'C:\agents\build-devops-01-01'
[2019-02-01 23:57:45Z INFO HostContext] Well known directory 'Work': 'C:\agents\build-devops-01-01\_work'
[2019-02-01 23:57:45Z INFO ProcessInvokerWrapper] Starting process:
[2019-02-01 23:57:45Z INFO ProcessInvokerWrapper]   File name: 'C:\Program Files\Docker\Docker\Resources\bin\docker.EXE'
[2019-02-01 23:57:45Z INFO ProcessInvokerWrapper]   Arguments: 'version --format '{{.Server.Version}}''
[2019-02-01 23:57:45Z INFO ProcessInvokerWrapper]   Working directory: 'C:\agents\build-devops-01-01\_work'
[2019-02-01 23:57:45Z INFO ProcessInvokerWrapper]   Require exit code zero: 'True'
[2019-02-01 23:57:45Z INFO ProcessInvokerWrapper]   Encoding web name:  ; code page: ''
[2019-02-01 23:57:45Z INFO ProcessInvokerWrapper]   Force kill process on cancellation: 'False'
[2019-02-01 23:57:45Z INFO ProcessInvokerWrapper]   Lines to send through STDIN: '0'
[2019-02-01 23:57:45Z INFO ProcessInvokerWrapper]   Persist current code page: 'False'
[2019-02-01 23:57:45Z INFO ProcessInvokerWrapper] Process started with process id 5412, waiting for process exit.
[2019-02-01 23:57:45Z INFO JobServerQueue] Try to append 1 batches web console lines for record 'd2afea5c-6cfc-4b65-bee5-c8de2111587e', success rate: 1/1.
[2019-02-01 23:57:45Z INFO JobServerQueue] Try to append 1 batches web console lines for record 'edf04652-f639-4565-9a89-6483baccf877', success rate: 1/1.
[2019-02-01 23:57:45Z INFO JobServerQueue] Try to upload 1 log files or attachments, success rate: 1/1.
[2019-02-01 23:57:46Z INFO ProcessInvokerWrapper] STDOUT/STDERR stream read finished.
[2019-02-01 23:57:46Z INFO ProcessInvokerWrapper] STDOUT/STDERR stream read finished.
[2019-02-01 23:57:46Z INFO ProcessInvokerWrapper] Finished process 5412 with exit code 1, and elapsed time 00:00:00.4793308.
[2019-02-01 23:57:46Z INFO JobServerQueue] Try to append 1 batches web console lines for record 'edf04652-f639-4565-9a89-6483baccf877', success rate: 1/1.
[2019-02-01 23:57:46Z ERR  StepsRunner] Caught exception from step: Microsoft.VisualStudio.Services.Agent.Util.ProcessExitCodeException: Exit code 1 returned from process: file name 'C:\Program Files\Docker\Docker\Resources\bin\docker.EXE', arguments 'version --format '{{.Server.Version}}''.
   at Microsoft.VisualStudio.Services.Agent.Util.ProcessInvoker.ExecuteAsync(String workingDirectory, String fileName, String arguments, IDictionary`2 environment, Boolean requireExitCodeZero, Encoding outputEncoding, Boolean killProcessOnCancel, IList`1 contentsToStandardIn, Boolean inheritConsoleHandler, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.Agent.ProcessInvokerWrapper.ExecuteAsync(String workingDirectory, String fileName, String arguments, IDictionary`2 environment, Boolean requireExitCodeZero, Encoding outputEncoding, Boolean killProcessOnCancel, IList`1 contentsToStandardIn, Boolean inheritConsoleHandler, CancellationToken cancellationToken)
   at Microsoft.VisualStudio.Services.Agent.Worker.Container.DockerCommandManager.ExecuteDockerCommandAsync(IExecutionContext context, String command, String options)
   at Microsoft.VisualStudio.Services.Agent.Worker.Container.DockerCommandManager.DockerVersion(IExecutionContext context)
   at Microsoft.VisualStudio.Services.Agent.Worker.ContainerOperationProvider.StartContainerAsync(IExecutionContext executionContext, Object data)
   at Microsoft.VisualStudio.Services.Agent.Worker.JobExtensionRunner.RunAsync()
   at Microsoft.VisualStudio.Services.Agent.Worker.StepsRunner.RunStepAsync(IStep step, CancellationToken jobCancellationToken)
[2019-02-01 23:57:46Z INFO StepsRunner] Step result: Failed
[2019-02-01 23:57:46Z INFO StepsRunner] Update job result with current step result 'Failed'.
[2019-02-01 23:57:46Z INFO StepsRunner] Current state: job state = 'Failed'
[2019-02-01 23:57:46Z INFO StepsRunner] Processing step: DisplayName='Get sources', ContinueOnError=False, Enabled=True
[2019-02-01 23:57:46Z INFO ExpressionManager] Evaluating: SucceededNode()
[2019-02-01 23:57:46Z INFO ExpressionManager] Result: False

[TingluoHuang]

@dtzar the agent tries to run C:\Program Files\Docker\docker.EXE version --format '{{.Server.Version}}'
and it received error from docker daemon.
error during connect: Get http://%2F%2F.%2Fpipe%2Fdocker_engine/v1.39/version: open //./pipe/docker_engine: Access is denied. In the default daemon configuration on Windows, the docker client must be run elevated to connect. This error may also indicate that the docker daemon is not running.

you can try run the same command as the user that agent process running as and see whether you are able to get output from docker version, i am wonder the agent is running as different user and don't have enough permission.

[dtzar]

We changed the agent to run as the "devops" user and now it works. At a minimum documentation needs updating. Ideally it would be nice to do something like the agent install to check what user the docker daemon is running as and ask or try to run as this user automatically.

[TingluoHuang]

@dtzar what's devops user?
i assume your docker daemon is running as devops and your agent runs as FOO originally, which cause the problem?
I will update the documentation.

[TingluoHuang]

I am going to close this issue since we get a doc update.
https://docs.microsoft.com/en-us/azure/devops/pipelines/process/container-phases?view=azure-devops&tabs=yaml#requirements

[dtzar]

If the update was adding "Be sure your pipelines agent has permission to access the Docker daemon." IMO we should provide a little more guidance.

[TingluoHuang]

Will this enough? Be sure your pipelines agent has permission to access the Docker daemon.

[ctaggart]

How do we change the agent to run as the devops user? I opened up https://github.com/MicrosoftDocs/vsts-docs/issues/3815