Azure-pipelines-tasks: Docker Build and Push Command has flaky output (sometimes)

[worldspawn]

Required Information

Bug
Type: Build fails. It shouldn't

Enter Task Name: Docker v2.170.2

Environment

• Server: dev.azure.com
• Agent - Self hosted vsts container running in azure aks
• Image: microsoft/vsts-agent:ubuntu-16.04-docker-18.06.1-ce-standard

Issue Description

We recently began receiving a rather meaningless error from a docker - build and push task. Somewhere in the dockerfile, around step 4 or 5, it starts doing node stuff, yarn install, yarn build etc. Theres no evidence in that output that the docker build process even began.

Task logs

Starting: Build and Push Bullclip Web
==============================================================================
Task         : Docker
Description  : Build or push Docker images, login or logout, or run a Docker command
Version      : 2.170.2
Author       : Microsoft Corporation
Help         : https://aka.ms/azpipes-docker-tsg
==============================================================================
##[error]Unhandled: Failed find: ENOENT: no such file or directory, stat '/vsts/agent/_work/2/s/node_modules/@babel/plugin-proposal-async-generator-functions/node_modules/@babel/plugin-syntax-async-generators/node_modules/@babel/core/node_modules/.bin/json5'
Finishing: Build and Push Bullclip Web

I was able to build other pipelines with docker files (that nothing had nothing to do with node) at the time. I finally logged into the vsts container and ran docker prune (we still had 14gb of free space). After that the build began to work... I can't really confirm that that was the fix for it as I wasn't out of space.

Whatever the underlying problem was the output I got from the task was really baffling (still is) and is not actionable. If the error occurred in the docker build process i should see docker build output. If the docker environment was broken i should see an appropriate message. Instead I got a ENOENT seemingly in relation to some node operation.

There isn't even the /usr/local/bin/docker build -f /vsts/agent/_work/3/s/Dockerfile ... line.

[worldspawn]

And obviously we ran this same thing locally, on windows 10 -> wsl 2 ubuntu and it built fine.

[worldspawn]

2 days later and its back. Same extremely terse output.

[sonayak]

@worldspawn : Sorry for the trouble. The issue seems to be coming from the task runtime and not from docker build. But I am not able to repro it. Can you try a run with 'system.debug' variable set and share the log with us? Are there multiple versions of of node installed in the system?

[worldspawn]

Can you try a run with 'system.debug' variable set and share the log with us?

That would have been a good idea 🤦‍♂️

I ended up deleting the whole node pool and starting again. It hasn't resurfaced yet. I have recently added a Node.js tool installer task to set the node version too 14.4.0 where before I guess it just used the default in the agent image. I suppose this means there are multiple version of node installed?

Will update if it occurs again with a debug run.

[worldspawn]

Hi. Its back :(

Here's the debug run output: https://gist.github.com/worldspawn/fcb2cf30b914c355f330a72f59a6d640

Have confirmed this happens for both agents (which run on the same node).

@sonayak

[sonayak]

Thanks @worldspawn for sharing the logs. The issue is very puzzling. We are trying to provide a fix.
In the meantime, you can avoid this by using direct path for Dockerfile like ./Dockerfile to unblock yourself. Please let us know if the workaround works for you.

[worldspawn]

Probably should have included the yaml:

parameters:
  - name: region
    type: string

jobs:
  - job: BuildPush
    displayName: Build and Push
    steps:
      - task: Docker@2
        displayName: Login to ACR
        inputs:
          command: login
          containerRegistry: drawboard${{ parameters.region }}
      - task: Docker@2
        displayName: Build and Push Bullclip Web
        inputs:
          command: buildAndPush
          repository: drawboard-teams-web
          tags: |
            $(Build.BuildNumber)

@sonayak I added the Dockerfile element to the task and the build worked. I then removed and the problem resurfaced 👍 You're on to something :)

[koushdey]

@worldspawn Is this issue still occurring during build when you don't provide an absolute path? The issue seems to be from the tasklib module that we use to detect files. It might be due to the large repo size.

[koushdey]

Closing the issue as there is no response on this.