Skaffold: Skaffold v1.17.1 unable to stream build output when copying from directory of larger context

[ahmad-musleh]

Expected behavior

skaffold build 
Generating tags...
 - service-1 -> 1.2.3.4:32000/service-1:latest
 - service-2 -> 1.2.3.4:32000/service-2:latest
 - service-3 -> 1.2.3.4:32000/service-3:latest
Checking cache...
 - service-1: Not found. Building
 - service-2: Not found. Building
 - service-3: Not found. Building
Building [service-3]...
Sending build context to Docker daemon   5.12kB
Step 1/4 : FROM python:3
Step 2/4 : ARG service_dir
Step 3/4 : COPY $service_dir /general
Step 4/4 : CMD ["python3", "-c", "while True: continue"]
Successfully built 3abe1820e6ec
Successfully tagged 10.4.6.71:32000/service-3:latest
Building [service-1]...
Sending build context to Docker daemon   5.12kB
Step 1/4 : FROM python:3
Step 2/4 : ARG service_dir
Step 3/4 : COPY $service_dir /general
Step 4/4 : CMD ["python3", "-c", "while True: continue"]
Successfully built ca63b88113c0
Successfully tagged 10.4.6.71:32000/service-1:latest
Building [service-2]...
Sending build context to Docker daemon   5.12kB
Step 1/4 : FROM python:3
Step 2/4 : ARG service_dir
Step 3/4 : COPY $service_dir /general
Step 4/4 : CMD ["python3", "-c", "while True: continue"]
Successfully built 8a6e49f61653

This was using v1.16.0

Actual behavior

skaffold build 
Generating tags...
 - service-1 -> 1.2.3.4:32000/service-1:latest
 - service-2 -> 1.2.3.4:32000/service-2:latest
 - service-3 -> 1.2.3.4:32000/service-3:latest
Checking cache...
 - service-1: Not found. Building
 - service-2: Not found. Building
 - service-3: Not found. Building
Building [service-3]...
Sending build context to Docker daemon   5.12kB
Step 1/4 : FROM python:3
Step 2/4 : ARG service_dir
Step 3/4 : COPY $service_dir /general
Building [service-1]...
Building [service-2]...
unable to stream build output: COPY failed: stat /var/lib/docker/tmp/docker-builder804227384/service_3: no such file or directory

Information

• Skaffold version: version v1.17.0
• Operating system: Ubuntu 16.04.5 LTS
• Contents of skaffold.yaml:

apiVersion: skaffold/v2alpha3
kind: Config
build:
  local: {}
  tagPolicy:
    sha256: {}
  artifacts:
    - image: service-1
      context: .
      docker:
        dockerfile: general_dir/Dockerfile
        buildArgs:
          service_dir: service_1

    - image: service-2
      context: .
      docker:
        dockerfile: general_dir/Dockerfile
        buildArgs:
          service_dir: service_2

    - image: service-3
      context: .
      docker:
        dockerfile: general_dir/Dockerfile
        buildArgs:
          service_dir: service_3

Steps to reproduce the behavior

1. Install Skaffold v1.17.0
2. Create dir with the following tree structure:

.
|-- general_dir
|   `-- Dockerfile
|-- service_1
|   `-- Chart.yaml
|-- service_2
|   `-- Chart.yaml
|-- service_3
|   `-- Chart.yaml
`-- skaffold.yaml

where each chart.yaml contains the following:

apiVersion: v2
appVersion: "1.0"
name: service-i
version: 0.1.0

and Dockerfile the has an ARG followed by a COPY

FROM python:3

ARG service_dir
COPY $service_dir /general

CMD ["python3", "-c", "while True: continue"]

1. skaffold build

[UPDATE] simplified scenario to only include skaffold build

[tejal29]

Thanks @ahmad-musleh for the detailed issue.
I was able to produce this issue locally.

Looks like, this is due to "dockerfile path" being the key for caching dependencies.
Since service_1, service_2 and service_3 use the same dockerfiles.
The GetDependency when CreateTarContext is called returns cached values for both service_1 and service_2 artifact which is relative to service_3. Hence the docker command fails.

[tejal29]

Related to #5110

[tejal29]

https://github.com/GoogleContainerTools/skaffold/releases/tag/v1.17.2 is out!