tilt 0.10.12 failing to build some pods

[sssilver]

Hi,

Tilt 0.10.12 is failing to build some repos that Tilt 0.10.8 builds successfully.

The error says:

ERROR: ImageBuild: failed to solve with frontend dockerfile.v0: failed to build LLB: failed to compute cache key: "/dev-stack/origintunneld/config.yaml" not found: not found
got unexpected error during build/deploy: ImageBuild: failed to solve with frontend dockerfile.v0: failed to build LLB: failed to compute cache key: "/dev-stack/origintunneld/config.yaml" not found: not found

Curiously, 12 of the 18 pods in our dev stack are fine. The other 6 fail with the above error.

Downgrading to 0.10.8 makes the problem go away.

This is on the latest macOS with Docker for Mac and its Kubernetes cluster.

[jazzdan]

Hey @sssilver, can you paste the output of tilt doctor?

Thanks!

[sssilver]

With pleasure!

$ tilt doctor
Tilt: v0.10.12, built 2019-10-02
System: darwin-amd64
---
Docker
- Host: [default]
- Version: 1.40
- Builder: 2
---
Kubernetes
- Env: docker-for-desktop
- Context: docker-desktop
- Cluster Name: docker-desktop
- Namespace: default
- Container Runtime: docker
- Version: v1.14.6
---
Thanks for seeing the Tilt Doctor!
Please send this info along when filing bug reports. 💗

[jazzdan]

@sssilver interesting, thanks! I have the exact same setup and can't reproduce, hmm.

Can you try running tilt like so?

DOCKER_BUILDKIT=0 tilt up

And see if the problem still reproduces?

[jazzdan]

A similar report in #tilt got this when running with DOCKER_BUILDKIT=0:

ERROR: ImageBuild: COPY failed: stat /var/lib/docker/tmp/docker-builder267372699/config: no such file or directory
stormtroope…┊ got unexpected error during build/deploy: ImageBuild: COPY failed: stat /var/lib/docker/tmp/docker-builder267372699/config: no such file or directory

[jazzdan]

Also anything you could detect @sssilver about what makes the builds that are failing similar to each other and different from the ones that are working just fine?

Maybe something to do with what kinds of files they are COPY'ing in?

If you try to do RUN ls -altrh path/to/file before copying it, what does Tilt display after it executes that RUN?

[sssilver]

Ran tilt with $ DOCKER_BUILDKIT=0 tilt up. The same set of services fail with:

...
ERROR: ImageBuild: COPY failed: stat /var/lib/docker/tmp/docker-builder916971671/dev-stack/origintunneld/origintunneld: no such file or directory
got unexpected error during build/deploy: ImageBuild: COPY failed: stat /var/lib/docker/tmp/docker-builder916971671/dev-stack/origintunneld/origintunneld: no such file or directory

The kinds of files that are being COPYed in are just the service binaries.

What they all have in common is that they're all services written in Go, built in the same repo. What baffles me is that with tilt 0.10.8 there's no problem -- everything works great.

[sssilver]

Do you happen to use a messaging app of some sort? Happy to help debug in a more real-time mode.

[nicks]

hi @sssilver, there's a #tilt channel in the kubernetes slack where we all hang out (go to https://slack.k8s.io for the invite and find us in https://kubernetes.slack.com/messages/CESBL84MV/)

[nicks]

I have a strong suspicion that this is caused by some subtle changes in how Tilt interprets dockerignore files.

If you have time, it would really help me out if you confirmed that this worked in v0.10.9 (https://github.com/windmilleng/tilt/releases/tag/v0.10.9) but broke in v0.10.10 (https://github.com/windmilleng/tilt/releases/tag/v0.10.10)

[jazzdan]

@sssilver We released a new version this morning (v0.10.13) which we hope addresses this issue. Let us know here if it does? Thanks so much for the bug report and your help!

[sssilver]

Everything works now -- y'all are amazing! Thanks so much!

[chuckha]

I'm seeing this as well. Here is the output of tilt doctor.

chuckh-a02:capi-dev cha$ tilt doctor
Tilt: v0.10.13, built 2019-10-04
System: darwin-amd64
---
Docker
- Host: [default]
- Version: 1.40
- Builder: 2
---
Kubernetes
- Env: unknown
- Context: kubernetes-admin@kubernetes
- Cluster Name: kubernetes
- Namespace: default
- Container Runtime: containerd
- Version: v1.15.5
---
Thanks for seeing the Tilt Doctor!
Please send this info along when filing bug reports. 💗

Here is the exact error

ERROR: ImageBuild: failed to solve with frontend dockerfile.v0: failed to build LLB: executor failed running [/bin/sh -c CGO_ENABLED=0 GOOS=linux GOARCH=amd64 GO111MODULE=on go build -a -o manager main.go]: exit code: 1
got unexpected error during build/deploy: ImageBuild: failed to solve with frontend dockerfile.v0: failed to build LLB: executor failed running [/bin/sh -c CGO_ENABLED=0 GOOS=linux GOARCH=amd64 GO111MODULE=on go build -a -o manager main.go]: exit code: 1

[sourcec0de]

Hi all, I'm running into a similar issue.

https://cloud.tilt.dev/snapshot/AaSD8NsLjPADUOHCm1o=
https://github.com/sourcec0de/tilt-mono

This error does not occur outside of tilt.
I can run docker build --file docker/node_base.docker --build-arg SERVICE_NAME=foo . and the container build succeeds.

foo         ┊ ERROR: ImageBuild: failed to solve with frontend dockerfile.v0: failed to build LLB: failed to compute cache key: "/src/js/lib" not found: not found
foo         ┊ got unexpected error during build/deploy: ImageBuild: failed to solve with frontend dockerfile.v0: failed to build LLB: failed to compute cache key: "/src/js/lib" not found: not found
Build Failed: ImageBuild: failed to solve with frontend dockerfile.v0: failed to build LLB: failed to compute cache key: "/src/js/lib" not found: not found

Other things I tried before commenting.

• docker system prune
• Updated to the latest stable docker version
• Reset docker to factory

tilt doctor

Tilt: v0.10.17, built 2019-11-01
System: darwin-amd64
---
Docker
- Host: [default]
- Version: 1.40
- Builder: 2
---
Kubernetes
- Env: docker-for-desktop
- Context: docker-desktop
- Cluster Name: docker-desktop
- Namespace: default
- Container Runtime: docker
- Version: v1.14.7
---
Thanks for seeing the Tilt Doctor!
Please send the info above when filing bug reports. 💗

The info below helps us understand how you're using Tilt so we can improve,
but is not required to ask for help.
---
Analytics Settings
- User Mode: opt-in
- Machine: 1b4e5db88519a532a9e689d4f35e224f
- Repo:

Error Logs

Starting Tilt (v0.10.17, built 2019-11-01)…
Beginning Tiltfile execution
Successfully loaded Tiltfile (30.957155ms)
foo         ┊ 
foo         ┊ ──┤ Building: foo ├──────────────────────────────────────────────
foo         ┊ STEP 1/3 — Building Dockerfile: [docker.io/library/foo]
foo         ┊ Building Dockerfile:
foo         ┊   FROM node:10
foo         ┊   
foo         ┊   ARG SERVICE_NAME
foo         ┊   ARG APP_BASE=/opt/app
foo         ┊   ARG SERVICE_LIB=src/js/lib
foo         ┊   ARG SERVICE_DIR=src/js/services/${SERVICE_NAME}
foo         ┊   
foo         ┊   WORKDIR ${APP_BASE}
foo         ┊   
foo         ┊   RUN mkdir -p ${APP_BASE}/${SERVICE_LIB}
foo         ┊   RUN mkdir -p ${APP_BASE}/${SERVICE_DIR}
foo         ┊   
foo         ┊   COPY ${SERVICE_DIR} ${APP_BASE}/${SERVICE_DIR}
foo         ┊   COPY ${SERVICE_LIB} ${APP_BASE}/${SERVICE_LIB}
foo         ┊   
foo         ┊   WORKDIR ${APP_BASE}/${SERVICE_DIR}
foo         ┊   
foo         ┊   RUN npm i
foo         ┊   
foo         ┊   CMD npm run start:watch
foo         ┊ 
foo         ┊   │ Tarring context…
foo         ┊   │ Building image
foo         ┊     ╎ copy /context /
foo         ┊     ╎ copy /context / done | 1.277s
foo         ┊     ╎ [1/8] FROM docker.io/library/node:10@sha256:dabc15ad36a9e0a95862fbdf6ffdad439edc20aa27c7f10456644464e3fb5f08
foo         ┊     ╎ [cached] [2/8] WORKDIR /opt/app
foo         ┊     ╎ [cached] [3/8] RUN mkdir -p /opt/app/src/js/lib
foo         ┊     ╎ [cached] [4/8] RUN mkdir -p /opt/app/src/js/services/foo
foo         ┊     ╎ [cached] [5/8] COPY src/js/services/foo /opt/app/src/js/services/foo
foo         ┊     ╎ [6/8] COPY src/js/lib /opt/app/src/js/lib
foo         ┊     ╎ 
foo         ┊     ╎ ERROR IN: [6/8] COPY src/js/lib /opt/app/src/js/lib
foo         ┊     ╎ 
foo         ┊     ╎ ERROR IN: [1/8] FROM docker.io/library/node:10@sha256:dabc15ad36a9e0a95862fbdf6ffdad439edc20aa27c7f10456644464e3fb5f08
foo         ┊     ╎ 
foo         ┊     ╎ ERROR IN: [6/8] COPY src/js/lib /opt/app/src/js/lib
foo         ┊ 
foo         ┊ ERROR: ImageBuild: failed to solve with frontend dockerfile.v0: failed to build LLB: failed to compute cache key: "/src/js/lib" not found: not found
foo         ┊ got unexpected error during build/deploy: ImageBuild: failed to solve with frontend dockerfile.v0: failed to build LLB: failed to compute cache key: "/src/js/lib" not found: not found
Build Failed: ImageBuild: failed to solve with frontend dockerfile.v0: failed to build LLB: failed to compute cache key: "/src/js/lib" not found: not found

[nicks]

@sourcec0de The important part of the error message is "/src/js/lib" not found

Docker is saying "you're trying to copy src/js/lib from the host machine, but i couldn't find src/js/lib in your docker context"

In this particular case, the docker_build call here has an only=src/js/services/foo, so Tilt does not include src/js/lib in your docker context.

Let me know if any of that is unclear.

As a side note, I feel like this error message is super-confusing, so people are googling it, and finding this unrelated issue. I filed an issue to make the error message better.