Tilt: ERROR: ImageBuild: COPY failed

[kalistace]

Hi guys,

I have updated Tilt to the latest version (0.9.7) today and since then I'm experiencing this type of errors for all my docker images :

ERROR: ImageBuild: COPY failed: stat /var/lib/docker/tmp/docker-builder817293152/build/libs/app-1.0-SNAPSHOT.jar: no such file or directory

DockerFile looks like this :

FROM adoptopenjdk/openjdk11:jdk-11.0.2.9-slim

COPY ./build/libs/app-1.0-SNAPSHOT.jar /app.jar

CMD ["java", "-jar", "-XX:-OmitStackTraceInFastThrow", "/app.jar"]

It is my understand it's bad practice to have relative paths in DockerFile, I've tried using an absolute path but the same issue occurred.

I have downgraded Tilt to version 0.9.4 and it works again. As a work around, I have configured brew to not auto update tilt but as you can imagine I would prefer to stay up to date with upgrades.

DOCKER_BUILDKIT=0 docker build -t x . and DOCKER_BUILDKIT=1 docker build -t x . both work locally.

I'm running on MacOs Mojave 10.14.6.
here is the tilt doctor :

```Tilt: v0.9.7, built 2019-07-30

System: darwin-amd64

Docker

• Host: tcp://192.168.64.2:2376
• Version: 1.39

- Builder: 1

Kubernetes

• Env: minikube
• Context: minikube
• Cluster Name: minikube
• Namespace: default
• Container Runtime: docker

- Version: v1.15.0

Thanks for seeing the Tilt Doctor!
Please send this info along when filing bug reports. 💗
```
Thanks !

[maiamcc]

Yikes, thanks for the report. We'll get right on trying to repo this. (Also, 💯 for a _very_ thorough bug report!)

[maiamcc]

If you could push up a git repo with a minimal repro case, that would be a great help!

[maiamcc]

Also: can you share your Tiltfile? At a guess, that's the docker daemon failing to find the file in the docker context, so I'm curious to know what your docker_build call looks like.

[landism]

If you could push up a git repo with a minimal repro case, that would be a great help!

The error repros for me just given the Dockerfile above, so we're good on this!

[landism]

Hm, except the error also repros for me on Tilt 0.9.4, and even with docker build, outside of Tilt:

$ DOCKER_BUILDKIT=0 docker build .
Sending build context to Docker daemon  8.192kB
Step 1/3 : FROM adoptopenjdk/openjdk11:jdk-11.0.2.9-slim
 ---> 9a223081d1a1
Step 2/3 : COPY ./build/libs/app-1.0-SNAPSHOT.jar /app.jar
COPY failed: stat /var/lib/docker/tmp/docker-builder833448662/build/libs/app-1.0-SNAPSHOT.jar: no such file or directory

[landism]

This looks like it's a bad Dockerfile. There is no ./build in adoptopenjdk/openjdk11:jdk-11.0.2.9-slim, so the error is the behavior I'd expect:

$ docker run --entrypoint ls adoptopenjdk/openjdk11:jdk-11.0.2.9-slim
bin
boot
dev
etc
home
lib
lib64
media
mnt
opt
proc
root
run
sbin
srv
sys
tmp
usr
var

FWIW, here's the sha256 for the image that ended up on my host:

$ docker inspect adoptopenjdk/openjdk11:jdk-11.0.2.9-slim
[
    {
        "Id": "sha256:9a223081d1a1b0b6b0f135a911e42dacc2573f4e9abeefff5634e2cf42877fa9",

This leaves two mysteries:

1. why docker build works for you outside of tilt
2. why the 0.9.4 and 0.9.7 behavior differ

It's possible that figuring out either (1) or (2) will give us hints as to why the other is the case. I don't think I can do much for (1) from here with the info at hand.

For (2), git log v0.9.7...v0.9.4 --oneline has few commits that, afaict, could even theoretically affect the docker build behavior, and they're all ignore (.dockerignore, .tiltignore)-related. If you don't have either of those files specifying paths that might cause this difference, then I don't see any other leads there, and would suggest focusing on (1).

[kalistace]

Thanks for looking into this.

Here are a few pointers that might help us :

• My docker daemon point to minikube, can be reproduced by eval $(minikube docker-env) (if you have minikube)

• I am not trying to copy from openjdk11, the jar I'm copying is relative to the Dockerfile on my filesystem. It's packaged via Gradle.

• Before opening a ticket here, I Googled the same Error + mac related issues, I came across this https://github.com/docker/for-mac/issues/1922#issuecomment-321840224, which would make me think a change in .dockerignore might actually have an impact on the docker build behaviour, at least for mac users ? I'm far from being an expert on the topic though

[landism]

I am not trying to copy from openjdk11, the jar I'm copying is relative to the Dockerfile on my filesystem. It's packaged via Gradle.

whoops, I feel silly.

[nicks]

interesting! can you post your dockerignore file?

[kalistace]

I don't have one ah ! :)

I think I have narrowed it down a little bit more, it seems related to the only parameter from the docker_build function.

I have removed it for try-and-error debug and it worked.

Here is a snippet from my Tiltfile to illustrate my point :

Ends up with the error:

[docker_build(service, path_to(service), only='**/build/libs/*.jar') for service in services]

Doesn't end up with an error, but also if not fit for my purpose:

[docker_build(service, path_to(service)) for service in services]

If you're curious as to why I want this behaviour: Allows me to only rebuild manually when I want it, that is when I'm happy with my code change, I run gradle clean install, it builds my jar and deploys it to my local k8s.

Hope that helps.

[nicks]

thanks! I think I have a repro case. It has something to do with the only=

[nicks]

Short answer: @kalistace : you can work around this by changing the docker_build call to only='build/libs'

Long answer: Tilt does not currently support * globs in only=.... That's because Tilt implements only='foo' as a dockerignore with the contents:

**
!foo

If you have only='**/build/libs', Tilt creates a dockerignore like:

**
!**/build/libs

Unfortunately, Docker doesn't support this (https://github.com/moby/moby/issues/30018). If you'd like to try it out yourself, there's a repro case in this temporary repo: https://github.com/nicks/tilt-issue1982.

This broke because we were updating Tilt's dockerignore implementation to be more consistent with Docker's dockerignore implementation. For a bunch of reasons, it's super difficult to implement exclude-globs in a performant way (which is why I suspect this hasn't been fixed in Docker core)

There are a couple possible ways we could move forward here. Maybe we could fix this, and find a performant way to implement it. Alternatively, Tilt should error if you try to use globs in only= parameters.

I could see arguments for or against being consistent with Docker, even when it means replicating their bugs. @jazzdan what do you think?

[kalistace]

Thanks @nicks, I appreciate the long answer. It's quite informative !

I will use your fix, for my usage it patches the problem entirely.

Sounds like having a warning or error might be a good first step, you can always iterate on that. But of course, I will leave that to you :)

[nicks]

OK, talked a bit offline. For now, we're going to do the lazy solution and emit an error when Tilt sees a * glob in only=. We're hoping this will be fine for most use-cases. If people complain about the error, we can revisit this.

[nicks]

check added in https://github.com/windmilleng/tilt/pull/2017