Tilt: Tilt syncing on Kind cluster produces an error and resorts to Docker image rebuild

[matthiasak]

Using KIND (http://kind.sigs.k8s.io) and Docker v18.09-ce on Ubuntu Cosmic.

Tilt file:

k8s_yaml(helm('helm'))

# web
docker_build(
    'docker.io/flyboy/web',
    '.',
    dockerfile='Dockerfile.web',
    live_update=[
        sync('.', '/usr/app')
    ]
)

k8s_resource('flyboy-web', port_forwards=9000)

Dockerfile for web app:

FROM node:10-alpine
ARG NODE_ENV=production
WORKDIR /usr/app
COPY package.json .
COPY package-lock.json .
RUN npm ci
COPY . .
CMD npm run serve

Modifying a file produces:

updating 1 files [/home/matthiasak/Code/flyboy-web/src/main.ts]                                                                                                                                                      
tar: lseek: Invalid seek                                                                                                                                                                                             
got unexpected error during build/deploy: command terminated with exit code 1                                                                                                                                        
STEP 1/3 — Building Dockerfile: [docker.io/flyboy/web]                                  

[nicks]

Thanks for the report! Let me try to repro this...

[matthiasak]

Awesome, thanks @nicks. For the record, even though I'm using Docker CE, these days they are using containerd under the hood. Could that be the issue?

[nicks]

ok, i've got a repro case! I think it might be a KIND bug.

[matthiasak]

:+1: thanks for digging in - can you help me report to KIND?

[nicks]

don't worry! We're on it. It might also be a Tilt bug!

The protocol for copying things to a kubernetes cluster is not well-specified, and figuring out who is doing the wrong thing can be tricky.

[nicks]

Repro steps so far (for other Tilt engineers)
1) Check out https://github.com/windmilleng/tilt-frontend-demo
2) Change the FROM image to alpine
3) Run tilt up --update-mode=exec
4) Edit src/App.js

This shows up on kind because kind implicitly turns on --update-mode=exec, but I was able to reproduce the problem with other clusters.

[jazzdan]

Hey @matthiasak I spent some time looking in to this this morning. I think this problem is due to an incompatibility between the way Tilt is creating the tar archive and the way that busybox tar, which ships with Alpine and is _not_ GNU or BSD tar, reads the tar archive. I'm not sure exactly what that incompatibility is yet.

You should be able to work around this by adding a RUN apk add tar to your Dockerfiles that you use with Tilt. This installs GNU tar. Let me know if that doesn't work for you!

In the meantime we're going to try to dig in to what exactly the incompatibility is between our code and busybox tar.

[jazzdan]

This article covers some of the differences between busybox and other tar implementations. Might be a good jumping off point.

[matthiasak]

@jazzdan someday, somehow I'm buying you a beer :beers:

[matthiasak]

Should I close this issue @jazzdan / @nicks?

[nicks]

Let's leave it open. I suspect that there's a small fix we can do to make the tarball compatible with busybox. (kubectl cp also uses tar to copy things, and the tarball it creates seems to work ok)