Skaffold: `skaffold dev` always rebuilds container from base image

[expz]

Expected behavior

If nothing has changed in the repository or base image, then the build images have the same SHA digest and hence can use the cache. (If the current behavior is a feature and not a bug, then perhaps a skaffold dev --use-cache flag could be added?)

I have a huge container that takes ~15 minutes to build on my laptop with most of it spent installing python libraries, etc that do not need to be changed often. I need a way to use skaffold without having to wait 15 minutes to see, e.g. if my one line change at the end of a Dockerfile has a bug.

Actual behavior

If I run skaffold dev --no-prune twice in a row without changing anything, then the SHA digests of all but the base image change, so the cache is not used.

Information

• Skaffold version: v0.28.0
• Operating system: Ubuntu 18.04 LTS
• Contents of skaffold.yaml:

apiVersion: skaffold/v1beta9
kind: Config
build:
  artifacts:
  - image: skaffold-test/postgres
    context: .
    docker:
      dockerfile: Dockerfile
      cacheFrom:
      - postgres:9.6
deploy:
  kubectl:
    manifests:
      - postgres.yaml

Steps to reproduce the behavior

Put these in the same directory as skaffold.yaml.

postgres.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: postgres-datahub
  labels:
    name: postgres
spec:
  containers:
  - name: postgres
    image: skaffold-test/postgres
    ports:
    - containerPort: 5432
---
apiVersion: v1
kind: Service
metadata:
  name: postgres
spec:
  type: LoadBalancer
  ports:
  - port: 5432
  selector:
    name: postgres

init.sh:

#!/bin/sh

psql postgres -c 'CREATE DATABASE test'

Dockerfile:

FROM postgres:9.6

COPY init.sh /docker-entrypoint-initdb.d/

RUN chmod a+x /docker-entrypoint-initdb.d/init.sh

Now run skaffold dev --no-prune twice in a row from the same directory.

[balopat]

Can you try --cache-artifacts?

[expz]

Thanks for your quick reply! The --cache-artifacts flag does solve the problem as I stated it, but it only caches the final built image. Yes, if I run skaffold dev --cache-artifacts --no-prune twice in a row now, it does not rebuild the container.

However, if I modify that last line of the Dockerfile, then it rebuilds from the base image:

FROM postgres:9.6

COPY init.sh /docker-entrypoint-initdb.d/

RUN chmod a+x /docker-entrypoint-initdb.d/init.sh \
  && echo rebuilds from base image

Docker's behavior is to use the cache for the image generated by the COPY command. It only rebuilds the image for the last line.

The cause seems to be that the SHA digest of the intermediate images changes when it should stay the same, so the cache cannot recognize that it already built those images.

This matters, because if I change a line in my code that is COPYed at the end of my Dockerfile, skaffold will rebuild from the base image (~15 minutes), including building all third-party libraries, etc that have not changed since the last run.

[expz]

Perhaps someone could retag this as a feature request?

[balopat]

@expz I'm a little bit confused. Can you clarify a little bit more what the problem is?

1. "skaffold will rebuild from the base image (~15 minutes)" what takes 15 minutes? In your examples everything is pretty small - it takes a couple of seconds to copy init.sh.
2. skaffold uses docker under the hood - so skaffold dev does make use of docker layer caching. Can you provide explicit example where the SHA digest changes for the intermediate layers without filesystem changes?

[balopat]

Ignore my previous comment:

@nkubala managed to reproduce the SHA change in the layer.
I think we understand the issue now and we should have a look at this. Any more help / debugging and PRs are appreciated :)

[nkubala]

ok sorry for the flurry of responses, figured this one out. when you specify images in cache-from to docker, it will use those images for matching cached images/layers, but crucially it will only use those images, ignoring everything else in your daemon. so in your case, you specified

cacheFrom:
- postgres:9.6

in your skaffold.yaml, which was passed to docker normally, and told the docker daemon to ignore all other layers in your daemon when checking for cached layers, which in turn caused docker to rebuild your intermediate layer even though nothing had changed.

the fix is to simply remove the cacheFrom clause from your skaffold.yaml and let docker figure out all the caching on its own :)

[AndiDog]

@balopat can you please reopen? There is no solution. cacheFrom should not prevent Docker caching of other, locally existing layers.

[briandealwis]

This seems to be a docker/moby issue: moby/moby#32612 moby/moby#34715