Skaffold: Skaffold rebuild image for no obvious reason

[gaetansnl]

Expected behavior

I expect skaffold to sync files without a full rebuild

Actual behavior

File change trigger rebuild (sometimes) for no obvious reason. It works most of the time

Information

• Skaffold version: v0.34.1
• Operating system: Windows 10
• Contents of skaffold.yaml:

apiVersion: skaffold/v1beta13
kind: Config
profiles:
  - name: dev
    build:
      artifacts:
        - image: custom-registry/custom-repo/NAME_C-dev
          context: .
          sync:
            manual:
              - src: packages/NAME_C/**
                dest: ./
                strip: packages/NAME_C/
          docker:
            dockerfile: ./k8s/image/Dockerfile
            target: NAME_C-dev
        - image: custom-registry/custom-repo/NAME_B-dev
          context: .
          sync:
            manual:
              - src: packages/NAME_B/**
                dest: ./
                strip: packages/NAME_B/
          docker:
            dockerfile: ./k8s/image/Dockerfile
            target: NAME_B-dev
        - image: custom-registry/custom-repo/NAME_B-proxy
          context: ./packages/NAME_B/image
          docker:
            dockerfile: fileNAME_CCache.Dockerfile
        - image: custom-registry/custom-repo/client-dev
          context: .
          sync:
            manual:
              - src: packages/NAME_D/**
                dest: ./
                strip: packages/NAME_D/
          docker:
            dockerfile: ./k8s/image/Dockerfile
            target: client-dev
        - image: custom-registry/custom-repo/NAME_A-dev
          context: .
          sync:
            manual:
              - src: packages/NAME_A/**
                dest: ./
                strip: packages/NAME_A/
          docker:
            dockerfile: ./k8s/image/Dockerfile
            target: NAME_A-dev
      local:
        useBuildKit: true
    deploy:
      kustomize:
        path: ./k8s/env-specific/internal-dev

Logs

time="2019-07-31T18:08:01+02:00" level=debug msg="Change detected notify.Write: \"**********\\app\\app\\packages\\NAME_D\\.idea\\workspace.xml\""
time="2019-07-31T18:08:01+02:00" level=debug msg="Change detected notify.Write: \"**********\\app\\app\\packages\\NAME_D\\.idea\\workspace.xml\""
...
time="2019-07-31T18:08:02+02:00" level=debug msg="Skipping excluded path: packages\\NAME_D\\.idea"

I can't see any explanation for the rebuild in the logs

[corneliusweig]

@gaetansnl It looks like this is a file owned by IntelliJ (or another JetBrains product), and IDEs do a lot of backround work including writing to their config files. Do you also see spurious rebuilds when the IDE is not running? Also, you could try to add your .idea folder to the .dockerignore file.

Does that make a difference?

[gaetansnl]

@corneliusweig The .idea folder is already in dockerignore. And in the logs it says it's ignored. So it should not trigger rebuild. I see no other indications of why it's rebuild in the logs.

When I try to edit a file in the .idea folder, it does not trigger rebuild.
The rebuild seems very random.

Is there a way to know what trigger the rebuild ?

EDIT: I think it was working fine before v0.34, but I'm not sure it's related because I also changed a lot of other things

[corneliusweig]

@gaetansnl If it worked before 0.34, that would be important to know. Can you check? Also, can you share the contents of your Dockerfile and .dockerignore? Or ideally, you can provide a repo with a minimal example demonstrating the bug.

[gaetansnl]

@corneliusweig I just tried on 0.33, the problem is present. But I think I found when it happens.

If I edit one file I get this in the logs :

time="2019-08-01T09:47:03+02:00" level=info msg="Changed file path\\pricing.js does not match any sync pattern. Skipping sync"
time="2019-08-01T09:47:03+02:00" level=info msg="Changed file path\\pricing.js does not match any sync pattern. Skipping sync"
time="2019-08-01T09:47:03+02:00" level=info msg="Changed file path\\pricing.js does not match any sync pattern. Skipping sync"
Syncing 1 files for repo/name-dev:v0.1.12-518-g779f86f8-dirty@sha256:89c5249c42f459124d5b21d79cbd7ed2504b0827bab9ab37b3fda7acb18ab519
time="2019-08-01T09:47:03+02:00" level=info msg="Copying files: map[path\\pricing.js:[/path/pricing.js]] to repo/name-dev:v0.1.12-518
-g779f86f8-dirty@sha256:89c5249c42f459124d5b21d79cbd7ed2504b0827bab9ab37b3fda7acb18ab519"

Sync is skipped in 3 artifacts, and file is copied for 1 artifact. Skaffold sync the file, everything is OK.
But now if I quickly edit the file multiple time. The logs become

time="2019-08-01T09:47:03+02:00" level=info msg="Changed file path\\pricing.js does not match any sync pattern. Skipping sync"
time="2019-08-01T09:47:03+02:00" level=info msg="Changed file path\\pricing.js does not match any sync pattern. Skipping sync"
time="2019-08-01T09:47:03+02:00" level=info msg="Changed file path\\pricing.js does not match any sync pattern. Skipping sync"
..... Skaffold start rebuild

As you can see the last sync is missing, and it triggers a rebuild instead.

It seems to be caused by quickly editing the file. It rebuilds maybe because the container doesn't have the time to launch and the sync fails ?

[gaetansnl]

I updated to 0.35.0 and I watched file changes (with FolderChangesView on windows). All changing files are in dockerignore except the one I modify. And the rebuild seems very random anyway, so I don't think it's related to dockerignore.

How can I debug the reason of the rebuild ?

[corneliusweig]

Sorry, I don't have a good idea here. The only thing you could try is if it happens with other projects as well. And if not, find a minimal example that demonstrates the problem.

Have you tried running your project without the IDE? Does the problem manifest there as well?

[gaetansnl]

I reproduced ! Steps to reproduce :

• Replace your registry url/secret
• run skaffold dev
• Start editing node1/src and node2/src. You might need between 10 and 50 save before it triggers a rebuild

I think maybe skaffold rebuild because the two images share code but only some of it is synchronized. But it should not randomly rebuild anyway I think ?

hot-reload - Copie.zip

To understand what Im doing in the Dockerfile and skaffold.yaml :

We are trying to find a way to organize the dockerfile and file sync to optimize performance. Currently we are doing something like in my reproduction. We have a base image to install the full monorepo (It takes several minutes). And then we have 10 images using this base image, it allows us to cache the installation.
Then I sync each image with one folder (like in the reproduction). It works fine most of the time

But we also have libraries shared between images , and we can't find a way to rebuild the library and send it to the deployed image... We tried building these libraries locally without docker but it causes too much architecture and dependency problems.... So currently we are rebuilding the whole monorepo 😄 Luckily it's faster than installing the dependencies...

[balopat]

But we also have libraries shared between images , and we can't find a way to rebuild the library and send it to the deployed image... We tried building these libraries locally without docker but it causes too much architecture and dependency problems....

Interesting - we are thinking about supporting bit more advanced sync workflows, to support Spring Dev Tools - that would require copying built libraries into the running container - this might fit in there too. cc @priyawadhwa

[balopat]

Instead of manual repro:

while true; do echo "" >> node1/src/index.js; sleep 1; done

will start syncing nicely, after a while it rebuilds instead of sync.
For some reason on node2 the rebuild happens much faster.

[dgageot]

I can't reproduce it either on master or on v0.34.1. @balopat can you still reproduce?

[dgageot]

Wait! I've reproduced it by running while true; do echo "" >> node1/src/index.js; sleep 1; done and while true; do echo "" >> node2/src/index.js; sleep 1; done in two different windows.

[dgageot]

Even simpler, this triggers the bug:

echo "" | tee -a node{1,2}/src/index.js

[dgageot]

So, I continued to play with your sample. And I ended up realizing that skaffold works as intended. Here's why:

• Both node1 and node2 images depend on the same base image that copies both node1/* and node2/*
• So when a change is made to both, node1/index.js and node2/index.js, both artifacts need to be updated.
• The issue is that node1 only knows how to sync node1/** files and
• node2 only knows how to sync node2/** files
• Because both artifact are given changes that they can't sync, both are rebuilt

The only two ways to fix that is:

• Either stop building everything is a huge base image and use one Dockerfile per artifact that depends only on a subset of the source files
• Or, for each artifact, let it sync all files, even if they don't actually need them. This will prevent rebuilds.

[gaetansnl]

Yes thats what I thought but the rebuild isn't always triggered. It should always trigger rebuild, no ?

[dgageot]

It should indeed. Let me take a look

[dgageot]

On master, it does rebuild each time. Wasn't the case on v0.34.1, so something has been fixed.

[dgageot]

I'm going to close this issue as it now works as expected.

[mtaylor567]

I have this problem too, Is it possible to turn off image rebuilds for dev? I only need to sync the files

[AlaaHamoudah]

I am having this issue too, and I think the reason is the target attribute.

[AlaaHamoudah]

@dgageot what was the solution for this issue?

[nkubala]

@AlaaHamoudah it seems like this issue was closed because this is the intended behavior based on the sync configurations that skaffold supports. @mtaylor567 it is possible to turn off image rebuilds - you can use the --auto-build=false flag which tells skaffold to wait for input through the control API before performing a build. you can then trigger this build (if you want to) by gRPC or curl, something like curl -X POST http://localhost:50052/v1/execute -d "{'build': true}"

[AlaaHamoudah]

@nkubala thank you for ur response, I tried adding the args --auto-build=false when running skaffold dev but it seems it is not watching the changes at all.

[nkubala]

@AlaaHamoudah that flag will turn off automatic image building in skaffold when file changes are made. so it will feel like skaffold isn't watching, but in reality it's waiting for user input to trigger a build. if you still feel like you're seeing skaffold not watching for changes, feel free to open another issue!