Skaffold: `dev` not invalidating image cache on lib change to multi-project config

[stevesea]

Expected behavior

on code change, skaffold cache should have been invalidated and not used.

Actual behavior

cached docker images used and rebuild isn't triggered.

Information

• Skaffold version: 1.0.0
• Operating system: macos mojave 10.14.6

skaffold (via cloud code plugin) in continuous deploy mode is intermittently not picking up source changes and rebuilding the docker images.

multi-module POM that's using jib to build containers. project includes two micronaut microservices in different submodules. Update source code (refactored method name), and maven/jib wasn't used to re-build containers.

Same issue happens outside of Cloud Code if I just do skaffold dev from command line.

I can work-around it by deleting ~/.skaffold/cache , or running skaffold dev --cache-artifacts=false

[orkonano]

I have the same issue since I updated to the last version 1.0.0

[dgageot]

Hi @stevesea @orkonano do you have a small project that we can use to reproduce the issue?

[dgageot]

@stevesea @orkonano any news?

[orkonano]

Sorry. It's working again. I didn't know what happened.

El mié., 27 de noviembre de 2019 11:22, David Gageot <
[email protected]> escribió:

@stevesea https://github.com/stevesea @orkonano
https://github.com/orkonano any news?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/GoogleContainerTools/skaffold/issues/3210?email_source=notifications&email_token=AAXWSNLJCSRKLOF2BZTUE7TQVZ7D5A5CNFSM4JKPXW2KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEFJU5KA#issuecomment-559107752,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAXWSNOZYOLGE5IAZNFY4F3QVZ7D5ANCNFSM4JKPXW2A
.

[stevesea]

@dgageot My use-case has changed a bit since I initially created the ticket, but I think I'm still seeing similar issues. I _think_ this is all due to my using a library shared by multiple apps within a multi-project build.

created this test case: https://github.com/stevesea/test-micronaut-skaffold

There's a multi-project Gradle file and three subprojects -- myapp-a and myapp-b, and a library 'mylib' shared by each app

Skaffold builds and deploys myapp-a and myapp-b. On source changes to either myapp-a or myapp-b, skaffold rebuilds and redeploys.

If I change source code in mylib, neither service is rebuilt. To force the apps to rebuild/redeploy, I've got to go in and change source code in myapp-a/myapp-b.

For example... say I change a log statement in mylib (edit 1). To see the new log message, I've got to edit a file in myapp-a -- I add some whitespace to a file.

I change the log message in mylib again (edit 2). To attempt to get it to rebuild, I undo my previous whitespace change in myapp-a. skaffold "finds" the build in the cache.

However, it found its previous build of the myapp-a source that matched that hash... but, it's deployed the version of myapp-a that was built using mylib prior to either of my edits

In my Skaffold yaml, I've got

      artifacts:
        - &artifactA
          context: .
          image: myapp-a
          jib:
            project: myapp-a
            args:
              - -DskipTests

        - &artifactB
          context: .
          image: myapp-b
          jib:
            project: myapp-b
            args:
              - -DskipTests

I'd expect all files under the context root to be included in the hash computed for the cached docker image, and not only the files under the jib.project for that artifact.

To work-around the issue with this multi-project build, I can use skaffold dev --no-prune=false --cache-artifacts=false --port-forward=true

[ridakaddir]

have the same issue even when start dev using skaffold dev --no-prune=false --cache-artifacts=false

skaffold always deploy the first generated image by jib.

My skaffold.yml

apiVersion: skaffold/v1 kind: Config build: local: push: false artifacts: - image: skolov/spring-jib-member context: member-service jib: {} - image: skolov/spring-jib-campaign context: campaign-service jib: {} deploy: kubectl: manifests: - support/ingress.yml - member-service/spring-jib-member.yaml - campaign-service/spring-jib-campaign.yaml

[asegretto]

I agree with ridakaddir. "skaffold always deploy the first generated image by jib." When skaffold builds and pushes with jib to the latest tag in my local registry, the sha shows the image unchanged. If I do a manual mvn jib:build on the same code then the local registry shows the updated sha.

The only workaround that I have found is to do a minikube delete and then minikube start. This deletes everything including the local registry and its contents.

[tstromberg]

I'm working on the reproduction for this one. From the sound of it, we're not propagating dependencies upwards across artifacts.

[loosebazooka]

This might have something to do with the version of jib you're working with? Can you confirm this still occurs with 2.2.0?

[loosebazooka]

Also you can run ./gradlew jib:_skaffoldFilesV2 to inspect what jib is watching. Can you add the output of that to your issues?

[briandealwis]

While I was waiting for a slow build, I tried @stevesea's example with gradlew myapp-a:_jibSkaffoldFilesV2 and got the following:

{
  "build": [
    "/private/tmp/test-micronaut-skaffold/build.gradle",
    "/private/tmp/test-micronaut-skaffold/settings.gradle",
    "/private/tmp/test-micronaut-skaffold/gradle.properties",
    "/private/tmp/test-micronaut-skaffold/myapp-a/build.gradle",
    "/private/tmp/test-micronaut-skaffold/myapp-a/settings.gradle"
  ],
  "inputs": [
    "/private/tmp/test-micronaut-skaffold/myapp-a/src/main/resources",
    "/private/tmp/test-micronaut-skaffold/myapp-a/src/main/java"
  ],
  "ignore": []
}

So there is no mention of mylib. But myapp-a's build.gradle does have an implementation dependency on my lib:

dependencies {
    implementation project(':mylib')

@loosebazooka has a much better understanding of how this works than I though.

[loosebazooka]

Yeah, it's a bug in < 1.8.0, fixed in https://github.com/GoogleContainerTools/jib/pull/2188 (A change in gradle made our logic invalid resulting in dep projects not being picked up.)
I would suggest just upgrading to the latest 2.2.0 which comes with a whole host of fixes and new integrations with skaffold.

@briandealwis moving to 2.2.0 fixes the output of @stevesea's example

[briandealwis]

Ah I should have checked the version. Thanks @loosebazooka.

[asegretto]

I also confirmed upgrading to jib version 2.2.0 fixed my issue. skaffold is deploying the latest build with jib now. Thanks!

[briandealwis]

Thanks for confirming @asegretto