Skaffold: Wrong image is tagged on subsequent skaffold build

[demisx]

Expected behavior

Subsequent skaffold build executions upon unchanged codebase do not tag wrong images.

Actual behavior

The first time I run skaffold build -p dev command I get 3 images built and tagged as expected. When I run the same command for the second time, wrong image gets retagged with "init-db" tag. Later when my app pulls these images from local repo it fails to bootstrap because the wrong image is tagged with "init-db:..." tag now. This does not happen in v0.31

Information

• Skaffold version: 0.36
• Operating system: MacOs 10.4.6
• Contents of skaffold.yaml:

apiVersion: skaffold/v1beta13
kind: Config
profiles:
  - name: dev
    activation:
      - command: dev
    deploy:
      kubectl:
        manifests:
          - api/app/k8s/dev/api.k8s-config.yaml
          ... 
    build:
      artifacts:
        # Logstash data ingestor service
        - image: dl-logstash
          context: .
          sync:
            manual:
              - src: 'api/app/logstash/config/dev/pipeline/*'
                dest: pipeline
          docker:
            dockerfile: api/app/logstash/Dockerfile
            target: dev-env
       # Bootstrapping DB
        - image: init-db
          context: .
          docker:
            dockerfile: api/Dockerfile
            target: init-db
            buildArgs:
              DL_ORG_DEV_NAMESPACE: '{{.DL_ORG_DEV_NAMESPACE}}'
        # API service
        - image: dl-api.dev
          context: .
          sync:
            manual:
              - src: 'api/app/src/**/*.ts'
                dest: .
                strip: 'api'
              - src: 'api/app/src/**/*.json'
                dest: .
                strip: 'api'
          docker:
            dockerfile: api/Dockerfile
            target: dev-env
            buildArgs:
              # These vars are different per each developer and cannot be part of ConfigMap in git
              DL_ORG_DEV_NAMESPACE: '{{.DL_ORG_DEV_NAMESPACE}}'
              DL_ORG_AWS_ACCOUNT_ID: '{{.DL_ORG_AWS_ACCOUNT_ID}}'
              DL_ORG_AWS_IAM_APP_USER_ACCESS_KEY_ID: '{{.DL_ORG_AWS_IAM_APP_USER_ACCESS_KEY_ID}}'
              DL_ORG_AWS_IAM_APP_USER_SECRET_ACCESS_KEY: '{{.DL_ORG_AWS_IAM_APP_USER_SECRET_ACCESS_KEY}}'

Additional docs

Skaffold Run log:

double-skaffold-build-output.txt

[balopat]

Thank you for filing, @demisx!
This looks like a bug in artifact caching. Is it an option for you turn it off with --cache-artifacts=false?

[demisx]

Hi @balopat. Thank you for your usual prompt response! I thought caching was off by default. Looking at the skaffold build -h output:

--cache-artifacts=true: Set to true to enable caching of artifacts

it sounds like I need to pass this option to enable caching. Am I reading it wrong?

[demisx]

I've confirmed that this issue doesn't happen in v0.36 if I run with --cache-artifacts=false flag:

skaffold build --cache-artifacts=false -p dev

[balopat]

Yeah, we switched it on by default with v0.36 - thank you for verifying the behavior!

[demisx]

Got it. Maybe the help output should be updated globally as well to be a little more clear:

--cache-artifacts=false: Set to false to disable default caching of artifacts

If this is just a matter of search & replace (I'm not a Go programmer yet), I can do it myself and submit a PR. Let me know

[balopat]

I think that's a good idea to make the default value a bit clearer! ideally an automated way would be the best - i.e. the text should pick up the default value from the Flag and we should use that. It's a good first issue to tackle as a go programmer ;) It's a bit tricky though to do it uniformly because not all default values make sense to put in the usage, e.g.:

https://github.com/GoogleContainerTools/skaffold/blob/master/cmd/skaffold/app/cmd/flags.go#L232-L240

We'll have to think about that a bit - but maybe "" -> "not set" and we could prefix all Usage with something like fmt.Sprintf("(default value: %s) %s", Def, Usage) in a loop

[demisx]

Agreed. That would be nice. In the meantime, please see if this update is still better than the status quo GH-2711

[balopat]

Thanks for that, it's going in now!

[dgageot]

Here's the issue:

• two of the images are created from the same sources but built with a different target.
• The caching uses the digest of the input files as a key
• So, skaffold thinks the first and second images should be the same...

Let me fix that.