Flux: Image with zero created timestamp

[slamdev]

Describe the bug

Flux fails to update a deployment with new image. Docker inspect shows valid created value: "Created": "2020-03-22T22:35:12.928157704Z"

Images that causes the problem are open for public access:

docker image inspect docker.io/slamdev/backend
docker image inspect docker.io/slamdev/frontend

To Reproduce

Flux is installed using this bash script: https://github.com/slamdev/multicluster-gitops/blob/master/hacks/install-flux.sh

Repo with all the configuration: https://github.com/slamdev/multicluster-gitops

Steps to reproduce the behaviour:

Expected behavior

Flux sync images created via docker build.

Logs

flux-7f557bf5d8-mgls6 flux ts=2020-03-22T22:40:26.501525942Z caller=images.go:106 component=sync-loop workload=dev:deployment/backend-app container=app repo=docker.io/slamdev/backend pattern=glob:* current=docker.io/slamdev/backend warning="image with zero created timestamp" current="docker.io/slamdev/backend (0001-01-01 00:00:00 +0000 UTC)" latest="docker.io/slamdev/backend:fd9085 (2020-03-22 22:34:51.819427717 +0000 UTC)" action="skip container"
flux-7f557bf5d8-mgls6 flux ts=2020-03-22T22:40:26.501872881Z caller=images.go:106 component=sync-loop workload=dev:deployment/frontend-app container=app repo=docker.io/slamdev/frontend pattern=glob:* current=docker.io/slamdev/frontend warning="image with zero created timestamp" current="docker.io/slamdev/frontend (0001-01-01 00:00:00 +0000 UTC)" latest="docker.io/slamdev/frontend:9f5d89 (2020-03-22 22:34:59.677510478 +0000 UTC)" action="skip container"

Additional context

• Flux version: 1.18.0
• Kubernetes version: v1.14.10-gke.24
• Git provider: github
• Container registry provider: dockerhub

[hiddeco]

If the workload currently does not have a tag set, can you please add this to see if it resolves the issue? Flux will not automate resources that do not have a tag defined (and are thus running on latest) as it only operates on immutable tags.

[slamdev]

@hiddeco thanks, that helped. my bad: didn't read the instruction properly.

Can you convert this issue to an improvement, so flux gives a proper message when the tag is "latest" or (as I found out experimentally) when the specified tag is not exist in the registry at all.

[dmarkey]

I have a similar issue.

1. I have a deployment that is tracking glob:foo-* that is on foo-123
2. foo-123 gets deleted on the image store. but the deployment is still running
3. Cluster gets rebooted(on a schedule) and is now ImagePullBackoff
4. foo-456 is built but flux refuses to roll out due to "image with zero created timestamp" even though the deployment is already broken.

I've been trying to establish why my image wouldnt have a timestamp but I can't figure out why.

Only way to remedy the above scenario is to override the image directly in the manifest.

[renanqts]

Same issue here. I'm trying to find a way at least to monitor this issue without check logs.

[mzupan]

So I ran into something sort of the same.

I had this

  values:
    image:
      repository: #####.dkr.ecr.us-east-1.amazonaws.com/worker
      tag: 10

As the current release running. I did a sync and pushed a new tag 11 out and was getting the timestamp issue

I just added quotes around the tag to force it as a string and then ran sync again. Then flux picked up and did the update from 10->11

[inful]

I have a similar issue.

1. I have a deployment that is tracking glob:foo-* that is on foo-123
2. foo-123 gets deleted on the image store. but the deployment is still running
3. Cluster gets rebooted(on a schedule) and is now ImagePullBackoff
4. foo-456 is built but flux refuses to roll out due to "image with zero created timestamp" even though the deployment is already broken.

I've been trying to establish why my image wouldnt have a timestamp but I can't figure out why.

Only way to remedy the above scenario is to override the image directly in the manifest.

I can confirme the observed behaviour running flux 1.19.0. I can see why you do not want to consider images with zero created timestamp for replacement of existing images - but replacing images without a timestamp with a "newer" timestamp (I guess these would be equivalent with removed or no-existing images) would to me be the expected behaviour.