Flux: Invalid error message manifest for tag testgomake missing in repository metallb/controller

[runningman84]

Describe the bug
Flux reports this error:

ts=2019-09-03T12:39:55.436504901Z caller=repocachemanager.go:223 component=warmer canonical_name=index.docker.io/metallb/controller auth={map[]} warn="manifest for tag testgomake missing in repository metallb/controller" impact="flux will fail to auto-release workloads with matching images, ask the repository administrator to fix the inconsistency"

According to the metallb project, the tags are fine:
https://github.com/danderson/metallb/issues/473

To Reproduce
Steps to reproduce the behaviour:

• install flux
• install metallb helm chart

Additional context
Add any other context about the problem here, e.g

• Flux version: 1.14.0
• Helm Operator version: na
• Kubernetes version: 1.14
• Git provider: github
• Container registry provider: docker.io

[hiddeco]

It is quite likely that the DockerHub UI filters out the image because the manifest is corrupt, while it is still accessible (and listed as a tag) for Docker API calls.

Removing the tag through the Docker API or overwriting the tag with a non corrupt manifest will probably resolve the issue. This is not something we can do.

[bootc]

Given that these broken tags/repos are often beyond the control of the operators of Flux, is there anything that can be done in terms of Flux simply ignoring the broken tags and carrying on anyway?

[foot]

Does fluxd hit up the https://hub.docker.com/v2/repositories/metallb/controller/tags?page_size=100 api or is it another one?

[foot]

TOKEN=`curl "https://auth.docker.io/token?service=registry.docker.io&scope=repository:metallb/controller:pull" | jq .token | tr -d \"`
curl  -H "Authorization: Bearer $TOKEN" https://registry-1.docker.io/v2/metallb/controller/tags/list

And testgomake is indeed there in the response:

Response

{"name":"metallb/controller","tags":["0.8.0-rc1-amd64","0.8.0-rc1-arm","0.8.0-rc1-arm64","0.8.0-rc1-ppc64le","0.8.0-rc1-s390x","bug-114-amd64","bug-114-arm","bug-114-arm64","bug-114-ppc64le","bug-114-s390x","bug-114","bug-137-amd64","bug-137-arm","bug-137-arm64","bug-137-ppc64le","bug-137-s390x","bug-137","bug-250-amd64","bug-250-arm","bug-250-arm64","bug-250-ppc64le","bug-250-s390x","bug-250","bug-52-amd64","bug-52-arm","bug-52-arm64","bug-52-ppc64le","bug-52-s390x","bug-52","circleci-test-tag","latest","leadership-amd64","leadership-arm","leadership-arm64","leadership-ppc64le","leadership-s390x","leadership","live-website-amd64","live-website-arm","live-website-arm64","live-website-ppc64le","live-website-s390x","live-website","main-amd64","main-arm","main-arm64","main-ppc64le","main-s390x","main","master-amd64","master-arm","master-arm64","master-election-amd64","master-election-arm","master-election-arm64","master-election-ppc64le","master-election-s390x","master-election","master-ppc64le","master-s390x","master","restart-leader-amd64","restart-leader-arm","restart-leader-arm64","restart-leader-ppc64le","restart-leader-s390x","restart-leader","testgomake","update-versions-amd64","update-versions-arm","update-versions-arm64","update-versions-ppc64le","update-versions-s390x","update-versions","v0.3-amd64","v0.3-arm","v0.3-arm64","v0.3-ppc64le","v0.3-s390x","v0.3.0-amd64","v0.3.0-arm","v0.3.0-arm64","v0.3.0-ppc64le","v0.3.0-s390x","v0.3.0","v0.3.1-amd64","v0.3.1-arm","v0.3.1-arm64","v0.3.1-ppc64le","v0.3.1-s390x","v0.3.1","v0.3","v0.4-amd64","v0.4-arm","v0.4-arm64","v0.4-ppc64le","v0.4-s390x","v0.4.0-amd64","v0.4.0-arm","v0.4.0-arm64","v0.4.0-ppc64le","v0.4.0-s390x","v0.4.0","v0.4.1-amd64","v0.4.1-arm","v0.4.1-arm64","v0.4.1-ppc64le","v0.4.1-s390x","v0.4.1","v0.4.2-amd64","v0.4.2-arm","v0.4.2-arm64","v0.4.2-ppc64le","v0.4.2-s390x","v0.4.2","v0.4.3-amd64","v0.4.3-arm","v0.4.3-arm64","v0.4.3-ppc64le","v0.4.3-s390x","v0.4.3","v0.4.4-amd64","v0.4.4-arm","v0.4.4-arm64","v0.4.4-ppc64le","v0.4.4-s390x","v0.4.4","v0.4.5-amd64","v0.4.5-arm","v0.4.5-arm64","v0.4.5-ppc64le","v0.4.5-s390x","v0.4.5","v0.4.6-amd64","v0.4.6-arm","v0.4.6-arm64","v0.4.6-ppc64le","v0.4.6-s390x","v0.4.6","v0.4","v0.5-amd64","v0.5-arm","v0.5-arm64","v0.5-ppc64le","v0.5-s390x","v0.5.0-amd64","v0.5.0-arm","v0.5.0-arm64","v0.5.0-ppc64le","v0.5.0-s390x","v0.5.0","v0.5","v0.6-amd64","v0.6-arm","v0.6-arm64","v0.6-ppc64le","v0.6-s390x","v0.6.0-amd64","v0.6.0-arm","v0.6.0-arm64","v0.6.0-ppc64le","v0.6.0-s390x","v0.6.0","v0.6.1-amd64","v0.6.1-arm","v0.6.1-arm64","v0.6.1-ppc64le","v0.6.1-s390x","v0.6.1","v0.6.2-amd64","v0.6.2-arm","v0.6.2-arm64","v0.6.2-ppc64le","v0.6.2-s390x","v0.6.2","v0.6","v0.7-amd64","v0.7-arm","v0.7-arm64","v0.7-ppc64le","v0.7-s390x","v0.7.0-amd64","v0.7.0-arm","v0.7.0-arm64","v0.7.0-ppc64le","v0.7.0-s390x","v0.7.0","v0.7.1-amd64","v0.7.1-arm","v0.7.1-arm64","v0.7.1-ppc64le","v0.7.1-s390x","v0.7.1","v0.7.2-amd64","v0.7.2-arm","v0.7.2-arm64","v0.7.2-ppc64le","v0.7.2-s390x","v0.7.2","v0.7.3-amd64","v0.7.3-arm","v0.7.3-arm64","v0.7.3-ppc64le","v0.7.3-s390x","v0.7.3","v0.7","v0.8-amd64","v0.8-arm","v0.8-arm64","v0.8-ppc64le","v0.8-s390x","v0.8.0-amd64","v0.8.0-arm","v0.8.0-arm64","v0.8.0-ppc64le","v0.8.0-s390x","v0.8.0","v0.8.1-amd64","v0.8.1-arm","v0.8.1-arm64","v0.8.1-ppc64le","v0.8.1-s390x","v0.8.1","v0.8.2-amd64","v0.8.2-arm","v0.8.2-arm64","v0.8.2-ppc64le","v0.8.2-s390x","v0.8.2","v0.8.3-amd64","v0.8.3-arm","v0.8.3-arm64","v0.8.3-ppc64le","v0.8.3-s390x","v0.8.3","v0.8"]}

[2opremio]

That's the tag, but not the metadata associated to the tag, let me try to gather what specific api we are using for that

[foot]

That's the tag, but not the metadata associated to the tag

Ah, I see.

curl  -H "Authorization: Bearer $TOKEN" https://registry-1.docker.io/v2/metallb/controller/manifests/testgomake

Gives
{"errors":[{"code":"MANIFEST_UNKNOWN","message":"manifest unknown","detail":{"Name":"metallb/controller","Revision":"sha256:f87bf4a1f0846b226886089890cea78d651e97ba284edc26a018ed88d72d2d89"}}]}

[2opremio]

Exactly

[runningman84]

Could you give me a final verdict here, is that something that metallb or flux has to fix?

[2opremio]

@runningman84 The problem is that the registry is corrupted. It contains a tag with no metadata. As a result, Flux disables any operations involving that tag (e.g. automatic upgrades in which that tag is a candidate). Otherwise, Flux will operate normally.

Flux is conservative and prefers not to do anything since ignoring the corrupted tag (e.g. during a transient error) could lead to upgrading to the wrong tag.

[foot]

I had a quick look around but couldn't figure out how to easily remove a tag from a docker registry.

It would be handy to have some steps to provide metallb / other maintainers on how to resolve this.

[foot]

Known bad manifests reported by users:

• metallb/controller:testgomake
• drone/drone:1.0.0-rc.6

[hiddeco]

@foot the best way to remove a tag that is no longer bound to any manifests is to overwrite the tag with a new (unique) image, and subsequently deleting this image.

[demisx]

I also constantly see this issue with sorintlab/stolon. I guess it can be ignored if I am not automating release of those images:

ts=2020-01-07T19:54:55.26575477Z caller=repocachemanager.go:223 component=warmer canonical_name=index.docker.io/sorintlab/stolon auth={map[]} warn="manifest for tag master missing in repository sorintlab/stolon" impact="flux will fail to auto-release workloads with matching images, ask the repository administrator to fix the inconsistency"

[rafaribe]

Is there some argument that can be specified in order for flux to stop outputting this warnings? Maybe ignore this warnings for specific image repos?

[marshallford]

I haven't validated this myself but Flux 1.18.0 added a --registry-disable-scanning flag, perhaps that or the other registry related flags would prevent the errors (assuming you aren't using registry scanning to commit back).

[2opremio]

We are discussing options here https://github.com/fluxcd/flux/issues/2516#issuecomment-584705523

A new issue will be created, please comment there.