Flux: err="decoded credential has wrong number of fields (expected 2, got 1)

[sunnoy]

err="decoded credential has wrong number of fields (expected 2, got 1)

--docker-config=config.sjon

config.json

{"auths":{"harbor.XXXXX.com":{"auth":"xxxxxxx"}}}

[sunnoy]

i want to know how the config to wirte ?

[squaremo]

It looks like you have run into #1596: credentials created with kubectl 1.13.1 (I think) are defective.
Depending on how you got the credentials, you may be able to fix the problem by recreating them with another version of kubectl.

[sunnoy]

but i create it with 1.14.7 ， it do not work

[daemonio]

Same problem here and I'm using k8s 1.16.1. Apparently this error is associated with another error "err=requesting tags: Get https://[url]: no basic auth credentials". I'm using a private repo (gitlab) and flux can't get any info from this private repo.

[squaremo]

@daemonio Do you imagePullSecrets (or docker config YAML) also only have an auth field, like the snippet given in the original report?

[marcus-sa]

Can't get it to work with any of the proposed "fixes" for ACR throughout all the issues as well :-1:

caller=loop.go:127 component=sync-loop event=refreshed url=[git repo url omitted] branch=dev HEAD=684cea532a27fdedb64f373bd2e1853ac6b6de30
caller=images.go:107 resource=kube-system:daemonset/azure-cni-networkmonitor err="decoded credential has wrong number of fields (expected 2, got 1)"

[hiddeco]

@marcus-sa your comment would be more helpful for us maintainers if it provided more information than what we already know. E.g. by providing the information @squaremo requested:

Do your imagePullSecrets (or docker config YAML) also only have an auth field, like the snippet given in the original report?

The sooner we have a confirmation of this, the sooner we can start reproducing the issue, the sooner a "fix" will be available.

[marcus-sa]

@hiddeco my bad.
It contains an additional field identitytoken which is generated when logging in to the registry using az acr login -n REGISTRY

{
  "auths": {
    "[repo name omitted].azurecr.io": {
      "auth": "[token omitted]",
      "identitytoken": "[token omitted]"
    }
  }
}

[squaremo]

@marcus-sa So just to confirm: the Azure tooling deliberately creates docker configs with that form, and it's not a quirk of some bit of rogue machinery (e.g., an old version of kubectl)?

[marcus-sa]

@squaremo nope, the Azure CLI uses that exactly for authorization.
It just populates the auths field in your $HOME/.docker/config.json

https://docs.microsoft.com/en-us/azure/container-registry/container-registry-authentication

You specifically attach an ACR to an AKS cluster, which means it'll have access to ACR without requiring any manual setup.

[marcus-sa]

Any updates on this?
Really want to use FluxCD, but this is quite blocking me (and probably several other future users from Azure)

[davidtom]

Any updates on this?
Really want to use FluxCD, but this is quite blocking me (and probably several other future users from Azure)

Just want to echo the ask for updates. Also trying to set up FluxCD on Azure and this is currently blocking me from moving forward with it.

[sunnoy]

I just use argcd ,it easy to use

[2opremio]

Reopening since this still seems to be a problem

[squaremo]

Examining the problem reported vs the code again, I see that the code will baulk if the auth field has a value but the value does not decode to username:password, which is a different problem to #1596.

@marcus-sa and @davidtom

Looks like there's a bunch of ways to get credentials for ACR, and the docs aren't clear about what each one actually does :-/

So may I ask: which method are you using (or hoping to use with flux) -- "Integrate with AKS", "Repository-scoped access token", or another? And, do are you able to describe what [whichever method] ends up doing -- does it put a file on the host, or does it give you a docker config.json to use as an image secret, or something else?

@marcus-sa When you create a config.json, does the auth field decode to a <username>:<password> string?

[stevegszabo]

I'm unable to get this working with ACR as Flux seems to expect a user:pass:

https://github.com/fluxcd/flux/blob/master/pkg/registry/credentials_test.go#L11

ACR seems to expect an auth token which is based on user:pass:

https://github.com/Azure/acr/blob/master/docs/Token-BasicAuth.md

[davidtom]

@squaremo apologies on the long response time, I missed this notification somehow.

I've been trying to use the "Integrate with AKS" method, I believe. Based on info from the troubleshooting document, I tried setting registry.acr.enabled=True in my Helm chart, but that didn't fix the issue.

I've since realized we probably don't need registry scanning, at least as we start to use Flux, so I've moved on from this for now. However, knowing its possible to get working would be great so we could consider further automating our deployments in the future.

[stevegszabo]

Using these helm options I was able to enable ACR scanning:

https://github.com/fluxcd/flux/blob/master/pkg/registry/azure.go#L25

registry.disableScanning=false
registry.acr.enabled=true

According to the logs flux was able to detect the new tag however it did not update the deployment resource:

ts=2020-03-20T12:10:30.842861743Z caller=warming.go:198 component=warmer info="refreshing image" image=myreponame.azurecr.io/namespace/app tag_count=10 to_update=1 of_which_refresh=0 of_which_missing=1

What is the significance of these fields:

tag_count=10 <= current tag count?
to_update=1
of_which_refresh=0
of_which_missing=1