Flux: Azure ACR Helm repository auth doesn't work

[timwebster9]

PR #1671 allows you to mount a repositories.yaml file inside the flux helm operator container in order to authenticate with private repositories. It appears to only 'sort of' work though.

After preparing and mounting my repositories file and deploy flux, I see this in the flux-helm-operator logs:

ts=2019-02-11T17:00:35.327615047Z caller=chartsync.go:343 component=chartsync info="chart download failed" releaseName=acr-java resource=acr-java:helmrelease/acr-java err="Looks like \"https://hmcts.azurecr.io/helm/v1/repo/\" is not a valid chart repository or cannot be reached: Failed to fetch https://hmcts.azurecr.io/helm/v1/repo/index.yaml : 401 Unauthorized"

However, I also see this in the same logs (edited for brevity):

ts=2019-02-11T16:40:56.961576439Z caller=chartsync.go:333 component=chartsync warning=...Successfully got an update from the \"hmcts\" chart repository\n..."

So it is able to update charts from the repository apparently.

Furthermore, if I kubectl exec into the flux-helm-operator pod and run helm install on the chart from the private repo, it works perfectly fine - the chart installs.

This is using an Azure ACR Helm repository. Open to any suggestions on how to troubleshoot this further if anyone has any...

[sbocz]

Are you reauthenticating with ACR every hour? You token you have mounted in your helm operator may be outdated. https://github.com/MicrosoftDocs/azure-docs/blob/master/articles/container-registry/container-registry-authentication.md

[timwebster9]

Hi @seanbocz

No, but:

• I logged in with a service principal instead of my user account (see this issue from here: https://github.com/MicrosoftDocs/azure-docs/issues/18288#issuecomment-437472556). I'm hoping this will help with staying logged in, but need to verify this after a while...
• If that was the problem, how would that explain the fact that the helm install worked from inside the pod?

[sbocz]

I'm guessing you were authenticated for an hour and it worked then. Maybe I'm not understanding the issue though. I'm trying to work through a similar situation with Helm on ACR (hence the original PR) so maybe when I get it figured out I can assist more.

[timwebster9]

I don't think it's the token expiring. From inside the helm operator pod, running helm install from the CLI actually works with the private ACR repo. I just tried it again. So the mounted credentials are working. But when Flux does it, it fails.

I think it's this bit that is failing: https://github.com/weaveworks/flux/blob/4586fb515ad9575c00981067012f1e0844d65094/integrations/helm/chartsync/download.go#L94

[timwebster9]

A couple of other things I've tried:

• I can confirm you can just do curl -L -u "username:password" https://repo.azurecr.io/helm/v1/repo/index.yaml and that will download the index.yaml file, where 'username' and 'password' are those of a service principal
• I also tried re-creating the secret for repositories.yaml using plaintext username and password instead of whatever format they were in previously. Not sure what format it was - the username was 00000000-0000-0000-0000-000000000000 and the password was a huge string. Either way, using plaintext instead didn't make any difference.
• as of now (3 hours later) I can still deploy helm charts using helm install from inside the flux-helm-operator pod, so the token expiry thing after one hour doesn't seem to be an issue (again, I'm using a service principal)

So it doesn't appear to be a problem with the credentials or even ACR at this point. It's difficult to troubleshoot further without more verbose debugging.

[timwebster9]

OK I got this working. Basically it was a string comparison problem and an issue with /.

Here's what I did to get this to work with ACR:

Create a service principal for ACR. See here for details around this.

Create a repositories.yaml file that looks like the one below, substituting your own URL and Service Principal credentials. Use plain text for username and password. Using the default format resulting from az acr helm repo add doesn't work. As this will end up in a secret, using SealedSecrets is recommended.

Also note the forward slash at the end of the URL. Make sure this is present.

apiVersion: v1
repositories:
- caFile: ""
  cache: myrepo-index.yaml
  certFile: ""
  keyFile: ""
  name: myrepo
  password: <password>
  url: https://myrepo.azurecr.io/helm/v1/repo/
  username: <username>
- caFile: ""
  cache: stable-index.yaml
  certFile: ""
  keyFile: ""
  name: stable
  password: ""
  url: https://kubernetes-charts.storage.googleapis.com
  username: ""

Take this file and mount it into the flux-helm-operator container using the process described here.

In your HelmRelease in your Flux config, use the same URL (including the forward slash) for your chart configuration. e.g.

spec:
  releaseName: my-release
  chart:
    repository: https://myrepo.azurecr.io/helm/v1/repo/
    name: chart-name
    version: 0.0.12

If the the forward slash is left out here, the repo bit will be left off by Flux and the chart will fail to download. Also this string needs to match exactly what is in repositories.yaml, so that's why it needs to be there as well.

So - not sure if there's a bug in here somewhere - maybe the code should be able to compare URLs a bit better (e.g. ignore trailing slash, etc).

[squaremo]

Using the default format resulting from az acr helm repo add doesn't work

Can you elaborate a bit on this -- the cache entries will be wrong (iirc), but is the URL wrong as well? (and is it wrong _only_ by a missing /?)

[squaremo]

Thanks for chasing the problem down, by the way, @timwebster9. Going on the comments it must have been infuriating to debug!

[timwebster9]

Using the default format resulting from az acr helm repo add doesn't work

Can you elaborate a bit on this -- the cache entries will be wrong (iirc), but is the URL wrong as well? (and is it wrong _only_ by a missing /?)

So there were 2 issues:

• The two URLs (the one in repositories.yaml and the one in the HelmRelease) didn't match - one of them had a trailing /, which caused the code that looks for a matching repo to not find the entry in repositories.yaml and therefore attempt authentication with no credentials. This was obviously 'user error', but seems like the kind of thing the code should be able to deal with.
• When the trailing / was missing from the URL in the HelmRelease, the chart download failed because it seemed to cut off the repo part of the URL. Instead it used a URL like this: https://myrepo.azurecr.io/helm/v1/_blobs/mychart-0.0.1.tgz. As soon as I added the slash back in it worked.

It wasn't too bad to debug once I figured out the build and was able to add some debugging to the code :-)

[timwebster9]

Sorry forgot to address your original question! If you use Azure CLI to log into ACR for helm like this:

az acr helm repo add

You will end up with an entry like this in your local repositories.yaml

- caFile: ""
  cache: /Users/timw/.helm/repository/cache/myrepo-index.yaml
  certFile: ""
  keyFile: ""
  name: myrepo
  password: 0I6HzNdeo2l1FW5pvQ5ZiOQpQg1XOIeBgFyA5WNHvMtCboKxLRjGBEk8sFDWk7IKoeK44lFSa0BuyNZr09y71Ohn19O1a27oTPq
  url: https://myrepo.azurecr.io/helm/v1/repo
  username: 00000000-0000-0000-0000-000000000000

Note the username and password. If you mount this file into the helm operator (and get all the URLs right like I described above), then it won't work. I had to use plaintext versions of the username and password, which were those for the Service Principal

[squaremo]

Got it, thanks @timwebster9. So I think we can say:

• the repositories.yaml you get from az acr helm repo add is not really usable -- you have to prepare one by hand, after creating a service principal; (-> #1733)
• Missing a trailing slash from the repository URL breaks things in non-obvious way, and could be better accounted for. (-> #1732)

[timwebster9]

Got it, thanks @timwebster9. So I think we can say:

• the repositories.yaml you get from az acr helm repo add is not really usable -- you have to prepare one by hand, after creating a service principal; (-> #1733)
• Missing a trailing slash from the repository URL breaks things in non-obvious way, and could be better accounted for. (-> #1732)

yeah that's a good summary 👍

[sbocz]

How are you getting a password for the service principal?

[timwebster9]

How are you getting a password for the service principal?

It will output it when you create it. See here: https://docs.microsoft.com/en-us/azure/container-registry/container-registry-auth-service-principal#create-a-service-principal

I'm not entirely clear what role assignments are required for Helm (this example is mostly concerned with image push/pull), but I think it will work as-is.

[sbocz]

Ah thanks. I did not know it was possible to authenticate a SP using a password.

[hiddeco]

@timwebster9 if you have time, can you confirm what I wrote down in #1744 is correct?

[timwebster9]

@hiddeco yup looks good 👍

[raycrawford]

FYI, I was experiencing the same problem with 2.0.60. Adding localhost to the 127.0.0.1 line of my host file fixed it. Thanks for the detailed explanation above which lead me to this fix!

[pavel-kurnosov]

I am trying to connect helm operator to azure acr and receive the same issue. I was following official docs, but it is doesn't work. Is there any way to better debug issue to see why am I receiving it?

[hiddeco]

@pavel-kurnosov what version of Helm are you using, what are the configured Helm operator arguments, and where did you mount the file?

[pavel-kurnosov]

@hiddeco Just right now I found the problem.
I did not specify an argument to help-operator with location to my repositories list:

• helm-repository-import=v3:/var/fluxd/helm/repository/repositories.yaml
  I think tutorial about using helm-operator need to be updated. Because it is saying how to use and add custom repositories, but not saying that this need to be specified manually.