Helm-operator: helm3 with private repository 401 error - not finding the mounted repository.yml

[domg123]

Describe the bug
I'm using helm3, I have a helmRelease for a chart which is in an jfrog artifactory repository. I've set up the access via the mounted repository.yml. The chart release fails with a '"failed to upgrade chart for release' and a 401, indicating it can't authenticate.

If I jump into the operator container, and use helm3 I can't see the repos (with helm3 repo list), but I can with helm2 (repo list). A bit of searching indicates that helm3 (with its changes in config path), doesn't find the mounted repository.yml. If I move the repository.yaml manually to where it is looking (from /var/fluxd/helm/repository/repositories.yaml to /root/.config/helm/repositories.yaml) - the operator, and the chart work..

To Reproduce
Steps to reproduce the behaviour:
helm3 upgrade -i helm-operator fluxcd/helm-operator --namespace dp-system --set helm.versions=v3 --set configureRepositories.enable=true

Add following repository.yaml according to instructions ...

apiVersion: v1
repositories:

• caFile: ""
  cache: dphelm-index.yaml
  certFile: ""
  keyFile: ""
  name: dphelm
  password: ARedactedPassword
  url: https://dpbuild.jfrog.io/dpbuild/helm
  username: my.name

And then apply the following ...

apiVersion: helm.fluxcd.io/v1
kind: HelmRelease
metadata:
name: nginx-ingress-dphelm
namespace: default
spec:
releaseName: nginx-ingress-dphelm
chart:
repository: https://dpbuild.jfrog.io/dpbuild/helm
version: 1.28.0
name: nginx-ingress

Expected behavior
Chart is installed

Logs
After applying - I get the following in the logs

ts=2020-01-08T15:02:54.800480049Z caller=release.go:217 component=release release=stweb-global targetNamespace=dp-system resource=dp-system:helmrelease/stweb-global helmVersion=v3 error="Helm release failed" revision=316aeda090536d6ae6c605e17454434b4379e4d0 err="failed to upgrade chart for release [stweb-global]: an error occurred while uninstalling the release. original install error: no objects visited: uninstallation completed with 1 error(s): object not found, skipping delete"
ts=2020-01-08T15:02:55.052568141Z caller=release.go:142 component=release release=nginx-ingress targetNamespace=default resource=default:helmrelease/nginx-ingress helmVersion=v3 error="chart unavailable: looks like \"https://dpbuild.jfrog.io/dpbuild/helm\" is not a valid chart repository or cannot be reached: failed to fetch https://dpbuild.jfrog.io/dpbuild/helm/index.yaml : 401 Unauthorized"

Additional context
Add any other context about the problem here, e.g

• Helm operator version: 1.0.0-rc5
• Targeted Helm version: helm 3
• Kubernetes version: 1.14.9
• Container registry provider: artifactory cloud

As mentioned above, if manually (in the container, or via editing the deployment mount points), I move the repository mount to /root/.config/helm/repositories.yaml (and also the cache to /root/.cache/helm/repository/ ) it works. I tried manipulating the HELM_HOME and XDG_CONFIG_HOME so it looked in the right place, without luck.

[stefanprodan]

I think this is fixed by https://github.com/fluxcd/helm-operator/pull/171 please give it a try, the image is fluxcd/helm-operator-prerelease:master-48f6b7d8

[domg123]

Thanks - I gave it a try, but no luck, I get the same

"ts=2020-01-08T16:43:11.013643508Z caller=release.go:142 component=release release=nginx-ingress-dphelm targetNamespace=default resource=default:helmrelease/nginx-ingress-dphelm helmVersion=v3 error="chart unavailable: looks like \"https://dpbuild.jfrog.io/dpbuild/helm\" is not a valid chart repository or cannot be reached: failed to fetch https://dpbuild.jfrog.io/dpbuild/helm/index.yaml : 401 Unauthorized""

I understand the operator doesn't use the cli tool in the container, so the following is more to show that it is the right version, but ... here is the output when I jump into the container.

/home/flux # helm-operator --version
master-48f6b7d8
/home/flux # helm3 repo list
Error: no repositories to show
/home/flux # helm2 repo list
NAME    URL
stable  https://kubernetes-charts.storage.googleapis.com
dphelm  https://dpbuild.jfrog.io/dpbuild/helm
/home/flux #

[hiddeco]

I think the fix you are waiting for is #173 (but unfinished). What you can do while this PR is finalized is use the additionalArgs value offered by the chart to set the following flag: --helm-repository-import=v3:/var/fluxd/helm/repository/repositories.yaml. This will import the repositories configured by the chart (for Helm v2) to Helm v3.

[gaieges]

Running into the same issue with a helm v2 repo (may not be the same issue, but figured I'd report on the above solution).

Seems as though the short term approach of using the additionalArgs has an issue as well - it appears that there is no additionalArgs value to be set anymore (I can see in previous commits it exists in the helm-operator deploy, but doesn't seem like it's there any longer)

I presume you mean set it via the helm-operator chart like so?

helm upgrade -i helm-operator fluxcd/helm-operator \
    --namespace fluxcd \
    --set additionalArgs=--helm-repository-import=v2:/var/fluxd/helm/repository/repositories.yaml

[domg123]

@gaieges yes, I had the same issue with the additionalArgs workaround too - wasn't sure what I was missing, looks like it is gone.

[gaieges]

I'm not sure if this matters, but I just noticed that trying a regular http request (with basic auth of the provided credentials) sends me to a 307 redirect (using Azure ACR) on the actual url provided in the repositories.yaml file

Happy to help test any new branches / pr's

[hiddeco]

The above linked PR contains a fix to the chart that has been confirmed to be working for me w/re to imports. It would be appreciated if someone could test this PR with a repository requiring authentication _using the latest 1.0.0-rc6 release_.

[gaieges]

Ah! That does seem to get past the issue described above:

ts=2020-01-08T22:18:45.918271194Z caller=repository.go:43 component=helm version=v2 info="successfully got an update from the chart repository" url=https://repo.azurecr.io/helm/v1/repo

However it seems as though it fails on the next step of pulling down one of those packages with the same credentials:

ts=2020-01-08T22:18:48.996810844Z caller=release.go:142 component=release release=myapp-prod targetNamespace=prod resource=prod:helmrelease/myapp helmVersion=v2 error="chart unavailable: Failed to fetch https://repo.azurecr.io/helm/v1/repo/_blobs/myapp-0.1.7.tgz : 401 Unauthorized"

[hiddeco]

@gaieges can you confirm the following image resolves all issues for you? hiddeco/helm-operator:helm-repository-auth-f70f5717

I have tested it against a Helm repository that requires basic auth.

[domg123]

I can confirm that my use-case (original issue) is fixed in master - many thanks!

[gaieges]

That seems to get me passed the next issue as well! Thank you!

I have some other configuration issues I need to address but will do that on my end.

[ghost]

Describe the bug
I'm using helm3, I have a helmRelease for a chart which is in an jfrog artifactory repository. I've set up the access via the mounted repository.yml. The chart release fails with a '"failed to upgrade chart for release' and a 401, indicating it can't authenticate.

If I jump into the operator container, and use helm3 I can't see the repos (with helm3 repo list), but I can with helm2 (repo list). A bit of searching indicates that helm3 (with its changes in config path), doesn't find the mounted repository.yml. If I move the repository.yaml manually to where it is looking (from /var/fluxd/helm/repository/repositories.yaml to /root/.config/helm/repositories.yaml) - the operator, and the chart work..

To Reproduce
Steps to reproduce the behaviour:
helm3 upgrade -i helm-operator fluxcd/helm-operator
--namespace dp-system
--set helm.versions=v3
--set configureRepositories.enable=true

Add following repository.yaml according to instructions ...

apiVersion: v1
repositories:

• caFile: ""
  cache: dphelm-index.yaml
  certFile: ""
  keyFile: ""
  name: dphelm
  password: ARedactedPassword
  url: https://dpbuild.jfrog.io/dpbuild/helm
  username: my.name

And then apply the following ...

apiVersion: helm.fluxcd.io/v1
kind: HelmRelease
metadata:
name: nginx-ingress-dphelm
namespace: default
spec:
releaseName: nginx-ingress-dphelm
chart:
repository: https://dpbuild.jfrog.io/dpbuild/helm
version: 1.28.0
name: nginx-ingress

Expected behavior
Chart is installed

Logs
After applying - I get the following in the logs

ts=2020-01-08T15:02:54.800480049Z caller=release.go:217 component=release release=stweb-global targetNamespace=dp-system resource=dp-system:helmrelease/stweb-global helmVersion=v3 error="Helm release failed" revision=316aeda090536d6ae6c605e17454434b4379e4d0 err="failed to upgrade chart for release [stweb-global]: an error occurred while uninstalling the release. original install error: no objects visited: uninstallation completed with 1 error(s): object not found, skipping delete"
ts=2020-01-08T15:02:55.052568141Z caller=release.go:142 component=release release=nginx-ingress targetNamespace=default resource=default:helmrelease/nginx-ingress helmVersion=v3 error="chart unavailable: looks like "[https://dpbuild.jfrog.io/dpbuild/helm](https://dpbuild.jfrog.io/dpbuild/helm%5C)" is not a valid chart repository or cannot be reached: failed to fetch https://dpbuild.jfrog.io/dpbuild/helm/index.yaml : 401 Unauthorized"

Additional context
Add any other context about the problem here, e.g

• Helm operator version: 1.0.0-rc5
• Targeted Helm version: helm 3
• Kubernetes version: 1.14.9
• Container registry provider: artifactory cloud

As mentioned above, if manually (in the container, or via editing the deployment mount points), I move the repository mount to /root/.config/helm/repositories.yaml (and also the cache to /root/.cache/helm/repository/ ) it works. I tried manipulating the HELM_HOME and XDG_CONFIG_HOME so it looked in the right place, without luck.

Hi @domg123 @stefanprodan , I am kind of stuck on the same issue and want to understand as its says certFile and keyfile, how are we mounting these cert files on helm-operator container? or can we straightaway list the contents of certs as value?

certFile: ""
keyFile: ""

[kingdonb]

@surjsingh1 From the README in helm-operator, the default values when these settings are omitted:

tls.keyFile | tls.key | Name of the key file within the k8s secret
tls.certFile | tls.crt | Name of the certificate file within the k8s secret

The kubernetes secret creation doc describes the --from-file method:

https://kubernetes.io/docs/tasks/configmap-secret/managing-secret-using-kubectl/#create-a-secret eg:

kubectl create secret generic db-user-pass \
  --from-file=./tls.crt \
  --from-file=./tls.key