Argo-cd: Failing to deploy services due to "helm dependency build" failure

[zonnie]

Checklist:

• [x] I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• [x] I've included steps to reproduce the bug.
• [x] I've pasted the output of argocd version.

Describe the bug

Some Applications based on helm fail to deploy due to somekind of internal filesystem issue.

For example, one of the apps that are in Unknown states

Name:               redis-rule-hit-count-buffer-test-2
Project:            default
Server:             https://kubernetes.default.svc
Namespace:          test-2
URL:                https://35.232.222.64/applications/redis-rule-hit-count-buffer-test-2
Repo:               [email protected]:gc-org/gc-saas-prod.git
Target:             HEAD
Path:               prod/common/infra/redis
SyncWindow:         Sync Allowed
Sync Policy:        Automated (Prune)
Sync Status:        Unknown
Health Status:      Healthy

CONDITION        MESSAGE                                                                                                                                                                                                                                                  LAST TRANSITION
ComparisonError  rpc error: code = Unknown desc = Manifest generation error (cached): `helm dependency build` failed exit status 1: Error: unable to move current charts to tmp dir: link error: cannot rename charts to tmpcharts: rename charts tmpcharts: file exists  2020-12-22 10:14:56 +0200 IST

This doesn't eventually resolve itself, it's stays this way...

To Reproduce

I'm not sure how to reproduce, this happens from time to time and causes complete deadlock
My Chart.yaml

name: redis
version: 0.1.0
apiVersion: v2
dependencies:
  - name: redis
    version: 11.2.2
    repository: https://charts.bitnami.com/bitnami

My app-of-apps

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: redis-servers-test-2
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: default
  source:
    repoURL: [email protected]:gc-org/gc-saas-prod.git
    targetRevision: HEAD
    path: prod/cluster_1/customers/customer_2/redis
    helm:
      releaseName: redis-servers-test-2
  destination:
    server: https://kubernetes.default.svc
    namespace: test-2
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
      allowEmpty: true

My template

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: redis-dashboard-top-widgets-cache-test-2
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
  annotations:
    argocd.argoproj.io/sync-wave: "3"
spec:
  project: default
  source:
    repoURL: [email protected]:gc-org/gc-saas-prod.git
    targetRevision: HEAD
    path: prod/common/infra/redis
    helm:
      values: |
        redis:
          fullnameOverride: redis-dashboard-top-widgets-cache
          redisPort: 6413
          master:
            nodeSelector:
              cus: test-2
            service:redis-rule-hit-count-buffer-master
              port: 6413
          image:
            tag: 6.0.6
          cluster:
            enabled: false
          existingSecret: "redis"
          existingSecretPasswordKey: redis-password
      releaseName: redis-dashboard-top-widgets-cache
  destination:
    server: https://kubernetes.default.svc
    namespace: test-2
  syncPolicy:
    syncOptions:
    - CreateNamespace=true
    retry:
      limit: -1 # unlimited
      backoff:
        duration: 5s
        factor: 2
        maxDuration: 5m
    automated:
      prune: true
      selfHeal: true
      allowEmpty: true

Expected behavior

The Application should be deployed successfully

Screenshots

Version

argocd: v1.7.10+bcb05b0.dirty
  BuildDate: 2020-11-21T00:34:29Z
  GitCommit: bcb05b0c2e0f8006aa2d2abaf780e73c9e73c945
  GitTreeState: dirty
  GoVersion: go1.15.5
  Compiler: gc
  Platform: darwin/amd64
argocd-server: v1.8.1+c2547dc
  BuildDate: 2020-12-10T02:59:21Z
  GitCommit: c2547dca95437fdbb4d1e984b0592e6b9110d37f
  GitTreeState: clean
  GoVersion: go1.14.12
  Compiler: gc
  Platform: linux/amd64
  Ksonnet Version: v0.13.1
  Kustomize Version: v3.8.1 2020-07-16T00:58:46Z
  Helm Version: v3.4.1+gc4e7485
  Kubectl Version: v1.17.8

Logs

Logs from the argocd-application-controller

time="2020-12-22T08:14:58Z" level=info msg="Normalized app spec: {\"status\":{\"conditions\":[{\"lastTransitionTime\":\"2020-12-22T08:14:56Z\",\"message\":\"rpc error: code = Unknown desc = Manifest generation error (cached): `helm dependency build` failed exit status 1: Error: unable to move current charts to tmp dir: link error: cannot rename charts to tmpcharts: rename charts tmpcharts: file exists\",\"type\":\"ComparisonError\"}]}}" application=redis-rule-hit-count-buffer-test-2

Logs from argocd-repo-server

time="2020-12-22T08:40:44Z" level=error msg="finished unary call with code Unknown" error="Manifest generation error (cached): `helm dependency build` failed exit status 1: Error: unable to move current charts to tmp dir: link error: cannot rename charts to tmpcharts: rename charts tmpcharts: file exists" grpc.code=Unknown grpc.method=GenerateManifest grpc.request.deadline="2020-12-22T08:41:43Z" grpc.service=repository.RepoServerService grpc.start_time="2020-12-22T08:40:44Z" grpc.time_ms=474.433 span.kind=server system=grpc
time="2020-12-22T08:40:44Z" level=info msg="manifest cache hit: &ApplicationSource{RepoURL:[email protected]:gc-org/gc-saas-prod.git,Path:prod/common/infra/redis,TargetRevision:HEAD,Helm:&ApplicationSourceHelm{ValueFiles:[],Parameters:[]HelmParameter{},ReleaseName:,Values:redis:\n  redisPort: 6380\n  master:\n    service:\n      port: 6380\n  image:\n    tag: 6.0.6\n  nodeSelector:\n    cus: test-1\n  cluster:\n    enabled: false\n  existingSecret: \"redis\"\n  existingSecretPasswordKey: redis-password\n,FileParameters:[]HelmFileParameter{},Version:,},Kustomize:nil,Ksonnet:nil,Directory:nil,Plugin:nil,Chart:,}/cead2aa7818699b6c3ef04fc7e35390ee0fcbee0"

[lcostea]

Might be because of this: https://helm.sh/blog/new-location-stable-incubator-charts/ - the address for stable repo has changed.
@alexmt maybe it would be good to have a 1.7.12 with helm 2.17 included, that has the new address of stable updated in the binary, otherwise upgrade to 1.8 is mandatory

[JasP19]

I believe another fix for this may be to update your stable repo in the argocd-cm ConfigMap as follows:

data:
  helm.repositories: |
    - url: https://charts.helm.sh/stable
      name: stable

After this, refresh your Argo applications.

[lcostea]

@zonnie sorry, now I saw you already are on 1.8.1 server, so it is not that

[lcostea]

@JasP19 does it work for helm 2? It has the stable repo embedded the article above says you need to remove stable first

[JasP19]

@lcostea it worked for us with Helm 2 and ArgoCD v1.6.1. I believe this is similar to overriding the stable repo when using ArgoCD without internet access as in #1412.

[jessesuen]

Is there an example git location which can reproduce this easily? It's hard to fix this without being able to reproduce it.

[zonnie]

I'm not sure when this reproduces.
If you can explain what causes the symptom I might be able to try and reproduce.
The error is completely internal and means nothing to me - maybe more details can help me find a way.

[JasP19]

I believe another fix for this may be to update your stable repo in the argocd-cm ConfigMap as follows:

data:
  helm.repositories: |
    - url: https://charts.helm.sh/stable
      name: stable

After this, refresh your Argo applications.

@zonnie did you get a chance to test the solution I proposed above?

[zonnie]

No, I didn't - can u please explain briefly how this is related?

I'll check this solution anyway and report back

[JasP19]

Might be because of this: https://helm.sh/blog/new-location-stable-incubator-charts/ - the address for stable repo has changed.
@alexmt maybe it would be good to have a 1.7.12 with helm 2.17 included, that has the new address of stable updated in the binary, otherwise upgrade to 1.8 is mandatory

@zonnie this is related to the fact that Helm recently changed the location of the stable repo as mentioned by @lcostea, above. I believe the Helm 2 binary is now failing to resolve/access the old stable repo. By updating the ConfigMap, you override the stable repo to point at the new URL with the new location. We were having the same issue in our cluster. We made this change and noted that it solved the issue when we refreshed our apps.

[zonnie]

Might be because of this: https://helm.sh/blog/new-location-stable-incubator-charts/ - the address for stable repo has changed.

@alexmt maybe it would be good to have a 1.7.12 with helm 2.17 included, that has the new address of stable updated in the binary, otherwise upgrade to 1.8 is mandatory

@zonnie this is related to the fact that Helm recently changed the location of the stable repo as mentioned by @lcostea, above. I believe the Helm 2 binary is now failing to resolve/access the old stable repo. By updating the ConfigMap, you override the stable repo to point at the new URL with the new location. We were having the same issue in our cluster. We made this change and noted that it solved the issue when we refreshed our apps.

Thanks @JasP19 I will give it a shot as I can

[gzur]

Is 1.7.11 also affected by this?

[JasP19]

@gzur unfortunately I am not certain. If you are using Helm 2 and have suddenly started seeing apps in an Unkown state with a helm dependency build failure then it may be worth attempting the solution I proposed. It worked for us on ArgoCD v1.6.1 and it's only a minor change.

[gzur]

@gzur unfortunately I am not certain. If you are using Helm 2 and have suddenly started seeing apps in an Unkown state with a helm dependency build failure then it may be worth attempting the solution I proposed. It worked for us on ArgoCD v1.6.1 and it's only a minor change.

I'm running v1.7.6 as is - I'm going to give updating to 1.7.11 the old college try and report back.

[gzur]

I'm running v1.7.6 as is - I'm going to give updating to 1.7.11 the old college try and report back.

I was able to upgrade to 1.7.11 - but it had no effect.

I ended up applying the following configuration through a custom values.yaml file:

argocd:
  server:
    config:
      repositories: |
        [...]
        - type: helm
          name: stable
          url: https://charts.helm.sh/stable
        [...]

After this, I had to helm template [...] | kubectl apply [...] for it to kick in, since ArgoCD is non-functional.

[gzur]

Might be because of this: https://helm.sh/blog/new-location-stable-incubator-charts/ - the address for stable repo has changed.

I believe the error described in this issue is unrelated to the Helm stable repository migration. I experienced that first hand, and the error message produced looks like this:

helm2 dependency build` failed exit status 1: 
Error: open /tmp/helm769321715/repository/cache/stable-index.yaml: no such file or directory

Which I successfully addressed with a configuration change described above: https://github.com/argoproj/argo-cd/issues/5107#issuecomment-751783856

The error described in the issue:
rename charts tmpcharts: file exists
is some helm-related issue (that I actually remember seeing before, but I can't remember what caused it)