Helmfile: Confusing error message on chart verification error

[andrewnazarov]

I'm facing the following issue. I defined a bunch of external services in my helmfile. Then I tried to sync everything. Doing helm sync I got

Upgrading stable/cert-manager
Upgrading gitlab/gitlab-runner
Upgrading stable/minio
Upgrading stable/prometheus-operator
Upgrading stable/chartmuseum
Error: failed to download "gitlab/gitlab-runner" (hint: running helm repo update may help)

Error: failed to download "stable/prometheus-operator" (hint: running helm repo update may help)

Error: failed to download "stable/cert-manager" (hint: running helm repo update may help)

Error: failed to download "stable/chartmuseum" (hint: running helm repo update may help)

Error: failed to download "stable/minio" (hint: running helm repo update may help)

err: release "runner" in "helmfile.yaml" failed: exit status 1
err: release "prom" in "helmfile.yaml" failed: exit status 1
err: release "cert-manager" in "helmfile.yaml" failed: exit status 1
err: release "chartmuseum" in "helmfile.yaml" failed: exit status 1
err: release "s3" in "helmfile.yaml" failed: exit status 1
exit status 1

What it's all about? In docs it's stated that helmfile sync updates chart repositories and dependencies under the hood.

Btw, I have another helmfile.yaml located in a different folder, it has a mix of internal (with local charts) and external services. helmfile sync works ok inside this folder.

helmfile version v0.43.2

[mumoshu]

Hey! Thanks for the report.

Would you mind rerunning it with helmfile --log-level=debug and share me
the result?
Perhaps that helps investigating the underlying issue.
2019年2月6日(水) 20:03 Andrew Nazarov [e-mail]:

I'm facing the following issue. I defined a bunch of external services in
my helmfile. Then I tried to sync everything. Doing helm sync I got

Upgrading stable/cert-manager
Upgrading gitlab/gitlab-runner
Upgrading stable/minio
Upgrading stable/prometheus-operator
Upgrading stable/chartmuseum
Error: failed to download "gitlab/gitlab-runner" (hint: running helm repo
update may help)

Error: failed to download "stable/prometheus-operator" (hint: running helm
repo update may help)

Error: failed to download "stable/cert-manager" (hint: running helm repo
update may help)

Error: failed to download "stable/chartmuseum" (hint: running helm repo
update may help)

Error: failed to download "stable/minio" (hint: running helm repo update
may help)

err: release "runner" in "helmfile.yaml" failed: exit status 1
err: release "prom" in "helmfile.yaml" failed: exit status 1
err: release "cert-manager" in "helmfile.yaml" failed: exit status 1
err: release "chartmuseum" in "helmfile.yaml" failed: exit status 1
err: release "s3" in "helmfile.yaml" failed: exit status 1
exit status 1

What it's all about? In docs it's stated that helmfile sync updates chart
repositories and dependencies under the hood.

Btw, I have another helmfile.yaml located in a different folder, it has a
mix of internal (with local charts) and external services. helmfile sync
works ok inside this folder.

helmfile version v0.43.2

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/roboll/helmfile/issues/467, or mute the thread
https://github.com/notifications/unsubscribe-auth/AABV-b_D_ATloI6l0wsYZVF-jxhWfujhks5vKraOgaJpZM4alABX
.

[andrewnazarov]

Sure. Here it is (sensitive info is removed):

processing file "helmfile.yaml" in directory "."
second-pass rendering result of "helmfile.yaml":
 0: repositories:
 1:   - name: gitlab
 2:     url: https://charts.gitlab.io
 3:   - name: stable
 4:     url: https://kubernetes-charts.storage.googleapis.com
 5: 
 6: helmDefaults:           
 7:   tillerNamespace: gitlab-managed-apps
 8:   kubeContext: gke_xxxx-XXXX_europe-west3-a_cluster-xxxx
 9:   verify: true
10:   wait: true
11:   timeout: 600
12:   recreatePods: false
13:   force: false
14: 
15: releases:
16:   - name: runner
17:     namespace: gitlab-managed-apps
18:     chart: gitlab/gitlab-runner
19:     version: ~0.1.45
20:     values:
21:     - "./gitlab-runner/gitlab-runner-values.yml"
22: 
23:   - name: prom
24:     namespace: monitoring
25:     chart: stable/prometheus-operator
26:     version: ~2.1.1
27:     values:
28:     - "./prometheus/values.yml"
29: 
30:   - name: s3
31:     namespace: gitlab-managed-apps
32:     chart: stable/minio
33:     version: ~2.4.1
34:     values:
35:     - "./minio/minio-values.yml"
36: 
37:   - name: cert-manager
38:     namespace: default
39:     chart: stable/cert-manager
40:     version: ~0.5.0
41:     set:
42:       - name: rbac.create
43:         value: false
44: 
45:   - name: chartmuseum
46:     namespace: gitlab-managed-apps
47:     chart: stable/chartmuseum
48:     version: ~1.9.0
49:     set:
50:       - name: env.open.STORAGE
51:         value: local
52:       - name: persistence.enabled
53:         value: true
54:       - name: persistence.accessMode
55:         value: ReadWriteOnce
56:       - name: persistence.size
57:         value: 10Gi

Adding repo gitlab https://charts.gitlab.io
exec: helm repo add gitlab https://charts.gitlab.io
"gitlab" has been added to your repositories

Adding repo stable https://kubernetes-charts.storage.googleapis.com
exec: helm repo add stable https://kubernetes-charts.storage.googleapis.com
"stable" has been added to your repositories

Updating repo
exec: helm repo update
Hang tight while we grab the latest from your chart repositories...
...Skip local chart repository
...Successfully got an update from the "gitlab" chart repository
...Successfully got an update from the "stable" chart repository
Update Complete. ⎈ Happy Helming!⎈ 

worker 1/5 started
worker 5/5 started
worker 3/5 started
worker 2/5 started
worker 4/5 started
worker 4/5 finished
worker 2/5 finished
successfully generated the value file at /Users/andrewnazarov/Repo/xxxx-4/infra/init/prometheus/values.yml. produced:
coreDns:
  enabled: false

kubeDns:
  enabled: true

alertmanager:
  alertmanagerSpec:
    storage:
      volumeClaimTemplate:
        spec:
          accessModes: ["ReadWriteOnce"]
          resources:
            requests:
              storage: 10Gi

prometheus:
  prometheusSpec:
    storage:
      volumeClaimTemplate:
        spec:
          accessModes: ["ReadWriteOnce"]
          resources:
            requests:
              storage: 10Gi

grafana:
  ingress:
    enabled: true
    annotations:
      kubernetes.io/ingress.class: "nginx"
      certmanager.k8s.io/cluster-issuer: letsencrypt-prod
    hosts:
      - grafana.test.itc-engineering.com
    tls:
      - secretName: grafana-myingress-cert
        hosts:
          - grafana.test.itc-engineering.com
  persistence:
    enabled: true
    accessModes: ["ReadWriteOnce"]
    size: 10Gi
successfully generated the value file at /Users/andrewnazarov/Repo/xxxx/infra/init/gitlab-runner/gitlab-runner-values.yml. produced:
# This file only contains subset of available configuration, see https://gitlab.com/charts/gitlab-runner/blob/master/values.yaml


## The GitLab Server URL (with protocol) that want to register the runner against
## ref: https://docs.gitlab.com/runner/commands/README.html#gitlab-runner-register
##
gitlabUrl: https://gitlab.itc-engineering.com

## The Registration Token for adding new Runners to the GitLab Server. This must
## be retreived from your GitLab Instance.
## ref: https://docs.gitlab.com/ce/ci/runners/README.html#creating-and-registering-a-runner
##
runnerRegistrationToken: "XXXXXXXXX"

## Unregister all runners before termination
##
## Updating the runner's chart version or configuration will cause the runner container
## to be terminated and created again. This may cause your Gitlab instance to reference
## non-existant runners. Un-registering the runner before termination mitigates this issue.
## ref: https://docs.gitlab.com/runner/commands/README.html#gitlab-runner-unregister
##
unregisterRunners: false

## Configure the maximum number of concurrent jobs
## ref: https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-global-section
##
concurrent: 3

## Defines in seconds how often to check GitLab for a new builds
## ref: https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-global-section
##
checkInterval: 15

## Configuration for the Pods that that the runner launches for each new job
##
runners:

  ## Run all containers with the privileged flag enabled
  ## This will allow the docker:dind image to run if you need to run Docker
  ## commands. Please read the docs before turning this on:
  ## ref: https://docs.gitlab.com/runner/executors/kubernetes.html#using-docker-dind
  ##
  privileged: true

  ## Distributed runners caching
  ## ref: https://gitlab.com/gitlab-org/gitlab-runner/blob/master/docs/configuration/autoscale.md#distributed-runners-caching
  ##
  ## Create a secret 's3access' containing 'accesskey' & 'secretkey'
  ## ref: https://aws.amazon.com/blogs/security/wheres-my-secret-access-key/
  ##
  ## $ kubectl create secret generic s3access --\
  ##   --from-literal=accesskey="YourAccessKey" \
  ##   --from-literal=secretkey="YourSecretKey"
  ## ref: https://kubernetes.io/docs/concepts/configuration/secret/
  ##
  cache:
    cacheType: s3
    s3ServerAddress: s3-minio:9000
    s3BucketName: cache
    s3CacheInsecure: true
    cacheShared: true

  ## Build Container specific configuration
  ##
  builds: 
    memoryLimit: 10Gi
    cpuRequests: 1000m
    memoryRequests: 8Gi

  ## Service Container specific configuration
  ##
  services: {}
    # cpuLimit: 200m
    # memoryLimit: 256Mi
    # cpuRequests: 100m
    # memoryRequests: 128Mi

  ## Helper Container specific configuration
  ##
  helpers: {}
    # cpuLimit: 200m
    # memoryLimit: 256Mi
    # cpuRequests: 100m
    # memoryRequests: 128Mi
    # image: gitlab/gitlab-runner-helper:x86_64-latest


worker 5/5 finished
successfully generated the value file at /Users/andrewnazarov/Repo/xxxx-4/infra/init/minio/minio-values.yml. produced:
# This file only contains subset of available configuration, see https://github.com/kubernetes/charts/blob/master/stable/minio/values.yaml

## Set default accesskey, secretkey
accessKey: "XXXXXXX"
secretKey: "XXXXXXX"

## Create a bucket after minio install
##
defaultBucket:
  enabled: true
  ## If enabled, must be a string with length > 0
  name: cache
  ## Can be one of none|download|upload|public
  policy: none
  ## Purge if bucket exists already
  purge: false

worker 3/5 finished
worker 1/5 finished
worker 5/5 started
worker 3/5 started
worker 4/5 started
Upgrading stable/cert-manager
Upgrading stable/prometheus-operator
Upgrading stable/chartmuseum
exec: helm upgrade --install --reset-values cert-manager stable/cert-manager --version ~0.5.0 --verify --wait --timeout 600 --namespace default --set rbac.create=false --tiller-namespace=gitlab-managed-apps --kube-context=gke_xxxx-XXXXXX_europe-west3-a_cluster-xxxx
exec: helm upgrade --install --reset-values prom stable/prometheus-operator --version ~2.1.1 --verify --wait --timeout 600 --namespace monitoring --values /var/folders/pl/k4wqlgfj3cjf6b2p7qblq1200000gn/T/values637418716 --tiller-namespace=gitlab-managed-apps --kube-context=gke_xxxx-XXXXXX_europe-west3-a_cluster-xxxx
exec: helm upgrade --install --reset-values chartmuseum stable/chartmuseum --version ~1.9.0 --verify --wait --timeout 600 --namespace gitlab-managed-apps --set env.open.STORAGE=local --set persistence.enabled=true --set persistence.accessMode=ReadWriteOnce --set persistence.size=10Gi --tiller-namespace=gitlab-managed-apps --kube-context=gke_xxxx-XXXXXX_europe-west3-a_cluster-xxxx
worker 1/5 started
worker 2/5 started
Upgrading stable/minio
exec: helm upgrade --install --reset-values s3 stable/minio --version ~2.4.1 --verify --wait --timeout 600 --namespace gitlab-managed-apps --values /var/folders/pl/k4wqlgfj3cjf6b2p7qblq1200000gn/T/values565451374 --tiller-namespace=gitlab-managed-apps --kube-context=gke_xxxx-XXXXXX_europe-west3-a_cluster-xxxx
Upgrading gitlab/gitlab-runner
exec: helm upgrade --install --reset-values runner gitlab/gitlab-runner --version ~0.1.45 --verify --wait --timeout 600 --namespace gitlab-managed-apps --values /var/folders/pl/k4wqlgfj3cjf6b2p7qblq1200000gn/T/values120155275 --tiller-namespace=gitlab-managed-apps --kube-context=gke_xxxx-XXXXXX_europe-west3-a_cluster-xxxx
Error: failed to download "gitlab/gitlab-runner" (hint: running `helm repo update` may help)

worker 2/5 finished
Error: failed to download "stable/cert-manager" (hint: running `helm repo update` may help)

worker 3/5 finished
Error: failed to download "stable/minio" (hint: running `helm repo update` may help)

worker 1/5 finished
Error: failed to download "stable/prometheus-operator" (hint: running `helm repo update` may help)

worker 4/5 finished
Error: failed to download "stable/chartmuseum" (hint: running `helm repo update` may help)

worker 5/5 finished
err: release "runner" in "helmfile.yaml" failed: exit status 1
err: release "cert-manager" in "helmfile.yaml" failed: exit status 1
err: release "s3" in "helmfile.yaml" failed: exit status 1
err: release "prom" in "helmfile.yaml" failed: exit status 1
err: release "chartmuseum" in "helmfile.yaml" failed: exit status 1
exit status 1

[andrewnazarov]

My first assumption is probably it has something to do with the version of one of the releases. For example, the version of cert-manager is pinned not to the most recent.

[andrewnazarov]

My first assumption is probably it has something to do with the version of one of the releases. For example, the version of cert-manager is pinned not to the most recent.

No, I've just checked it out. Versions are not the case.

[andrewnazarov]

I found it. helmDefaults.verify=true causes the issue. I set it to false and helmfile sync started working.

By the way, it seems for some reason -i flag doesn't work with --log-level=debug. I'll double check that.

[mumoshu]

Thanks for your effort!

I found it. helmDefaults.verify=true causes the issue. I set it to false and helmfile sync started working.

Interesting. All it does should be just adding --verify to helm commands called by helmfile. Do you have any idea how it resulted in such errors...?

By the way, it seems for some reason -i flag doesn't work with --log-level=debug. I'll double check that.

How the whole set of flags are ordered in your command? I thought both --log-level and -i flags must precede before sync so that the command looks like helm --log-level=debug -i sync.

[andrewnazarov]

Interesting. All it does should be just adding --verify to helm commands called by helmfile. Do you have any idea how it resulted in such errors...?

Probably it has somethng to do with:

helm upgrade --dry-run --debug --install --reset-values chartmuseum stable/chartmuseum --version ~1.9.0 --verify --wait --timeout 600 --namespace gitlab-managed-apps --set env.open.STORAGE=local --set persistence.enabled=true --set persistence.accessMode=ReadWriteOnce --set persistence.size=10Gi --tiller-namespace=gitlab-managed-apps --kube-context=gke_iris-206888_europe-west3-a_cluster-iris
[debug] Created tunnel using local port: '60206'

[debug] SERVER: "127.0.0.1:60206"

Error: Failed to fetch provenance "https://kubernetes-charts.storage.googleapis.com/chartmuseum-1.9.0.tgz.prov"

In that case the error message Error: failed to download "stable/chartmuseum" (hint: running 'helm repo update' may help) is misleading.

[andrewnazarov]

How the whole set of flags are ordered in your command? I thought both --log-level and -i flags must precede before sync so that the command looks like helm --log-level=debug -i sync

The order of flags is right.

history | grep -E 'helmfile.*sync'
  752  helmfile --log-level=debug -i sync
  753  helmfile --log-level=debug -i sync
  754  helmfile -i sync
  756  helmfile -i sync
  757  helmfile -i sync
  761  helmfile -i sync
  767  helmfile -i sync
  768  helmfile -i sync
  777  helmfile -i sync
  778  helmfile -i sync
  779  helmfile --log-level=debug -i sync

But it seems that helmfile sync doesn't respect -i flag at all, even without --log-level flag. I don't want to create a new issue at the moment as there are too much noise and false -positives coming from me)).

[mumoshu]

@andrewnazarov Thanks a lot for your support!

Agreed that helmfile should somehow notice the user about the verification error.

Regarding -i, yes, sorry about that! We had a dedicated issue NOT to accept -i for unsupported helmfile commands #416.

[Xtigyro]

Yesterday with helmfile v0.130 I started seeing the same problem. Still need to validate it completely whether it's the exact same root cause.

[4c74356b41]

@mumoshu just hit this same error, started yesterday as well, same fix helped.

[mumoshu]

@Xtigyro @4c74356b41 On which charts did you start seeing the error? (I doubt if it has something to do with the deprecation of stable and incubator chart repositories

[4c74356b41]

with stable, yeah, but helm pull stable/grafana works fine, so it looks like you are correct here.