Helmfile: How to deal with helm errors due to invalid/missing provenance

[apurvdeodhar]

Tool set versions

• kubectl

Client Version: v1.19.3
Server Version: v1.19.2

• helm

v3.5.3+g041ce5a

• helmfile

helmfile version v0.138.7

• minikube

minikube version: v1.14.2

Configuration files

• helmfile.yaml

---
repositories:
  - name: stable
    url: https://charts.helm.sh/stable
  - name: elastic
    url: https://helm.elastic.co
  - name: incubator
    url: https://charts.helm.sh/incubator

helmDefaults:
  kubeContext: 'minikube'
  cleanupOnFail: true
  verify: true
  timeout: 600
  createNamespace: true

commonLabels:
  name: elk-stack-test

releases:
  - name: logs-elasticsearch
    namespace: logs
    chart: elastic/elasticsearch
    version: 7.12.1
    missingFileHandler: Warn
#    values:
#      - elasticsearch/values.yaml
#    secrets:
#      - elasticsearch/secrets.yaml
    verify: true
    timeout: 600
    atomic: true
    cleanupOnFail: true
    disableValidation: true
    disableValidationOnInstall: true

  - name: logs-logstash
    namespace: logs
    chart: elastic/logstash
    version: 7.12.1
    missingFileHandler: Warn
#    values:
#      - logstash/values.yaml
#    secrets:
#      - logstash/secrets.yaml
    verify: true
    timeout: 600
    atomic: true
    cleanupOnFail: true
    disableValidation: true
    disableValidationOnInstall: true

  - name: logs-kibana
    namespace: logs
    chart: elastic/kibana
    version: 7.12.1
    missingFileHandler: Warn
#    values:
#      - kibana/values.yaml
#    secrets:
#      - kibana/secrets.yaml
    verify: true
    timeout: 600
    atomic: true
    cleanupOnFail: true
    disableValidation: true
    disableValidationOnInstall: true

  - name: logs-filebeat
    namespace: logs
    chart: elastic/filebeat
    version: 7.12.1
    missingFileHandler: Warn
#    values:
#      - filebeat/values.yaml
#    secrets:
#      - filebeat/secrets.yaml
    verify: true
    timeout: 600
    atomic: true
    cleanupOnFail: true
    disableValidation: true
    disableValidationOnInstall: true

How to reproduce

helmfile apply

Expected behaviour

Release "logs-filebeat" has been installed.

Release "logs-logstash" has been installed.

Release "logs-kibana" has been installed.

Release "logs-elasticsearch" has been installed.

Actual behaviour

COMBINED OUPUT:
  Release "logs-kibana" does not exist. Installing it now.
  Error: failed to download "elastic/kibana" at version "7.12.1" (hint: running `helm repo update` may help)
err 3: failed processing release logs-elasticsearch: command "/usr/local/bin/helm" exited with non-zero status:
.
.
.
.
.
COMBINED OUPUT:
  Release "logs-filebeat" does not exist. Installing it now.
  Error: failed to download "elastic/filebeat" at version "7.12.1" (hint: running `helm repo update` may help)
err 3: failed processing release logs-elasticsearch: command "/usr/local/bin/helm" exited with non-zero status:

Directory structure

ELK
├── environments
│   └── dev
│       ├── secrets.yaml
│       └── values.yaml
└── helmfile.yaml

My thoughts

Since I am using charts like filebeat, kibana, logstash from same repository elastic I think only because of this reason I need to helm repo update. Obviously this will work if I separate my mono helmfile.yaml and break it into chunks like mentioned here but IMO helmfile should have an option to update repo perhaps it could be a part of helmDefaults or releases

helmDefaults:
  kubeContext: 'minikube'
  cleanupOnFail: true
  updateRepo: true

releases:
  - name: logs-elasticsearch
    namespace: logs
    updateRepo: true

What I tried

• I followed the open issue here and used --force-update, --skip-deps but it gives me same error as mentioned above.

Looking forward for your reply @mumoshu !!!

[mumoshu]

@apurvdeodhar Hey! This seems to be due to that all the charts from "elastic" are broken in terms of provenance:

$ helm fetch elastic/kibana --version 7.12.1 --verify
Error: failed to fetch provenance "https://helm.elastic.co/helm/kibana/kibana-7.12.1.tgz.prov"

Try it without --verify like helm fetch elastic/kibana --version 7.12.1 and see it works.

That said, you should omit all the verify: true settings you've written in helmfile.yaml

[mumoshu]

@apurvdeodhar Also, it would be a good idea to submit a feature request to helm to improve this error happen on helm upgrade --install:

Error: failed to download "elastic/kibana" at version "7.12.1" (hint: running `helm repo update` may help)

This doesn't mention --verify and the fact that it failed due to an invalid/missing provenance.

[apurvdeodhar]

@apurvdeodhar Hey! This seems to be due to that all the charts from "elastic" are broken in terms of provenance:

$ helm fetch elastic/kibana --version 7.12.1 --verify
Error: failed to fetch provenance "https://helm.elastic.co/helm/kibana/kibana-7.12.1.tgz.prov"

Try it without --verify like helm fetch elastic/kibana --version 7.12.1 and see it works.

That said, you should omit all the verify: true you've written in hlmfile.yaml

Thanks @mumoshu removing verify: true worked.

[apurvdeodhar]

@apurvdeodhar Also, it would be a good idea to submit a feature request to helm to improve this error happen on helm upgrade --install:

Error: failed to download "elastic/kibana" at version "7.12.1" (hint: running `helm repo update` may help)

This doesn't mention --verify and the fact that it failed due to an invalid/missing provenance.

I was thinking the same. Will run some tests first and then file an issue. Thanks !!