Helmfile: Helmfile doesn't work when no internet access

Created on 30 May 2018  Â·  26Comments  Â·  Source: roboll/helmfile

Hi there,

we are using helmfile to deploy our stack to kubernetes on premise.
I have downloaded charts offline and constructed:

context: kubernetes-admin@kubernetes                # kube-context (--kube-context)

releases:

  # Prometheus deployment
  - name: prom-helmf-ns-monitoring              # name of this release
    namespace: monitoring                       # target namespace
    chart: /opt/kubernetes/stable/prometheus                    # the chart being installed to create this release, referenced by `repository/chart` syntax
    values: ["values/values_prometheus_ns_monitoring.yaml"]
    set:                                        # values (--set)
      - name: rbac.create
        value: true
  # Grafana deployment
  - name: graf-helmf-ns-monitoring              # name of this release
    namespace: monitoring                       # target namespace
    chart: /opt/kubernetes/stable/grafana
    values: ["values/values_grafana_ns_monitoring.yaml"]

  # Controller pod (Nginx)
  - name: controller-pod-nginx                  # name of this release
    namespace: ingress-nginx                    # target namespace
    chart: /opt/kubernetes/stable/nginx-ingress                 # the chart being installed to create this release, referenced by `repository/chart` syntax
    values: ["values/values_nginx_ns_ingress-nginx.yaml"]
    set:                                        # values (--set)
      - name: rbac.create
        value: true

Everything is happening offline
My kubernetes cluster is up and running

kubectl create serviceaccount --namespace kube-system tiller
kubectl create clusterrolebinding tiller-cluster-rule --clusterrole=cluster-admin --serviceaccount=kube-system:tiller
# kubectl patch deploy --namespace kube-system tiller-deploy -p '{"spec":{"template":{"spec":{"serviceAccount":"tiller"}}}}'
helm init   --service-account tiller --skip-refresh

When I run helmfile:

 helmfile -f monitoring_deployment.yaml sync
exec: helm repo update --kube-context kubernetes-admin@kubernetes
Hang tight while we grab the latest from your chart repositories...
...Skip local chart repository
...Unable to get an update from the "stable" chart repository (https://kubernetes-charts.storage.googleapis.com):
        Get https://kubernetes-charts.storage.googleapis.com/index.yaml: dial tcp: lookup kubernetes-charts.storage.googleapis.com on 10.0.2.3:53: read udp 10.0.2.15:53788->10.0.2.3:53: i/o timeout
Update Complete. ⎈ Happy Helming!⎈
exec: helm dependency update /opt/kubernetes/stable/prometheus --kube-context kubernetes-admin@kubernetes
No requirements found in /opt/kubernetes/stable/prometheus/charts.
exec: helm dependency update /opt/kubernetes/stable/grafana --kube-context kubernetes-admin@kubernetes
No requirements found in /opt/kubernetes/stable/grafana/charts.
exec: helm dependency update /opt/kubernetes/stable/nginx-ingress --kube-context kubernetes-admin@kubernetes
No requirements found in /opt/kubernetes/stable/nginx-ingress/charts.
exec: helm upgrade --install --reset-values prom-helmf-ns-monitoring /opt/kubernetes/stable/prometheus --namespace monitoring --values /opt/kubernetes/monitoring/values/values_prometheus_ns_monitoring.yaml --set rbac.create=true --kube-context kubernetes-admin@kubernetes
exec: helm upgrade --install --reset-values controller-pod-nginx /opt/kubernetes/stable/nginx-ingress --namespace ingress-nginx --values /opt/kubernetes/monitoring/values/values_nginx_ns_ingress-nginx.yaml --set rbac.create=true --kube-context kubernetes-admin@kubernetes
exec: helm upgrade --install --reset-values graf-helmf-ns-monitoring /opt/kubernetes/stable/grafana --namespace monitoring --values /opt/kubernetes/monitoring/values/values_grafana_ns_monitoring.yaml --kube-context kubernetes-admin@kubernetes
Error: UPGRADE FAILED: "graf-helmf-ns-monitoring" has no deployed releases
Error: UPGRADE FAILED: "prom-helmf-ns-monitoring" has no deployed releases
Error: UPGRADE FAILED: "controller-pod-nginx" has no deployed releases
err: exit status 1
err: exit status 1
err: exit status 1

VERSIONS:

 helm version
Client: &version.Version{SemVer:"v2.9.1", GitCommit:"20adb27c7c5868466912eebdf6664e7390ebe710", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.9.1", GitCommit:"20adb27c7c5868466912eebdf6664e7390ebe710", GitTreeState:"clean"}

and

 helmfile --v
helmfile version v0.18.0
enhancement
Source
picture xjantoth
👍1

Most helpful comment

@mumoshu we have found perhaps workaround

1) remove these lines from: /root/.helm/repository/repositories.yaml
so it will look like this in the end

 cat  /root/.helm/repository/repositories.yaml
apiVersion: v1
generated: 2018-05-30T22:05:42.031075224Z
repositories:
- caFile: ""
  cache: /root/.helm/repository/cache/local-index.yaml
  certFile: ""
  keyFile: ""
  name: local
  password: ""
  url: http://127.0.0.1:8879/charts
  username: ""

2) always need to check heml list -a

  • because there might be some failing deployments - which in my case - there were exactly two
  • run helm delete <deployment_name> --purge

Note: I run
helmfile -f monitoring_deployment.yaml sync many times and many deployments were failed.

Well so the point is:

if you are OFFLINE and you either try you will fail:
1) helmfile -f monitoring_deployment.yaml delete
keep in mind that you upcoming
helmfile -f monitoring_deployment.yaml sync will FAIL
because because you have some entries helm list -a
You need to delete them manually!!!
2) or if you run helmfile -f monitoring_deployment.yaml sync for the very first time
offline and your deployment is going to fail from some reasons
you HAVE to clean up these failed deployments first helm list -a
before you going to execute again helmfile -f monitoring_deployment.yaml sync

Cleaning:

helm delete --purge <failed_deployment_name>
picture xjantoth on 31 May 2018
👍2

All 26 comments

@xjantoth Thanks for the report!

Just curious but how can you run a vanilla helm upgrade --install without internet connection? An example helm command would be appreciated.

If helm is able to handle that, it would be possible for helmfile to support it, too.

mumoshu picture mumoshu on 30 May 2018

Well actually in my original post I posted this:

kubectl create serviceaccount --namespace kube-system tiller
kubectl create clusterrolebinding tiller-cluster-rule --clusterrole=cluster-admin --serviceaccount=kube-system:tiller
# kubectl patch deploy --namespace kube-system tiller-deploy -p '{"spec":{"template":{"spec":{"serviceAccount":"tiller"}}}}'
helm init   --service-account tiller --skip-refresh

Please notice last command. You can actually initialize helm with tiller pod offline
Please advise :)

Then you should be able to use helm charts in offline mode - but I am having problem that helm/helmfile is trying to get something from internet :(

xjantoth picture xjantoth on 30 May 2018

@xjantoth Thanks for the info!

I have not tried that myself, so I'm still trying to understand if that's how helm is designed to work.

Are you sure you don't have failed releases as discussed in https://github.com/kubernetes/helm/issues/3353?

mumoshu picture mumoshu on 30 May 2018

Also: Do you actually use the same kubeconfig context kubernetes-admin@kubernetes to run kubectl, helm and helmfile? You seem to miss an explicit context in your kubectl create and helm init examples.

mumoshu picture mumoshu on 30 May 2018

Yes sure I just grepped kubernetes-admin@kubernetes in my:

  • ~/.kube/config file
  • my helmfile

I just want to convince helmfile that is does not need to go on INTERNET
to search for something - because it has everything in place locally.

xjantoth picture xjantoth on 30 May 2018

@mumoshu when it comes to heml version itself - I have installed helm in the following fashion:

wget   https://storage.googleapis.com/kubernetes-helm/helm-v2.9.1-linux-amd64.tar.gz
tar -xvzf helm-v2.9.1-linux-amd64.tar.gz
cp  linux-amd64/helm /bin/

So I do not think tahat I am using buggy version.

xjantoth picture xjantoth on 30 May 2018

@mumoshu take a look at this:

 helm install --name grafanax /opt/kubernetes/stable/grafana --values values/values_grafana_ns_monitoring.yaml --namespace monitoring

no errors and when I run kubectl get svc,pods -n monitoring

kubectl get svc,pod -n monitoring
NAME               TYPE       CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
service/grafanax   NodePort   11.113.119.216   <none>        80:30777/TCP   1m

NAME                            READY     STATUS    RESTARTS   AGE
pod/grafanax-7cff99d885-9kwn7   1/1       Running   0          1m

So it means like the pure command helm works as expected.

xjantoth picture xjantoth on 30 May 2018

@xjantoth Thanks. After downloading the grafana chart locally, the following helmfile worked, without internet connection:

charts:
  - name: grafana
    namespace: grafana
    chart: /Users/mumoshu/go/src/github.com/roboll/helmfile/examples/deployments/published/grafana

$ tar zxvf ~/.helm/cache/archive/grafana-1.10.0.tgz
$ pwd
/Users/mumoshu/go/src/github.com/roboll/helmfile/examples/deployments/published
$ helmfile -f charts.yaml sync
exec: helm repo update
Hang tight while we grab the latest from your chart repositories...
...Skip local chart repository
...Unable to get an update from the "stable" chart repository (https://kubernetes-charts.storage.googleapis.com):
    Get https://kubernetes-charts.storage.googleapis.com/index.yaml: dial tcp: lookup kubernetes-charts.storage.googleapis.com: no such host
Update Complete. ⎈ Happy Helming!⎈
exec: helm upgrade --install grafana /Users/munmoshu/go/src/github.com/roboll/helmfile/examples/deployments/published/grafana --namespace grafana
Release "grafana" has been upgraded. Happy Helming!
LAST DEPLOYED: Thu May 31 10:49:09 2018
NAMESPACE: grafana
STATUS: DEPLOYED
...

So, it seems to work locally :)

mumoshu picture mumoshu on 31 May 2018

Well @mumoshu that's great - but we knew about it :), The question is however the same as it was in the begining - it does not work for me OFFLINE - without internet. Could you please provide the commands how did you install helm and helmfile ?

xjantoth picture xjantoth on 31 May 2018

I installed it with brew install kubernetes-helm and from helmfiles
github releases respectively onto my macOS machine.
2018年5月31日(木) 13:21 Jan Toth notifications@github.com:

Well @mumoshu https://github.com/mumoshu that's great - but we knew
about it :), The question is however the same as it was in the begining -
it does not work for me OFFLINE - without internet. Could you please
provide the commands how did you install helm and helmfile ?

—
You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub
https://github.com/roboll/helmfile/issues/155#issuecomment-393397135,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AABV-W7rdZsNoPx5C477JFYlLtB3mnNhks5t32-3gaJpZM4UTOX3
.

mumoshu picture mumoshu on 31 May 2018

Btw, are you sure you don't have already failed helm releases in your
cluster? The helm issue linked from my prev comment, and helm list could
help.
2018年5月31日(木) 13:27 Yusuke KUOKA ykuoka@gmail.com:

I installed it with brew install kubernetes-helm and from helmfiles
github releases respectively onto my macOS machine.
2018年5月31日(木) 13:21 Jan Toth notifications@github.com:

Well @mumoshu https://github.com/mumoshu that's great - but we knew
about it :), The question is however the same as it was in the begining -
it does not work for me OFFLINE - without internet. Could you please
provide the commands how did you install helm and helmfile ?

—
You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub
https://github.com/roboll/helmfile/issues/155#issuecomment-393397135,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AABV-W7rdZsNoPx5C477JFYlLtB3mnNhks5t32-3gaJpZM4UTOX3
.

mumoshu picture mumoshu on 31 May 2018

Well I have downgraded helm version from 2.9.1 to 2.9.0

 helmfile -f monitoring_deployment.yaml sync
exec: helm repo update --kube-context kubernetes-admin@kubernetes
Hang tight while we grab the latest from your chart repositories...
...Skip local chart repository
...Unable to get an update from the "stable" chart repository (https://kubernetes-charts.storage.googleapis.com):
        Get https://kubernetes-charts.storage.googleapis.com/index.yaml: dial tcp: lookup kubernetes-charts.storage.googleapis.com on 10.0.2.3:53: read udp 10.0.2.15:39649->10.0.2.3:53: i/o timeout
Update Complete. ⎈ Happy Helming!⎈
exec: helm dependency update /opt/kubernetes/stable/prometheus --kube-context kubernetes-admin@kubernetes
No requirements found in /opt/kubernetes/stable/prometheus/charts.
exec: helm dependency update /opt/kubernetes/stable/grafana --kube-context kubernetes-admin@kubernetes
No requirements found in /opt/kubernetes/stable/grafana/charts.
exec: helm upgrade --install --reset-values prom-helmf-ns-monitoring /opt/kubernetes/stable/prometheus --namespace monitoring --values /opt/kubernetes/monitoring/values/values_prometheus_ns_monitoring.yaml --set rbac.create=true --kube-context kubernetes-admin@kubernetes
exec: helm upgrade --install --reset-values graf-helmf-ns-monitoring /opt/kubernetes/stable/grafana --namespace monitoring --values /opt/kubernetes/monitoring/values/values_grafana_ns_monitoring.yaml --kube-context kubernetes-admin@kubernetes
Error: UPGRADE FAILED: "graf-helmf-ns-monitoring" has no deployed releases
Error: UPGRADE FAILED: "prom-helmf-ns-monitoring" has no deployed releases
err: exit status 1
err: exit status 1

I think the problem is this part:

..Unable to get an update from the "stable" chart repository (https://kubernetes-charts.storage.googleapis.com):
        Get https://kubernetes-charts.storage.googleapis.com/index.yaml: dial tcp: lookup kubernetes-charts.storage.googleapis.com on 10.0.2.3:53: read udp 10.0.2.15:39649->10.0.2.3:53: i/o timeout
xjantoth picture xjantoth on 31 May 2018

well now I nocited something strange:

exec: helm upgrade --install --reset-values prom-helmf-ns-monitoring /opt/kubernetes/stable/prometheus --namespace monitoring --values /opt/kubernetes/moni

I do not see --reset-values in your output. Perhaps - that would be the problem. Why helmfile is putting that parameter to its command?

xjantoth picture xjantoth on 31 May 2018

@xjantoth I'm still unsure if that affects these upgrade failures:

Error: UPGRADE FAILED: "graf-helmf-ns-monitoring" has no deployed releases
Error: UPGRADE FAILED: "prom-helmf-ns-monitoring" has no deployed releases

But anyway, suppose you don't have a fancy http proxy, you need internet connection to run helm repo update. It should try to download repository indices at minimum, which requires internet connection.

Also, help dependency update may require internet connection, as it should fetch chart dependencies reading requirements.yml.

Can you write some golang? I guess you can fix that by introducing an another flag which skips helm repo update and helm dependency update. In other words, make this block conditional:

https://github.com/roboll/helmfile/blob/37f6ae8557e93f3d45be3989acf67b28e7265772/main.go#L195-L207

mumoshu picture mumoshu on 31 May 2018

Unfortunately, I can't write golang - I am sure that the issue with offline
helmfile usage is just because helmfile is trying to ask for some updates from
Internet - which is not needed at all because it literally has

  • all docker container
  • all helm charts

stored locally.

Look if helm binary itself is able to proceed deployment OFFLINE with no issues
as it has been proved

helm install --name grafanax /opt/kubernetes/stable/grafana --values values/values_grafana_ns_monitoring.yaml --namespace monitoring

it means that we need to find a way how to tell helmfile do the same.

Something like - please HEMFILE - do not check fro anything from Internet
because this SERVER has no access to internet.

xjantoth picture xjantoth on 31 May 2018

Thanks for the response and the explanation! I understood this as a feature request. Not sure when I can work on this, but I am open to accept PRs relevant to this feature.

mumoshu picture mumoshu on 31 May 2018

Adding an flag like helm sync --skip-dependency-update to skip helm repo update and helm dependency update would be good. Of course I'm open to suggestions.

mumoshu picture mumoshu on 31 May 2018

Well so now we would need to find someone who could help to write this simple code ? if I got your comment right?

xjantoth picture xjantoth on 31 May 2018

@mumoshu yes that what we need 100% positive.

xjantoth picture xjantoth on 31 May 2018

@xjantoth Unfortunately, yes!

Personally saying, I already have a plenty of feature requests discussed in this project and waiting to be implemented.

Also, this is more or less an open-source project with no financial sponsorship or no contracted developer. Even I can't force anyone to work on something specific.

But you can expect me to do my best to spend my spare time to maintain this project :) It is just that there's no guarantee.

mumoshu picture mumoshu on 31 May 2018

@mumoshu personally I'm very happy how this project is maintained. I am grateful for this amazing
piece of software you are working at. I probably need to lean golang to be able to implement these
hopefully simple feature to helmfile binary. Much appreciate your help!

xjantoth picture xjantoth on 31 May 2018

@mumoshu perhaps you could help me find some workaround for now:

I have removed few lines from: /root/.helm/repository/repositories.yaml
file and now it looks like this:

 cat  /root/.helm/repository/repositories.yaml
apiVersion: v1
generated: 2018-05-30T22:05:42.031075224Z
repositories:
- caFile: ""
  cache: /root/.helm/repository/cache/local-index.yaml
  certFile: ""
  keyFile: ""
  name: local
  password: ""
  url: http://127.0.0.1:8879/charts
  username: ""

Which seems to be a little improvement however, 

helmfile -f monitoring_deployment.yaml sync
exec: helm repo update --kube-context kubernetes-admin@kubernetes
Hang tight while we grab the latest from your chart repositories...
...Skip local chart repository
Update Complete. ⎈ Happy Helming!⎈
exec: helm dependency update /opt/kubernetes/stable/prometheus --kube-context kubernetes-admin@kubernetes
No requirements found in /opt/kubernetes/stable/prometheus/charts.
exec: helm dependency update /opt/kubernetes/stable/grafana --kube-context kubernetes-admin@kubernetes
No requirements found in /opt/kubernetes/stable/grafana/charts.
exec: helm upgrade --install --reset-values prom-helmf-ns-monitoring /opt/kubernetes/stable/prometheus --namespace monitoring --values /opt/kubernetes/monitoring/values/values_prometheus_ns_monitoring.yaml --kube-context kubernetes-admin@kubernetes
exec: helm upgrade --install --reset-values graf-helmf-ns-monitoring /opt/kubernetes/stable/grafana --namespace monitoring --values /opt/kubernetes/monitoring/values/values_grafana_ns_monitoring.yaml --kube-context kubernetes-admin@kubernetes
Error: UPGRADE FAILED: "graf-helmf-ns-monitoring" has no deployed releases
Error: UPGRADE FAILED: "prom-helmf-ns-monitoring" has no deployed releases
err: exit status 1
err: exit status 1

I am trying to figure out why do I get 

Error: UPGRADE FAILED: "graf-helmf-ns-monitoring" has no deployed releases
Error: UPGRADE FAILED: "prom-helmf-ns-monitoring" has no deployed releases
err: exit status 1
err: exit status 1
xjantoth picture xjantoth on 31 May 2018

@mumoshu we have found perhaps workaround

1) remove these lines from: /root/.helm/repository/repositories.yaml
so it will look like this in the end

 cat  /root/.helm/repository/repositories.yaml
apiVersion: v1
generated: 2018-05-30T22:05:42.031075224Z
repositories:
- caFile: ""
  cache: /root/.helm/repository/cache/local-index.yaml
  certFile: ""
  keyFile: ""
  name: local
  password: ""
  url: http://127.0.0.1:8879/charts
  username: ""

2) always need to check heml list -a

  • because there might be some failing deployments - which in my case - there were exactly two
  • run helm delete <deployment_name> --purge

Note: I run
helmfile -f monitoring_deployment.yaml sync many times and many deployments were failed.

Well so the point is:

if you are OFFLINE and you either try you will fail:
1) helmfile -f monitoring_deployment.yaml delete
keep in mind that you upcoming
helmfile -f monitoring_deployment.yaml sync will FAIL
because because you have some entries helm list -a
You need to delete them manually!!!
2) or if you run helmfile -f monitoring_deployment.yaml sync for the very first time
offline and your deployment is going to fail from some reasons
you HAVE to clean up these failed deployments first helm list -a
before you going to execute again helmfile -f monitoring_deployment.yaml sync

Cleaning:

helm delete --purge <failed_deployment_name>
xjantoth picture xjantoth on 31 May 2018
👍2

Hey. Thanks a lot for sharing your workaround!

I just got sometime to review pending issues like this, and noticed that you could use helmfile -f helmfile.yaml charts to do upgrade only, i.e. skip dependency updates.

mumoshu picture mumoshu on 22 Aug 2018

If that works for you, could this be a documentation issue?

mumoshu picture mumoshu on 22 Aug 2018

@mumoshu if you could document this workaround - it would be awesome :)

xjantoth picture xjantoth on 22 Aug 2018
👍1
Was this page helpful?
0 / 5 - 0 ratings

Related issues

klebediev picture klebediev  Â·  3Comments
ivandardi picture ivandardi  Â·  3Comments
GoldenMouse picture GoldenMouse  Â·  3Comments
pavdmyt picture pavdmyt  Â·  3Comments
willejs picture willejs  Â·  4Comments