Helmfile: Cannot install istio 1.2.2 from helm chart with helmfile
I am trying to install the istio-1.2.2 istio helm chart in istio-1.2.2/install/kubernetes/helm/istio. I have copied both it and the istio-init chart to the root directory of my helmfile repo.
The relevant portion of my helmfile.yaml is:
- chart: ./istio-init
name: istio-init
namespace: istio-system
values:
- certmanager:
enabled: false
- chart: ./istio
name: istio
namespace: istio-system
values:
- certmanager:
enabled: false
grafana:
enabled: false
prometheus:
enabled: false
tracing:
enabled: false
kiali:
enabled: false
When I run helmfile apply with just the istio-init chart in my helmfile.yaml file, that chart installs succesfully. When I run helmfile apply with the istio chart added back to my helmfile.yaml file, I see the following error:
$ helmfile apply
Building dependency istio-init
No requirements found in istio-init/charts.
Building dependency istio
in ./helmfile.yaml: helm exited with status 1:
Error: no 'repository' field specified for dependency: "sidecarInjectorWebhook"
I can succesfully install it with helm install, however.
$ helm install ./istio --name istio --namespace istio-system --set certmanager.enabled=false --set grafana.enabled=false --set prometheus.enabled=false --set tracing.enabled=false --set kiali.enabled=false
<lots of output elided>
NOTES:
Thank you for installing istio.
Your release is named istio.
To get started running application with Istio, execute the following steps:
1. Label namespace that application object will be deployed to by the following command (take default namespace as an example)
$ kubectl label namespace default istio-injection=enabled
$ kubectl get namespace -L istio-injection
2. Deploy your applications
$ kubectl apply -f <your-application>.yaml
For more information on running Istio, visit:
https://istio.io/
What is the correct way to install the istio chart with helmfile? Is this a bug or misconfiguration on my part?
I am using the following versions running on macOS Mojave (10.14.6) against an AWS EKS 1.13 cluster:
$ helm version
Client: &version.Version{SemVer:"v2.14.3", GitCommit:"0e7f3b6637f7af8fcfddb3d2941fcc7cbebb0085", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.14.3", GitCommit:"0e7f3b6637f7af8fcfddb3d2941fcc7cbebb0085", GitTreeState:"clean"}
$ helmfile --version
helmfile version v0.80.2
Many thanks in advance!
mojochao
All 11 comments
@mojochao Hey!
I have not yet tried 1.2.2 but I do successfully installed 1.1.3 with helmfile.
I'd say that the order of installation is the key here. Could you try moving istio-init to a dedicated helmfile for ordering?
For instance your root helmfiel.yaml woud look like this:
helmfiles:
- istio-init.yaml
- istio.yaml
Where istio-init.yaml is:
repositories:
- name: istio.io
url: https://storage.googleapis.com/istio-release/releases/1.1.3/charts/
releases:
- name: istio-init
chart: istio.io/istio-init
version: 1.1.3
And istio.yaml is:
repositories:
- name: istio.io
url: https://storage.googleapis.com/istio-release/releases/1.1.3/charts/
releases:
- name: istio
chart: istio.io/istio
version: 1.1.3
namespace: istio-system
values:
# snip
This way you can ensure that helmfile installs istio-init and istio in this order.
mumoshu
on 7 Aug 2019
The only major hurdle you might face (which I work around) is that you need to wait for the Istio init CRD's to finish applying.
I'm doing this with a post sync script at the minute.
The script I have at present is:
#!/usr/bin/env bash
attempts=1
# Wait for the `istio-init` chart to finish and setup at least 3 of the CRD jobs. CHECK ON NEW ISTIO RELEASES.
while (( attempts <= 300 )); do
result=$(kubectl get pods -n "${1}" --selector job-name -o jsonpath="{range.items[?(@.status.phase == 'Succeeded')]}{.metadata.name}:{end}" | tr ':' "\n" | wc -l | awk '{ print $1 }')
echo "[A${attempts}] [istio-init] Jobs succeeded: ${result}"
if [[ ${result} -eq 3 ]]; then
echo "[A${attempts}] [istio-init] All jobs completed successfully. Exiting."
exit 0
fi
((++attempts))
sleep 1
done
echo "[A${attempts}] [istio-init] Timed out waiting for jobs to complete. Exiting."
exit 1
templates:
istio-init: &istio-init
name: istio-init
namespace: istio-system
installed: true
chart: istio/istio-init
version: 1.2.0
recreatePods: false
tillerNamespace: {{ requiredEnv "TILLER_NAMESPACE" }}
values:
- global:
imagePullPolicy: Always
tag: 1.2.0
hooks:
- events:
- postsync
showlogs: true
command: "/bin/bash"
args:
- "./wait-for-init.sh"
- "{{`{{.Release.Namespace}}`}}"
releases:
- <<: *istio-init
then of course I'm following the top level helmfile as @mumoshu suggested.
The only issue I'm having at the moment is that obviously you need kubectl installed where you run it (is there a possibility of bundling some extras into the docker image? jq and kubectl would be most welcome), and at present it doesn't check how many jobs actually exist to wait for; it's just hardcoded to 3.
naphta
on 7 Aug 2019
Wow, so does istio-init now contain something other than CRDs? That's annoying for sure 馃槩
Maybe it worth enhancing helmfile by adding a dedicated hook type for waiting until a job succeeds without needing kubectl and jq. Would you mind writing a feature request for that? I'm imagining something like this:
hooks:
- events:
- postsync
# not sure how showlogs should work for this type of hook...
#showlogs: true
type: waiter
# maybe we can provide a sensible default apiVersion per kind
#apiVersion: v1/apps
kind: Job
# do you wanna use go template in name?
name: job-name
is there a possibility of bundling some extras into the docker image? jq and kubectl would be most welcome
I'm unsure how much effort I can put to maintain those dependencies to be up-to-date, but I'm willing to accept any contributions to include them.
mumoshu
on 7 Aug 2019
@mumoshu It contains job pods which apply the CRDs, so sort of yes and no!
I did create an issue for adding in jq and kubectl but I'll create a PR with the changes, I'll see if I can pin kubectl to using latest stable on each build (which I guess is what helm is doing)
naphta
on 7 Aug 2019
Thanks for all the suggestions, but I'm still having the same problem. I've created two independent helmfiles:
$ cat istio-init.yaml
environments:
default:
values:
- istio_version: 1.2.4
helmDefaults:
wait: true
templates:
istio-init: &istio-init
chart: ./external/istio-{{ .Environment.Values.istio_version }}/install/kubernetes/helm/istio-init
name: istio-init
namespace: istio-system
installed: true
version: {{ .Environment.Values.istio_version }}
recreatePods: false
tillerNamespace: kube-system
values:
- global:
imagePullPolicy: Always
tag: {{ .Environment.Values.istio_version }}
hooks:
- events:
- postsync
showlogs: true
command: "/bin/bash"
args:
- "./wait-for-istio-init.sh"
- "{{`{{.Release.Namespace}}`}}"
releases:
- <<: *istio-init
and
$ cat istio.yaml
environments:
default:
values:
- istio_version: 1.2.4
helmDefaults:
wait: true
releases:
- chart: ./external/istio-{{ .Environment.Values.istio_version }}/install/kubernetes/helm/istio
version: {{ .Environment.Values.istio_version }}
name: istio
namespace: istio-system
values:
prometheus:
enabled: false
Also created the shell script to wait for init completion mentioned above by @naphta
#!/usr/bin/env bash
# Wait for the `istio-init` chart to finish and setup at least 3 of the CRD jobs. CHECK ON NEW ISTIO RELEASES.
attempts=1
while (( attempts <= 300 )); do
result=$(kubectl get pods -n "${1}" --selector job-name -o jsonpath="{range.items[?(@.status.phase == 'Succeeded')]}{.metadata.name}:{end}" | tr ':' "\n" | wc -l | awk '{ print $1 }')
echo "[A${attempts}] [istio-init] Jobs succeeded: ${result}"
if [[ ${result} -eq 3 ]]; then
echo "[A${attempts}] [istio-init] All jobs completed successfully. Exiting."
exit 0
fi
((++attempts))
sleep 1
done
echo "[A${attempts}] [istio-init] Timed out waiting for jobs to complete. Exiting."
exit 1
Modified my root helmfile.yaml with sub-helmfiles.
...
helmfiles:
- istio-init.yaml
- istio.yaml
...
Applying changes always results in:
位 helmfile apply
Building dependency external/istio-1.2.4/install/kubernetes/helm/istio-init
No requirements found in external/istio-1.2.4/install/kubernetes/helm/istio-init/charts.
Comparing istio-init external/istio-1.2.4/install/kubernetes/helm/istio-init
No affected releases
Building dependency external/istio-1.2.4/install/kubernetes/helm/istio
in ./helmfile.yaml: in .helmfiles[1]: in ./istio.yaml: helm exited with status 1:
Error: no 'repository' field specified for dependency: "sidecarInjectorWebhook"
Does anyone have a repo with a working helmfile of istio that I could try out?
Many thanks everyone!
mojochao
on 14 Aug 2019
As another data point, just using helm install of both istio-init and istio charts, with no pause in between, results in success:
$ helm install $ISTIO_CHARTS_DIR/istio-init --name istio-init --namespace istio-system && helm install $ISTIO_CHARTS_DIR/istio --name istio --namespace istio-system --set prometheus.enabled=false
NAME: istio-init
LAST DEPLOYED: Wed Aug 14 15:18:19 2019
NAMESPACE: istio-system
STATUS: DEPLOYED
RESOURCES:
==> v1/ClusterRole
NAME AGE
istio-init-istio-system 1s
==> v1/ClusterRoleBinding
NAME AGE
istio-init-admin-role-binding-istio-system 1s
==> v1/ConfigMap
NAME DATA AGE
istio-crd-10 1 1s
istio-crd-11 1 1s
istio-crd-12 1 1s
==> v1/Job
NAME COMPLETIONS DURATION AGE
istio-init-crd-10 0/1 1s 1s
istio-init-crd-11 0/1 1s 1s
istio-init-crd-12 0/1 1s 1s
==> v1/Pod(related)
NAME READY STATUS RESTARTS AGE
istio-init-crd-10-6nhnr 0/1 ContainerCreating 0 1s
istio-init-crd-11-8zfmp 0/1 ContainerCreating 0 1s
istio-init-crd-12-6txr8 0/1 ContainerCreating 0 1s
==> v1/ServiceAccount
NAME SECRETS AGE
istio-init-service-account 1 1s
NAME: istio
LAST DEPLOYED: Wed Aug 14 15:18:21 2019
NAMESPACE: istio-system
STATUS: DEPLOYED
RESOURCES:
==> v1/ClusterRole
NAME AGE
istio-citadel-istio-system 16s
istio-galley-istio-system 16s
istio-mixer-istio-system 16s
istio-pilot-istio-system 16s
istio-reader 16s
istio-sidecar-injector-istio-system 16s
==> v1/ClusterRoleBinding
NAME AGE
istio-citadel-istio-system 16s
istio-galley-admin-role-binding-istio-system 16s
istio-mixer-admin-role-binding-istio-system 16s
istio-multi 16s
istio-pilot-istio-system 16s
istio-sidecar-injector-admin-role-binding-istio-system 16s
==> v1/ConfigMap
NAME DATA AGE
istio 2 16s
istio-galley-configuration 1 16s
istio-security-custom-resources 2 16s
istio-sidecar-injector 2 16s
==> v1/Deployment
NAME READY UP-TO-DATE AVAILABLE AGE
istio-citadel 1/1 1 1 16s
istio-galley 0/1 1 0 16s
istio-ingressgateway 0/1 1 0 16s
istio-pilot 0/1 1 0 16s
istio-policy 1/1 1 1 16s
istio-sidecar-injector 0/1 1 0 16s
istio-telemetry 1/1 1 1 16s
==> v1/Pod(related)
NAME READY STATUS RESTARTS AGE
istio-citadel-657c84d86f-8bmqz 1/1 Running 0 16s
istio-galley-6d4c54fc76-bnbmp 0/1 ContainerCreating 0 16s
istio-ingressgateway-7f768f54c7-f62pz 0/1 Running 0 16s
istio-pilot-6b65d765b5-gx8mw 0/2 Pending 0 16s
istio-policy-5d7d7d557d-c89gb 2/2 Running 0 16s
istio-sidecar-injector-78949dd945-t9tsq 0/1 Running 0 15s
istio-telemetry-77797d4d8-kfxdn 2/2 Running 0 16s
==> v1/Role
NAME AGE
istio-ingressgateway-sds 16s
==> v1/RoleBinding
NAME AGE
istio-ingressgateway-sds 16s
==> v1/Service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
istio-citadel ClusterIP 10.100.54.218 <none> 8060/TCP,15014/TCP 16s
istio-galley ClusterIP 10.100.2.32 <none> 443/TCP,15014/TCP,9901/TCP 16s
istio-ingressgateway LoadBalancer 10.100.196.148 aa9ddfd6dbed0... 15020:32451/TCP,80:31380/TCP,443:31390/TCP,31400:31400/TCP,15029:31270/TCP,15030:30381/TCP,15031:32106/TCP,15032:32187/TCP,15443:31895/TCP 16s
istio-pilot ClusterIP 10.100.218.234 <none> 15010/TCP,15011/TCP,8080/TCP,15014/TCP 16s
istio-policy ClusterIP 10.100.8.126 <none> 9091/TCP,15004/TCP,15014/TCP 16s
istio-sidecar-injector ClusterIP 10.100.164.17 <none> 443/TCP 16s
istio-telemetry ClusterIP 10.100.251.193 <none> 9091/TCP,15004/TCP,15014/TCP,42422/TCP 16s
==> v1/ServiceAccount
NAME SECRETS AGE
istio-citadel-service-account 1 16s
istio-galley-service-account 1 16s
istio-ingressgateway-service-account 1 16s
istio-mixer-service-account 1 16s
istio-multi 1 16s
istio-pilot-service-account 1 16s
istio-security-post-install-account 1 16s
istio-sidecar-injector-service-account 1 16s
==> v1alpha2/attributemanifest
NAME AGE
istioproxy 15s
kubernetes 15s
==> v1alpha2/handler
NAME AGE
kubernetesenv 15s
prometheus 15s
==> v1alpha2/instance
NAME AGE
attributes 15s
requestcount 15s
requestduration 15s
requestsize 15s
responsesize 15s
tcpbytereceived 15s
tcpbytesent 15s
tcpconnectionsclosed 15s
tcpconnectionsopened 15s
==> v1alpha2/rule
NAME AGE
kubeattrgenrulerule 15s
promhttp 15s
promtcp 15s
promtcpconnectionclosed 15s
promtcpconnectionopen 15s
tcpkubeattrgenrulerule 15s
==> v1alpha3/DestinationRule
NAME AGE
istio-policy 16s
istio-telemetry 16s
==> v1beta1/ClusterRole
NAME AGE
istio-security-post-install-istio-system 16s
==> v1beta1/ClusterRoleBinding
NAME AGE
istio-security-post-install-role-binding-istio-system 16s
==> v1beta1/MutatingWebhookConfiguration
NAME AGE
istio-sidecar-injector 15s
==> v1beta1/PodDisruptionBudget
NAME MIN AVAILABLE MAX UNAVAILABLE ALLOWED DISRUPTIONS AGE
istio-galley 1 N/A 0 16s
istio-ingressgateway 1 N/A 0 16s
istio-pilot 1 N/A 0 16s
istio-policy 1 N/A 0 16s
istio-sidecar-injector 1 N/A 0 16s
istio-telemetry 1 N/A 0 16s
==> v2beta1/HorizontalPodAutoscaler
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
istio-ingressgateway Deployment/istio-ingressgateway <unknown>/80% 1 5 1 16s
istio-pilot Deployment/istio-pilot <unknown>/80% 1 5 0 15s
istio-policy Deployment/istio-policy <unknown>/80% 1 5 0 15s
istio-telemetry Deployment/istio-telemetry <unknown>/80% 1 5 0 15s
NOTES:
Thank you for installing istio.
Your release is named istio.
To get started running application with Istio, execute the following steps:
1. Label namespace that application object will be deployed to by the following command (take default namespace as an example)
$ kubectl label namespace default istio-injection=enabled
$ kubectl get namespace -L istio-injection
2. Deploy your applications
$ kubectl apply -f <your-application>.yaml
For more information on running Istio, visit:
https://istio.io/
@mojochao From the error message, I think you're encountering https://github.com/helm/helm/issues/5531 and https://github.com/helm/helm/issues/3742.
How is your local istio charts under ./external obtained?
Anyways, try using the official helm charts repository like:
repositories:
- name: istio.io
url: https://storage.googleapis.com/istio-release/releases/1.1.3/charts/
releases:
- name: istio-init
chart: istio.io/istio-init
version: 1.1.3
repositories:
- name: istio.io
url: https://storage.googleapis.com/istio-release/releases/1.1.3/charts/
releases:
- name: istio
chart: istio.io/istio
version: 1.1.3
namespace: istio-system
You should change 1.1.3 to whatever version you'd like to use. It is just that I tested it with 1.1.3 before :)
mumoshu
on 15 Aug 2019
If you do need to use the istio charts downloaded locally, try this workaround mentioned in the upstream issue
mumoshu
on 15 Aug 2019
I decided to do as you suggest @mumoshu, and just use the Istio helm charts repo and everything works now with latest Istio v1.2.4. I did not have to use a shell script to wait for completion, and hope that future installs will be reliable without one.
Many thanks guys! So far, I'm finding helmfile a great fit for my needs, and appreciate all who've contributed to its development.
Cheers!
mojochao
on 19 Aug 2019
Didn't work for me, unless I run helmfile apply twice. Reason is, by the time istio being installed, the istio-init pods may have not been completed, which causes the installation to fail due to missing CRDs.
Is it possible to wait a specific amount of time after a chart is installed?
TarekAS
on 23 Oct 2019
Related issues
willejs
路
4Comments
ppawiggers
路
3Comments
marianogg9
路
3Comments
pavdmyt
路
3Comments
GoldenMouse
路
3Comments