helmfile --args is a historical artifact that only worked in the very beginning of the helmfile project. At that time helmfile sync only ran a single helm command.
Nowadays a helmfile run involves many helm commands so --args doesn't make sense. Which custom args should be passed to helm template, helm template, helm diff, helm repo up, helm dep build, etc? Impossible to deduce.
Just drop it from helmfile, and alternatively provide more specific flags if necessary.
mumoshu
All 35 comments
one thing which it might affect is an argoCD plugin https://github.com/travisghansen/argo-cd-helmfile/blob/master/src/argo-cd-helmfile.sh#L242
not sure though
vavdoshka
on 26 Apr 2021
@vavdoshka Thanks for sharing!
And... Whoa! I've never expected it to work like that. A helmfile template command involves many commands depending on your configuration, not only the helm template to render the final result but also helm repo up, helm dep up, helm dep build, and even helm template used internally by chartify.
So, the usage of args there somehow passed the args to the final helm template only? That sounds like a miracle.
@travisghansen What was your primary motivation to use args? Can you remove the use of args if we added dedicated flags like --kube-version, --api-versions,, --is-upgrade, and --no-hooks to helmfile template?
I think the use of --args --skip-crds can be replaced with helmfile template --include-crds=false today.
mumoshu
on 27 Apr 2021
I think it's nice to have an ability to pass through generic args to the helm template command (there was an issue about this specifically...I'll see if I can find it). For example just today I had to use this to pass --skip-tests down helm using the project linked above with argocd.
Likely unrelated, but I ran into issues using the chartify feature with pure templates. It seemed to puke on some of the arguments being passed in the argocd integration bit but don't recall what those were either.
travisghansen
on 27 Apr 2021
travisghansen
on 27 Apr 2021
Likely unrelated, but I ran into issues using the chartify feature with pure templates. It seemed to puke on some of the arguments being passed in the argocd integration bit but don't recall what those were either.
Yes, it is very likely to break. We have no complete list mapping that knows "which helm command supports which args", neither in helmfile nor in chartify, which makes implementing your desired feature correctly very hard.
mumoshu
on 27 Apr 2021
For example just today I had to use this to pass --skip-tests down helm using the project linked above with argocd.
Can we just add it to helmfile-template as a dedicated flag?
mumoshu
on 27 Apr 2021
Well, let me be more clear. I know it is a very "nice to have" feature. What I'm saying is implementing it correctly seems too hard from my perspective.
If anyone can come up with a nice way to implement it in a way that works through the helmfile's code, including chartify, that would be great.
Otherwise, adding dedicated flags to helmfile-template should be the way to go. I can easily see that I'll be able to maintain it that way.
mumoshu
on 27 Apr 2021
Yeah, I want to say it blew up on the --api-version business...I just never got around to reporting it.
A dedicated flag to template seems ideal honestly to keep pollution down. I'm only using it with template operation anyway.
Appreciate the dialog and willingness to work jointly!
travisghansen
on 27 Apr 2021
BTW, I'm using that plugin fairly heavily with argocd at this point and it all works beautifully...managing many clusters using it jointly with appset etc to orchestrate at many layers...I'm even running a silly controller to push rancher clusters automatically down to argocd as clusters at that layer.
travisghansen
on 27 Apr 2021
@travisghansen Thanks for the constructive feedbacks! I very much appreciate it. I've still some idea to try on args so you can expect me to not delete args anytime soon, at least.
Also, I've created #1812 to discuss about Helmfiel with ArgoCD. If you could come up with any topics around that, I'd appreciate your comments.
Lastly... shouldn't --api-version be within helmfile.yaml, rather than the command-line flags, if it's up to the chart being used which K8s API versions are supported?
mumoshu
on 27 Apr 2021
Thank you for the great tool! I know it can get tiring maintaining such a project for various reasons.
api-versions shouldn't be in the helmfile.yaml because it's technically generated per-cluster. I'm using the same set of helmfiles deployed across many clusters while using 2 high-level required envs ENVIRONMENT_ID and CLUSTER_ID. A whole slew of values are derived from those 2 values driving automated ingress URLs, dns keys for external-dns etc. Technically if cluster versions (or api-versions) differ the produced output of the chart may differ slightly from cluster to cluster. Or said differently, if I upgrade a cluster from one version to another I may see updates in produced output.
Does that make sense why I keep it out of the helmfiles?
I can provide some concrete examples from my own charts if that's helpful..
travisghansen
on 27 Apr 2021
@travisghansen Thanks for confirming! It's now crystal clear that it shouldn't be in helmfile.yaml 馃憤
mumoshu
on 27 Apr 2021
It鈥檚 not a bad feature to have by the way, there are use cases when the templating tool may not have access to live cluster info so having a failswitch for it is nice.
travisghansen
on 27 Apr 2021
Hello!
First of all, thank you @mumoshu for a wonderful tool and @travisghansen for a script integrating helmfile into ArgoCD.
I completely agree with the need to support the --args flag for helmfile, especially in scenarios where helmfile is used as a Helm chart compose tool to render/template manifests, e.g when it is invoked by ArgoCD, Spinnaker etc.
I was trying to use this possibility (via the integration script from @travisghansen ) to dynamically propagate the target cluster API capabilities, which are made available by ArgoCD as build environment - to the helm for correct render of templates.
However, it seems like it is still not working.
Helm template command expects to get the api versions passed as string array, e.g something like this:
helm template --api-versions=cert-manager.io/v1alpha3 --api-versions=cert-manager.io/v1alpha2.
However, helmfile seems to pass only the first of the specified flags. This problem was already mentioned in the following issue: https://github.com/roboll/helmfile/issues/457
Here is example of this behaviour:
````sh
snippet from Helm chart using capabilities
....
{{- if .Capabilities.APIVersions.Has "cert-manager.io/v1alpha3" -}}
apiVersion: cert-manager.io/v1alpha3
{{- else -}}
apiVersion: certmanager.k8s.io/v1alpha1
{{- end }}
invoke template with args passing a single api-versions flag
helmfile -l chart=cert-issuer template --args "--api-versions=cert-manager.io/v1alpha3"
Source: cert-issuer/templates/self_signed_issuer.yaml
apiVersion: cert-manager.io/v1alpha3
kind: ClusterIssuer
metadata:
name: kfrfpr-issuer-ca
spec:
ca:
secretName: ca-secret
invoke template with args passing several api-versions flags, where first is not the required one
helmfile -l chart=cert-issuer template --args "--api-versions=cert-manager.io/v1alpha2 --api-versions=cert-manager.io/v1alpha3"
Source: cert-issuer/templates/self_signed_issuer.yaml
apiVersion: certmanager.k8s.io/v1alpha1
kind: ClusterIssuer
metadata:
name: kfrfpr-issuer-ca
spec:
ca:
secretName: ca-secret
running the later command in debug mode, you can see that only the first flag was passed downward to the helm executable
helmfile --debug -l chart=cert-issuer template --args "--api-versions=cert-manager.io/v1alpha2 --api-versions=cert-manager.io/v1alpha3"
....
Templating release=cert-issuer, chart=../cert-issuer
exec: helm template cert-issuer ../cert-issuer --version 1.2.1 --namespace cert-manager --values /var/folders/j8/4x2hx6_13sq78dh_2cqkz99m0000gq/T/values595035734 --api-versions=cert-manager.io/v1alpha2 --debug
````
BTW, possibility to pass the args for templating can also become pretty useful if the Helm tool community will finally agree to re-introduce the --kube-version into Helm3 (there is a pending issue on that: https://github.com/helm/helm/pull/9040).
As helm template command does not communicate with target cluster, currently it uses the embedded K8s Go library in order to identify the Kubernetes Version to use. As such, for Helm starting with version 3.5.X, the default target K8s version become 1.20 which broke a lot of Helm charts rendering with Ingress objects dynamically identifying the correct API version to use like this:
{{- if semverCompare "<1.14-0" .Capabilities.KubeVersion.GitVersion -}}
{{- print "extensions/v1beta1" -}}
{{- else if semverCompare "<1.19-0" .Capabilities.KubeVersion.GitVersion -}}
{{- print "networking.k8s.io/v1beta1" -}}
{{- else -}}
{{- print "networking.k8s.io/v1" -}}
{{- end -}}
Hopefully, the Helm3 will allow to explicitly specify target K8s version for template command and than we will need to somehow propagate it down from helmfile. The --args parameter is currently the only way that i know of to achieve it.
GolubevV
on 11 May 2021
Interesting, are you saying that 鈥攁rgs is filtering duplicate keys but generally allows passing multiple args/options? That support in my script was added via external PR so I didn鈥檛 spend much time testing it and didn鈥檛 notice that limitation.
travisghansen
on 11 May 2021
The problem is coming not from the script (i was author of that PR:) ) but from the helmfile internal implementation which seem to keep only one of multiple args while helm tool requires to pass multiple args in this way:
https://github.com/roboll/helmfile/issues/457
As mentioned earlier and tested today, until this problem in helmfile is fixed (so that api-versions from command line are passed completely), the only workaround to dynamically pass the api-versions into helmfile logic is to use specially crafted snippet - in that case helmfile correctly propogates it down to helm:
````
# helmfile snippet to get KUBE_API_VERSIONS from ArgoCD build env and pass down to helm
apiVersions:
{{- range $apiV := ( requiredEnv "KUBE_API_VERSIONS" | split "," ) }}
- {{ $apiV }}
{{- end }}
helmfile debug output running snippet above and passing KUBE_API_VERSIONS as if it was ArgoCD runtime:
KUBE_API_VERSIONS="v1,apiregistration.k8s.io/v1,apiregistration.k8s.io/v1beta1,extensions/v1beta1,apps/v1,cert-manager.io/v1alpha3,..." helmfile --debug -l chart=cert-issuer template
...
second-pass rendering result of "common.yaml.part.0":
...
16:
17: apiVersions:
18: - v1
19: - apiregistration.k8s.io/v1
20: - authorization.k8s.io/v1beta1
21: - autoscaling/v1
22: - autoscaling/v2beta1
23: - autoscaling/v2beta2
24: - batch/v1
25: - batch/v1beta1
26: - certificates.k8s.io/v1
27: - certificates.k8s.io/v1beta1
28: - networking.k8s.io/v1
29: - networking.k8s.io/v1beta1
30: - apiregistration.k8s.io/v1beta1
31: - policy/v1beta1
32: - rbac.authorization.k8s.io/v1
33: - rbac.authorization.k8s.io/v1beta1
34: - storage.k8s.io/v1
35: - storage.k8s.io/v1beta1
36: - admissionregistration.k8s.io/v1
37: - admissionregistration.k8s.io/v1beta1
38: - apiextensions.k8s.io/v1
39: - apiextensions.k8s.io/v1beta1
40: - scheduling.k8s.io/v1
41: - extensions/v1beta1
42: - scheduling.k8s.io/v1beta1
43: - coordination.k8s.io/v1
44: - coordination.k8s.io/v1beta1
45: - node.k8s.io/v1beta1
46: - discovery.k8s.io/v1beta1
47: - catalog.cattle.io/v1
48: - crd.projectcalico.org/v1
49: - monitoring.coreos.com/v1
50: - acme.cert-manager.io/v1alpha3
51: - acme.cert-manager.io/v1alpha2
52: - apps/v1
53: - cert-manager.io/v1alpha3
54: - cert-manager.io/v1alpha2
55: - cluster.cattle.io/v3
56: - management.cattle.io/v3
57: - metrics.k8s.io/v1beta1
58: - events.k8s.io/v1
59: - events.k8s.io/v1beta1
60: - authentication.k8s.io/v1
61: - authentication.k8s.io/v1beta1
62: - authorization.k8s.io/v1
...
Templating release=cert-issuer, chart=
exec: helm template cert-issuer
helm:hxKQF> install.go:172: [debug] Original chart version: "1.2.1"
helm:hxKQF> install.go:189: [debug] CHART PATH: /Users/vgolubev/Library/Caches/helm/repository/cert-issuer-1.2.1.tgz
helm:hxKQF> ---
Source: cert-issuer/templates/self_signed_issuer.yaml
apiVersion: cert-manager.io/v1alpha3
kind: ClusterIssuer
metadata:
name: kfrfpr-issuer-ca
spec:
ca:
secretName: ca-secret
````
Currently, i have to add that snippet as common behaviour to all helmfiles even though using the metnioned script. Onec issue 457 is fixed and if --args behaviour is preserved, the argocd-helmfile script will automatically inject those params and there will be no longer need to copy-paste that snippet when api-versions dynamic templating is required.
GolubevV
on 11 May 2021
Awesome thanks for the contribution and the snippet!
I don鈥檛 think the intent is to remove the functionality of 鈥攁rgs entirely but rather remove it as a global option from helmfile and only support it for subcommands that make sense (or rather have subcommand specific implementations).
travisghansen
on 11 May 2021
@travisghansen Thanks for the comment! Yeah, something like that. As long as args makes clear sense in a sub-command, I'm fine to preserve it.
@GolubevV Your usecase makes sense a lot. But shouldn't we just enhance Helmfile to support your use-cases natively? Do you need to set apiVersions via envvars dynamically on helmfile run, and is that all you need?
What if helmfile.yaml's apiVersion automatically populated from KUBE_API_VERSIONS, without writing the template snippet you've shared?
mumoshu
on 12 May 2021
If I understand your suggestion about the env var that would actually work quite nicely yes. May want to make that an argument to the template sub command to allow folks to opt-in/out.
EDIT: or do something like helm secrets does and automatically alter behavior based on detection of running in argocd - https://github.com/jkroepke/helm-secrets#argocd-support
travisghansen
on 12 May 2021
I agree that env var can help but question is - what should be the name of that var?
Should it be aligned by default with the name used by ArgoCD e.g KUBE_API_VERSIONS ?
Or probably make the name of that env var configurable?
In that regard, I like the idea of @travisghansen to implement runtime detection to be able to automatically propagate/map some ArgoCD build variables down to helmfile internal variables, like:
- ARGOCD_APP_NAMESPACE
- KUBE_VERSION
- KUBE_API_VERSIONS
It can be one of those improvements for better integration that we want to track in https://github.com/roboll/helmfile/issues/1812
BTW, In defence of the --args parameter, i continue to find more and more use-cases when helmfile is used as templating engine. Recently, it was very useful for propagating the parameter --skip-testsdown to helm to exclude Helm chart tests manifests from output. But maybe there is already native support for such behaviour in helmfile?
GolubevV
on 13 May 2021
@GolubevV Hey!
Or probably make the name of that env var configurable?
Probably.
ARGOCD_APP_NAMESPACE
KUBE_VERSION
KUBE_API_VERSIONS
All these seem to make sense to me.
But I'd perhaps prefer it being an opt-in feature. What if helmfile template --argocd resulted in reading those envvars appropriately?
Also, would you mind writing a more detailed feature request with how you'd exactly like those envvars to be mapped to helmfile settings, and link it from https://github.com/roboll/helmfile/issues/1812 for reference?
That way, designing/implementing the feature would be more approachable and feasible. Currently, everyone writes many ideas about ArgoCD integration in various places, which is starting to kill me
BTW, In defence of the --args parameter, i continue to find more and more use-cases when helmfile is used as templating engine. Recently, it was very useful for propagating the parameter --skip-testsdown to helm to exclude Helm chart tests manifests from output. But maybe there is already native support for such behaviour in helmfile?
As I've been repeatedly saying to everyone, I'd like everyone to submit feature requests to add necessary and concrete flags to helmfile-template. That said, writing a feature request to add --skip-tests to helmfile-template does make sense.
I'd remove --args only after I'm confident we have all the necessary flags added.
I consider the use of --args is just a work-around, not even a necessary evil.
mumoshu
on 14 May 2021
Another voice on args https://github.com/roboll/helmfile/issues/1853#issuecomment-842931450
mumoshu
on 18 May 2021
As I've been repeatedly saying to everyone, I'd like everyone to submit feature requests to add necessary and concrete flags to helmfile-template.
It won't be easy to have an exhaustive list of necessary and concrete flags since the upstream helm will evolve. Currently I rely on --args to pass --kube-version and --include-crds and etc and it works fine on CLI.
There is bug in state file directive though.
helmDefaults:
args:
- "--set k=v"
will produce
ARGS:
0: helm (4 bytes)
1: template (8 bytes)
2: mysql (5 bytes)
3: bitnami/mysql (13 bytes)
4: --namespace (11 bytes)
5: mysql (5 bytes)
6: --debug (7 bytes)
7: set (3 bytes)
8: k=v (3 bytes)
It seems stripping -- from the string and there is no way to apply args to specific command only.
My 2 cents:
- +1 for removing the buggy
argsfrom state file - -1 for removing
--argsfrom CLI unless there is an alternative solution for passing arbitrary arguments to helm as is.
yujunz
on 1 Jun 2021
@yujunz Hey! Thanks for the feedback. But I'm feeling like I'm repeating the same call again and again.
alternative solution for passing arbitrary arguments to helm as is.
This turns out to be impossible to me. If anyone has some time to speculate it and able to come up with a concrete idea on how it should look like, please help me.
The hardest part if that we really don't have a concrete spec around how current args SHOULD work and even the understanding of how it's currently working, which makes it very hard to decide where it should go.
Someone uses it in a way, another uses it in another way. Everyone brings in various ideas. I try to summarize, balance all and try to come up with the best idea to move forward. And that turned out to be just removing args while adding concrete flags/config keys for necessary helm flags.
mumoshu
on 1 Jun 2021
It is pretty common to pass args as is in tool chains. For example bazel run will pass whatever followed by a bare -- to the target binary. See example in https://stackoverflow.com/questions/54175366/bazel-run-passing-main-arguments
It doesn't have to be called --args if that is confusing some users. We just need something to distinguish between options/flags for helmfile and helm commands. -- is a common separator for such purpose.
After all, this will not conflict with concrete flags/config which can be parsed by helmfile but complement them for flexibility and extensibility.
Does it make sense?
yujunz
on 1 Jun 2021
@yujunz Sorry but no. I raed bazel run to pass the args to the your main application that bazel built. bazel runs a single application to pass args to. That makes sense. Helmfile commands and helmfile commands don't map 1:1.
For example helmfile diff runs helm repo add, helm template, helm fetch, helm dep build, helm dep up, and finally helm diff, depending on the scenario and the configuration. "Perhaps" helmfile --args $ARGS diff is somehow passing args to helm diff only today? But how could I pass to it helm template only so that I can inject some args for use by chartify? That's the problem I have with args. I know it accidentally works as in a way someone expects today, but not for another. It's fundamentally flawed. The fact that it works sometimes doesn't negate the fact that it is fundamentally flawed.
mumoshu
on 2 Jun 2021
For example helmfile diff runs helm repo add, helm template, helm fetch, helm dep build, helm dep up, and finally helm diff
TIL.
I was expecting --args to apply to the main helm command only (normally the last one). IMO that would cover most cases and seems an acceptable trade-off of implementation complexity.
For arguments specific to each steps, either implement it as what you have suggested
I'd like everyone to submit feature requests to add necessary and concrete flags to helmfile-template.
Or with specific flags ,e.g. --diff-args, --repo-add-args, which might be overkilled IMO.
What do you think? @mumoshu
yujunz
on 2 Jun 2021
I was expecting --args to apply to the main helm command only (normally the last one). IMO that would cover most cases and seems an acceptable trade-off of implementation complexity.
@yujunz Thanks. Yeah, maybe. I understand that it might cover most cases.
Or with specific flags ,e.g. --diff-args, --repo-add-args, which might be overkilled IMO.
Yeah, this might work too. The only concern for me is the possibility that people start asking various questions for various combinations of yaml settings and --foo-args combinations that I have to answer, which is a maintenance burden :)
Maybe --args will remain as-is. I have no right to remove anything that people are still using, anyway. Hopefully, I can just tolerate it.
mumoshu
on 2 Jun 2021
@mumoshu if --args is to be removed all of the helm possible arguments should be ported into helmfile, otherwise it breaks current functionality.
I agree that --args should probably stay as it is.
We for example are using the --dryrun and --debug args
dudicoco
on 3 Jun 2021
Thanks, everyone!
mumoshu
on 4 Jun 2021
@mumoshu I think it might still be good to address array style parameters and what should be done if anything about that.
travisghansen
on 4 Jun 2021
@travisghansen Hey! Sry but what was "array style parameters` thing? Where did we discuss about that before
mumoshu
on 4 Jun 2021
I don鈥檛 recall which issue it landed on but its the problem where specifying the same arg name multiple times results in only the last variant of it being passed down to helm (ie: 鈥攁pi-versions).
travisghansen
on 4 Jun 2021
@travisghansen Thanks! Sry but I can't recall either. Would you mind creating a dedicated issue for it, with a short concrete example of how you would use it, so that I can try to implement and test it. It's currently too vague for me to work on.
mumoshu
on 4 Jun 2021
@mumoshu start here I guess: https://github.com/roboll/helmfile/issues/1060#issuecomment-704582234
travisghansen
on 4 Jun 2021
Related issues
mumoshu
路
4Comments
sstarcher
路
3Comments
klebediev
路
3Comments
marianogg9
路
3Comments
RafalMaleska
路
3Comments
Most helpful comment
https://github.com/roboll/helmfile/issues/1060