Cluster-api: [clusterctl] Support pluggable templating engines for provider yaml (core, bootstrap, kcp, infra)

[CecileRobertMichon]

To follow up on #3115 which adds a templating interface for clusterctl cluster config, I would like to be able to process templates for the providers as well (infrastructure-components.yaml, core-components.yaml, bootstrap-components.yaml, control-plane-components.yaml).

To give concrete use cases:

• As a provider, I would like to be able to specify a default for an environment variable that is part of the provider config, such as AZURE_ENVIRONMENT which should default to AzurePublicCloud (which came out of #3189)
• As a user, I would like to be able to easily pass in a flag on any provider manager such as a feature flag (eg. MachinePool) or a concurrency flag (eg. machine-concurrency). This is related to https://github.com/kubernetes-sigs/cluster-api/issues/3045.

/kind feature

[wfernandes]

/area clusterctl

[vincepri]

If we do #3189, can we rely on component config in the future to configure controllers?

[CecileRobertMichon]

If we do #3189, can we rely on component config in the future to configure controllers?

I think that makes sense as the templating needs for the provider aren't as complex as the cluster templates (config is mostly controlled by env vars and container args, there is no need to change what the template structure itself looks like, if/else statements, etc.) Component config is also versioned as you pointed out in Slack and can be reloaded at runtime.

[vincepri]

That said, I'm in favor of having templating for the infrastructure components as a quicker solution

[CecileRobertMichon]

@vincepri I actually just remembered a case where component config wouldn't be enough (I think): as @devigned mentioned in https://github.com/kubernetes-sigs/cluster-api/issues/2339#issuecomment-619136970:

One concrete example of this is the desire to use a different identity type when initializing a CAPZ provider. Many of the cloud providers offer credentialed access or usage of an implicit identity provisioned on the node. If the configuration was templated, it would be easy to produce one template that allowed a user of clusterctl init to specify the desired configuration as input and produce a yaml with the desired identity configuration.

Since this is infra specific, I don't think it would quite fit in the component config. It also requires conditionals in the infra components (ie. create the bootstrap credentials secret iff not using managed identity), and wouldn't be fixed by simple variable substitutions.

[fabriziopandini]

I'm +1 to use #3189 to have the first set of use cases addressed without changing the UX, and to proceed in enabling usage of templating engines in a second step.

WRT to the change required for using a full-fledged templating engine, my suggestion is to add the declaration of the template engine for each provider in the clusterctl config file / in the embedded provider config, so it will be possible to have each providers using different templating engines w/o entangling the UX of clusterctl with additional flags.

[wfernandes]

I think we can start by implementing a backward compatible change for #3189. I was hoping to try it with drone/envsubst. See https://github.com/kubernetes-sigs/cluster-api/issues/3189#issuecomment-646293367

To @fabriziopandini last point, I had actually proposed it in the CAEP at one point when we were considering of adding the extensible yaml processors in clusterctl directly via a contrib folder. I think it was along the lines of

providers:
  # add a custom provider
  - name: "my-infra-provider"
    url: "https://github.com/myorg/myrepo/releases/latest/infrastructure_components.yaml"
    type: "InfrastructureProvider"
    processor: <ytt, cue, dhall, whatever..>

But we could do something embedded in the provider config itself.

[fabriziopandini]

@wfernandes I'm fine with your proposal for the config, TBD how to integrate new processors in clusterctl

[vincepri]

/milestone Next

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fabriziopandini]

/lifecycle frozen