Flux: Also update submodules on checkout/update

[Xaroth]

Describe the feature
Currently flux does not seem to be checking out submodules and/or updating them on sync runs. This is a request to add that feature.

What would the new user story look like?
How would the new interaction with Flux look like? E.g.
Allowing submodules to be updated/fetched, this allows one to abstract away some reusable stuff in separate repositories, or allows one to spread out the infrastructure-as-a-code layout between teams responsible for specific areas, by separating it on a repository level.

Expected behavior
During repository init/update, the --recurse-submodules option can be passed to git clone and git pull ( as per https://git-scm.com/book/en/v2/Git-Tools-Submodules ) to automaticall also update submodules. Alternatively git submodule update --init --recursive should also work.

[benyanke]

We would also use this heavily - would love to see it added.

Specifically, I'd like to use this to include a common cluster config repo in many individual cluster repos, allowing them to share common configs, but all be independently versioned.

Also open to alternative suggestions on how to accomplish a similar workflow.

Is there anything I could do to help push this forward @Xaroth ?

[jgalliers]

+1 for this, I'm not an expert but I would hope adding something like a --recurse-submodules=true command to fluxd would not be a massive change and would really help add some flexibility to repository structures

[Abhishek-Srivastava]

This is a must have feature. Supporting On-prem deployment thru gitops with existing architecture of Flux requires to have multiple git repos or multiple branches, if you have multiple on-prem deployments. This certainly can't scale.

[snwfdhmp]

+1 for this feature, it's a must-have for big repositories with multiple sub repos.

[jgalliers]

If you are in a total bind, I have used this in my .flux.yaml to emulate the same behaviour:

version: 1
commandUpdated:
generators:
- command: git submodule init >> /dev/null
- command: git submodule update --remote >> /dev/null
- command: kustomize build .

[sshakeri]

The command in .flux.yamlapproach almost worked for me. Except the deploy key could not be added to my submodule repo because it already was added to my main repo. I ended up creating a new flux ns for the submodule.

[artazar]

A small amendment to @jgalliers comment.

I was not using kustomize native layout on my repo, so building up all possible manifests from the repository led me to the following instead:

version: 1
commandUpdated:
  generators:
    - command: git submodule init >> /dev/null
    - command: git submodule update --remote >> /dev/null
    - command: kustomize edit add resource $(find -not -path "./_charts/*" -mindepth 2 -type f \( -name "*.yaml" -o -name "*.yml" \) 2>/dev/null)
    - command: kustomize build .

The kustomize edit line is quite a hack to build a list of all possible manifests, and it gets important to know how to structure your repo with this workaround in place. But if that is tolerable - this works.