Argo-cd: Support git submodules

[RudyBone]

Is your feature request related to a problem? Please describe.
I have a number of helm charts that all make use of a common values.yaml file. Currently there is no way to share a values yaml file in ArgoCD unless all of the helm charts are gathered under a single git project, in which case I can put the common yaml file in the project directory and reference it as ../values.yaml in the application yaml of each helm charts in the project. Since I am just evaluating ArgoCD, rather then re-arrange my git repos to do this I tried to create a top level git project with submodules referencing my helm charts. This does not work, probably because when you git clone a project with submodules the submodules directories are not immediately populated unless you run git submodule init OR add --recurse-submodules to the git clone command.

Describe the solution you'd like
Support for git submodules

[alexec]

Good idea, great for community contribution!

[jannfis]

This is a great idea. I have talked to colleagues with similar use-cases in the past, who were asking me whether it's possible to include Helm values.yaml files from different repositories for the same application with ArgoCD. This feature would indeed enable implementation of such use-cases.

At a quick glance, implementation seems not too complex. However, if we're digging deeper, questions arise, especially regarding on how to detect out-of-sync state for ArgoCD as well as repository authentication and connections of included submodules.

Alas, I'm not very familiar with the sub module concept in Git. In fact, now is the first time I ever heard of it and am currently reading up on it. I would suppose that the commit SHA of any given branch in the superproject would not change if an included submodule's commit SHA changes, right?

However, I'm very interested in this feature, or in the problem it would solve respectively the use cases it would enable. I will conduct some simple proof-of-concepts locally, and then decide whether I dare for an implementation or rather pass.

[yann-soubeyrand]

Submodules can function in two modes:

• You can stick your submodule to a commit. Thus, each time you update your submodule, you'll have to commit in your parent repository.
• Your submodule can be configured to follow a branch. This way, the submodule can be updated without committing in the parent repository.

You can compare these two modes to the functioning of Argo CD applications: your target revision can be a tag or a branch.

[adamjohnson01]

@jannfis , are you planning on implementing this or can I?

[jannfis]

@adamjohnson01 Please go ahead!

[adamjohnson01]

I have this working in my test environment with a very small change. I check if there are submodules and then run git submodule update --init --recursive

In order for any updated files in your submodules to be picked up you will need to have updated your repo to point to the changes using git submodule update --remote --merge etc. I do not think that applications should change if there is no revision change in the parent repo.

I am still testing out the authentication side of things

[jannfis]

@adamjohnson01 That's some great news. I also boggled my mind around authentication when thinking about this idea, and I was concluding we would have to live with the constraint that the sub-modules need to support the same set of credentials for authentication as the parent repo (i.e. same SSH key or HTTPS user/pass combination). I'd love to hear if you come up with different ideas!

[adamjohnson01]

@jannfis, I came to the same conclusion. I looked into the idea of adding an extra field to ApplicationSource that would contain a list of submodule repos and use the credentials from those and a combination of git submodule foreach to do the update but it would require a lot of changes and I am not sure that it is worth it. How about we start off with supporting git submodule with the condition that the credentials need to be the same on parent and submodule and we can always add support for different credentials in the future if enough people are interested in it?

[jannfis]

Yep, I think this would be a good start, @adamjohnson01 and fully agree that "benefit-cost ratio" would not be good without a strong demand.

I think different authentication tokens would be a real edge-case for this scenario, as I think most people would use another repository on the same server with sub-modules anyway. At least that's the use-cases I know from colleagues that want this support.

[xrow]

Hi, we also need this feature. Thanks

[guhcampos]

I believe there can be three clear milestones for such a feature, all of three add significant value and enable a number of use cases:

• A. support only publicly accessible submodules
• B. support submodules that are accessible using the same credentials as the parent repository
• C. full support for sumodules as described

A alone would enable a multitude of possibilities using public projects. Let's take nginx-ingress as the first example: I can add https://github.com/kubernetes/ingress-nginx.git as a submodule of my project and automatically refer to the YAML files for the ingress CRDs in my project, something that is currently difficult to achieve with ArgoCD alone (I usually just copy the YAML files into my repo, but then I have to manually track their updates forever)

B enables most of the private use cases. Most small and mid-sized organizations will have a single "group" in their git environments with pretty much global acessible credentials. Sometimes different groups may have different read/write permissions to particular projects, but it's very rare to make a project completely private in such environments.

C finally would enable larger organizations on which permissions to git repositories are fine grained and one credential has access to a really small subset of projects. In this case, and only in this case, a deployment may need more than one credential to achieve what it needs. I think this is a "nice-to-have" scenario and should not be a deterrent for the other two.

Those are my two cents. Thanks!

[adamjohnson01]

@guhcampos, the current pull request fulfils A and B.

[alexec]

Fixed in v1.4