Argo-cd: Document the patterns for cluster bootstrapping

[alexec]

Is your feature request related to a problem? Please describe.

We know there's a need to bootstrap clusters with a set of applications.

Describe the solution you'd like

Document patterns that teams use today.

[jon-walton]

The way I've done it so far, is to use an Application of Applications

In the root of our config repo, we've got an argocd folder. It's just a nested folder structure defining where the k8s cluster is and it's function (Azure Dev in East Asia).

Currently, when a new cluster comes online, it's a manual process to use the argocd CLI to create a new
project, cluster and an Application pointing it to the folder (dev-ea or ci-ea in this instance). We should in the near future have an Application manifest defined in the argocd to look in these folders and define the project/cluster details, etc

├───argocd
│   └─── << in the future, Application manifests for each cluster should go here >>
│   └───applications
│       └───azure
│           ├───ci-ea
│           │       argocd-ingress.yaml
│           │       argocd-install.yaml
│           │       cert-manager.yaml
│           │       nginx-ingress.yaml
│           │       
│           └───dev-ea
│                   app1.yaml
│                   cert-manager.yaml
│                   external-dns.yaml
│                   keyvault-flex-volume.yaml
│                   linkerd.yaml
│                   nginx-ingress.yaml

All of the Application manifests in the argocd folder (eg dev-ea) point to either a helm chart or a directory. Each helm chart then has a values folder with overrides for each cluster (ideally I would like these values files to live under the argocd/applications/... hierarchy)

│                   
├───infrastructure
│   ├───azure
│   │   ├───ci-ea
│   │   │   └───cert-manager
│   │   │           ci-domain.yaml
│   │   │           
│   │   ├───dev-ea
│   │   │   └───cert-manager
│   │   │           AzureIdentity.yaml
│   │   │           AzureIdentityBinding.yaml
│   │   │           dev-domain.yaml
│   │   │           
│   │   ├───keyvault-flexvol
│   │   │       deployment.yaml
│   │   │       
│   │   └───pod-identity
│   │           deployment.yaml
│   │           
│   ├───cert-manager
│   │   ├───charts
│   │   ├───templates
│   │   └───values
│   │           azure-ci-ea.yaml
│   │           azure-dev-ea.yaml
│   │           values.yaml
│   │           
│   ├───linkerd
│   │   └───azure
│   │       └───dev-ea
│   │               ingress.yaml
│   │               install.yaml
│   │               service-profiles.yaml
│   │                
│   └───nginx-ingress
│       ├───templates
│       └───values
│               argocd.yaml
│               default.yaml
│               internal.yaml
│               public.yaml

And we put our own applications in their own root folder, with a base helm chart overridden for specific uses

│               
└───MyApps
    ├───microservice
    │   └───templates
    │
    └───values
        ├───app1
        │       azure-dev-ea.yaml
        │       default.yaml
        │       
        └───app2
                azure-dev-ea.yaml
                default.yaml

And as an example, if I want to upgrade nginx-ingress for the dev environment but not prod. I would pin prod's targetRevision to a commit. I can then upgrade the nginx-ingress chart without issues

What would be awesome is if I could pin all of prod's Application manifests to a specific targetRevision in 1 go, without having to update all the files

[stevesea]

We're also using declarative setup to create an 'application of applications'.

We have 3 argocd deployments -- one for a dev playground (envs: per-team, and each dev can also create their own), one for nonprod (envs; test, staging, demo, etc), and one for production (envs: prod).

argocd is initially deployed to through Kustomize -- there are overlays for for each of the above deployments. the kustomize setup deploys argocd, and creates an 'argocd' Application that keeps itself up to date. Included in that deployment is another ArgoCD Project/Application per 'environment' for our collection of services.

e.g. the argocd-nonprod overlay references 'PROJ-test' and 'PROJ-staging' Project/Applications that point to a different repo that's a helm chart that creates more ArgoCD Projects/Applications

layout of that argocd kustomize:

 -- argocd
    |-- base
    |   |-- applications
    |   |   |-- argocd
    |   |   |   |-- app-argocd.yaml
    |   |   |   |-- kustomization.yaml
    |   |   |   `-- proj-argocd.yaml
    |   |   |-- kube-janitor
    |   |   |   |-- kj-app.yaml
    |   |   |   |-- kj-ns.yaml
    |   |   |   `-- kustomization.yaml
    |   |   |-- nginx-ingress
    |   |   |   |-- ingress-app.yaml
    |   |   |   |-- ingress-ns.yaml
    |   |   |   |-- ingress-proj.yaml
    |   |   |   `-- kustomization.yaml
    |   |   |-- PROJ-production
    |   |   |   |-- kustomization.yaml
    |   |   |   |-- PROJ-apps.yaml
    |   |   |   `-- PROJ-infra.yaml
    |   |   |-- PROJ-staging
    |   |   |   |-- kustomization.yaml
    |   |   |   |-- PROJ-apps.yaml
    |   |   |   `-- PROJ-infra.yaml
    |   |   `-- PROJ-test
    |   |       |-- kustomization.yaml
    |   |       |-- PROJ-apps.yaml
    |   |       `-- PROJ-infra.yaml
    |   |-- argocd
    |   |   |-- argocd-cm.yaml
    |   |   `-- kustomization.yaml
    |   `-- argocd_ha
    |       |-- argocd-cm.yaml
    |       `-- kustomization.yaml
    `-- overlays
        |-- argocd-eng
        |   |-- app-argocd.yaml
        |   |-- argo-server-ingress.yaml
        |   |-- argocd-cm.yaml
        |   |-- argocd-rbac-cm.yaml
        |   |-- argocd-server-service.yaml
        |   |-- kj-app.yaml
        |   |-- kustomization.yaml
        |   `-- regional-pd-app.yaml
        |-- argocd-nonprod
        |   |-- app-argocd.yaml
        |   |-- argo-server-ingress.yaml
        |   |-- argocd-cm.yaml
        |   |-- argocd-rbac-cm.yaml
        |   |-- argocd-server-service.yaml
        |   |-- kustomization.yaml
        |   `-- regional-pd-app.yaml
        `-- argocd-prod
            |-- app-argocd.yaml
            |-- argo-server-ingress.yaml
            |-- argocd-cm.yaml
            |-- argocd-rbac-cm.yaml
            |-- argocd-server-service.yaml
            `-- kustomization.yaml

the set of manifests that we deploy as an environment is grouped into either 'applications' or 'infrastructure', and each has a helm chart:

|-- README.md
|-- PROJ-applications
|   |-- Chart.yaml
|   |-- dev-values.yaml
|   |-- production-values.yaml
|   |-- staging-values.yaml
|   |-- templates
|   |   |-- applications.yaml
|   |   `-- projects.yaml
|   |-- test-values.yaml
|   `-- values.yaml
`-- PROJ-infrastructure
    |-- Chart.yaml
    |-- dev-values.yaml
    |-- production-values.yaml
    |-- staging-values.yaml
    |-- templates
    |   |-- applications.yaml
    |   `-- projects.yaml
    |-- test-values.yaml
    `-- values.yaml

in PROJ-applications and PROJ-infrastructure the values.yaml file has a map of which ArgoCD projects & applications should be created -- these reference the manifest repos that actually contain the manifests that'll be deployed to the destination clusters.

Each microservice has two git repos -- a source repo and a deployment repo. Devs interact with ArgoCD through merge requests.

• when dev's MR is approved and merged to master in their source repo, the build pipeline creates a docker container then triggers an update of the deployment repo to update the image tag. It also merges to the 'test' branch in the deployment repo.
• when dev/qa wants to promote from the test to the staging env, they'll create an MR from the test to the staging branch in the deployment repo.
• similarly, when dev/qa wants to promote from the staging to production env, they'll create an MR to go from the staging to the production branch in the deployment repo.

[alexec]

@jon-walton @stevesea I note it seems you use plain YAML for your manifests, rather than templating them?

[stevesea]

One thing I really like about ArgoCD is that I can pick and choose which deployment manifest strategy best fits with the complexity of my task -- some things don't need templating at all, our home grown services use Helm at the moment but feels like we can transition many of them to kustomize if they don't really need the complexity of templating.

[alexec]

I've added this example app:

https://github.com/argoproj/argocd-example-apps/tree/master/applications

Thoughts?

[alexec]

Any chance of a review of this document please?

https://github.com/argoproj/argo-cd/pull/1530

[jon-walton]

@alexec I use both helm and plain manifests

[jon-walton]

and the list of applications deployed to each cluster may be different, therefore I haven't templated that part

[denniswebb]

@alexec I've read the document in #1530 and as a new user of Argo, it is very helpful. More visuals or a walkthrough would help, but that's something for another PR because everything can be made better.

[dbachrach]

One thing I think would be helpful is guidance on when to split something into its own argocd application. Currently, we have all cluster yaml resources defined in a single repo. When getting started with argocd, it wasn't clear whether we should create a single application that syncs the entire repo or many small applications (for each individual component) that all are configured with the same git repo. We opted to go with the single application for now.
I think a formal recommendation (or a pro/con breakdown) in the argocd docs would be helpful.

[alexec]

I'm not sure what the recommendation for applications is. Typically you might have one project per team, but multiple applications per team. A lot of this would depend on the size of your organization.

[alexec]

So guys, one of the outcomes of this issue is to answer the hypothesis, do we have good enough patterns for bootstrapping, or do we need to provide some first-class support.

The mooted solution is to add the ability to auto-detect applications within a repository, e.g by looking for argocd.yaml files. Along with some configuration, this would automatically create applications.

This is a big piece of work, so I'm not keen to do it unless there's a strong need.

Thoughts?

[stevesea]

I think having a first-class support of a particular bootstrapping/deployment pattern would be a weakness -- a distraction and resource sink if you tried to add it to the core of ArgoCD.

How teams want to do it will be very dependent on the CI/CD infrastructure of a particular organization, and their own opinions on weighing the cost/benefit of how to organize their applications, how to track git repos for changes, etc.

I'd say to leave it up to the community to help document common patterns and guide best practices. Maybe there will be add-on projects or plugins that might help support the best practices that are distilled through community engagement.