Argo-cd: Create ArgoCD cluster with declarative setup using existing secret containing kubeconfig

[MPV]

Summary

Does declarative setup for ArgoCD already support creating new clusters using an existing secret containing a kubeconfig file?

If this isn't already supported: this is what I'm after.

Motivation

I'd like to use ArgoCD + Cluster-API for:

1. Creating new clusters using Cluster-API + ArgoCD.
2. Letting ArgoCD become aware of (get access to) those clusters.
3. Letting ArgoCD deploy apps into it's newly created clusters.

Proposal

Allowing argocd.argoproj.io/secret-type: cluster secrets to refer to another secret containing a kubeconfig (as part of/instead of it's config section) and use it for reaching that target cluster.

I'm referring to this functionality:
https://argoproj.github.io/argo-cd/operator-manual/declarative-setup/#clusters

So instead of:

apiVersion: v1
kind: Secret
metadata:
  name: mycluster-secret
  labels:
    argocd.argoproj.io/secret-type: cluster
type: Opaque
stringData:
  name: mycluster.com
  server: https://mycluster.com
  config: |
    {
      "bearerToken": "<authentication token>",
      "tlsClientConfig": {
        "insecure": false,
        "caData": "<base64 encoded certificate>"
      }
    }

You could do this:

apiVersion: v1
kind: Secret
metadata:
  name: mycluster-secret
  labels:
    argocd.argoproj.io/secret-type: cluster
type: Opaque
stringData:
  name: mycluster.com
  server: https://mycluster.com
  config: |
    {
      "existingKubeconfigSecret": "mycluster-kubeconfig"
    }

Or:

apiVersion: v1
kind: Secret
metadata:
  name: mycluster-secret
  labels:
    argocd.argoproj.io/secret-type: cluster
type: Opaque
stringData:
  name: mycluster.com
  server: https://mycluster.com
  existingKubeconfigSecret: mycluster-kubeconfig

[alexmt]

It seems strange to reference secret from another secret. Can you please clarify why don't create argocd compatible secret? Do you need to share one secret with Kubeconfig between Argo CD and some other tool?

[MPV]

It seems strange to reference secret from another secret. Can you please clarify why don't create argocd compatible secret? Do you need to share one secret with Kubeconfig between Argo CD and some other tool?

Yes, that kubeconfig secret is created/managed by another tool:

The new cluster is created dynamically from custom resources. After its creation, the "Cluster API" controller (which created the cluster) creates a new secret which contains just the kubeconfig.

[alexmt]

Thank you for the information @MPV ! So config is created by Cluster API.

In the past, we've made changes to get smooth integration with other opensource project. E.g. changed secret key to tls.crt/tks.key in order to simplify using cert manager.

How do you think, is it possible to do something similar in this case? E.g. Argo CD might support a new secret like kube.config and "Cluster API" would create secret with that field.

[MPV]

I'm looking for something like what is shown in this video (which I already have working), but with the addition of automatically deploying apps into the newly created cluster, just after its creation:

https://youtu.be/9pFBgxDa7Aw

...and without needing to push more than one git commit, containing all of:

• the cluster definition (in cluster API resources) for creating the cluster.
• the ArgoCD cluster config/secret for the new cluster.
• multiple ArgoCD apps instructed to be deployed into the new cluster (as soon as it would become created + ArgoCD becoming configured to be aware of it).

[MPV]

How do you think, is it possible to do something similar in this case? E.g. Argo CD might support a new secret like kube.config and "Cluster API" would create secret with that field.

@alexmt Yeah, let's investigate. I opened up a mirror issue over at https://github.com/kubernetes-sigs/cluster-api/issues/3866

[janwillies]

I'd love to see this too. Currently you have to jump through hoops to make this work, e.g. https://github.com/janwillies/crossargo-sync, https://github.com/infracloudio/crossplane-secrets-controller, ...

It would be very convenient if argocd would take a regular kubeconfig

[MPV]

For reference, this is how such a secret looks like (in the Cluster-API example):

$ kubectl -n mycluster get secret mycluster-1-kubeconfig -o yaml
apiVersion: v1
data:
  value: obfuscated-base-64-encoded
kind: Secret
metadata:
  creationTimestamp: "2020-10-23T14:52:51Z"
  labels:
    cluster.x-k8s.io/cluster-name: mycluster
  name: mycluster-1-kubeconfig
  namespace: mycluster
  ownerReferences:
  - apiVersion: controlplane.cluster.x-k8s.io/v1alpha3
    blockOwnerDeletion: true
    controller: true
    kind: KubeadmControlPlane
    name: mycluster-kubeadm-control-plane-1
    uid: aa9353fd-25b6-49b4-afa7-c7924191baa1
  resourceVersion: "779511"
  selfLink: /api/v1/namespaces/mycluster/secrets/mycluster-1-kubeconfig
  uid: <obfuscated>
type: Opaque

[MPV]

4600 talks about making ExecProvider available in the Cluster config.

Maybe it could become a way/workaround to solve this (by running something that gets the credentials from the existing secret and outputs it to stdout, as per default ExecConfig behaviour).

[ncdc]

Hi @alexmt! I'm one of the Cluster API maintainers. I'm curious about your statement:

How do you think, is it possible to do something similar in this case? E.g. Argo CD might support a new secret like kube.config and "Cluster API" would create secret with that field.

Cluster API is currently creating a secret that contains a kubeconfig. However, this secret is only created after you declaratively request a new cluster (e.g. you create a Cluster, AWSCluster, AWSMachineTemplate, KubeadmControlPlane, KubeadmConfigTemplate, and MachineDeployment - this gets you a real cluster on AWS).

I think what @MPV is asking for is the ability for Argo CD to recognize, support, and use the Cluster API secret containing the kubeconfig. It sounds like you're perhaps ok with Argo CD supporting a secret containing a kubeconfig, based on what you wrote above? Would you be open to supporting the Cluster API kubeconfig secret? Argo CD would have to accept that the secret will not exist for a period of time, but it will eventually get created as Cluster API reconciles things.

[peterrosell]

The task that we would like to automate is this step (https://argoproj.github.io/argo-cd/getting_started/#5-register-a-cluster-to-deploy-apps-to-optional)

argocd cluster add cluster-created-by-cluster-api

ArgoCD shall monitor for secrets that is created by cluster-api. There is a label/annotation that can be used and the secret name ends with -kubeconfig.
When it finds a matching secret it shall extract the info just like the CLI.
When it notice a change/removal it shall remove the cluster from ArgoCD.

I guess it's possible to create a hack with the shell operator that do this, but it would of course be much nicer if ArgoCD had a built in support for this cluster-api kubeconfig secret.

[janwillies]

crossplanes kubeconfig secret look like this:

apiVersion: v1
kind: Secret
type: connection.crossplane.io/v1alpha1
metadata:
  name: foo
  namespace: crossplane-system
  ownerReferences:
  - apiVersion: eks.aws.crossplane.io/v1beta1
    controller: true
    kind: Cluster
    name: foo
    uid: cd04c529-8f91-4a1a-a827-2f6138ec4087
data:
  clusterCA: LS0tL...
  endpoint: aH...
  kubeconfig: YXB...

[abdennour]

guys! i can see only 2 in CRDs list :

k get crd | grep arg
applications.argoproj.io                             2020-09-24T14:52:28Z
appprojects.argoproj.io                              2020-09-24T14:52:29Z

there is nothing called clusters.argoproj.io.. We expect to have that.