Crossplane: Design for Crossplane Agent

[muvaf]

What problem are you facing?

There are 2 main modes of deployment of Crossplane that folks are mostly using:

• Local scenario.
  
  
  
  • In this model, you have a single beefy cluster with infrastructure CRs and you create Kubernetes primitive resources to deploy your apps (Deployment, Pod etc) that will consume that infrastructure.
  
  
• Remote scenario.
  
  
  
  • You have a single dedicated Crossplane cluster that you use to provision & manage all of your infrastructure from a single point, including the Kubernetes cluster that your app will run. This is challenging than the first scenario because secrets & your app will need to be delivered to another cluster and for that we have workload API, i.e. KubernetesApplication and KubernetesApplicationResources.
  
  

Here we're focusing on the second scenario. For reference, here is how the workload API looks like:

KubernetesApplication

apiVersion: workload.crossplane.io/v1alpha1
kind: KubernetesApplication
metadata:
  name: wp-app
  labels:
    app: wp
spec:
  resourceSelector:
    matchLabels:
      app: wp
  targetSelector:
    matchLabels:
      app: wp
  resourceTemplates:
    - metadata:
        name: wp-namespace
        labels:
          app: wp
      spec:
        template:
          apiVersion: v1
          kind: Namespace
          metadata:
            name: wp
            labels:
              app: wordpress
    - metadata:
        name: wp-deployment
        labels:
          app: wp
      spec:
        secrets:
          # This must match the writeConnectionSecretToRef field
          # on the database claim; it is the name of the secret to
          # pull from the crossplane cluster, from this Application's namespace.
          - name: sql
        template:
          apiVersion: apps/v1
          kind: Deployment
          metadata:
            namespace: wp
            name: wordpress
            labels:
              app: wordpress
          spec:
            selector:
              matchLabels:
                app: wordpress
            template:
              metadata:
                labels:
                  app: wordpress
              spec:
                containers:
                  - name: wordpress
                    image: "wordpress:4.6.1-apache"
                    env:
                      - name: WORDPRESS_DB_HOST
                        valueFrom:
                          secretKeyRef:
                            # This is the name of the secret to use to consume the secret
                            # within the managed cluster. The reason it's different from the
                            # name of the secret above is because within the managed cluster,
                            # a crossplane-managed secret is written as '{metadata.name}-{secretname}'.
                            # The metadata name is specified above for this resource, and so is
                            # the secret name.
                            name: wp-deployment-sql
                            key: endpoint
                      - name: WORDPRESS_DB_USER
                        valueFrom:
                          secretKeyRef:
                            name: wp-deployment-sql
                            key: username
                      - name: WORDPRESS_DB_PASSWORD
                        valueFrom:
                          secretKeyRef:
                            name: wp-deployment-sql
                            key: password
                    ports:
                      - containerPort: 80
                        name: wordpress
    - metadata:
        name: wp-service
        labels:
          app: wp
      spec:
        template:
          apiVersion: v1
          kind: Service
          metadata:
            namespace: wp
            name: wordpress
            labels:
              app: wordpress
          spec:
            ports:
              - port: 80
            selector:
              app: wordpress
            type: LoadBalancer

Alternatively:

3 KubernetesApplicationResources

---
apiVersion: workload.crossplane.io/v1alpha1
kind: KubernetesApplicationResource
metadata:
  name: wp-namespace
spec:
  targetSelector:
    matchLabels:
      app: wp
  template:
    apiVersion: v1
    kind: Namespace
    metadata:
      name: wp
      labels:
        app: wordpress
---
apiVersion: workload.crossplane.io/v1alpha1
kind: KubernetesApplicationResource
metadata:
  name: wp-deployment
spec:
  targetSelector:
    matchLabels:
      app: wp
  secrets:
    - name: wp-sql
  template:
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      namespace: wp
      name: wordpress
      labels:
        app: wordpress
    spec:
      selector:
        matchLabels:
          app: wordpress
      template:
        metadata:
          labels:
            app: wordpress
        spec:
          containers:
            - name: wordpress
              image: "wordpress:4.6.1-apache"
              env:
                - name: WORDPRESS_DB_HOST
                  valueFrom:
                    secretKeyRef:
                      # This is the name of the secret to use to consume the secret
                      # within the managed cluster. The reason it's different from the
                      # name of the secret above is because within the managed cluster,
                      # a crossplane-managed secret is written as '{metadata.name}-{secretname}'.
                      # The metadata name is specified above for this resource, and so is
                      # the secret name.
                      name: wp-deployment-wp-sql
                      key: endpoint
                - name: WORDPRESS_DB_USER
                  valueFrom:
                    secretKeyRef:
                      name: wp-deployment-wp-sql
                      key: username
                - name: WORDPRESS_DB_PASSWORD
                  valueFrom:
                    secretKeyRef:
                      name: wp-deployment-wp-sql
                      key: password
              ports:
                - containerPort: 80
                  name: wordpress
---
apiVersion: workload.crossplane.io/v1alpha1
kind: KubernetesApplicationResource
metadata:
  name: wp-service
spec:
  targetSelector:
    matchLabels:
      app: wp
  template:
    apiVersion: v1
    kind: Service
    metadata:
      namespace: wp
      name: wordpress
      labels:
        app: wordpress
    spec:
      ports:
        - port: 80
      selector:
        app: wordpress
      type: LoadBalancer

There are a few issues with the workload propagation mechanics we're providing to the users.

• You cannot interact with what you deploy directly, i.e. always have to use KubernetesApplicationResource as a proxy and that has its own set of challenges:
  
  
  
  • It's a template and gets deployed after you make the edit, so, you loose the admission check rejections in case something went wrong. Instead, you'll see them in the status but you won't be prevented from making the change as opposed to directly interacting.
  
  
  • Late initialization.
  
  
  • Let's say you deployed a Pod and spec.node is late-initialized. You will not see that because we only propagate the status back, not spec because the template is not strong-typed and it's hard to differentiate between user's actual desired spec and what's only a late-inited value.
  
  
  • If you have an element in an array that is late-inited or some elements are added after the creation, PATCH command will replace the whole array with what you got in your template. If the type is well-constructed to provide its own merge mechanics, this could be avoidable but that is usually not the case. For example, in some cases an element of an array in spec is late-inited for bookkeeping the IP and removing this causes its controller to provision new ones each time.
  
  
• Migration from a Helm chart that has a few Deployments and a StatefulSet for DB is harder.
  
  
  
  • You actually need to change each and every element to be in a KubernetesApplicationResource as well as StatefulSet of DB with a DB requirement CR.
  
  
• Operation experience. This is related to the first point. If you have an app using OAM or some other app model, then you always have that intermediate proxy of workload CRs. You're losing on some of the value that these models provide because of that proxy and in some cases it could be functionally detrimental.

Surely, it has its own advantages as well. For example, you can manage all of your apps from single point via KubernetesApplications targeting the right clusters. But as we see more usage patterns, we're more convinced that the thing users want to manage from a single central point is infrastructure (including policy, audit etc) but not necessarily applications.

How could Crossplane help solve your problem?

In the current model, the general approach is that main cluster pushes some applications together with some secrets that are populated by infrastructure CRs. What we'd like to achieve is to keep the experience of single control plane that you can view, manage, audit your infrastructure and define your own APIs to be used by different people as well as to improve the mechanics of how apps can consume that infrastructure from different clusters in a way that they don't have to make big changes and/or sacrifices from their original UX experience.

In its essence, the proposed model is that there will be an agent that you deploy into _any_ cluster, configure it to talk to a cluster that has Crossplane installed and make the APIs and infrastructure that exists in this single Crossplane cluster available for consumption in the remote cluster so that people directly create Deployment, MySQLInstanceRequirement and mount latter's secret to the former. It could sound that Crossplane gets a bit harder to operate with a new component and that is true for setup phase. However, the trade-off is that it gets a little bit harder to setup but once you do it as platform-owner, many of your users won't need to think about workload APIs anymore; they will just look at what infrastructure is defined and made available to them and use them in their YAMLs without having to migrate all to be KubernetesApplicationResource. So, the aim is to make end-users/app developers/app operators more happy while keeping the power of central control plane that lets you see & manage all the infrastructure used in the organization.

[prasek]

@muvaf having multiple agents (in different k8s app/target clusters) connect to a namespace in a single control plane instance is likely something worth supporting. let's discuss. //cc @grantgumina

[prasek]

@resouer @hongchaodeng @ryanzhang-oss @artursouza here are the Crossplane Connector Agent UX slides we shared at the community meeting yesterday:
https://docs.google.com/presentation/d/1pzVecSx7BLfZ5CS6kGhDJiaJ26KzpBof6thQNNEOX0c/edit#slide=id.g8801599ecb_2_169

//cc @muvaf @negz @hasheddan @kasey

[hongchaodeng]

Hi @muvaf

I don't quite understand this point:

Migration from a Helm chart that has a few Deployments and a StatefulSet for DB is harder.
You actually need to change each and every element to be in a KubernetesApplicationResource as well as StatefulSet of DB with a DB requirement CR.

Would you help provide more details and examples to clarify this point more? Highly appreciated!

[muvaf]

@hongchaodeng sure. Let's say you deploy Wordpress and have a MySQL server running as StatefulSet. Your Helm chart would have 1 Deployment that runs Wordpress image, 1 StatefulSet that runs your MySQL server and 1 Service that exposes your Wordpress, maybe also 1 Namespace. If you want to use Crossplane for your database, you can deploy it and change StatefulSet with a MySQLRequirement in the Helm chart, which is pretty straightforward, in fact you can fulfill MySQLRequirement with different Compositions and maybe one of them is in-cluster StatefulSet. It's all nice if app is in the cluster with Crossplane.

Now let's say you decided to manage all your infrastructure from one central point, meaning Crossplane is deployed in one cluster to be used by many people but actual Wordpress runs in another remote cluster. Now, you'll need to change StatefulSet to MySQLRequirement and wrap both Deployment and Service into KubernetesApplicationResources to schedule them to the remote cluster and also you need to include KubernetesClusterRequirement CR because it needs to know which cluster to provision to or even be provisioned to a cluster. At this point, your Helm chart changed completely and the experience that KubernetesApplicationResource provides is not the best one because of reasons listed above.

In the new model, instead of managing the apps from one place together with infra, we're moving towards a model where infra is managed from one place but users consume it from separate clusters independently. The agent helps them to do self-service in their own cluster. So, with the agent we keep the good UX of local mode while providing the power of having all the infrastructure in one place to be managed by the SRE/platform builders. Instead of pushing the app from center, we pull the infra in remote cluster from the central one. You can see details in https://github.com/crossplane/crossplane/pull/1633