Crossplane: Composition to reference fields from another base

[janwillies]

Looking at solving https://github.com/crossplane/provider-aws/issues/212 with Composition I ran into the following:

1. need to reference a fieldpath from another base
2. fill a placeholder in a free form spec.forProvider.assumeRolePolicyDocument: |
3. only a substring of the string from 1. is needed

Here is the YAML file I came up with:

apiVersion: apiextensions.crossplane.io/v1alpha1
kind: Composition
metadata:
  name: ekscluster-irsa
  labels:
    provider: aws
spec:
  from:
    apiVersion: example.org/v1alpha1
    kind: EKSClusterInstance
  to:
  - base:
      apiVersion: eks.aws.crossplane.io/v1beta1
      kind: Cluster
      spec:
        forProvider:
          roleArn: arn:aws:iam::1234567890:role/crossplane-eks-cluster-role
          resourcesVpcConfig:
            subnetIds:
              - subnet-abc
              - subnet-def
          version: "1.16"
        reclaimPolicy: Delete
        providerRef:
          name: aws  
  - base:
      apiVersion: identity.aws.crossplane.io/v1beta1
      kind: IAMRole
      spec:
        forProvider:
          roleName: ekscluster-velero
          assumeRolePolicyDocument: |
            {
              "Version": "2012-10-17",
              "Statement": [
                {
                  "Effect": "Allow",
                  "Principal": {
                    "Federated": "arn:aws:iam::${AWS_ACCOUNT_ID}:oidc-provider/${OIDC_PROVIDER}"
                  },
                  "Action": "sts:AssumeRoleWithWebIdentity",
                  "Condition": {
                    "StringEquals": {
                      "${OIDC_PROVIDER}:sub": "system:serviceaccount:${SERVICE_ACCOUNT_NAMESPACE}:${SERVICE_ACCOUNT_NAME}"
                    }
                  }
                }
              ]
            }
        reclaimPolicy: Delete
        providerRef:
          name: aws
    patches:
    # need to get a field from the other base (`Cluster`), put it into this base spec.forProvider.assumeRolePolicyDocument... AND transform it
    # status.atProvider.identity.oidc.issue outputs the following URL:
    # https://oidc.eks.eu-central-1.amazonaws.com/id/F00BAR1C01C4F51A59436D9096A1A
    - fromFieldPath: "status.atProvider.identity.oidc.issuer"
      # the information needs to go into both ${OIDC_PROVIDER} placeholders (after transformation)
      toFieldPath: "spec.forProvider.assumeRolePolicyDocument...."
      transforms:
      - type: string
        string:
          fmt: "%s" # how to get a substring, only the last part after "..amazonaws.com/id/": F00BAR1C01C4F51A59436D9096A1A
  - base:
      apiVersion: identity.aws.crossplane.io/v1alpha1
      kind: IAMPolicy
      spec:
        forProvider:
          name: eksluster-name-s3-mybucket-admin
          document: |
            {
              "Version": "2012-10-17",
              "Statement":[{
                "Effect": "Allow",
                "Action": "s3:*",
                "Resource": ["arn:aws:s3:::my_bucket",
                             "arn:aws:s3:::my_bucket/*"]
                }
              ]
            }
        reclaimPolicy: Delete
        providerRef:
          name: aws
  - base:
      apiVersion: identity.aws.crossplane.io/v1beta1
      kind: IAMRolePolicyAttachment
      metadata:
        name: ekscluster-velero
      spec:
        forProvider:
          roleNameRef:
            name: ekscluster-velero
          # https://github.com/crossplane/provider-aws/issues/232
          policyArn: arn:aws:iam::aws:policy/s3-mybucket-admin 
        reclaimPolicy: Delete
        providerRef:
          name: aws-provider
  - base:
      apiVersion: workload.crossplane.io/v1alpha1
      kind: KubernetesApplication
      metadata:
        name: velero-service-account
        # should the namespace be left out here since Composition is cluster-scoped?
        namespace: xyz
      spec:
        # should match the EKS Cluster from above
        targetRef:
          matchControllerRef: true
        resourceSelector:
          matchLabels:
            app: velero
        resourceTemplates:
        - metadata:
            name: velero-namespace
            labels:
              app: velero
          spec:
            template:
              apiVersion: v1
              kind: Namespace
              metadata:
                name: velero
        - metadata:
            name: velero-service-account
            labels:
              app: velero
          spec:
            template:
              apiVersion: v1
              kind: ServiceAccount
              metadata:
                name: velero
                annotations:
                  eks.amazonaws.com/role-arn: arn:aws:iam::$AWS_ACCOUNT_ID:role/$IAM_ROLE_NAME
    # need to reference the IAMRole from above
    - fromFieldPath: "spec.forProvider.roleName"
      # the information needs to go into the $IAM_ROLE_NAME placeholder in the velero-service-account
      toFieldPath: "spec.resourceTemplates[1].spec.template.metadata.annotations['eks.amazonaws.com/role-arn']"
      transforms:
      - type: string
        string:
          fmt: "arn:aws:iam::1234567890:role/%s" # with static AWS Account
  writeConnectionSecretsToNamespace: crossplane-system
  reclaimPolicy: Delete

[negz]

Hrm, I wonder if this could be a case for cross resource references? I'm not sure it's a great fit given that it's a bit weird for a policy document to reference an EKS cluster, but we could allow spec.forProvider.assumeRolePolicyDocumentRef to reference an EKS cluster...

The substring transform should be straightforward to add if we need it.

[turkenh]

I think this has other use cases and would be a powerful feature to consider adding:

Use case 1:

I want to provision a private GKE cluster using crossplane and define this as a composition together with other resources (NodePools etc) that my application needs.

To be able to provision private GKE clusters, I need to set privateClusterConfig.masterIpv4CidrBlock in GKECluster spec. Instead of exposing this as a parameter to my composite resource as suggested in this comment, which would require another entity (e.g. operator) to automate setting this, I would like to define a custom type and its controller, e.g. AvailableCIDR, and feed what is in its status to spec of GKECluster inside my composition.

Use case 2:

My application requires an ingress controller deployed to the cluster and a DNS record being provisioned for the ip of LoadBalancer type Service created.

I want to create a composite resource which includes:

• a GKECluster
• a Provisioner resource (e.g. HelmRelease) for nginx-ingress
• a custom resource (e.g. FieldReporter) which reads the provisioned load balancer ip address in the cluster and reports it in status
• a CR for provisioning DNS which consumes status of FieldReporter in its spec.

This way I can represent my application as a composite resource without needing to implement yet another controller on top of composite resource. Yes, we may still need to implement custom types and controllers (e.g. AvailableCIDR FieldReporter) but they could be reusable building blocks which could be part of crossplane ( or some provider) instead of a custom controller only for my application.

[janwillies]

Good points. Would be great to have a crossplane model for the DNS end to end. Maybe something like the Gardener solution: https://github.com/gardener/external-dns-management#the-model

[muvaf]

I believe back propagation to composite resource will solve this issue by using composite resource as common data store which can accept values from composed resources https://github.com/crossplane/crossplane/issues/1639

However, I believe there is also the case where you'd like to have the value before creating that composed resource, i.e. an empty value is valid but you'd like that additional information to be supplied during creation so that you don't miss that configuration window of the field. I believe there could be an option in patch object to specify block until source field has a value.