Crossplane: managed resource connection secret updates do not get propagated to the claim namespace

[jbw976]

What happened?

I have a cluster composition that uses EKS as the underlying managed resource. The auth tokens given out by EKS have a TTL of 15 min, so the connection secret for the EKS managed resources gets updated often.

It appears that updates to the connection secret do not get propagated back up the composition hierarchy and to the claim's namespace. So when trying to use the claim's connection secret, it is stale and results in error: You must be logged in to the server (Unauthorized).

How can we reproduce it?

The AWS reference platform is an example of this behavior. The composition hierarchy is:

• Cluster claim (namespaced)
  
  
  
  • CompositeCluster (composite resource)
  
  
  • eks.aws.platformref.crossplane.io (composite resource)
    
    
    
    
    
    • cluster.eks.aws.crossplane.io (managed resource)
    
    
    
  
  

When examining the connection secrets at those 4 levels, the only diff is between the claim and the CompositeCluster, so it appears the managed resource secret update is getting propagated through the compositions, but not the final hop to the namespaced claim.

What environment did it happen in?

Crossplane version: v0.14.0-rc.102.g6e1b4ce2

[negz]

As you point out, the propagation from the actual credentials in AWS to the claim secret works as follows:

1. The EKS credentials change inside AWS.
2. Every 30 to 60 seconds the provider-aws EKS controller polls the AWS API for new changes. If the credentials have changed they will be propagated to the EKS cluster's connection secret.
3. Every 30 to 60 seconds the composite resource controller polls all of its composed resources. If any composed resource credential secret keys have changed (and those secret keys are mapped to the composite's connection details) they will be propagated to the composite's connection secret.
4. On every reconcile the claim controller propagates its composite resource's connection secret.

I believe the culprit here is the fact that we don't queue claim reconciles every 30-60 seconds, because the controller watches both the claim and composite and thus assumes a reconcile will be queued if anything changes about those resources. Unfortunately we don't watch the composite's connection secret.

There's a few fixes here:

1. Configure the claim reconciler to speculatively requeue every 60 seconds. This would result in an almost 3 minute propagation delay from the AWS API to the claim secret in the pathological case; i.e. at each propagation step we happen to not requeue until almost a minute after the previous step's credentials changed.
2. Annotate the composite resource's connection secret with a reference to the composite resource, and have the claim reconciler watch for secrets with such an annotation. This would reduce the pathological propagation delay to 2 minutes, because we'd instantly propagate changes to the composite resource secret.
3. Take it one step further and annotate the connection secrets of all composed resources such that the composite resource could watch them. This would reduce the pathological propagation delay to a minute.

I'm inclined to compromise and go with option 2 for the time being.

[muvaf]

Annotate the composite resource's connection secret with a reference to the composite resource, and have the claim reconciler watch for secrets with such an annotation. This would reduce the pathological propagation delay to 2 minutes, because we'd instantly propagate changes to the composite resource secret.

@negz are we able to use the existing OwnerReference to know that what this Secret belongs to? Annotation works, too, but it seems the information is already there.

[negz]

@muvaf I misspoke in that comment; what we actually need is to resolve the composite connection secret to a claim that should have a reconcile queued when it changes. So we either need to annotate composite resource secrets with their claim's details (at least namespace and name, maybe GVK for informational purposes) or use an EventHandler that uses a client to get the secret's composite resource via its controller ref, then enqueues a request for the composite's claim. Neither approach feels super elegant to me, but I'm leaning toward using a controller ref approach close to what you suggest.

[muvaf]

I think I confused what's available in the owner ref. I thought it's what you were going to annotate, i.e. claim name/namespace, but it doesn't seem to be the case. It's the composite name, which is only indirectly useful to us as you mentioned. So, I don't really think what I proposed is a simpler/better approach than annotation.