Kubebuilder: Support for old object access

[crimsonfaith91]

Kubebuilder should support access to old objects. Currently, the Reconcile loop only provides a key. There is no easy way to access the old objects after update operations. A workaround is to save the object state periodically via global variables, but there should be a better way to access the old objects.

@pwittrock
/cc @liyinan926

[liyinan926]

I looked into the kubebuilder code a bit. It appears to me that the ResourceEventHandlerFuncs used internally simply enqueues for all three types of events. Particularly, for updates, the updated objects are enqueued, while the old ones are simply ignored. Also because Reconcile is called on objects retrieved from the work queue, there's no way for users to access the old objects through Reconcile.

Another issue or question is how to differentiate between and react differently to added objects, updated objects, and deleted objects, with only the Reconcile function. Deleted objects can be detected with the DeletionTimestamp. But how can I tell newly added objects from updated ones, through ResourceVersion?

[pwittrock]

This is by design. Why do you need to access old objects? Controllers should be designed as level-based not edge based.

But how can I tell newly added objects from updated ones, through ResourceVersion?

You probably don't want to do this as it probably wouldn't be a self-healing system. e.g. if you do initialization on create only, and then something comes in and deletes the initialized components, it should self heal and recreate those.

[andrewrynhard]

I still think access to the old object should be made available. How would I, for example, update an external resource created by the controller that depends on a field in the resource spec? If a field changes, I want to update the external resource with the new field value.

[bingosummer]

I have the same request.
The scenario is:
I have a controller which is responsible for creating/updating/deleting external resource. These operations normally need some time to complete. So, the controller should re-queue it after sending the creating request to external components. When the controller gets the request again from queue, it should check the state of the external resource.

As https://book-v1.book.kubebuilder.io/basics/what_is_a_controller.html mentioned, the controller should be a level-based API. The Controller should immediately stop rolling out the old values and start the rollout for the new values. But can I assume that the request in the Reconcile function is in order? If not, there may be a problem.
For example, the controller is handling a request whose resourceVersion is 1. At the same time, user is applying a new change in spec and a new request with a new resourceVersion 2 is put into the queue. Then, for some reason, the controller decides to re-queue the request (resourceVersion: 1). Then the controller will get the request (resourceVersion: 2) and the request (resourceVersion: 1). Without old object, I think, the controller can't tell which request is newer.

@pwittrock Can you please help on this?

[DirectXMan12]

I replied on the linked issue.

[DirectXMan12]

For posterity, you should never see things going back in time.

[skonto]

@DirectXMan12 @andrewrynhard's scenario is valid. The solution advertised in other operator implementations is to reconstruct state from the env to compare with the object in cache. The latter is the recommended approach from what I see but may need a lot of work depending on the operator's complexity. I have one such case btw. Knowing that your spec has changed is pretty useful.
Maybe another hack would be to use a hash value at the status level as kubelet does: https://github.com/kubernetes/kubernetes/issues/53644 which has some corner cases.

[shenlanse]

I really want access old object. Sometimes, reconcile is a really time-consuming operation. And I only need reconcile on some specific field update. If i can't get old object, i will not distinguish that and have to reconcile on all update event. But the reconcile in my scenario is really time-consuming.

[DirectXMan12]

You can accomplish the "I only want to enqueue if a specific field changes" via a predicate (https://godoc.org/sigs.k8s.io/controller-runtime/pkg/predicate). Be careful when doing so, however.