Kubernetes-client: creatOrReplace() should ignore if identical to last submitted:

[chirlo]

Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.3", GitCommit:"721bfa751924da8d1680787490c54b9179b1fed0", GitTreeState:"clean", BuildDate:"2019-02-01T20:00:57Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}

I've had this problem with a PersistentVolumeClaim but it might apply to others. I initally createOrReplace the following resource from a yaml file:

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 20Gi

which succeeds and everything runs fine. Now If i try it again I right in the following error:

  io.fabric8.kubernetes.client.KubernetesClientException: ...  Received status: Status(apiVersion=v1, code=422, details=StatusDetails(causes=[StatusCause(field=spec, message=Forbidden: is immutable after creation except resources.requests for bound claims, reason=FieldValueForbidden, additionalProperties={})], group=null, kind=PersistentVolumeClaim, name=postgres, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=PersistentVolumeClaim "postgres" is invalid: spec: Forbidden: is immutable after creation except resources.requests for bound claims, metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=Invalid, status=Failure, additionalProperties={}).

This I understand is normal following a PUT, the issue is that it should haven't been PUT as it is the same resource that already exists.
I tracked the problem down to

https://github.com/fabric8io/kubernetes-client/blob/22bc59dc1caa08bb8e0a8a7850273a24802e380d/kubernetes-client/src/main/java/io/fabric8/kubernetes/client/dsl/internal/NamespaceVisitFromServerGetWatchDeleteRecreateWaitApplicableImpl.java#L144

The ResourceCompare.equals compares the item to be created with the reloaded item, which in this case has been added some spec entries by the K8s cluster:

volumeMode=Filesystem
volumeName=my-pv

The equals then returns false because of this added entries, and the item is then pushed although it should have been ignored.

I've also observed the reloaded item includes a metadata.annotations with the last applied configuration in JSON form:

"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"v1\",\"kind\":\"PersistentVolumeClaim\",\"metadata\":{\"annotations\":{},\"name\":\"postgres\",\"namespace\":\"default\"},\"spec\":{\"accessModes\":[\"ReadWriteOnce\"],\"resources\":{\"requests\":{\"storage\":\"20Gi\"}}}}\n"

Without any knowledge of Kubernetes internals, I'd guess that that JSON string is what needs to be compared in the 'equals' method against the item to be deployed.
This seem to be confirmed here: How apply calculates differences and merges changes

I can try to go for a PR if you think this is the way to go

[nbn]

Hi, I may be facing the same issues, having a service with a spec.type=ClusterIP. I can create it, but subsequently it fails with immutable field spec.clusterIP, although the spec is unaltered and does not specify the clusterIP.

at io.fabric8.kubernetes.client.handlers.ServiceHandler.replace(ServiceHandler.java:39) at io.fabric8.kubernetes.client.dsl.internal.NamespaceVisitFromServerGetWatchDeleteRecreateWaitApplicableListImpl.createOrReplace(NamespaceVisitFromServerGetWatchDeleteRecreateWaitApplicableListImpl.java:271)

Not must else information to provide, other than I use the 4.10.0 version of the kubernetes-client and openshift-client jar.

[rohanKanojia]

@nbn : spec.clusterIP gets populated on the first time you create it by kubernetes apiserver. Next createOrReplace() call goes as a PUT which gets rejected. Am I missing something here?

[nbn]

@rohanKanojia I just ran a debug session. My input does not have a spec.clusterIP set explicitly, so yes it gets created the first time. Second time, it Use ResourceCompare.equals(r,meta) and I can see r having a spec.IP, while meta's is ´null´. Then it calls replace, which funny enough only seems to be updating the ´spec.clusterIP` and then it fails.

[rohanKanojia]

I remember I fixed an issue similar to this in https://github.com/fabric8io/kubernetes-client/pull/1931 . Are you saying client patching spec.clusterIP is causing problems?

[nbn]

I can only tell you what I observe. I try to roll the same spec through twice, and the second time it fails. I read a yaml list (Having it as a list) and then simply invoke it with:

client.resourceList(itemsList).inNamespace(aNamespace).createOrReplace()

The service is simple enough, though

apiVersion: v1
kind: Service
metadata:
  labels:
  name: my-service
spec:
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: http
  selector:
    artifact: mypod
  sessionAffinity: None
  type: ClusterIP

And it is an openshift v3.11.170

[rohanKanojia]

@nbn : Will try to reproduce

[rohanKanojia]

@chirlo : Your point seems valid to me. Feel free to create a PR to move this discussion forward

[nbn]

@rohanKanojia Great. Let me know if you need anything from me. I'll be happy to help you.

[longwa]

Any progress on this?

I'm having the same issue applying a YAML file with a PVC defined.

client.load(inputStream).inNamespace(namespace).createOrReplace()

When trying to apply for the second time.

Failure executing: PUT at: https://x.x.x.x/api/v1/namespaces/k8s-test/persistentvolumeclaims/my-pvc. Message: PersistentVolumeClaim "my-pvc" is invalid: spec: Forbidden: is immutable after creation except resources.requests for bound claims. Received status: Status(apiVersion=v1, code=422, details=StatusDetails(causes=[StatusCause(field=spec, message=Forbidden: is immutable after creation except resources.requests for bound claims, reason=FieldValueForbidden, additionalProperties={})], group=null, kind=PersistentVolumeClaim, name=my-pvc, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=PersistentVolumeClaim "my-pvc" is invalid: spec: Forbidden: is immutable after creation except resources.requests for bound claims, metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=Invalid, status=Failure, additionalProperties={}).

If I take the same file and run kubectl apply -f myfile.yml --namespace foo it works fine.

Since createOrReplace has replaced the deprecated apply function, it seems like it should follow the same behavior.

[manusa]

Related to #2292

[rohanKanojia]

@longwa : Hi, I'm not able to see the same behavior of kubectl in case of PVCs. Suppose I have this PVC:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 20Gi

When I create if for the first time, it gets created( I checked with fabric8-client SDK too, it gets created):

~/work/k8-resource-yamls : $ kubectl create -f ~/work/k8-resource-yamls/test-pvc.yml 
persistentvolumeclaim/my-pvc created

However, when I try to do kubectl apply -f again, it gives me error:

~/work/k8-resource-yamls : $ kubectl create -f ~/work/k8-resource-yamls/test-pvc.yml 
Error from server (AlreadyExists): error when creating "/home/rohaan/work/k8-resource-yamls/test-pvc.yml": persistentvolumeclaims "my-pvc" already exists

I think we should drop this ResourceHandler.equals() method which doesn't seem to be comparing objects properly, we should probably leave it to Kubernetes to calculate whether the object has changed or not.

[rohanKanojia]

oh, okay. It's kubectl apply not kubectl create. Sorry for noise.

[longwa]

Hi yes, it’s with apply. My understanding is that the createOrReplace should work like apply. IMO it was more clear with the now deprecated ‘apply’ function as that indicated a direct analog with k8s.

[rohanKanojia]

I think this issue should be closed by https://github.com/fabric8io/kubernetes-client/pull/2372

I wrote a test for this case:
https://github.com/fabric8io/kubernetes-client/blob/14cae945ee20ce1ac980d492e53d388aa93eb371/kubernetes-tests/src/test/java/io/fabric8/kubernetes/client/mock/PersistentVolumeClaimTest.java#L180-L225

[rohanKanojia]

I'm closing this, for now. But feel free to reopen in case it's not working as expected.

[chongyangxue]

Does this issue solved in latest version of kubernetes-client?
I'm having the same issue in 4.11.1.
@rohanKanojia

[rohanKanojia]

We had reverted it's fix due to this regression #2445 . Now we're discussing strategies on how to achieve this in #2454 . Please share your thoughts on #2454

[stale[bot]]

This issue has been automatically marked as stale because it has not had any activity since 90 days. It will be closed if no further activity occurs within 7 days. Thank you for your contributions!