Kubernetes-client: Unable to update status on Custom Resources

[kolorful]

Hello, I was recently work on a CRD, but I was not able to set the Status on the CR. I've tried a couple of things and this is what I find:

• I'm able to create and patch the Spec
• I'm able to get Status (via list) set by my controller written in go
• I'm not able to create a Resource with Status or patch the Resource with Status via my Java client.

I'm clueless right now, and any help would be awesome, thanks!

Anacron.java

public class Anacron extends CustomResource {
    private AnacronSpec spec;
    private AnacronStatus status;
    public AnacronSpec getSpec() { return spec; }
    public void setSpec(final AnacronSpec spec) { this.spec = spec; }
    public AnacronStatus getStatus() { return status; }
    public void setStatus(final AnacronStatus status) { this.status = status; }
}

AnacronList.java

public class AnacronList extends CustomResourceList<Anacron> {
}

AnacronSpec.java

@JsonPropertyOrder({"schedule", "jobTemplate", "concurrencyPolicy", "jobTemplate"})
@JsonDeserialize(
        using = JsonDeserializer.None.class
)
public class AnacronSpec implements KubernetesResource {
    @JsonProperty(value = "schedule")
    private String schedule;
    @JsonProperty(value = "concurrencyPolicy")
    private String concurrencyPolicy;
    @JsonProperty(value = "startingDeadlineSeconds")
    private int startingDeadlineSeconds;
    @JsonProperty(value = "jobTemplate")
    private JobTemplateSpec jobTemplate;

    public AnacronSpec() {
    }

    public String getConcurrencyPolicy() { return concurrencyPolicy; }
    public void setConcurrencyPolicy(final String concurrencyPolicy) { this.concurrencyPolicy = concurrencyPolicy; }
    public int getStartingDeadlineSeconds() { return startingDeadlineSeconds; }
    public void setStartingDeadlineSeconds(final int startingDeadlineSeconds) { this.startingDeadlineSeconds = startingDeadlineSeconds; }
}

AnacronStatus.java

@JsonInclude(JsonInclude.Include.NON_NULL)
@JsonPropertyOrder({"apiVersion", "kind", "metadata", "active", "lastScheduleTime"})
@JsonDeserialize(
        using = JsonDeserializer.None.class
)
public class AnacronStatus implements KubernetesResource {
    @JsonProperty(value = "active")
    private List<ObjectReference> active = new ArrayList();
    @JsonProperty(value = "lastScheduleTime")
    private String lastScheduleTime;
    public AnacronStatus() {
    }

    public List<ObjectReference> getActive() { return active; }
    public void setActive(final List<ObjectReference> active) { this.active = active; }
    public String getLastScheduleTime() { return lastScheduleTime; }
    public void setLastScheduleTime(final String lastScheduleTime) { this.lastScheduleTime = lastScheduleTime; }
}

DoneableAnacron.java

public class DoneableAnacron extends CustomResourceDoneable<Anacron> {
    public DoneableAnacron(final Anacron resource, final Function function) {
        super(resource, function);
    }
}

Client Initialization

private static final CustomResourceDefinition ANACRON_CRD = new CustomResourceDefinitionBuilder()
            .withApiVersion("apiextensions.k8s.io/v1beta1")
            .withNewMetadata().withName(CRD_NAME).endMetadata()
            .withNewSpec().withGroup(CRD_GROUP).withVersion(CRD_VERSION).withScope(CRD_SCOPE)
            .withNewNames().withKind(CRD_KIND).withSingular(CRD_SINGULAR).withPlural(CRD_PLURAL).withShortNames(CRD_SHORT_NAMES).endNames().endSpec()
            .build();

final MixedOperation<Anacron, AnacronList, DoneableAnacron, Resource<Anacron, DoneableAnacron>> anacronClient = kubernetesClient.customResources(ANACRON_CRD, Anacron.class, AnacronList.class, DoneableAnacron.class);

[rohanKanojia]

@kolorful : Have u tried our raw custom resource api? It's somewhat less typed and allows you to get, delete, create and edit custom resources without pojos. Here is an example:

https://github.com/fabric8io/kubernetes-client/blob/c78cc4d6de12309b477b0f3295955c3fd3cd2a26/kubernetes-examples/src/main/java/io/fabric8/kubernetes/examples/RawCustomResourceExample.java#L36-L50

[rohanKanojia]

Are you facing problems with creating Custom Resource Definition or Custom Resource?

[kolorful]

I haven't tried creating the CRD using this client, I created the CRD via kubectl with a yaml file I wrote.

[rohanKanojia]

So you're facing problems with creating custom resource. Would you like to try this new raw custom resource api which was added recently and see if it helps. It's available in v4.2.2

[kolorful]

I'm able to create the custom resource, but cannot update its status via patch. I'll give the new API a try

[kolorful]

I tried the latest endpoint and I converted my pojo to string using MAPPER.writeValueAsString(anacron). The json seems to be correct.

json:

{  
   "kind":"Anacron",
   "metadata":{ // hidden },
   "spec":{  
      "schedule":"* */1 * * *",
      "jobTemplate":{  
         "metadata":{ // hidden },
         "spec":{ // hidden }
      },
      "concurrencyPolicy":"Forbid",
      "startingDeadlineSeconds":100
   },
   "status":{  
      "active":[  
      ],
      "lastScheduleTime":"2019-05-23T16:58:00Z"
   }
}

But I got an error when trying to create the CR, and it didn't give much details

May 23 14:21:47 Caused by: java.lang.IllegalStateException: Internal Server Error
May 23 14:21:47: at io.fabric8.kubernetes.client.dsl.internal.RawCustomResourceOperationsImpl.makeCall(RawCustomResourceOperationsImpl.java:150)
May 23 14:21:47: at io.fabric8.kubernetes.client.dsl.internal.RawCustomResourceOperationsImpl.validateAndSubmitRequest(RawCustomResourceOperationsImpl.java:159)
May 23 14:21:47: at io.fabric8.kubernetes.client.dsl.internal.RawCustomResourceOperationsImpl.create(RawCustomResourceOperationsImpl.java:55)
May 23 14:21:47 dev-mcontrol1 runCronjob.sh[20719]: ... 12 more

Hmm, I tried Cronjob instead of my CR, looks like it doesn't work either.

I went back and checked client-go and it seems linke only the UpdateStatus endpoint works.

func (c *cronJobs) UpdateStatus(cronJob *v1beta1.CronJob) (result *v1beta1.CronJob, err error) {
    result = &v1beta1.CronJob{}
    err = c.client.Put().
        Namespace(c.ns).
        Resource("cronjobs").
        Name(cronJob.Name).
        SubResource("status").
        Body(cronJob).
        Do().
        Into(result)
    return
}

[adam-sandor]

We didn't have problems with setting status on our CR using the typed API. Status isn't a special thing in the Kubernetes API so the same rules should apply to it than any other field. I do see that our classes look quite different from your CR mapping classes:

• We don't specify the JSON mapping, just leave it to defaults.
• Our class mapping Status doesn't implement KubernetesResource.
  I'm guessing the second point could be the cause of the issue. Status is not something that has it's own metadata.

public class OpsRepo extends CustomResource {
    private OpsRepoSpec spec;
    private OpsRepoStatus status;

    public OpsRepoSpec getSpec() {
        return spec;
    }

    public void setSpec(OpsRepoSpec spec) {
        this.spec = spec;
    }

    public OpsRepoStatus getStatus() {
        return status;
    }

    public void setStatus(OpsRepoStatus status) {
        this.status = status;
    }
}

public class OpsRepoStatus {

    private OpsRepoState state;
    private String repoUrl;

    public OpsRepoState getState() {
        return state;
    }

    public void setState(OpsRepoState state) {
        this.state = state;
    }

    public String getRepoUrl() {
        return repoUrl;
    }

    public void setRepoUrl(String repoUrl) {
        this.repoUrl = repoUrl;
    }
}

[kolorful]

@adam-sandor thanks for your input. I tried that and doesn't work either. I then did a round of tests and I think the problem lies in my CRD yaml file.

The only difference appear to be the working one doesn't have:

  subresources:
    status: {}

Once I have this set, my status patching no longer works.

According to https://kubernetes.io/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definitions/#status-subresource

This seems to be an expected behavior, because PUT/POST/PATCH requests to the custom resource ignore changes to the status stanza.
What I want is PUT requests to the /status subresource take a custom resource object and ignore changes to anything except the status stanza., but I can't find this UpdateStatus function in fabric8io/kubernetes-client, am I wrong? @rohanKanojia I don't see the new api has this functionally either, and I'm confused by what does edit do.

[ampie]

If I can add to this. According to https://blog.openshift.com/kubernetes-custom-resources-grow-up-in-v1-10/, one of the key benefits of defining the status property as the standard '/status' subresource is that updates to the status won't increment the metadata.generation property. This property will only get updated when the spec gets updated. This is extremely useful for cases where the custom controller needs to update the status without triggering an unintended sync.

@adam-sandor I would therefor say that perhaps the /status subresource, as well as the /scale subresource is something special in K8S, and from the above document it would seem that they even have their own resource URL's where they can be updated, e.g. "/apis/example.com/v1/namespaces/default/databases/mysql/status". From what I can see it would seem that the Fabric8 Java client doesn't support PUTs to the url of the subresource.

[adam-sandor]

Ah that is very interesting @ampie ! @csviri check this out ☝️

[saturnism]

https://github.com/fabric8io/kubernetes-client/issues/417 seems related, i'm also looking for this :)

[stale[bot]]

This issue has been automatically marked as stale because it has not had any activity since 90 days. It will be closed if no further activity occurs within 7 days. Thank you for your contributions!

[sbaier1]

this is still relevant, is it not? there still is no proper subresource support

[sbaier1]

i am currently using a quick and dirty workaround for this:

            final String statusUri = URLUtils.join(client.getMasterUrl().toString(), "apis", "myclass.com", "v1", "namespaces",
                    namespace, "my-objects", name, "status");
            infoClass.setStatus(status);
            final RequestBody requestBody = RequestBody.create(MediaType.parse("application/json"), mapper.writeValueAsBytes(infoClass));
            baseClient.getHttpClient().newCall(new Request.Builder()
                    .method("PUT", requestBody)
                    .url(statusUri)
                    .build())
                    .execute()
                    .close();

and then ignoring watcher MODIFY operations when the generation is unchanged.
Still think this should be properly implemented...

[rohanKanojia]

Hmm, I see. Let me try to prioritize this.

[kolorful]

Thank you!

[novakov-alexey]

Trying to use this feature.
Could someone help me to understand status update better? My question is: does status update trigger Modify event in watcher once CR status is updated or such event should not occur any more using new kubernetes-client version? Thanks in advance.

[kolorful]

@novakov-alexey I believe updating the Status of an CR or in general updating a subresouce of any type of resources won't trigger a Modify event in watcher, because the whole point is not needing to update the generation of the resource.
I could be wrong and one could simply test it locally.

[novakov-alexey]

@kolorful it corresponds to my understanding as well. However, I am getting Modify event on live Kubernetes (not mock server) with the same generation number, when I am updating the CR Status, which is strange. 😞 I will try to prepare abstract reproducible code example.

[rohanKanojia]

While implementation i just followed https://kubernetes.io/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definitions/#subresources i would check for the modify event and report back

[novakov-alexey]

@rohanKanojia There is an example where I get Modified event, which I would like to not have:

import com.fasterxml.jackson.databind.DeserializationFeature
import com.fasterxml.jackson.module.scala.DefaultScalaModule
import io.fabric8.kubernetes.api.model.apiextensions.{CustomResourceDefinition, CustomResourceDefinitionBuilder}
import io.fabric8.kubernetes.api.model.{HasMetadata, ObjectMetaBuilder}
import io.fabric8.kubernetes.client._
import io.fabric8.kubernetes.client.dsl.{NonNamespaceOperation, Resource}
import io.fabric8.kubernetes.client.utils.Serialization
import io.fabric8.kubernetes.internal.KubernetesDeserializer
import io.fabric8.kubernetes.api.builder.Function
import io.fabric8.kubernetes.client.CustomResourceDoneable
import com.fasterxml.jackson.databind.annotation.JsonDeserialize
import io.fabric8.kubernetes.client.CustomResourceList
import io.fabric8.kubernetes.client.CustomResource

class AnyCrDoneable(val resource: AnyCustomResource, val f: Function[AnyCustomResource, AnyCustomResource])
    extends CustomResourceDoneable[AnyCustomResource](resource, f)

@JsonDeserialize(using = classOf[KubernetesDeserializer]) class AnyCrList
    extends CustomResourceList[AnyCustomResource]

class AnyCustomResource extends CustomResource {
  private var spec: AnyRef = _
  private var status: AnyRef = _

  def getSpec: AnyRef = spec

  def setSpec(spec: AnyRef): Unit =
    this.spec = spec

  def getStatus: AnyRef = status

  def setStatus(status: AnyRef): Unit =
    this.status = status

  override def toString: String =
    super.toString + s", spec: $spec, status: $status"
}


def createWatch(
    kerbClient: NonNamespaceOperation[
      AnyCustomResource,
      AnyCrList,
      AnyCrDoneable,
      Resource[AnyCustomResource, AnyCrDoneable]
    ]
  ): Watch = {
    kerbClient.watch(new Watcher[AnyCustomResource]() {
      override def eventReceived(action: Watcher.Action, resource: AnyCustomResource): Unit =
        println(s"received: $action for $resource")

      override def onClose(cause: KubernetesClientException): Unit =
        println(s"watch is closed, $cause")
    })
  }

private def newCr(crd: CustomResourceDefinition, spec: AnyRef) = {
    val anyCr = new AnyCustomResource
    anyCr.setKind(crd.getSpec.getNames.getKind)
    anyCr.setApiVersion(s"${crd.getSpec.getGroup}/${crd.getSpec.getVersion}")
    anyCr.setMetadata(
      new ObjectMetaBuilder()
        .withName("test-kerb")
        .build()
    )
    anyCr.setSpec(spec)
    anyCr
  }

    Serialization.jsonMapper().registerModule(DefaultScalaModule)
    Serialization.jsonMapper.configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false)

    val prefix = "io.github.novakov-alexey"
    val version = "v1"
    val apiVersion = prefix + "/" + version
    val kind = classOf[Kerb].getSimpleName
    val plural = "kerbs"

    KubernetesDeserializer.registerCustomKind(apiVersion, kind, classOf[AnyCustomResource])
    KubernetesDeserializer.registerCustomKind(apiVersion, s"${kind}List", classOf[CustomResourceList[_ <: HasMetadata]])

    val client = new DefaultKubernetesClient
    val crd = new CustomResourceDefinitionBuilder()
      .withApiVersion("apiextensions.k8s.io/v1beta1")
      .withNewMetadata
      .withName(plural + "." + prefix)
      .endMetadata
      .withNewSpec
      .withNewNames
      .withKind(kind)
      .withPlural(plural)
      .endNames
      .withGroup(prefix)
      .withVersion(version)
      .withScope("Namespaced")
      .withPreserveUnknownFields(false)
      .endSpec()
      .build()

    val kerbClient = client
      .customResource(crd, classOf[AnyCustomResource], classOf[AnyCrList], classOf[AnyCrDoneable])
      .inNamespace("test")
    val watch = createWatch(kerbClient)

    val kerb = Kerb("test.realm", Nil, failInTest = true)
    val anyCr = newCr(crd, kerb.asInstanceOf[AnyRef])

    kerbClient.delete(anyCr)
    Thread.sleep(1000)

    val cr = kerbClient.createOrReplace(anyCr)
    Thread.sleep(1000)

    anyCr.setStatus(Status(true).asInstanceOf[AnyRef])
    anyCr.getMetadata.setResourceVersion(cr.getMetadata.getResourceVersion)
    kerbClient.updateStatus(anyCr)
    Thread.sleep(1000)

    watch.close()

Result in the watcher:

received: DELETED for CustomResourceSupport{kind='Kerb', apiVersion='io.github.novakov-alexey/v1', metadata=ObjectMeta(annotations=null, clusterName=null, creationTimestamp=2020-01-28T21:32:58Z, deletionGracePeriodSeconds=0, deletionTimestamp=2020-01-28T21:33:15Z, finalizers=[orphan], generateName=null, generation=2, labels=null, managedFields=[], name=test-kerb, namespace=test, ownerReferences=[], resourceVersion=8586413, selfLink=/apis/io.github.novakov-alexey/v1/namespaces/test/kerbs/test-kerb, uid=c08b468e-4215-11ea-8fc9-be6a45512a41, additionalProperties={})}, spec: Map(failInTest -> true, principals -> List(), realm -> test.realm), status: Map(ready -> true)

received: ADDED for CustomResourceSupport{kind='Kerb', apiVersion='io.github.novakov-alexey/v1', metadata=ObjectMeta(annotations=null, clusterName=null, creationTimestamp=2020-01-28T21:33:16Z, deletionGracePeriodSeconds=null, deletionTimestamp=null, finalizers=[], generateName=null, generation=1, labels=null, managedFields=[], name=test-kerb, namespace=test, ownerReferences=[], resourceVersion=8586414, selfLink=/apis/io.github.novakov-alexey/v1/namespaces/test/kerbs/test-kerb, uid=cb69ba21-4215-11ea-8fc9-be6a45512a41, additionalProperties={})}, spec: Map(failInTest -> true, principals -> List(), realm -> test.realm), status: null

# why MODIFIED is triggered?

received: MODIFIED for CustomResourceSupport{kind='Kerb', apiVersion='io.github.novakov-alexey/v1', metadata=ObjectMeta(annotations=null, clusterName=null, creationTimestamp=2020-01-28T21:33:16Z, deletionGracePeriodSeconds=null, deletionTimestamp=null, finalizers=[], generateName=null, generation=1, labels=null, managedFields=[], name=test-kerb, namespace=test, ownerReferences=[], resourceVersion=8586417, selfLink=/apis/io.github.novakov-alexey/v1/namespaces/test/kerbs/test-kerb, uid=cb69ba21-4215-11ea-8fc9-be6a45512a41, additionalProperties={})}, spec: Map(failInTest -> true, principals -> List(), realm -> test.realm), status: Map(ready -> true)
watch is closed, null

[novakov-alexey]

@rohanKanojia do you think you could help with above question?

[rohanKanojia]

@novakov-alexey: Hi, I also tested this locally on my minikube instance with Kubernetes 1.17.0 and looks like you're right. I'm also seeing Modified event being triggered when I do status update. I think this is something handled by kubernetes and Fabric8 client doesn't play any role in it. We just hit k8s server with /status/ endpoint of resource and Kubernetes handles the rest.

[novakov-alexey]

@rohanKanojia thank you for confirmation. I see that even your Kubernetes version is newer. I also tend to think that this behaviour is up to Kubernetes.