Kubernetes-client: Custom resources cannot be created and deleted with mock server

[eminn]

Creation of custom resources are silently failing whilst doing metadata extraction. And it results the CRD being saved with missing/empty attributes.

Imagine we do have following custom resource

{
  "apiVersion": "appmesh.k8s.aws/v1beta2",
  "kind": "VirtualNode",
  "metadata": {
    "name": "xxxx",
    "namespace": "xxxx"
  },
  "spec": {
    "podSelector": {
      "matchLabels": {
        "app": "xxxx"
      }
    },
    "listeners": [
      {
        "portMapping": {
          "port": 80,
          "protocol": "http"
        }
      }
    ],
    "serviceDiscovery": {
      "dns": {
        "hostname": "xxxx"
      }
    }
  }
}

The following test fails:

  @Test
  void shouldHandleRawCrdsWithSpec() {
    KubernetesCrudAttributesExtractor extractor = new KubernetesCrudAttributesExtractor();
    String resource = "{\"apiVersion\":\"appmesh.k8s.aws/v1beta2\",\"kind\":\"VirtualNode\",\"metadata\":{\"name\":\"xxxx\",\"namespace\":\"xxxx\"},\"spec\":{\"podSelector\":{\"matchLabels\":{\"app\":\"xxxx\"}},\"listeners\":[{\"portMapping\":{\"port\":80,\"protocol\":\"http\"}}],\"serviceDiscovery\":{\"dns\":{\"hostname\":\"xxxx\"}}}}\n";

    AttributeSet attributes = extractor.extract(resource);

    AttributeSet expected = new AttributeSet();
    expected = expected.add(new Attribute("kind", "VirtualNode"));
    expected = expected.add(new Attribute("namespace", "xxxx"));
    expected = expected.add(new Attribute("name", "xxxx"));
    assertTrue(attributes.matches(expected));
  }

The AttributeSet returned from extractor is empty. The reason for that is when it tries to deserialize given custom resource into FallbacHasMetadata during extraction process jackson fails to map given custom resource to FallbacHasMetadata because there are more fields than expected.

com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "spec" (class io.fabric8.kubernetes.client.server.mock.FallbackHasMetadata), not marked as ignorable (3 known properties: "kind", "apiVersion", "metadata"])

So possible solution would be adding @JsonIgnoreProperties(ignoreUnknown = true) to the FallbackHasMetadata class.

[eminn]

When applying the self-suggested solution above, added some tests and I've hit something else. The only difference between KubernetesCrudAttributesExtractor and KubernetesAttributesExtractor is that the former excludes the kind attribute. Is there a specific reason why it omits the kind attribute?. I've traced the changes back to https://github.com/fabric8io/kubernetes-client/pull/2097 and https://github.com/fabric8io/kubernetes-client/issues/2019#issuecomment-597539647 but cannot see how they are related.

I'm happy to create a PR fixing the issue if I could get a feedback on this.

[rohanKanojia]

Hi, Thanks a lot for reporting this issue.

I think you're right. Kind is an important attribute and should be considered while extracting AttributeSet from any Kubernetes resource. I also tried removing extract() method in KubernetesCrudAttributesExtractor to include kind too but it doesn't seem to make any difference(all tests except KubernetesCrudAttributesExtractorTest#shouldHandleResource pass).

[eminn]

I believe that test is testing the wrong behavior. It explicitly expects the extractor to not include Kind attribute. We can fix that test if we are fine with the approach.

[manusa]

Is there a specific reason why it omits the kind attribute?

I remember there was something that prevented CRUD from working with custom resources (can't remember exactly what now).

https://github.com/fabric8io/kubernetes-client/issues/1109#issuecomment-604949318

Let me check it some more, see if I remember.

[manusa]

However, is the issue fixed by adding the @JsonIgnoreProperties(ignoreUnknown = true)?

If it is, why do you want to change the AttributeSet extraction logic?

Especially since there is a test that is specifically verifying that.

[eminn]

Yeah the annotation fixes the issue but to verify that I've written the following test case which fails with the kind attribute being not found. Then I started looking for answers. So wanted to double check whether there is a sound reason for omitting the kind attribute(if so I can just fix my test by omitting the kind field to follow the convention?.) or not.

  @Test
  void shouldHandleRawCrdsWithSpec() {
    KubernetesCrudAttributesExtractor extractor = new KubernetesCrudAttributesExtractor();
    String resource = "{\"apiVersion\":\"appmesh.k8s.aws/v1beta2\",\"kind\":\"VirtualNode\",\"metadata\":{\"name\":\"xxxx\",\"namespace\":\"xxxx\"},\"spec\":{\"podSelector\":{\"matchLabels\":{\"app\":\"xxxx\"}},\"listeners\":[{\"portMapping\":{\"port\":80,\"protocol\":\"http\"}}],\"serviceDiscovery\":{\"dns\":{\"hostname\":\"xxxx\"}}}}\n";

    AttributeSet attributes = extractor.extract(resource);

    AttributeSet expected = new AttributeSet();
    expected = expected.add(new Attribute("kind", "VirtualNode"));
    expected = expected.add(new Attribute("namespace", "xxxx"));
    expected = expected.add(new Attribute("name", "xxxx"));
    assertTrue(attributes.matches(expected));
  }

[manusa]

I'm recovering an old branch I had for this analysis.
I think the problem with CRUD mock server, AttributeExtractor and Kind had to do with the kind being added to the path and breaking stuff.

[manusa]

I'll try to recover that branch and the test.

[manusa]

if so I can just fix my test by omitting the kind field to follow the convention?

I'm mostly sure that the Kind field was omitted specifically to avoid problems when retrieving objects via CRUD it somehow tampered with the path.

In fact your test is the complete opposite of the test Rohan reported failing.

I created #2664 to recover one of the tests that reproduced the issue that was fixed.

I think you can go ahead and create the @JsonIgnoreProperties(ignoreUnknown = true) fix if you want.

[manusa]

I can confirm that reverting KubernetesCrudDispatcher to use KubernetesAttributesExtractor instead of KubernetesCrudAttributesExtractor:

  public KubernetesCrudDispatcher(List<CustomResourceDefinitionContext> crdContexts) {
    this(new KubernetesAttributesExtractor(crdContexts), new KubernetesResponseComposer());
  }

  public KubernetesCrudDispatcher(KubernetesAttributesExtractor attributeExtractor, ResponseComposer responseComposer) {
    super(new Context(Serialization.jsonMapper()), attributeExtractor, responseComposer);
  }

breaks the test added in #2664

doesn't match:

The thing is that the extract method you're referencing is used to extract attributes from Resources only (e.g. handling the POST/Create operation) (AttributeExtractor.fromResource). Other attributes are inferred from the path (maybe kind is a misleading/not totally accurate name for the attribute).
On the other hand, during GET/list operation, the attributes are extracted using the normal extractor AttributeExtractor.fromPath. Just considering the path.
The thing is that the Kind field value in the Resource, is wrong (sometimes) and doesn't translate OK to the real path (see screenshots foo-bar vs. foobar). So this value must be ignored when inferring attributes form the Resource and considered from the POST/create request path.

TL;DR kind is important, and is being considered when dealing with CRDs in the Mockserver. However, since the plural and other information may be wrong or not available to the MockServer, these parts are inferred from the HTTP requests (GET/PUT/POST).