Kubernetes-client: JsonMappingException "No resource type found for: tekton.dev/v1alpha1#PipelineList" in intellij plugins

[adietish]

Hi guys,

When retrieving the list of pipelines I hit an exception when PipelineList is being deserialized in jackson:

client.pipelines().inNamespace(namespace).list().getItems()

results in:

com.fasterxml.jackson.databind.JsonMappingException: No resource type found for:tekton.dev/v1alpha1#PipelineList"

io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:64) 
 io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:53)
 io.fabric8.kubernetes.client.utils.Serialization.unmarshal(Serialization.java:248)
 io.fabric8.kubernetes.client.utils.Serialization.unmarshal(Serialization.java:199)
 io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:474)
 io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:430)
 io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:412)
 io.fabric8.kubernetes.client.dsl.base.BaseOperation.listRequestHelper(BaseOperation.java:151)
 io.fabric8.kubernetes.client.dsl.base.BaseOperation.list(BaseOperation.java:621)
 io.fabric8.kubernetes.client.dsl.base.BaseOperation.list(BaseOperation.java:70)

For the background:
I'm writing an IntelliJ plugin to display, operate, etc. kubernetes resources in Intellij at https://github.com/redhat-developer/intellij-kubernetes

I'm trying to separate kubernetes- and tekton-pieces into 2 different plugins:
kubernetes base plugin: https://github.com/redhat-developer/intellij-kubernetes/pull/26
tekton extension plugin: https://github.com/redhat-developer/intellij-tekton/pull/36

The kubernetes (base) plugin holds all kubernetes & jackson jars:

• "io.fabric8:kubernetes-client:4.9.0",
• "io.fabric8:kubernetes-model:4.9.0",
• "io.fabric8:kubernetes-model-common:4.9.0",
• "io.fabric8:openshift-client:4.9.0"
• "com.fasterxml.jackson.core:jackson-annotations"
• "com.fasterxml.jackson.core:jackson-core"
• "com.fasterxml.jackson.corejackson-databind"
• "com.fasterxml.jackson.module:jackson-module-jaxb-annotations"

The tekton (extension) plugin holds the tekton jars:

• "io.fabric8:tekton-client:4.9.0"
• "io.fabric8:tekton-model:4.9.0"

IntelliJ plugins each have their own classloader. The kubernetes (base) plugin doesn't "see" the classes in the tekton (extension) plugin. The tekton plugin on the other hand depends on the kubernetes plugin and can thus access all it's classes

In the tekton (extension) plugin I manually register the custom type for the tekton pipeline since I found out that ServiceLoader wouldn't find the TektonResourceMappingProvider (which would do this automatically in a non-segregated classpath:

KubernetesDeserializer.registerCustomKind("tekton.dev/v1alpha1", "Pipeline", Pipeline.class)

I'm obviously missing bits for this to work in intellij plugins (which have different classloaders for each plugin).

I'd highly appreciate some help here. Thanks in advance!

[adietish]

Manually registering the PipelineList as follows

KubernetesDeserializer.registerCustomKind("tekton.dev/v1alpha1", "PipelineList", PipelineList.class);

results in a stackoverflow and obviously is not the right thing to do:

java.lang.StackOverflowError
    at com.fasterxml.jackson.databind.deser.std.BaseNodeDeserializer.deserializeObject(JsonNodeDeserializer.java:249)
    at com.fasterxml.jackson.databind.deser.std.JsonNodeDeserializer.deserialize(JsonNodeDeserializer.java:68)
    at com.fasterxml.jackson.databind.deser.std.JsonNodeDeserializer.deserialize(JsonNodeDeserializer.java:15)
    at com.fasterxml.jackson.databind.ObjectMapper._readValue(ObjectMapper.java:4189)
    at com.fasterxml.jackson.databind.ObjectMapper.readTree(ObjectMapper.java:2586)
    at com.fasterxml.jackson.core.JsonParser.readValueAsTree(JsonParser.java:1818)
    at io.fabric8.kubernetes.internal.KubernetesDeserializer.deserialize(KubernetesDeserializer.java:81)
    at io.fabric8.kubernetes.internal.KubernetesDeserializer.deserialize(KubernetesDeserializer.java:41)
    at com.fasterxml.jackson.databind.ObjectMapper._readValue(ObjectMapper.java:4189)
    at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2476)
    at com.fasterxml.jackson.databind.ObjectMapper.treeToValue(ObjectMapper.java:2929)
    at io.fabric8.kubernetes.internal.KubernetesDeserializer.fromObjectNode(KubernetesDeserializer.java:109)
    at io.fabric8.kubernetes.internal.KubernetesDeserializer.deserialize(KubernetesDeserializer.java:83)
    at io.fabric8.kubernetes.internal.KubernetesDeserializer.deserialize(KubernetesDeserializer.java:41)
    at com.fasterxml.jackson.databind.ObjectMapper._readValue(ObjectMapper.java:4189)
    at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2476)
    at com.fasterxml.jackson.databind.ObjectMapper.treeToValue(ObjectMapper.java:2929)
    at io.fabric8.kubernetes.internal.KubernetesDeserializer.fromObjectNode(KubernetesDeserializer.java:109)
    at io.fabric8.kubernetes.internal.KubernetesDeserializer.deserialize(KubernetesDeserializer.java:83)
    at io.fabric8.kubernetes.internal.KubernetesDeserializer.deserialize(KubernetesDeserializer.java:41)

[adietish]

Interestingly enough the InternalResourceMappingProvider isn't found either

[adietish]

I suspect that the extensions are not found because the lookup for KubernetesResourceMappingProviders is done in the static initializer for KubernetesDeserializer. The lookup seems to be done far too early. I suspect this because the lookup for adapters to the KubernetesClient (ex. OpenShiftClient) works and is triggered lazily upon calling Adapters.get(type)

[rohanKanojia]

@adietish : Thanks for your bug report. Will try to reproduce your issue. We have recently upgraded tekton model to v0.11.0 with v1beta1 apiGroup support. Could you please try out our SNAPSHOT version and see if your problem is reproducible or not?

Actually, I'm a bit surprised to see your error since we have a test which validates `pipelines().list() call: https://github.com/fabric8io/kubernetes-client/blob/555c3790b02b910d62f72e95350dd563c97359a8/extensions/tekton/tests/src/test/java/io/fabric8/tekton/test/crud/PipelineCrudTest.java#L60

[adietish]

@rohanKanojia thanks for chiming in so quickly!
Sure I'll try the updated, snapshot artifact.
I'm very sure that this bug is specific to my setup where I have 2 intelliJ plugins with segregated classpaths (a base for kubernetes which doesnt see the tekton model classes and an extension plugin for tekton which does see the kubernetes & tekton model classes). I know that things work fine in a setup with a flat classpath.
My current guess is that the kubernetes deserializer (which is in the kubernetes plugin) doesnt see the tekton mapping bcs it is doing the lookup for mappings in it's static initializer. I thus guess that if the lookup for mappers was triggered in my tekton plugin (ex. analogous to how the lookup is done for adapters) things could work. Makes sense?
I'll try a validate this assumption in a PR.

One thing I dont fully get is what mapping enables the PipelineList mapping (and all other list mapping like ex. KubernetesResourceList). Is it the List mapping ins InternalResourceMappingProvider?

[rohanKanojia]

Umm, Not sure. All tekton related mappings are here: https://github.com/fabric8io/kubernetes-client/blob/master/extensions/tekton/model/src/main/java/io/fabric8/tekton/TektonResourceMappingProvider.java

But I don't see List objects there. Does adding List objects in this resolves your issue?

[adietish]

@rohanKanojia that's exactly what lead me to this question: I dont see any of the KubernetesResourceMapping providers (that are registered via ServiceLoader and looked up in KubernetesDeserializer providing list types. The only one that does is the InternalResourceMappingProvider. I'm thus wondering if this mapper is how all list types (including the PipelineList) are mapped?

[rohanKanojia]

Hmm, I think you're right. Maybe @iocanel can help you out here since he wrote these extensions and knows all about them.

[adietish]

@rohanKanojia I tried manually registering the mappings in InternalResourceMappingProvider and still hit the same error:

KubernetesDeserializer.registerCustomKind("List", KubernetesList.class);
KubernetesDeserializer.registerCustomKind("v1#List", KubernetesList.class);
KubernetesDeserializer.registerCustomKind("tekton.dev/v1alpha1", "Pipeline", Pipeline.class);

Digging into it I see KubernetesDeserializer trying to instantiate PipelineList in #getInternalTypeForName trying to instantiate PipelineList in the internal, hard-coded packages

"io.fabric8.kubernetes.api.model.", 
"io.fabric8.kubernetes.api.model.admissionregistration.", 
"io.fabric8.kubernetes.api.model.apiextensions.",
"io.fabric8.kubernetes.api.model.apps.",
"io.fabric8.kubernetes.api.model.authentication.",
"io.fabric8.kubernetes.api.model.authorization.",
"io.fabric8.kubernetes.api.model.batch.",
"io.fabric8.kubernetes.api.model.extensions.",
"io.fabric8.kubernetes.api.model.networking.",
"io.fabric8.kubernetes.api.model.policy.",
"io.fabric8.kubernetes.api.model.rbac.",
"io.fabric8.kubernetes.api.model.storage.",
"io.fabric8.openshift.api.model."

Of course this fails bcs PipeLineList is in the tekton adapter and thus it's package is io.fabric8.tekton.pipeline.v1alpha1

[adietish]

@iocanel could you please enlighten me on how list types (especially PipelineList) are registered? Thanks in advance!

[iocanel]

We implement a ResourceMappingsProvider per extension. These classes are
responsible for mapping API group and version to classes.

On Mon, Apr 6, 2020, 15:30 Andre Dietisheim [e-mail]
wrote:

@iocanel https://github.com/iocanel could you please enlighten me on
how list types (especially PipelineList) are registered? Thanks in advance!

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/fabric8io/kubernetes-client/issues/2109#issuecomment-609764785,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AADCEWFC7CJ6RBEB6RKFH53RLHDPXANCNFSM4L4NW4RA
.

[adietish]

Hi @iocanel

thanks for chiming in!

We implement a ResourceMappingsProvider per extension. These classes are responsible for mapping API group and version to classes.

That's the part i already know about. My question is about the mapping of the different list types, especially PipelineList which I run into troubles with.
The different ResourceMappingProviders dont have list types - ex. TektonResourceMappingProvider doesnt map PipelineList:

        this.mappings.put("tekton.dev/v1alpha1#PipelineResource", PipelineResource.class);
        this.mappings.put("tekton.dev/v1alpha1#Condition", Condition.class);
        this.mappings.put("tekton.dev/v1beta1#Pipeline", Pipeline.class);
        this.mappings.put("tekton.dev/v1beta1#PipelineRun", PipelineRun.class);
        this.mappings.put("tekton.dev/v1beta1#Task", Task.class);
        this.mappings.put("tekton.dev/v1beta1#TaskRun", TaskRun.class);
        this.mappings.put("tekton.dev/v1beta1#TaskRef", TaskRef.class);
        this.mappings.put("tekton.dev/v1beta1#ClusterTask", ClusterTask.class);
        this.mappings.put("tekton.dev/v1beta1#SidecarState", SidecarState.class);

The only provider that has is InternalResourceMappingProvider

        mappings.put("List", KubernetesList.class);
        mappings.put("v1#List", KubernetesList.class);

How is PipelineList mapped?

[adietish]

I traced things down and found out that list types TaskList, PipelineList etc. are created without mapping via classpath lookup in jackson's TypeFactory. Given that IntelliJ doesn't have a flat but a classpath that's segregated across plugins makes this fail when separating the (base) kubernetes model from it's tekton extension into 2 different plugins.
I'll try to figure out how to solve this in jackson.

[adietish]

Hi @iocanel

I managed to have things working by setting the tekton-plugin classloader in the jackson TypeFactory. I consider this like a dirty workaround.
The better solution imho is to have the TektonResourceMappingProvider register PipelineList, TaskList etc.
As pointed out above, when doing this, I end up with a stackoverflow. It is obvious that kubernetes-client wasn't designed to support this approach. I see this confirmed by the fact that no KubernetesResourceMappingProvider provides list types and tracing showed me that those are instantiated via classpath lookup (which doesnt work in my IJ plugin setup, as pointed out above).
I'd appreciate if you could explain the design and why this is how things are implemented so that I can provide a fix that you'd consider valid.

Thx, André

[iocanel]

@adietish: I don't remember why we don't add the List types. Most probably, because we didn't need to.

How is this possible?
For core types there is a safety net, so we wouldn't really see the issue there.

So, I propose to also register the List types and try to fix the SO bug.

[adietish]

@iocanel, @rohanKanojia: as far as I can see so far KubernetesDeserializer is not able to decode a KubernetesResource when it has further nested KubernetesResource(s). This ends up in the endless loop I reported above.
This does not happen in a setup with a flat classpath bcs all types with nested resources (ex. TaskList, PipelineList, NamespaceList, WatchEvent etc.) are deserialized by BeanDeserializer. There's therefore no recursive calling of the KubernetesDeserializer happening.

My suggestion where list types would get deserialized by KubernetesDeserializer thus causes these recursive calls.
To fix this I'll add support for this recursion. I'll either implement the missing pieces or subclass BeanDeserializer.

Makes sense?

[iocanel]

@adietish: I don't 100% get the issue. Feel free to implement the solution and we can discuss the details on the PR.

[adietish]

When trying to find out why the recursion happens I found a weird thing in deserialization that is causing my indefinite recursion when deserializing. You might maybe have an explanation for the following:
I am deserializing the class Task in a standalone application and in intelliJ. Interestingly the annotations that are found differ for both setups:

• in a standalone application:
  when Task is being deserialized, AnnotatedClassResolver#resolveClassAnnotations finds

@JsonDeserialize(
    using = None.class
)

• in intellij, where kubernetes and tekton are in separate plugins (with separate classlaoders), 2 annotations are found:

@JsonDeserialize(
    using = None.class
)

@JsonDeserialize(
    using = KubernetesDeserializer.class
)

KubernetesDeserializer is annotated in the superclass for Task: KubernetesResource. That explains where it's taken from. I dont understand though, how in standalone only None is found, while in intellij, both are found. Any idea?

[adietish]

Turns out that the classloader segregation in intellij is causing this:
The annotation in Task should override the annotation in KubernetesResource. The annotations collector in jackson-databind (AnnotationCollector#addOrOverride) is storing the @JsonDeserialize annotations in Task and its superclass KubernetesResource in a map and using the @JsonDeserialize annotation instance as key. In intellij the annotations are unfortunately loaded via different classloaders (a bug in intellij) and then causes the collector not to grock that the annotation in the subclass should override the annotation in the superclass.

[adietish]

I finally found a solution in intellij and the only changes left for the kubernetes-client that would help me are:

• lazy-loading of KubernetesResourceProviders
• lookup for KubernetesResourceProviders via current Classloader & current Thread classloader

I'll file a PR asap.

[rohanKanojia]

@adietish : cool, looking forward to your PR :rocket: