Quarkus: Could not deserialize watch event - exception only when run in native mode

[iamvvra]

My small k8s object watcher app - https://github.com/iamvvra/k8swatcher - throws the below deserialization exception only when the app is run as binary, I never faced this exception running with maven or with the uber jar. I am connecting to an Openshift cluster.

2019-08-22 21:52:06,559 ERROR [io.fab.kub.cli.dsl.int.WatchConnectionManager] (OkHttp https://mycluster.com/...) Could not deserialize watch event: {"type":"ADDED","object":{"kind":"Pod","apiVersion":"v1","metadata":{" ...

com.fasterxml.jackson.databind.JsonMappingException: No resource type found for:v1#Pod
 at [Source: (String)"{"type":"ADDED","object":{"kind":"Pod","apiVersion":"v1","metadata":{"name":"pod1-1-msbsr","generateName":"pod1-1-","namespace":"dev","selfLink":"/api/v1/namespaces/dev/pods/pod1-1-msbsr","uid":"5de8eead-c3e3-11e9-aac9-0050568753b2","resourceVersion":"117436928","creationTimestamp":"2019-08-21T07:14:51Z","labels":{"deployment":"deployment1-1","deploymentconfig":"deployment1-server","name":"app1"},"annotations":{"openshift.io/deployment-con"[truncated 2691 chars]; line: 1, column: 3189] (through reference chain: io.fabric8.kubernetes.api.model.WatchEvent["object"])
    at com.fasterxml.jackson.databind.DeserializationContext.mappingException(DeserializationContext.java:1718)
    at io.fabric8.kubernetes.internal.KubernetesDeserializer.deserialize(KubernetesDeserializer.java:79)
    at io.fabric8.kubernetes.internal.KubernetesDeserializer.deserialize(KubernetesDeserializer.java:33)
    at com.fasterxml.jackson.databind.deser.impl.MethodProperty.deserializeAndSet(MethodProperty.java:129)
    at com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:288)
    at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:151)
    at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4014)
    at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3005)
    at io.fabric8.kubernetes.client.dsl.internal.WatchHTTPManager.readWatchEvent(WatchHTTPManager.java:298)
    at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager$1.onMessage(WatchConnectionManager.java:229)
    at okhttp3.internal.ws.RealWebSocket.onReadMessage(RealWebSocket.java:323)
    at okhttp3.internal.ws.WebSocketReader.readMessageFrame(WebSocketReader.java:219)
    at okhttp3.internal.ws.WebSocketReader.processNextFrame(WebSocketReader.java:105)
    at okhttp3.internal.ws.RealWebSocket.loopReader(RealWebSocket.java:274)
    at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:214)
    at okhttp3.RealCall$AsyncCall.execute(RealCall.java:206)
    at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
    at com.oracle.svm.core.thread.JavaThreads.threadStartRoutine(JavaThreads.java:460)
    at com.oracle.svm.core.posix.thread.PosixJavaThreads.pthreadStartRoutine(PosixJavaThreads.java:193)

Expected behavior
Should deserialize and provide the received value as object

To Reproduce
Steps to reproduce the behavior:

1. Write a program to watch an object, say Pod.
2. Generate a binary, MAC
3. Connect to a Kubernetes cluster - possibly it can be an Openshift.

Configuration

quarkus.kubernetes-client.trust-certs=true
quarkus.log.console.enable=true
quarkus.log.level=INF

Screenshots

Environment (please complete the following information):

• Output of uname -a or ver: Darwin vvra-mac.local 18.7.0 Darwin Kernel Version 18.7.0: Thu Jun 20 18:42:21 PDT 2019; root:xnu-4903.270.47~4/RELEASE_X86_64 x86_64
• Output of java -version: openjdk version "1.8.0_222"
  OpenJDK Runtime Environment (build 1.8.0_222-20190711112007.graal.jdk8u-src-tar-gz-b08)
  OpenJDK 64-Bit GraalVM CE 19.1.1 (build 25.222-b08-jvmci-19.1-b01, mixed mode)
• GraalVM version (if different from Java): same as above
• Quarkus version or git rev: 0.20.0

[gsmet]

@geoand looks like an issue for you :)

[geoand]

Sure thing, I'll take a look!

[geoand]

Can you please try with Quarkus 0.21.1 and see if the problem is still there?
I remember there was some fix made that might be relevant, but I'm not sure what version it was added (although I can certainly dig it up)

Thanks

[geoand]

I'll also check your reproducer tomorrow and see how it's different from what I had used to test watch of pods (which worked)

[geoand]

I was able to reproduce the problem. Looking into it now

[iamvvra]

Can you please try with Quarkus 0.21.1 and see if the problem is still there?
I remember there was some fix made that might be relevant, but I'm not sure what version it was added (although I can certainly dig it up)

Thanks

The issue persists.

[geoand]

I have figured out the root cause of this, unfortunately it's tricky...

So to begin with, we need some context:

When building a native binary with GraalVM, we need to tell GraalVM which classes need reflection (in this cases Pod needs to be registered for reflection since it's used by Jackson). In a lot of cases Quarkus can determine this precicely and will gladly register everything for the developer automatically.
The Kubernetes model posses a unique challenge in that it is very large (400+ classes) so we absolutely don't want to register the whole thing (as this would significantly increase the native binary size and build time - see this for a showcase of what happens in such a case).
For implementations of io.fabric8.kubernetes.client.Watcher, Quarkus tries to register whatever class of the Kubernetes model the watcher handles. What this means practically is that if you write something like:

    client.pods().watch(new Watcher<Pod>() {
      @Override
      public void eventReceived(Action action, Pod pod) {
        System.out.println("Received " + action + " event for Pod " + pod.getMetadata().getName());
      }

      @Override
      public void onClose(KubernetesClientException e) {
      }
    });

Quarkus can successfully determine that Pod needs to be registered for reflection.

In your case you use your custom com.k8swatcher.ResourceWatcher that does bind a specific type to Watcher (nor is this class ever subclassed thus binding it to a specific type), therefore making it impossible for Quarkus to determine what Kubernetes model class is being handled by the io.fabric8.kubernetes.client.Watcher.

I will look into it more and see if there are any clever tricks we can apply, but I am not optimistic.
@gsmet I wonder if in cases like this it would make sense for the extension to provide a configuration property for the user to explicitly configure which classes are needed for reflection, WDYT?

[geoand]

Since frankly I see no elegant way around this problem that could be applied in Quarkus itself, I suggest that you add classes like the following to your application:

public class PodResourceWatcher extends ResourceWatcher<Pod> {

    public PodResourceWatcher(WatchConfig config, NotificationPublisher notificationPublisher) {
        super(config, notificationPublisher);
    }
}

I know that from a purely Java language perspective it shouldn't be needed, however it's the best compromise IMHO in order to efficiently leverage the capabilities (and work around the limitations) of GraalVM.

[geoand]

@gsmet @gastaldi since there is no good way around this issue and an explicit coding pattern is required, I wonder if it should be documented?

[iamvvra]

Since frankly I see no elegant way around this problem that could be applied in Quarkus itself, I suggest that you add classes like the following to your application:

public class PodResourceWatcher extends ResourceWatcher<Pod> {

    public PodResourceWatcher(WatchConfig config, NotificationPublisher notificationPublisher) {
        super(config, notificationPublisher);
    }
}

I know that from a purely Java language perspective it shouldn't be needed, however it's the best compromise IMHO in order to efficiently leverage the capabilities (and work around the limitations) of GraalVM.

Exactly what I had designed it earlier. As the generics had the maturity to infer the types automatically, and also the details I was interested exited in the parent class (io.fabric8.kubernetes.api.model.HasMetadata) I went with this design.

Anyhow, I am now testing implementing just for Pod and see if this work.

Beside, I wish certain things are documented explicitly.

[geoand]

Anyhow, I am now testing implementing just for Pod and see if this work.

It should, I tested it locally and it worked as expected.

Beside, I wish certain things are documented explicitly.

This is why I asked two colleagues what they think about documenting a specific coding pattern :)

[gastaldi]

@geoand +1 to document that. Is there any way to issue a warning during augmentation?

[geoand]

@gastaldi In this specific case, yes. The augmentation phase could warn that no generic type could be bound / matched (I don't know what the proper verb is here) for com.k8swatcher.ResourceWatcher during build time.
That's a very good idea actually. It might be a false warning in some cases, but probably best to warn anyway.

I'll open a PR later on for adding a warning and for documenting the coding pattern.

[iamvvra]

It should, I tested it locally and it worked as expected.

@geoand Could you tell me how you tested? I get deserialization errors because of no empty arg constructor found in the EventMessageDeserializer - though I have one explicitly defined.

Also, I have defined the Jackson's class com.fasterxml.jackson.databind.ObjectMapper in reflection-config.json and passed it to GRAAL in <additionalBuildArg>-H:ReflectionConfigurationFiles=${project.basedir}/reflection-config.json</additionalBuildArg> - is it correct? Did I miss anything?

[geoand]

All I did was add:

package com.k8swatcher;

import io.fabric8.kubernetes.api.model.HasMetadata;
import io.fabric8.kubernetes.api.model.Pod;

public class PodResourceWatcher extends ResourceWatcher<Pod> {

    public PodResourceWatcher(WatchConfig config, NotificationPublisher notificationPublisher) {
        super(config, notificationPublisher);
    }
}

and then changed WatchConfigurer to use it:

                case POD:
                    ResourceWatchMap.watchPods(new PodResourceWatcher(watchConfig, notificationPublisher)).apply(client,
                            ns);
                    break;

I tested against an Openshift cluster where I was adding and removing pods

[iamvvra]

Surprise! I get the below error, however I have changed the deserializer inherit from StdDeserializer and invoke super(EventMessage.class) in its constructor.

[geoand]

When I checked out your code there was no EventMessageDeserializer.
To get passed that problem, simply annotate that class with @RegisterForReflection