Kubernetes-client: shared index informer is watching resources in all namespaces even though they are intended to watch in particular namespace

Created on 4 Jul 2020  路  18Comments  路  Source: fabric8io/kubernetes-client

Below is code to create SharedIndexInformer, I wanted to watch in a particular namespace only but it is watching in all the namespaces.
Please suggest

public SharedIndexInformer<T> idmSharedIndexInformerFor() {
        CustomResourceDefinitionContext CrdContext = new CustomResourceDefinitionContext.Builder()
                .withVersion(this.version)
                .withScope(this.scope)
                .withGroup(this.group)
                .withPlural(this.plural)
                .build();
        return sharedInformerFactory.sharedIndexInformerForCustomResource(CrdContext, apiTypeClass, apiListTypeClass,new OperationContext().withNamespace("default"), 60 * 1000L);
    }

Starting it like below
sharedInformerFactory.startAllRegisteredInformers();

picture skpandey91

Most helpful comment

Okay, it works fine. Just added it, had custom resources in other namespaces and they were not pulled in from the indexinformer.

picture bframke on 23 Jul 2020
👍3

All 18 comments

@skpandey91 : Which version of client are you using?

rohanKanojia picture rohanKanojia on 6 Jul 2020

@skpandey91 : Which version of client are you using?

4.10.2 version I am using, As it is trying to list, watch CRs in all namespaces , it is mandatory to create cluster role binding, but we work with rolebinding only ..PLease suggest we are blocked..
Normal watcher worked with one namspace with only rolebinding. We are moving to informer from normal watcher as normal watcher gives Too old resource if number of resources are huge.

skpandey91 picture skpandey91 on 6 Jul 2020

This is quite strange since we have a test for this case :confused: . Would it be possible for you to share a reproducible test case ?

https://github.com/fabric8io/kubernetes-client/blob/ee9316464f1821f735ea0606a9ce064af48b719c/kubernetes-tests/src/test/java/io/fabric8/kubernetes/client/mock/DefaultSharedIndexInformerTest.java#L478-L520

rohanKanojia picture rohanKanojia on 6 Jul 2020

This is quite strange since we have a test for this case 馃槙 . Would it be possible for you to share a reproducible test case ?

https://github.com/fabric8io/kubernetes-client/blob/ee9316464f1821f735ea0606a9ce064af48b719c/kubernetes-tests/src/test/java/io/fabric8/kubernetes/client/mock/DefaultSharedIndexInformerTest.java#L478-L520

@Test
    public void testInfoIDMRoles() throws InterruptedException, IOException {
KubernetesClient client=getClient();
        SharedInformerFactory sharedInformerFactory=client.informers();
        CustomResourceDefinitionContext CrdContext = new CustomResourceDefinitionContext.Builder()
                .withVersion("v1")
                .withScope("Namespaced")
                .withGroup("idm.fndsec.amdocs.com")
                .withPlural("PodSetes")
                .build();
        SharedIndexInformer<PodSet> podInformer=sharedInformerFactory.sharedIndexInformerForCustomResource(CrdContext, PodSet.class, PodSet.PodSetList.class,new CustomResourceOperationContext().withNamespace("default") ,60 * 1000L);
        podInformer.addEventHandler(
                new ResourceEventHandler<PodSet>() {
                    @Override
                    public void onAdd(PodSet pod) {
                        logger.info("Pod " + pod.getMetadata().getName() + " got added");
                    }

                    @Override
                    public void onUpdate(PodSet oldPod, PodSet newPod) {
                        logger.info("Pod " + oldPod.getMetadata().getName() + " got updated");
                    }

                    @Override
                    public void onDelete(PodSet pod, boolean deletedFinalStateUnknown) {
                        logger.info("Pod " + pod.getMetadata().getName() + " got deleted");
                    }
                }
        );

        sharedInformerFactory.startAllRegisteredInformers();
}

    static KubernetesClient getClient() {

        Config k8SConfig = new ConfigBuilder().withMasterUrl("****")
            .withTrustCerts(true)
            .withNamespace("default")
            .withClientCertData(
                "****")
            .withClientKeyData(
                "****")
            .build();

        return new DefaultKubernetesClient(k8SConfig).inNamespace("default");
    }
skpandey91 picture skpandey91 on 6 Jul 2020

Hello, we just now updated to version 1.6.0 of Quarkus which uses 4.10.2 of Fabric8 and we can see that it looks at all Namespaces and not the one its intended to look at.

bframke picture bframke on 20 Jul 2020

Hello, we just now updated to version 1.6.0 of Quarkus which uses 4.10.2 of Fabric8 and we can see that it looks at all Namespaces and not the one its intended to look at.

So is it a defect or feature ??

skpandey91 picture skpandey91 on 20 Jul 2020

I would call it a defect, because we are defining a namespace that should be used by the Informer and it is ignored.

bframke picture bframke on 20 Jul 2020

Hi, I'm not able to reproduce this issue. For me having three pods in my default namespace:

~/work/repos/jkube/quickstarts/maven/karaf-camel-log : $ kubectl get pods -ndefault
NAME                            READY   STATUS      RESTARTS   AGE
docker-registry-1-65cbp         1/1     Running     2          3d
persistent-volume-setup-5bsnl   0/1     Completed   0          3d
router-1-2xs2f                  1/1     Running     2          3d

When I run a simple Pod namespace informer for default namespace:

        try (KubernetesClient client = new DefaultKubernetesClient()) {
            SharedInformerFactory sharedInformerFactory = client.informers();
            SharedIndexInformer<Pod> podInformer = sharedInformerFactory.sharedIndexInformerFor(
                    Pod.class,
                    PodList.class,
                    new OperationContext().withNamespace("default"),
                    30 * 1000L);
            logger.info("Informer factory initialized.");

            podInformer.addEventHandler(
                    new ResourceEventHandler<Pod>() {
                        @Override
                        public void onAdd(Pod pod) {
                            logger.info("Pod " + pod.getMetadata().getName() + " got added");
                        }

                        @Override
                        public void onUpdate(Pod oldPod, Pod newPod) {
                            logger.info("Pod " + oldPod.getMetadata().getName() + " got updated");
                        }

                        @Override
                        public void onDelete(Pod pod, boolean deletedFinalStateUnknown) {
                            logger.info("Pod " + pod.getMetadata().getName() + " got deleted");
                        }
                    });

            logger.info("Starting all registered informers");
            sharedInformerFactory.startAllRegisteredInformers();

            // Wait for 1 minute
            Thread.sleep(15 * 60 * 1000L);
            sharedInformerFactory.stopAllRegisteredInformers();
        } catch (InterruptedException e) {
            Thread.currentThread().interrupt();
            e.printStackTrace();
        }

I get this as output:

/usr/java/jdk-14.0.1/bin/java -javaagent:/opt/ideaIC-2019.3.3/idea-IC-193.6494.35/lib/idea_rt.jar=37767:/opt/ideaIC-2019.3.3/idea-IC-193.6494.35/bin -Dfile.encoding=UTF-8 -classpath /home/rohaan/work/repos/kubernetes-client-demo/target/classes:/home/rohaan/.m2/repository/io/fabric8/kubernetes-client/4.10-SNAPSHOT/kubernetes-client-4.10-SNAPSHOT.jar:/home/rohaan/.m2/repository/io/fabric8/kubernetes-model-core/4.10-SNAPSHOT/kubernetes-model-core-4.10-SNAPSHOT.jar:/home/rohaan/.m2/repository/io/fabric8/kubernetes-model-common/4.10-SNAPSHOT/kubernetes-model-common-4.10-SNAPSHOT.jar:/home/rohaan/.m2/repository/com/fasterxml/jackson/module/jackson-module-jaxb-annotations/2.10.3/jackson-module-jaxb-annotations-2.10.3.jar:/home/rohaan/.m2/repository/jakarta/xml/bind/jakarta.xml.bind-api/2.3.2/jakarta.xml.bind-api-2.3.2.jar:/home/rohaan/.m2/repository/jakarta/activation/jakarta.activation-api/1.2.1/jakarta.activation-api-1.2.1.jar:/home/rohaan/.m2/repository/javax/annotation/javax.annotation-api/1.3.2/javax.annotation-api-1.3.2.jar:/home/rohaan/.m2/repository/javax/xml/bind/jaxb-api/2.3.0/jaxb-api-2.3.0.jar:/home/rohaan/.m2/repository/io/fabric8/kubernetes-model-rbac/4.10-SNAPSHOT/kubernetes-model-rbac-4.10-SNAPSHOT.jar:/home/rohaan/.m2/repository/io/fabric8/kubernetes-model-admissionregistration/4.10-SNAPSHOT/kubernetes-model-admissionregistration-4.10-SNAPSHOT.jar:/home/rohaan/.m2/repository/io/fabric8/kubernetes-model-apps/4.10-SNAPSHOT/kubernetes-model-apps-4.10-SNAPSHOT.jar:/home/rohaan/.m2/repository/io/fabric8/kubernetes-model-autoscaling/4.10-SNAPSHOT/kubernetes-model-autoscaling-4.10-SNAPSHOT.jar:/home/rohaan/.m2/repository/io/fabric8/kubernetes-model-apiextensions/4.10-SNAPSHOT/kubernetes-model-apiextensions-4.10-SNAPSHOT.jar:/home/rohaan/.m2/repository/io/fabric8/kubernetes-model-batch/4.10-SNAPSHOT/kubernetes-model-batch-4.10-SNAPSHOT.jar:/home/rohaan/.m2/repository/io/fabric8/kubernetes-model-certificates/4.10-SNAPSHOT/kubernetes-model-certificates-4.10-SNAPSHOT.jar:/home/rohaan/.m2/repository/io/fabric8/kubernetes-model-coordination/4.10-SNAPSHOT/kubernetes-model-coordination-4.10-SNAPSHOT.jar:/home/rohaan/.m2/repository/io/fabric8/kubernetes-model-discovery/4.10-SNAPSHOT/kubernetes-model-discovery-4.10-SNAPSHOT.jar:/home/rohaan/.m2/repository/io/fabric8/kubernetes-model-events/4.10-SNAPSHOT/kubernetes-model-events-4.10-SNAPSHOT.jar:/home/rohaan/.m2/repository/io/fabric8/kubernetes-model-extensions/4.10-SNAPSHOT/kubernetes-model-extensions-4.10-SNAPSHOT.jar:/home/rohaan/.m2/repository/io/fabric8/kubernetes-model-networking/4.10-SNAPSHOT/kubernetes-model-networking-4.10-SNAPSHOT.jar:/home/rohaan/.m2/repository/io/fabric8/kubernetes-model-metrics/4.10-SNAPSHOT/kubernetes-model-metrics-4.10-SNAPSHOT.jar:/home/rohaan/.m2/repository/io/fabric8/kubernetes-model-policy/4.10-SNAPSHOT/kubernetes-model-policy-4.10-SNAPSHOT.jar:/home/rohaan/.m2/repository/io/fabric8/kubernetes-model-scheduling/4.10-SNAPSHOT/kubernetes-model-scheduling-4.10-SNAPSHOT.jar:/home/rohaan/.m2/repository/io/fabric8/kubernetes-model-settings/4.10-SNAPSHOT/kubernetes-model-settings-4.10-SNAPSHOT.jar:/home/rohaan/.m2/repository/io/fabric8/kubernetes-model-storageclass/4.10-SNAPSHOT/kubernetes-model-storageclass-4.10-SNAPSHOT.jar:/home/rohaan/.m2/repository/io/fabric8/openshift-model/4.10-SNAPSHOT/openshift-model-4.10-SNAPSHOT.jar:/home/rohaan/.m2/repository/com/squareup/okhttp3/okhttp/3.12.12/okhttp-3.12.12.jar:/home/rohaan/.m2/repository/com/squareup/okio/okio/1.15.0/okio-1.15.0.jar:/home/rohaan/.m2/repository/com/squareup/okhttp3/logging-interceptor/3.12.12/logging-interceptor-3.12.12.jar:/home/rohaan/.m2/repository/com/fasterxml/jackson/dataformat/jackson-dataformat-yaml/2.10.3/jackson-dataformat-yaml-2.10.3.jar:/home/rohaan/.m2/repository/org/yaml/snakeyaml/1.24/snakeyaml-1.24.jar:/home/rohaan/.m2/repository/com/fasterxml/jackson/datatype/jackson-datatype-jsr310/2.10.3/jackson-datatype-jsr310-2.10.3.jar:/home/rohaan/.m2/repository/com/fasterxml/jackson/core/jackson-annotations/2.10.3/jackson-annotations-2.10.3.jar:/home/rohaan/.m2/repository/com/fasterxml/jackson/core/jackson-databind/2.10.3/jackson-databind-2.10.3.jar:/home/rohaan/.m2/repository/com/fasterxml/jackson/core/jackson-core/2.10.3/jackson-core-2.10.3.jar:/home/rohaan/.m2/repository/io/fabric8/zjsonpatch/0.3.0/zjsonpatch-0.3.0.jar:/home/rohaan/.m2/repository/com/github/mifmif/generex/1.0.2/generex-1.0.2.jar:/home/rohaan/.m2/repository/dk/brics/automaton/automaton/1.11-8/automaton-1.11-8.jar:/home/rohaan/.m2/repository/io/fabric8/openshift-client/4.10-SNAPSHOT/openshift-client-4.10-SNAPSHOT.jar:/home/rohaan/.m2/repository/org/json/json/20190722/json-20190722.jar:/home/rohaan/.m2/repository/org/slf4j/slf4j-simple/1.7.28/slf4j-simple-1.7.28.jar:/home/rohaan/.m2/repository/org/slf4j/slf4j-api/1.7.5/slf4j-api-1.7.5.jar io.fabric8.NamespacedInformerDemo
Jul 20, 2020 8:19:40 PM io.fabric8.NamespacedInformerDemo main
INFO: Informer factory initialized.
Jul 20, 2020 8:19:40 PM io.fabric8.NamespacedInformerDemo main
INFO: Starting all registered informers
[informer-controller-Pod] INFO io.fabric8.kubernetes.client.informers.cache.Controller - informer#Controller: ready to run resync and reflector runnable
[informer-controller-Pod] INFO io.fabric8.kubernetes.client.informers.cache.Reflector - Started ReflectorRunnable watch for class io.fabric8.kubernetes.api.model.Pod
Jul 20, 2020 8:19:40 PM io.fabric8.NamespacedInformerDemo$1 onAdd
INFO: Pod docker-registry-1-65cbp got added
Jul 20, 2020 8:19:40 PM io.fabric8.NamespacedInformerDemo$1 onUpdate
INFO: Pod docker-registry-1-65cbp got updated
Jul 20, 2020 8:19:40 PM io.fabric8.NamespacedInformerDemo$1 onAdd
INFO: Pod persistent-volume-setup-5bsnl got added
Jul 20, 2020 8:19:40 PM io.fabric8.NamespacedInformerDemo$1 onUpdate
INFO: Pod persistent-volume-setup-5bsnl got updated
Jul 20, 2020 8:19:40 PM io.fabric8.NamespacedInformerDemo$1 onAdd
INFO: Pod router-1-2xs2f got added
Jul 20, 2020 8:19:40 PM io.fabric8.NamespacedInformerDemo$1 onUpdate
INFO: Pod router-1-2xs2f got updated
rohanKanojia picture rohanKanojia on 20 Jul 2020

What's more strange is that we have a test case already covering this scenario:

https://github.com/fabric8io/kubernetes-client/blob/12e258091e0a81b462a08aa89b9d7191230f472e/kubernetes-tests/src/test/java/io/fabric8/kubernetes/client/mock/DefaultSharedIndexInformerTest.java#L478-L528

Could anyone please share a reproducer project which we can try out and test?

rohanKanojia picture rohanKanojia on 20 Jul 2020

Then just deploy the pod you watch into another namespace or on more than two

bframke picture bframke on 20 Jul 2020

I do have pods running in other namespaces. This is my output with --all-namespaces argument:

~/work/repos/jkube/quickstarts/maven/docker-file-simple : $ oc get pods --all-namespaces
NAMESPACE                       NAME                                                      READY     STATUS              RESTARTS   AGE
default                         docker-registry-1-65cbp                                   1/1       Running             2          3d
default                         persistent-volume-setup-5bsnl                             0/1       Completed           0          3d
default                         router-1-2xs2f                                            1/1       Running             2          3d
kube-dns                        kube-dns-4d9nt                                            1/1       Running             2          3d
kube-proxy                      kube-proxy-cvtrh                                          1/1       Running             2          3d
kube-system                     kube-controller-manager-localhost                         1/1       Running             2          3d
kube-system                     kube-scheduler-localhost                                  1/1       Running             2          3d
kube-system                     master-api-localhost                                      1/1       Running             2          3d
kube-system                     master-etcd-localhost                                     1/1       Running             2          3d
myproject                       docker-file-simple-s2i-1-build                            0/1       Error               0          27s
myproject                       karaf-camel-log-7d84c79b4c-62cmw                          0/1       CrashLoopBackOff    4          2m
myproject                       karaf-camel-log-s2i-2-build                               0/1       Completed           0          5m
myproject                       xml-config-controller-b889999c4-hwvnn                     0/1       ContainerCreating   0          5m
myproject                       xml-config-controller-b889999c4-kqllk                     0/1       ContainerCreating   0          5m
openshift-apiserver             openshift-apiserver-6nd4w                                 1/1       Running             2          3d
openshift-controller-manager    openshift-controller-manager-69rbg                        1/1       Running             2          3d
openshift-core-operators        openshift-service-cert-signer-operator-6d477f986b-nqcn6   1/1       Running             2          3d
openshift-core-operators        openshift-web-console-operator-57986c9c4f-sjvjm           1/1       Running             2          3d
openshift-service-cert-signer   apiservice-cabundle-injector-8ffbbb6dc-9p2dp              1/1       Running             2          3d
openshift-service-cert-signer   service-serving-cert-signer-668c45d5f-dgrm2               1/1       Running             2          3d
openshift-web-console           webconsole-74646f4d98-qs85t                               1/1       Running             4          3d
~/work/repos/jkube/quickstarts/maven/docker-file-simple : $
rohanKanojia picture rohanKanojia on 20 Jul 2020

Okay, so I'am a using Custom Resource, but also a Cluster Role for the Service Account. The Kubernetes Client itself is set to a Namespace. Maybe you should try it out with a Custom Resource, maybe only that one has a problem in this case.

bframke picture bframke on 20 Jul 2020

ohk, I'm able to reproduce this. Let me try to fix it whenever I get time this week.

rohanKanojia picture rohanKanojia on 21 Jul 2020
👍1

ohk, now I'm able to see why informer was watching in all namespaces instead of a specified one. In 4.10.x versions we introduced a new interface called Namespaced:

https://github.com/fabric8io/kubernetes-client/blob/55cf52c4e27adafa52052cea9cbca6d1ede1b07b/kubernetes-model-generator/kubernetes-model-core/src/main/java/io/fabric8/kubernetes/api/model/Namespaced.java#L18-L19

Every resource in Kubernetes Model which is a namespace resource implements this interface, otherwise, it's considered to be a Cluster scoped resource.

Our test was passing because our PodSet class implements this interface:

https://github.com/fabric8io/kubernetes-client/blob/55cf52c4e27adafa52052cea9cbca6d1ede1b07b/kubernetes-tests/src/test/java/io/fabric8/kubernetes/client/mock/crd/PodSet.java#L22

I modified my CronTab POJO to implement this interface. After this informers started working as expected. You can check my test here.

Could you please check if adding making your Custom Resource classes implement this Namespaced interface solves your issue?

rohanKanojia picture rohanKanojia on 23 Jul 2020

Okay, so just adding implements Namespaced to the CR helps in this case? Just asking if I understood it correctly.

Edit: reading helps :D Trying to test it now

bframke picture bframke on 23 Jul 2020
👍2

@bframke : Yes, I think so. I've tested and it seemed to work for me.

rohanKanojia picture rohanKanojia on 23 Jul 2020

Please let me know if it works for you. I'll update the docs so that it's clearly visible.

rohanKanojia picture rohanKanojia on 23 Jul 2020

Okay, it works fine. Just added it, had custom resources in other namespaces and they were not pulled in from the indexinformer.

bframke picture bframke on 23 Jul 2020
👍3
Was this page helpful?
0 / 5 - 0 ratings

Related issues

chenchun picture chenchun  路  28Comments
kolorful picture kolorful  路  24Comments
iocanel picture iocanel  路  21Comments
WywTed picture WywTed  路  22Comments
zoujinhe picture zoujinhe  路  21Comments