Splunk-connect-for-kubernetes: Custom Resource Definition (CRD) Object Monitoring

[ekristen]

Is it possible to monitor CRDs with the object monitoring? From the configuration it seems possible, but it errors when trying to monitor custom resource definitions.

Error undefined method watch_<replace_with_crd_name>

[matthewmodestino]

I am not sure if the ruby client support it, but try:

custom_resource_definition as the object to watch.

It wouldnt be the name of your crd...

https://github.com/splunk/splunk-connect-for-kubernetes/blob/20e67c6f2dcf4d06f6efc7cc6763126946a6956f/manifests/splunk-kubernetes-objects/configMap.yaml#L10-L31

Try something like:

  <watch>
     resource_name custom_resource_definition
   </watch>

It is here you are trying to scrape in the objects configmap:

API Version: apiextensions.k8s.io/v1
Kind: CustomResourceDefinition

[ekristen]

I'll have to take a closer look at the ruby code then. Perhaps what you are suggesting would work for monitoring actual CustomResourceDefinition definition objects, but I'm looking to monitor instances of a CRD.

Like for example the Prometheus Operator has a CRD called Prometheus, it has it's own APIVersion and Kind type, I want to monitor it.

When I tried modifying the helm chart to manually add the version and the kind like the docs suggest you can with the built in types it just threw the above error watch_prometheus undefined.

[matthewmodestino]

Ah, well technically the CRDs label things in such a way that you may be able to tell your story in the data we collect already. The watch would simply allow you to act to start the logic to triggering a monitor on something, but without doing any of that we will already see all its objects meta and logs and should be able to stich together with labels/dimensions

[ekristen]

After reviewing the code it seems like it should work. However I get errors about the thread creation bit that watch_resource_name isn't valid. I can't seem to figure out why this would be.

[ekristen]

Ok upon further inspection, it doesn't work because the plugin is trying to use helper methods that do not exist on the default kubeclient gem for trying to get CRD information.

IE it tries to do get_nameofcrds which is invalid on the kubeclient gem.

[matthewmodestino]

gotcha!

Do you think we really need it or would the labels for the crd pods be enough?

example in splunk operator CRD we have:

$ kubectl -n splunk get pods splunk-mattymo-standalone-0 --show-labels
NAME                          READY   STATUS    RESTARTS   AGE   LABELS
splunk-mattymo-standalone-0   1/1     Running   2          37d   app.kubernetes.io/instance=splunk-mattymo-standalone,app.kubernetes.io/managed-by=splunk-operator,app.kubernetes.io/name=splunk-mattymo,app.kubernetes.io/part-of=splunk,app=splunk,controller-revision-hash=splunk-mattymo-standalone-56765df547,for=mattymo,statefulset.kubernetes.io/pod-name=splunk-mattymo-standalone-0,type=standalone

where things like managed-by are?

Object scrape would be more for discovery and watching events for changes and metadata collection about the crd, where the logs and metrics pod probably already collect what you need.

[ekristen]

I guess it depends on the goals of the object monitoring? I was looking at it from the perspective of using it to audit changes to the resources themselves.

Monitoring the core objects gets you only one piece of the picture.

[ekristen]

BTW I opened https://github.com/splunk/fluent-plugin-kubernetes-objects/issues/51 -- basically if it give it a CRD that it doesn't have helper methods for it essentially fork bombs the server running the object collection. It happens slowly, should be fixed regardless.

[matthewmodestino]

changes to the resources themselves can be "watched" with kube:apiserver-audit logs. Thats what most projects I see use for this kinda thing. The objects pod is more meta enrichment for the logs and metrics, but it also has a sourcetype called kube:objects:events that is ops focused and doesn't require you to configure audit, it may suit you better as well. The audit data below can be searched with labels and annotations, so you should be golden! Events has less metadata, but still should have enough to get by.

[ekristen]

RE: Audit -- this might work, I'll take a deeper look into it. Thanks.

RE: Events -- Not all CRD end up creating Pods. There are many uses for CRDs. Events are only useful if the controller or something is monitoring the CRD and actually writing events for the changes, even then it's only as good as the information written into the event, it doesn't actually record the object properties only that an object was involved.