Jaeger-operator: [Question] Custom annotations at the service level

[vassilvk]

According to the documentation, custom annotations can be provided at the deployment component level, however it is not clear if and how custom annotations can be set at the service level.
For example, when deploying Jaeger using production strategy, the operator creates a query service.
Is there a way to set custom annotations for that service object through the Jaeger CR?

[objectiser]

Currently annotations are not configurable for the Service objects. Would you be interested in contributing a PR? The annotations would need to be managed separately from the current set, e.g. ServiceAnnotations.

[vassilvk]

@objectiser - thanks for the response!

I'm definitely interested in contributing, but my current workload would make that difficult.

For the sake of the discussion and clarity, when you say "separately from the current set, eg ServiceAnnotations", is this what you envision:

apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
  ...
spec:
  ...
  serviceAnnotations:
    query:
      my.annotations.io/a: abc
      my.annotations.io/b: xyz

[jpkrohling]

@vassilvk what's your concrete use case?

The annotations would need to be managed separately from the current set, e.g. ServiceAnnotations.

I'm sure you have a great reason for that, but may I ask "why"? Services could indeed have the same set of annotations/labels as deployments, can't they?

[objectiser]

@jpkrohling That was my suggestion - the problem is if we just apply the same set of annotations/labels, then there is no option to use them just for deployments or just services. Don't have concrete cases, but was thinking from a flexibility perspective?

[jpkrohling]

Unless we have a specific use case for this, I'd rather have one single set of labels/annotations for everything related to the component (query, ingester, collector, ...)

[objectiser]

Ok, I guess that will keep the CR simpler.

[vassilvk]

@jpkrohling - my use case has to do with my environment.

I am running a controller which listens to k8s and automatically registers k8s services with a Consul instance running on the same cluster. The k8s => Consul registration is driven by annotations on the k8s service objects.

There is a long list of downstream services which revolve around that Consul instance - one of them is a reverse proxy which dynamically routes traffic to services registered with Consul.
I need access to the annotations of service jaeger-query in order for it to automatically appear behind that reverse proxy gateway.

I hope this makes it clear why setting the same annotations to all services exposed by the operator doesn't make sense when annotations are used to differentiate services and open them for downstream processing.

Please note that my workaround is to create another service which basically mimics the definition of jaeger-query, but allows me to set the annotations I need. This seems hacky to me, as I have a perfectly good service delivered to me by the operator, but I ignore it and run another one in parallel, because the one created by the operator is of no use to me.

@jpkrohling - I have seen your comments on other issues where you were concerned about exposing the services created by the operator into the jaeger custom resource definition. If I understand this correctly, your concern had to do with the fact that services are somehow internal to the workings of the operator and exposing them in the CR would be leaky.

I am not sure that I agree with the thinking that services are internal. What is the purpose of that jaeger-query service if not to be exposed to other parts of the system? And if exposed to other parts of the system, then obviously the system must know about that service and rely on its existence. In that respect, that service is not an internal object and maybe it should be considered an official part of the stack produced by the operator, hence subject to configuration through the CR.

[jpkrohling]

your concern had to do with the fact that services are somehow internal to the workings of the operator and exposing them in the CR would be leaky

Sort of. My concern is that we'll never be able to provide for 100% of the cases, and every new use case we provide for comes with a new complexity for everyone (new property in the CR, documentation, tests, maintenance, ...). For some use cases, I do believe that it's better to manage the service outside of the realm of the operator, either manually or via Helm/templating engine.

For this specific case, I believe it can be done today already, as we do allow annotations and labels to be specified for components (query, collector, ...), which then should propagate to all objects that are part of that (services, deployments, config maps, ...). Would this work for you?

apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
  ...
spec:
  ...
  query:
    annotations:
      my.annotations.io/a: abc
      my.annotations.io/b: xyz

I absolutely agree with you that the service itself is exposed to the whole cluster though.

[vassilvk]

@jpkrohling - yes, that would definitely work.

Unfortunately custom annotations are not propagated down to services (regardless of which strategy is used).

In addition, the documentation seems to be explicit about this.
In section Understanding Custom Resource Definitions, the comment about the custom annotation example reads as follows:

<10> Define annotations to be applied to all deployments (not services). These can be overridden by annotations defined on the individual components.

[mwasilew2]

+1

We ( @Gitlab ) had a similar need for applying annotations to a service. We wanted to bind a BackendConfig object with the query service using annotations and our current workaround is to manage the service and ingress objects outside of operator. It would be nice if it supported that.

[jpkrohling]

I missed the notification for the last comment from @vassilvk, sorry about that. I think the skip using the custom annotations for services as it might have influence on the pod selector, but we can certainly work around that.

Anyone willing to give it a try?

[vassilvk]

I ended up replacing the extra service workaround with a KubeMod ModRule which intercepts the deployment of the service created by Jaeger Operator and patches it with the annotations I need before it is deployed to the cluster:

apiVersion: api.kubemod.io/v1beta1
kind: ModRule
metadata:
  name: jaeger-query-service-modrule
spec:
  type: Patch

  match:
    - select: '$.kind'
      matchValue: Service
    - select: '$.metadata.labels.app'
      matchValue: jaeger
    - select: '$.metadata.labels["app.kubernetes.io/component"]'
      matchValue: service-query

  patch:
    - op: add
      path: /metadata/annotations
      value: |-
        my-annotation-1: 'my-annotation-1-value'
        my-annotation-2: 'my-annotation-2-value'

[jpkrohling]

I didn't know about KubeMod, that's a cool trick!

[vassilvk]

Thanks!
KubeMod is pretty new - in fact I just released its first beta last weekend :)

[lmickh]

In additional to some of the use cases already mentioned, it is also helpful to pass the labels down to the services so that they can be found by the prometheus-operator when it is configured to filter by labels. The jaeger-operator-metrics service and servicemonitor for example will not function if your Prometheus config uses label matching for service discovery. There isn't much in the way of metrics right now, but I would think that is going to be expanded on in the future.