Jaeger-operator: No identifying labels on ES Index Cleaner pod

[qbast]

I am trying to setup network policies for my 'monitoring' namespace. This includes restricting access to ElasticSearch to Jaeger.
I can easily create a policy for Jaeger itself by matching on app=jaeger annotation, but unfortunately pods created by ES cleaner cron job have no identifying annotations.
Please modify cronjob template to add annotations that can be used to identify cleaner pod.

[jpkrohling]

I think you probably meant labels, as that's where we place the app=jaeger key/value pairs. Looking at the source, it looks like we do set the same app=jaeger label:

https://github.com/jaegertracing/jaeger-operator/blob/5f250045063bc8490bceeb39d1aec4641c4c720a/pkg/cronjob/es_index_cleaner.go#L56-L63

Are you able to share the output of kubectl describe cronjob my-jaeger-es-index-cleaner, where the last part is the actual name of the cronjob?

[qbast]

Hello

Yes, I mean labels not annotations.
Here is output from describe cronjob:

Name:                          jaeger-es-index-cleaner
Namespace:                     monitoring
Labels:                        app=jaeger
                               app.kubernetes.io/component=cronjob-es-index-cleaner
                               app.kubernetes.io/instance=jaeger
                               app.kubernetes.io/managed-by=jaeger-operator
                               app.kubernetes.io/name=jaeger-es-index-cleaner
                               app.kubernetes.io/part-of=jaeger
Annotations:                   <none>
Schedule:                      55 23 * * *
Concurrency Policy:            Allow
Suspend:                       False
Successful Job History Limit:  3
Failed Job History Limit:      1
Starting Deadline Seconds:     <unset>
Selector:                      <unset>
Parallelism:                   1
Completions:                   <unset>
Pod Template:
  Labels:       <none>
  Annotations:  prometheus.io/scrape: false
                sidecar.istio.io/inject: false
  Containers:
   jaeger-es-index-cleaner:
    Image:      jaegertracing/jaeger-es-index-cleaner
    Port:       <none>
    Host Port:  <none>
    Args:
      7
      http://elasticsearch-master:9200
    Environment:     <none>
    Mounts:          <none>
  Volumes:           <none>
Last Schedule Time:  Tue, 15 Oct 2019 01:55:00 +0200
Active Jobs:         <none>
Events:              <none>

So it looks ok. However kubectl describe pod jaeger-es-index-cleaner-1570924500-dvnj6 shows:

Name:           jaeger-es-index-cleaner-1570924500-dvnj6
Namespace:      monitoring
Priority:       0
Node:           gke-dev-augnetmq-cluster-main-pool-f157e1fd-34cb/10.128.0.24
Start Time:     Sun, 13 Oct 2019 01:55:00 +0200
Labels:         controller-uid=b3a46cbf-ed4b-11e9-9fee-42010affffea
                job-name=jaeger-es-index-cleaner-1570924500
Annotations:    cni.projectcalico.org/podIP: 10.144.4.31/32
                prometheus.io/scrape: false
                sidecar.istio.io/inject: false
Status:         Succeeded
IP:             10.144.4.31
Controlled By:  Job/jaeger-es-index-cleaner-1570924500
Containers:
  jaeger-es-index-cleaner:
    Container ID:  docker://e81e6c7ee508b0b68d8be4f080f47bd7fb437d7316df6a8220a6ce5573e35d62
    Image:         jaegertracing/jaeger-es-index-cleaner
    Image ID:      docker-pullable://jaegertracing/jaeger-es-index-cleaner@sha256:f5b46b9528b61e8e7f57c116b76d024a11c035c9b6298472a92595ccf9185ddc
    Port:          <none>
    Host Port:     <none>
    Args:
      7
      http://elasticsearch-master:9200
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Sun, 13 Oct 2019 01:55:02 +0200
      Finished:     Sun, 13 Oct 2019 01:55:05 +0200
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-jgrbp (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  default-token-jgrbp:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-jgrbp
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:          <none>

So labels from the cronjob are not propagated to pods.

[jpkrohling]

Good catch. This is because the PodSpec is missing the labels:

https://github.com/jaegertracing/jaeger-operator/blob/5f250045063bc8490bceeb39d1aec4641c4c720a/pkg/cronjob/es_index_cleaner.go#L82-L98

[jpkrohling]

@qbast would you be open to sending a PR to add the labels to the PodSpec?

[yeya24]

@jpkrohling I' d like to open a PR today

[jpkrohling]

It's yours!

[yeya24]

@jpkrohling I see there are several labels for the cronjob. So what labels do I need to add in the pod besides app=jaeger? Do I need to add all of them?

            Labels: map[string]string{
                "app":                          "jaeger",
                "app.kubernetes.io/name":       name,
                "app.kubernetes.io/instance":   jaeger.Name,
                "app.kubernetes.io/component":  "cronjob-es-index-cleaner",
                "app.kubernetes.io/part-of":    "jaeger",
                "app.kubernetes.io/managed-by": "jaeger-operator",
            },

[jpkrohling]

They should all be added. Using the Deployment as reference, we have the same labels both at the Deployment's .ObjectMeta and at the Deployment's .Spec.Template.ObjectMeta.

Just make sure to store the labels in a common var, so that future changes will be applied to both places.