Jaeger-operator: Support TLS for spark dependencies

[pavolloffay]

Spark dependencies job uses Java keystore for certificates. The docker image allows to configure java opts with ssl configuration - JAVA_OPTS=-Djavax.net.ssl..

The certs can be mounted via volume and volume mounts which are part of JaegerCommonSpec. The issue that these certs have to be imported into java keystore/thruststore.

https://developers.redhat.com/blog/2017/11/22/dynamically-creating-java-keystores-openshift/ suggests using init container for the job.

Todos:

• [ ] allow configuring JAVA_OPTS to point to keystore/thruststore
• [ ] import certs into keystore/thruststore

[pavolloffay]

Dependent issue in spark-dependencies https://github.com/jaegertracing/jaeger-operator/issues/294

[pavolloffay]

@jpkrohling this is not ES specific. The dependency job supports multiple backends.

[jorgex1]

Hi @pavolloffay are there any updates on this?

[pavolloffay]

There aren't ane news. If anybody has free cycles feel free to take it.

[jorgex1]

Spent a bit doing some digging through things and came across this page https://hub.helm.sh/charts/jaegertracing/jaeger

The page mentions that in order to have a version of the spark job working one would need to get the certificates from the elasticsearch cluster and do

$ keytool -import -trustcacerts -keystore trust.store -storepass <some pass> -alias es-root -file <es.pem>

$ kubectl create configmap jaeger-tls --from-file=trust.store --from-file=<es.pem>

And then use the jaeger-tls configmap within the spark job definition as such:

spec:
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: jaeger-spark
            args:
              - --java.opts=-Djavax.net.ssl.trustStore=/tls/trust.store -Djavax.net.ssl.trustStorePassword=<some pass>
            volumeMounts:
              - name: jaeger-tls
                mountPath: /tls
                subPath: 
                readOnly: true
          volumes:
            - name: jaeger-tls
              configMap:
                name: jaeger-tls
          ...

Which all sounds like it is in accordance with the issue description.
Think it may be a bit better to define jaeger-tls as a secret though, but other than that, is all that more or less correct?

[pavolloffay]

Correct the certs have to be imported into java keystore in the spark image.

[pavolloffay]

My idea was to implement a script in the spark image that would do that if the certs are specified as env vars.

[jam01]

Hey guys, is there any work around for this atm? I'm trying to follow the thread of TLS related changes and options. Is there any skip-tls JAVA_OPT or something that can be used temporarily?

To maybe help others:
add the java options described above to the jaeger CR
manually create a JKS trust store from the elasticsearch-es-http-certs-public secret
downscale the operator replicaset to 0
edit the cronjob container template to mount a volume from the secret

[jpkrohling]

This would probably a question to ask in the repository that holds the code for the spark dependencies processor. https://github.com/jaegertracing/spark-dependencies

[aleksandrovpa]

Hi, maybe this is stupid question, but i couldn't run a spark job with following configuration:

spark:
  enabled: true
  cmdlineParams:
    java.opts: "-Djavax.net.ssl.trustStore=/tls/trust.store -Djavax.net.ssl.trustStorePassword=changeit"
  extraConfigmapMounts:
    - name: jaeger-tls
      mountPath: /tls
      subPath: ""
      configMap: jaeger-tls
      readOnly: true

Here is error:

/entrypoint.sh: 37: exec: --java.opts=-Djavax.net.ssl.trustStore=/tls/trust.store -Djavax.net.ssl.trustStorePassword=changeit: not found

Can someone show me, what i'm doing wrong? or maybe there are any workaround, how to get spark job works.
Thnx

[kevinearls]

@aleksandrovpa I don't have much time to look at this at the moment, but you may find what you need in these two items:

https://github.com/jaegertracing/jaeger-operator/issues/1332
https://github.com/jaegertracing/jaeger-operator/pull/1359

[jam01]

@aleksandrovpa we're just composing the cron job manually in order to bootstrap the trust store through an init container. You can check this out https://github.com/MS3Inc/tavros/blob/main/tests/integration/targets/playbooks/provision_playbook/example.com/platform/jaeger/default/cronjob-spark-dependencies.yaml#L39