Jaeger-operator: Allow storage.options.cassandra.password to have an `existingSecret` reference to pick up cassandra password from

[Hashfyre]

HI,

There is ability to connect Cassandra storage with username and password through jaeger-operator, but there is no way to use a kubernetes secret for the password field and the Jaeger CR would need to have it in plaintext. This is an issue since we would store the Jaeger CR definition in git.

Even with templating through helm/kustomize, it'd mean the operator needs to store the secret somewhere in their own env. A much cleaner way would be able to refer to an existing kubernetes secret.

a useExistingSecret: <kubernetes secret name> would help here, since we would spin up cassandra with either the bitnami/cassandra chart or scylla-operator (among others), both of which support the existingSecret paradigm to create the clusters.

[jpkrohling]

You can mount a secret into env vars: https://kubernetes.io/docs/concepts/configuration/secret/#using-secrets-as-environment-variables

Wouldn't that work for you?

[Hashfyre]

Secret mounting would not work for the Jaeger CustomResource.

apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
  name: jaeger-streaming
spec:
  ...
  storage:
    type: cassandra
    options:
      cassandra:
        servers: cassandra.cassandra.svc.cluster.local
        keyspace: test
        username: cassandra
        password: CASSANDRA_PASS // <- this is how the CR accepts passwords
    cassandraCreateSchema:
      datacenter: dc1
      mode: "test"
      timeout: "3m"

Secret mounting works on Standard K8S GVKs, not for CRs
env.SecretKeyRef is specific to PodSpec

[jpkrohling]

Sorry, you are absolutely correct, but we actually do support building env vars from secrets in a different way:

Excerpt from the docs:

apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
  name: simple-prod
spec:
  strategy: production
  storage:
    type: elasticsearch
    options:
      es:
        server-urls: https://quickstart-es-http.default.svc:9200
    secretName: jaeger-secret

The secret would then contain the env vars and values to be passed down to the collector/query/ingester. You can generate the secret like this:

$ kubectl create secret generic jaeger-secret --from-literal=ES_PASSWORD=changeme --from-literal=ES_USERNAME=elastic

While this shows the usage for ES, it should apply for Cassandra as well. I think we do need a better documentation for this, though.

[Hashfyre]

Yes, this should work. I'll try and submit a docs patch. Thanks.

[stale[bot]]

Is this still relevant? If so, what is blocking it? Is there anything you can do to help move it forward?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

[tobypeschel]

Sorry, you are absolutely correct, but we actually do support building env vars from secrets in a different way:

Excerpt from the docs:

apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
  name: simple-prod
spec:
  strategy: production
  storage:
    type: elasticsearch
    options:
      es:
        server-urls: https://quickstart-es-http.default.svc:9200
    secretName: jaeger-secret

The secret would then contain the env vars and values to be passed down to the collector/query/ingester. You can generate the secret like this:

$ kubectl create secret generic jaeger-secret --from-literal=ES_PASSWORD=changeme --from-literal=ES_USERNAME=elastic

While this shows the usage for ES, it should apply for Cassandra as well. I think we do need a better documentation for this, though.

Unless I'm doing something wrong, it doesn't seem to be working for me with Cassandra.

I'm doing the same as above with secretName: cassandra-creds which is a secret created like: kubectl create secret generic cassandra-creds --from-literal=CASSANDRA_USER=blah --from-literal=CASSANDRA_PASSWORD=blah

Yet still the connection fails and the logs fill with auth errors (AuthenticationFailed('Remote end requires authentication.')). Any ideas?

[jpkrohling]

Could you please double-check that you are using CASSANDRA_USERNAME and not CASSANDRA_USER? The CLI flag for the username is cassandra.username, which translates to the env var CASSANDRA_USERNAME.

[jpkrohling]

In time: @Hashfyre, did you manage to send a documentation PR?

[tobypeschel]

Yep, same issue with CASSANDRA_USERNAME too, unfortunately.

[jpkrohling]

@tobypeschel if you run a local Jaeger process with CASSANDRA_USERNAME/CASSANDRA_PASSWORD as env vars, does it work? If you set the flags directly in the CR as options, does it work?

[tobypeschel]

@jpkrohling It works if I supply the username and password as options, yes. Otherwise no, still it doesn't work. It looks like it simply doesn't behave in the same way as ES_PASSWORD/ES_USERNAME.

[jpkrohling]

If Jaeger does accept CASSANDRA_USERNAME/CASSANDRA_PASSWORD env vars, we should support the same as we have in ES in the operator. I'm not able to work on this for the next few weeks, but I'm open to review PRs fixing this.

[Gerrit-K]

I've deployed a cassandra instance from the bitnami chart and a jaeger instance using the operator chart and linked them to an existing secret as described above. I can confirm that it works using CASSANDRA_USERNAME and CASSANDRA_PASSWORD 👍

@tobypeschel it's been a while since you've posted your issues here, but in case they are still present, I'd be happy to help

[Gerrit-K]

Due to several other issues with the helm chart (invalid CRD, CR and labels), I switched over to deploying the manifests manually and now ran into this issue again. Upon further inspection it appears to me that the _schema job_ does not account for the secret:

https://github.com/jaegertracing/jaeger-operator/blob/5bf83e1024edb5963517d1a7419bd61184ca2c06/pkg/storage/cassandra_dependencies.go#L57-L58

Whereas the regular deployment does:

https://github.com/jaegertracing/jaeger-operator/blob/5bf83e1024edb5963517d1a7419bd61184ca2c06/pkg/deployment/all_in_one.go#L85-L94

I guess it just "accidentally" worked for me before (with the operator helm chart) because I initially deployed the custom resource using the plain username and password options, resulting in the schema being created successfully, and only after that switched to using a secret.

[jpkrohling]

Would you be interested in contributing a fix?

[Gerrit-K]

Would you be interested in contributing a fix?

In the spirit of being helpful and giving back to the open-source community, yes, always! However, I've not written much go code yet (especially production-grade), so chances are that you'll spend more time giving feedback to my contribution than you would need to implement it yourself. Additionally I would need to allocate enough spare time to get familiar with the contribution guidelines, set up my dev environment and submit the PR, so that would take considerably longer. If both of those arguments don't matter that much, I'd be happy to help :)

[jpkrohling]

chances are that you'll spend more time giving feedback to my contribution than you would need to implement it yourself

That's fine, I'll end up with one more person knowing this code, which is always a good thing.

Additionally I would need to allocate enough spare time to get familiar with the contribution guidelines, set up my dev environment and submit the PR, so that would take considerably longer

This will make it easier to contribute more features in the future ;-) The docs should be up to date, but let us know if you get blocked.

If both of those arguments don't matter that much, I'd be happy to help

Looking forward to your PR!