Jaeger-operator: CRD Validation Failed when add custom configMap volume

Created on 1 Jul 2020 · 13Comments · Source: jaegertracing/jaeger-operator

collector:
    affinity:
      nodeAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          nodeSelectorTerms:
          - matchExpressions:
            - key: intend
              operator: In
              values:
              - "system"
    resources:
      requests:
        memory: "128Mi"
        cpu: "100m"
      limits:
        memory: "2Gi"
        cpu: "700m"
    volumes:
    - name: sampling-configuration-volume
      configMap:
        name: sampling-configuration
        // ** remove the following items it works.
        items:
          - path: my-sampling.json
            key: sampling

kubectl apply xxx got The Jaeger "simple-prod" is invalid: type: Required value. However remove the items it works.

kubectl version

Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.2", GitCommit:"52c56ce7a8272c798dbc29846288d7cd9fbae032", GitTreeState:"clean", BuildDate:"2020-04-16T23:35:15Z", GoVersion:"go1.14.2", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"16+", GitVersion:"v1.16.8-eks-e16311", GitCommit:"e163110a04dcb2f39c3325af96d019b4925419eb", GitTreeState:"clean", BuildDate:"2020-03-27T22:37:12Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"linux/amd64"}

needs-info

Source

Catorpilor

All 13 comments

Are you able to get your CR applied to a recent minikube instance? The following CR works for me, and I do see the volume added to the pod:

apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
  name: simplest
spec:
  volumes:
    - name: sampling-configuration-volume
      configMap:
        name: custom-sampling-configuration
        items:
        - key: sampling
          path: my-sampling.json

And here's my ConfigMap definition:

apiVersion: v1
kind: ConfigMap
metadata:
  name: custom-sampling-configuration
  namespace: default
data:
  sampling: '{"default_strategy":{"param":1,"type":"probabilistic"}}'

Edit: I removed the collector: node from my first example, as I tried the volumes directly under the .Spec. I do expect the same to work with the volumes under the collector.

jpkrohling on 2 Jul 2020

I'll try it on minikube a.s.a.p.
I'm not familiar with the validation part. For test purpose I've removed all the required parts of key,path in crd's configMap definitions,but nothing changed still got the same error.

Catorpilor on 2 Jul 2020

@jpkrohling i tested on minikube with the latest kubernetes-version(v1.18.3) and it works. Then i use the same version as eks used (v1.16.8) got the same error.

The Jaeger "simplest" is invalid: type: Required value
/tmp
»
/tmp
» kubectl version
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.5", GitCommit:"e6503f8d8f769ace2f338794c914a96fc335df0f", GitTreeState:"clean", BuildDate:"2020-06-27T00:38:11Z", GoVersion:"go1.14.4", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.8", GitCommit:"ec6eb119b81be488b030e849b9e64fda4caaf33c", GitTreeState:"clean", BuildDate:"2020-03-12T20:52:22Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"linux/amd64"}

Catorpilor on 3 Jul 2020

👍1

@objectiser did we come to a conclusion about how many versions back we want to support? Current, Current-1 and Current-2?

jpkrohling on 3 Jul 2020

@jpkrohling The decision was to support n-2 - although were possible we would like to remain compatible with as many versions as possible. Unfortunately this hasn't been added to the docs yet, so that needs to be done - if you could do a PR for that.

objectiser on 3 Jul 2020

Created https://github.com/jaegertracing/documentation/pull/417

jpkrohling on 3 Jul 2020

@Catorpilor, I was able to reproduce the problem with minikube --kubernetes-version=v1.16.8, but I'm still unable to figure out what's going on.

I'll add this to my queue, but if you do find a workaround or if you are able to come up with a PR, let me know!

jpkrohling on 3 Jul 2020

@jpkrohling as i mentioned earlier

For test purpose I've removed all the required parts of key,path in crd's configMap definitions but nothing changed still got the same error.

i'm also confused about the validation part.
I may have found a workaround for this problem(kudos to the prometheus-operator).

The related configMap definitions

apiVersion: v1
data:
  sampling.json: '{"default_strategy":{"operation_strategies":[{"operation":"/health-check","param":0,"type":"probabilistic"},{"operation":"/metrics","param":0,"type":"probabilistic"},{"operation":"/healthz","param":0,"type":"probabilistic"}],"param":0.2,"type":"probabilistic"},"service_strategies":[{"operation_strategies":[{"operation":"/getTime","param":0.2,"type":"probabilistic"},{"operation":"/drivers","param":0.4,"type":"probabilistic"}],"param":0.1,"service":"jaeger-example","type":"probabilistic"}]}'
kind: ConfigMap
metadata:
  name: sampling-configuration
  namespace: observability

i changed key from sampling to sampling.json.

allInOne configuration

apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
  name: simplest
spec:
  strategy: allInOne
  allInOne:
    options:
      sampling.strategies-file: /etc/myconfig/sampling.json
      log-level: debug
    volumes:
      - name: sampling-configuration-volume
        configMap:
          name: sampling-configuration
    volumeMounts:
      - name: sampling-configuration-volume
        mountPath: /etc/myconfig

Add it works!!

bash-5.0# curl localhost:14268/api/sampling?service=xx | jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   405  100   405    0     0  25312      0 --:--:-- --:--:-- --:--:-- 27000
{
  "strategyType": "PROBABILISTIC",
  "probabilisticSampling": {
    "samplingRate": 0.2
  },
  "operationSampling": {
    "defaultSamplingProbability": 0.2,
    "defaultLowerBoundTracesPerSecond": 0,
    "perOperationStrategies": [
      {
        "operation": "/health-check",
        "probabilisticSampling": {
          "samplingRate": 0
        }
      },
      {
        "operation": "/metrics",
        "probabilisticSampling": {
          "samplingRate": 0
        }
      },
      {
        "operation": "/healthz",
        "probabilisticSampling": {
          "samplingRate": 0
        }
      }
    ]
  }
}
bash-5.0# curl localhost:5778/sampling?service=jaeger-example | jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   582  100   582    0     0  58200      0 --:--:-- --:--:-- --:--:-- 58200
{
  "strategyType": "PROBABILISTIC",
  "probabilisticSampling": {
    "samplingRate": 0.1
  },
  "operationSampling": {
    "defaultSamplingProbability": 0.1,
    "defaultLowerBoundTracesPerSecond": 0,
    "perOperationStrategies": [
      {
        "operation": "/getTime",
        "probabilisticSampling": {
          "samplingRate": 0.2
        }
      },
      {
        "operation": "/drivers",
        "probabilisticSampling": {
          "samplingRate": 0.4
        }
      },
      {
        "operation": "/health-check",
        "probabilisticSampling": {
          "samplingRate": 0
        }
      },
      {
        "operation": "/metrics",
        "probabilisticSampling": {
          "samplingRate": 0
        }
      },
      {
        "operation": "/healthz",
        "probabilisticSampling": {
          "samplingRate": 0
        }
      }
    ],
    "defaultUpperBoundTracesPerSecond": 0
  }
}

kubectl version

Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.5", GitCommit:"e6503f8d8f769ace2f338794c914a96fc335df0f", GitTreeState:"clean", BuildDate:"2020-06-27T00:38:11Z", GoVersion:"go1.14.4", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.8", GitCommit:"ec6eb119b81be488b030e849b9e64fda4caaf33c", GitTreeState:"clean", BuildDate:"2020-03-12T20:52:22Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"linux/amd64"}

Catorpilor on 3 Jul 2020

👍1

Interesting! I got kinda puzzled, because the same definition did work for Deployment resources but not for Jaeger:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 80
      volumes:
      - name: sampling-configuration-volume
        configMap:
          name: custom-sampling-configuration
          items:
          - key: sampling
            path: my-sampling.json

So, there seems to be a way to get the original definition to work as well. In any case, your workaround seems valid to me.

Do you think there's anything we can do at this operator's side?

jpkrohling on 3 Jul 2020

For now i think a more specific documentation would be fine.
For managed kubernetes it's always one or two versions behind.

Catorpilor on 3 Jul 2020

👍1

@Catorpilor would you be able to open a PR against our documentation, with your suggestion on how to avoid getting people in the same situation you were?

jpkrohling on 6 Jul 2020

ok, i’ll open a PR tomorrow.

Catorpilor on 6 Jul 2020

I'm closing this in favor of the docs issue.

jpkrohling on 7 Jul 2020

Was this page helpful?

0 / 5 - 0 ratings