Jaeger-operator: Recommended memory for jaeger-spark-dependencies (elasticsearch)

[palmerabollo]

I'm getting the following exception when the jaeger-spark-dependencies job runs:

$ kubectl logs jaeger-spark-dependencies-1572911700-2pf7x

19/11/04 23:55:20 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
19/11/04 23:55:21 ERROR SparkContext: Error initializing SparkContext.
java.lang.IllegalArgumentException: System memory 101384192 must be at least 471859200. Please increase heap size using the --driver-memory option or spark.driver.memory in Spark configuration.
    at org.apache.spark.memory.UnifiedMemoryManager$.getMaxMemory(UnifiedMemoryManager.scala:217)
    at org.apache.spark.memory.UnifiedMemoryManager$.apply(UnifiedMemoryManager.scala:199)
    at org.apache.spark.SparkEnv$.create(SparkEnv.scala:330)
...

I have two questions:

• What are the recommended memory requests/limits?
• What happens if you don't enable this job?

[chelomontilla]

Same problem here, any recommendation ?

[pavolloffay]

What happens if you don't enable this job?

Then you will not see dependencies on the dependencies tab. https://www.jaegertracing.io/docs/1.14/deployment/#aggregation-jobs-for-service-dependencies

What are the recommended memory requests/limits?

Have you specified resources in your jaeger CR, or are you using the default?

[chelomontilla]

Here is my config:

apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
  name: jaeger-staging
  namespace: staging
spec:
  strategy: streaming
  ingress:
    enabled: false
  collector:
    replicas: 2
    options:
      kafka:
        producer:
          topic: staging-jaeger-spans
          brokers: broker:9092
  ingester:
    replicas: 2
    options:
      kafka:
        consumer:
          topic: staging-jaeger-spans
          brokers: broker:9092
      ingester:
        deadlockInterval: 0
  query:
    replicas: 2
  storage:
    type: elasticsearch
    options:
      es-provision: false
      es:
        server-urls: elastic_url
        index-prefix: staging
      esIndexCleaner:
        enabled: false # turn the job deployment on and off
        numberOfDays: 30 # number of days to wait before deleting a record
        schedule: "55 23 * * *" # cron expression for it to run
        image: jaegertracing/jaeger-es-index-cleaner # image of the job
    dependencies:
      enabled: true
      elasticsearchNodesWanOnly: true
  resources:
    requests:
      memory: "64Mi"
      cpu: "100m"
    limits:
      memory: "2048Mi"
      cpu: "512m"

With this config it's still failing:

19/11/05 23:56:34 ERROR SparkContext: Error initializing SparkContext.
java.lang.IllegalArgumentException: System memory 129761280 must be at least 471859200. Please increase heap size using the --driver-memory option or spark.driver.memory in Spark configuration.
    at org.apache.spark.memory.UnifiedMemoryManager$.getMaxMemory(UnifiedMemoryManager.scala:217)
    at org.apache.spark.memory.UnifiedMemoryManager$.apply(UnifiedMemoryManager.scala:199)
    at org.apache.spark.SparkEnv$.create(SparkEnv.scala:330)
    at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:175)
    at org.apache.spark.SparkContext.createSparkEnv(SparkContext.scala:256)
    at org.apache.spark.SparkContext.<init>(SparkContext.scala:423)
    at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58)
    at io.jaegertracing.spark.dependencies.elastic.ElasticsearchDependenciesJob.run(ElasticsearchDependenciesJob.java:216)
    at io.jaegertracing.spark.dependencies.elastic.ElasticsearchDependenciesJob.run(ElasticsearchDependenciesJob.java:203)
    at io.jaegertracing.spark.dependencies.DependenciesSparkJob.run(DependenciesSparkJob.java:54)
    at io.jaegertracing.spark.dependencies.DependenciesSparkJob.main(DependenciesSparkJob.java:40)

[jpkrohling]

Looks like Spark requires at least 4GB. @kevinearls do we have any tests using Spark?

[kevinearls]

@jpkrohling There's this one: https://github.com/jaegertracing/jaeger-operator/blob/master/test/e2e/spark_dependencies_test.go It's called both here: https://github.com/jaegertracing/jaeger-operator/blob/e7c48398df2f7363f97896936c26c7ea93062a65/test/e2e/elasticsearch_test.go#L69-L78 and here https://github.com/jaegertracing/jaeger-operator/blob/e7c48398df2f7363f97896936c26c7ea93062a65/test/e2e/cassandra_test.go#L75-L83

[jpkrohling]

Looks like I can't reproduce this. Here's what I did:

CR:

apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
  name: simple-prod
spec:
  strategy: production
  storage:
    type: elasticsearch
    options:
      es:
        # Note: This assumes elasticsearch is running in the "default" namespace.
        server-urls: http://elasticsearch.default.svc:9200
        index-prefix: staging
      esIndexCleaner:
        enabled: false # turn the job deployment on and off
        numberOfDays: 30 # number of days to wait before deleting a record
        schedule: "55 23 * * *" # cron expression for it to run
        image: jaegertracing/jaeger-es-index-cleaner # image of the job
    dependencies:
      enabled: true
      elasticsearchNodesWanOnly: true
  resources:
    requests:
      memory: "64Mi"
      cpu: "100m"
    limits:
      memory: "2048Mi"
      cpu: "512m"

Once that gets stable, create a job from the cronjob:

$ kubectl create job --from=cronjob/simple-prod-spark-dependencies simple-prod-spark-dependencies-001
job.batch/simple-prod-spark-dependencies-001 created

It eventually completes and there's nothing wrong in the logs:

$ kubectl get pods
NAME                                       READY   STATUS              RESTARTS   AGE
elasticsearch-0                            1/1     Running             0          3m51s
simple-prod-collector-87cdbd564-tmdx7      1/1     Running             0          110s
simple-prod-query-5b5698c686-tzrg8         2/2     Running             0          110s
simple-prod-spark-dependencies-001-tt6g7   0/1     ContainerCreating   0          12s
$ kubectl logs simple-prod-spark-dependencies-001-tt6g7
19/11/13 15:13:01 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
19/11/13 15:13:07 INFO ElasticsearchDependenciesJob: Running Dependencies job for 2019-11-13T00:00Z, reading from staging-jaeger-span-2019-11-13 index, result storing to staging-jaeger-dependencies-2019-11-13
19/11/13 15:13:12 INFO ElasticsearchDependenciesJob: Done, 0 dependency objects created
19/11/13 15:13:12 INFO ElasticsearchDependenciesJob: Running Dependencies job for 2019-11-13T00:00Z, reading from staging:jaeger-span-2019-11-13 index, result storing to staging:jaeger-dependencies-2019-11-13
19/11/13 15:13:14 INFO ElasticsearchDependenciesJob: Done, 0 dependency objects created
$ kubectl get jobs
NAME                                 COMPLETIONS   DURATION   AGE
simple-prod-spark-dependencies-001   1/1           41s        93s

I'm running it all in minikube, set like this:

$ minikube start --vm-driver kvm2 --cpus 6 --memory 12288

The operator is running as make run, and ES is set with make es.

[wangycc]

shell:

docker run --rm -it -e STORAGE=elasticsearch -e JAVA_OPTS="-Xms30g -Xmx30g" -e ES_CLIENT_NODE_ONLY=false -e ES_NODES_WAN_ONLY=false -e ES_NODES=x.x.x.x jaegertracing/spark-dependencies

19/11/20 12:00:20 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
19/11/20 12:00:22 INFO ElasticsearchDependenciesJob: Running Dependencies job for 2019-11-20T00:00Z, reading from jaeger-span-2019-11-20 index, result storing to jaeger-dependencies-2019-11-20








19/11/20 12:48:36 WARN HeartbeatReceiver: Removing executor driver with no recent heartbeats: 173356 ms exceeds timeout 120000 ms
19/11/20 12:48:36 WARN NettyRpcEnv: Ignored failure: java.util.concurrent.TimeoutException: Cannot receive any reply from 876eed1b85e8:45656 in 10 seconds
19/11/20 12:48:36 ERROR TaskSchedulerImpl: Lost executor driver on localhost: Executor heartbeat timed out after 173356 ms
19/11/20 12:48:36 WARN Executor: Issue communicating with driver in heartbeater
org.apache.spark.rpc.RpcTimeoutException: Futures timed out after [10 seconds]. This timeout is controlled by spark.executor.heartbeatInterval

jaeger-span Index size: 36.6gb

green open jaeger-span-2019-11-20    lAAp46SpTeaUVmI5A6047g 5 1  650309235       0    76gb  36.6gb

jaeger-dependencies RSS memory size: 31.2g

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
22109 185       20   0   32.7g  31.2g   4212 S  2292 49.8 155:16.87 java

Frequently running failure. any recommendation ?

[pavolloffay]

Spark definitely will not start with memory around 1Gi. It requires at least 2Gb to start. The ideal configuration is 4/8GB at least.

Action items:

• I would remove merging of resources from the storage spec with dependencies resources. It does not make sense to merge as the spark requires much more memory. Maybe provide memory default for spark dependencies.
• document memory requirements

[pavolloffay]

Heads up: if the resources in Jaeger CR are specified at the root level it will be applied to spark job. However spark job requires at least 4Gi so be careful and override the resources in the dependencies spec.

It's going to be documented here: https://github.com/jaegertracing/documentation/pull/355