Spark-on-k8s-operator: extraJavaOptions config conflict

[andrewgdavis]

operator-version: 0.8.2

Problem Description:

  sparkConf:
    "spark.driver.extraJavaOptions": "-XshowSettings:vm -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap"
    "spark.executor.extraJavaOptions": "-XshowSettings:vm -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap"

settings are in conflict with

      #    prometheus:
      #      jmxExporterJar: "/prometheus/jmx_prometheus_javaagent-0.3.1.jar"
      #      #jmxExporterJar: "/opt/spark/jars/collector-0.3.1.jar"
      #      port: 8090

When both are applied within the same sparkapplication.yaml, java is invoked with:

-Xmx512m -javaagent:/prometheus/jmx_prometheus_javaagent-0.3.1.jar=8090:/etc/metrics/conf/prometheus.yaml

When the prometheus section is commented out, the sparkConf settings are applied correctly:

-Xmx512m -XshowSettings:vm -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap

validation of the settings were found by issuing the following when different settings were applied:
` kubectl exec -t spark-pi-driver ps aws

[EladDolev]

Still undocumented at the moment, but you should pass these using spec.driver.javaOptions and spec.executor.javaOptions and not through sparkConf

[andrewgdavis]

Confirmed https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/issues/489#issuecomment-489812486 was the correct way forward. Feel free to close (or keep open for documentation purposes).

[liyinan926]

Will keep this open until the documentation gets fixed. Thanks @EladDolev for pointing it out.

[liyinan926]

Documentation updated in https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/pull/513. Closing this.

[skonto]

@liyinan926 @EladDolev since the operator exposes SparkConf it should make this correct and support both in a correct manner. That means fixing the conflict if it is not fixed already. #513 mentions what is the preference but does not state the problem and new users will have to dig into the issue history to understand their options.