Spark-on-k8s-operator: How to pass in application arguments in yaml file?

Created on 25 Jan 2019 · 9Comments · Source: GoogleCloudPlatform/spark-on-k8s-operator

Using spark-submit, I can pass in application-arguments at the end:

./bin/spark-submit \
  --class <main-class> \
  --master <master-url> \
  --deploy-mode <deploy-mode> \
  --conf <key>=<value> \
  ... # other options
  <application-jar> \
  [application-arguments]

I am wondering what's the syntax of passing in these application-arguments in yaml files?

Thanks.

Source

goplusgo

Most helpful comment

--kafka-bootstrap-server: my-kafka:9092
--kafka-input-topic topicA
--kafka-output-topic topicB

goplusgo on 14 May 2020

👍2

All 9 comments

spec:
    arguments:
    - arg1
    - arg2

liyinan926 on 25 Jan 2019

@liyinan926 thanks. For instance, if my application-arguments is:

 --kafka-bootstrap-servers kafka:9092

in the spark-submit command. What's the best way to convert it to the arguments section? Like this?

spec:
    arguments:
    - --kafka-bootstrap-servers
    - kafka:9092

goplusgo on 25 Jan 2019

BTW, looks like it doesn't support multiple args. When I use:

- --kafka-bootstrap-servers kafka:9092 
- --kafka-input-topic
- output
- --kafka-output-topic
- output2

I saw the driver pod failed to start up. And the command it converted to is:

exec /sbin/tini -s -- /opt/spark/bin/spark-submit --conf spark.driver.bindAddress=<IP> --deploy-mode client --properties-file /opt/spark/conf/spark.properties --class <className> spark-internal --kafka-bootstrap-servers 'kafka:9092 - --kafka-input-topic - output - --kafka-output-topic - output2'

So what's the correct way to use multiple arguments in the yaml file? Thanks!

goplusgo on 25 Jan 2019

Now it works. Please close the issue.

goplusgo on 26 Jan 2019

👎2

How did you fix this issue.. @goplusgo
Thanks!!

prasad-1729 on 14 May 2020

In my case I needed to use "--" in front of the application arguments (long form) instead of "-" (short form) and I couldn't use the short form for some reason

kyprifog on 14 May 2020

Can i get example for that...? @kyprifog
Thanks!!

prasad-1729 on 14 May 2020

--kafka-bootstrap-server: my-kafka:9092
--kafka-input-topic topicA
--kafka-output-topic topicB

goplusgo on 14 May 2020

👍2

I've landed here with randomness googling because it was exactly what I needed, but I think this is missing in the documentation.
An integration in the docs would be greatly appreciated, if not on the older versions at least on the current stable.