Jaeger-operator: Spark-dependencies how to specify the nodeSelector option

[wangycc]

Spark-dependencies requires a lot of memory, I want to run on the specified node.

Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 1.0 failed 1 times, most recent failure: Lost task 0.0 in stage 1.0 (TID 5, localhost, executor driver): java.lang.OutOfMemoryError: GC overhead limit exceeded
    at java.lang.AbstractStringBuilder.<init>(AbstractStringBuilder.java:68)
    at java.lang.StringBuilder.<init>(StringBuilder.java:89)
    at java.io.ObjectInputStream$BlockDataInputStream.readUTFBody(ObjectInputStream.java:3404)
    at java.io.ObjectInputStream$BlockDataInputStream.readUTF(ObjectInputStream.java:3224)
    at java.io.ObjectInputStream.readString(ObjectInputStream.java:1903)
    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1562)
    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431)
    at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:75)
    at org.apache.spark.serializer.DeserializationStream.readKey(Serializer.scala:156)
    at org.apache.spark.serializer.DeserializationStream$$anon$2.getNext(Serializer.scala:188)
    at org.apache.spark.serializer.DeserializationStream$$anon$2.getNext(Serializer.scala:185)
    at org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:73)
    at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:438)
    at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
    at org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
    at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)
    at org.apache.spark.util.collection.ExternalAppendOnlyMap.insertAll(ExternalAppendOnlyMap.scala:153)
    at org.apache.spark.Aggregator.combineValuesByKey(Aggregator.scala:41)
    at org.apache.spark.shuffle.BlockStoreShuffleReader.read(BlockStoreShuffleReader.scala:90)
    at org.apache.spark.rdd.ShuffledRDD.compute(ShuffledRDD.scala:105)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)

[objectiser]

Unfortunately it is not possible currently - there is a more general open issue that relates to this: #414

[wangycc]

@objectiser @jpkrohling What should I do about this situation? Or is there any good advice?

[jpkrohling]

~@wangycc If you are comfortable with Go, the best option right now is to collaborate on #414 by preparing a PR that would support your case. It's easier to discuss/decide on something concrete.~

~If that's out of question, you can create your own spark-dependencies job that is managed outside of the Operator. Something like:~

(see next messages)

[objectiser]

@wangycc @jpkrohling Version 1.14.0 includes the ability to set affinity and tolerations(as discussed in https://www.jaegertracing.io/docs/1.14/operator/#finer-grained-configuration) for the dependencies as well as other jobs. So #414 should have been closed - will close now.

[jpkrohling]

I thought the issue was still open because there's something missing, but you are right: everything is there already :)