Docker-stacks: Apache Toree - Scala kernel times out with Spark 2.4.2 on all-spark-notebook

[sgloutnikov]

What docker image you are using?

jupyter/all-spark-notebook:ae5f7e104dd5 and previous Spark 2.4.2 images.

What complete docker command do you run to launch the container (omitting sensitive values)?

docker run -p 8888:8888 jupyter/all-spark-notebook:ae5f7e104dd5

What steps do you take once the container is running to reproduce the issue?

Launch a new Apache Toree - Scala kernel.

What do you expect to happen?

Start a new kernel sucessfully and connect to Apache Toree.

What actually happens?

Kernel times out and fails to connect when starting. Stack trace of each attempt:

[I 21:51:33.787 NotebookApp] Writing notebook-signing key to /home/jovyan/.local/share/jupyter/notebook_secret
Starting Spark Kernel with SPARK_HOME=/usr/local/spark
[I 21:51:34.891 NotebookApp] Kernel started: d04e3dc3-f04f-41b8-876c-139e2d0b731c
19/05/02 21:51:36 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Exception in thread "main" java.lang.NoClassDefFoundError: scala/App$class
    at org.apache.toree.Main$.<init>(Main.scala:24)
    at org.apache.toree.Main$.<clinit>(Main.scala)
    at org.apache.toree.Main.main(Main.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
    at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:849)
    at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)
    at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)
    at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
    at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: scala.App$class
    at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    ... 15 more
log4j:WARN No appenders could be found for logger (org.apache.spark.util.ShutdownHookManager).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
[I 21:51:37.887 NotebookApp] KernelRestarter: restarting kernel (1/5), new random ports
Starting Spark Kernel with SPARK_HOME=/usr/local/spark
19/05/02 21:51:39 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Exception in thread "main" java.lang.NoClassDefFoundError: scala/App$class
    at org.apache.toree.Main$.<init>(Main.scala:24)
    at org.apache.toree.Main$.<clinit>(Main.scala)
    at org.apache.toree.Main.main(Main.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
    at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:849)
    at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)
    at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)
    at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
    at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: scala.App$class
    at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    ... 15 more

I think this problem may have started with Spark 2.4.2, because when I reverted to the newest all-spark-notebook image with Spark 2.4.1 everything worked fine. No issues with jupyter/all-spark-notebook:4cdbc9cdb7d1

[parente]

It's possible Apache Toree is not Spark 2.4.2 compatible, though strange that a patch release would cause it.

Maybe @lresende knows?

[sgloutnikov]

The Scala upgrade to 2.12 may be the issue making Apache Toree incompatible? From the release announcement: Note that Scala 2.11 support is deprecated from 2.4.1 onwards. As of 2.4.2, the pre-built convenience binaries are compiled for Scala 2.12. Spark is still cross-published for 2.11 and 2.12 in Maven Central, and can be built for 2.11 from source.

[lresende]

@sgloutnikov by default, Apache Toree is built with Scala 2.11, in the case of running with Apache Spark using Scala 2.12 you will need to recompile Apache Toree properly. There is an open issue on the Toree JIRA to publish Scala 2.12 binaries with the next release.

[lhoss]

There is an open issue on the Toree JIRA to publish Scala 2.12 binaries with the next release.

@lresende you meant https://issues.apache.org/jira/browse/TOREE-423 ?

ps: In Toree someone had the same issue with Spark 2.4.2, now I cross linked the issues:
https://issues.apache.org/jira/browse/TOREE-500