Spark-nlp: Support for Scala 2.12.X

[oscarvarto]

Spark 2.4.0 introduced support for Scala 2.12. It would be great to have a release compatible with current stable version of Scala.

[dengelha]

Yes !!! Same here :+1:

[maziyarpanahi]

I am all with starting the conversation about what it takes to make this library support Scala 2.12, but let's not forget the support for Scala 2.12 in Spark 2.4.x is experimental as it is stated in the release notes. Also, almost the entire Spark community is still on 2.11 (some even 2.10) and they will stay on it for years to come. (That is why it took so long for Spark developers and maintainers to finally support 2.12 - Spark developers are less enthusiastic about Scala releases than real Scala developers out there)
https://contributors.scala-lang.org/t/spark-as-a-scala-gateway-drug-and-the-2-12-failure/1747

We have to also realize the maintainers/developers of this library have to double their resources and efforts to support two very different versions of Scala and it doesn't make sense if the community behind Spark is still stuck in 2.11.x.

So first, let's talk about what it takes to have this library in Scala 2.12? What are the difficulties? Where and how we can all help? Can we clone this library and compile it with 2.12 and see the list of errors? Would that be enough?

I am very happy to start the conversation here and make this a reference to what we all have to do. Let's make this place a todo list and we all cheap in and give it a shot :)

[dengelha]

Nice, so then, let's create TODOs :-)

My motivation concerning the upgrade is the permanent conversation with my DEV section. "No, we cannot use spark, as we need to have 2.12!". In consequence, sticking to CoreNLP.. and even considering spaCy (no offense but I am a Scala guy) for future tasks :(

So, let's rock'n'roll our TODOs..

[maziyarpanahi]

Great to hear that @dengelha! Since you are the Scala guy you might have some ideas as to where to start?

I have read this post, it is still helpful to this day:
https://www.scala-lang.org/news/2.12.0/
Also, keep an eye on the release notes for the newest changes:
https://github.com/scala/scala/releases

Update: The util library by Twitter has always tried to support multiple Scala versions. Right now they support crossScalaVersions := Seq("2.11.12", "2.12.7"): https://github.com/twitter/util/blob/develop/build.sbt

[dengelha]

Thanx for the advice @maziyarpanahi,

hopefully, I can make it work during my vacation in this month. I just forked. This will be my very first contribution to a github project, so please do not expect me to succeed. If you have any suggestions, please let me know. Hopefully, there are others trying the 2.12 upgrade as well and we can exchange thoughts.

[maziyarpanahi]

@dengelha You are very welcome!
Your contribution is highly appreciated here without a doubt.

We will come back here to share and see how we are doing in making spark-nlp compatible with Spark 2.4 and also Scala 2.12.

Thanks again mate,

[reynoldsm88]

Has anything happened in this regard? I see that the packages are still only up to Scala 2.11 in Maven Central so I assume something happened?

[maziyarpanahi]

@reynoldsm88 not yet. We have it in our todo list and waiting for Apache Spark to have a better 2.12 support as they released it initially as experimental, then released it officially, then a version later revert it back! But we are aware of this and looking forward to 2.12 ourselves.

[reynoldsm88]

@maziyarpanahi ah okay, I didn't realize they reverted it back 😞 . I read this article from earlier this year and assumed it had finally been stabilized : How Lightbend Got Apache Spark To Work With Scala 2.12.

Oh well, I guess then probably we'll have to wait to Spark 3.x because there isn't a 2.5 planned I don't think, right?

[maziyarpanahi]

Yes, they did a release and said they will deprecate 2.11 support and we thought this is it. But then in another minor release, they changed their mind:

Note that 2.4.3 switched the default Scala version from Scala 2.12 to Scala 2.11, which is the default for all the previous 2.x releases except 2.4.2.

My guess is they will drop the support for 2.11 in the version 3.0 and start support 2.12 officially either from 3.0 or something close to it before dropping 2.11 for good. Unfortunately, going back to 2.11 showed the community they are not ready to say goodbye to 2.11 and get fully behind Scala 2.12 which is sad but we are keeping an eye on it and fully embrace 2.12 when it becomes the default.

[reynoldsm88]

@maziyarpanahi thanks for the insite, good to hear 👍

[f1yegor]

Bump this up as spark 3.0 is released compiled for 2.12.

Dirty implementation to compile locally https://github.com/f1yegor/spark-nlp/tree/spark3.0_scala2.12

Hope to get somebody inspired)

[rohit-nlp]

Ok, I am taking this up to Support Spark 3.0 .. Just to quote as mentioned in Scala doc " Scala 2.11 and 2.12 are _mostly_ source compatible to facilitate cross-building, they are not binary compatible." which means just compiling the spark-NLP with 2.12 should work in scala 2.12 .. however, this means we would have 2 separate versions for Spark-NLP.. let's see I will come back with a list of issues and then will plan how to fix them.

[wjuncdl]

May I ask what's going on in this regard? Do we have a plan to make this issue done in near future? In our product, we have a text mining function that depends on spark-nlp, now our product intends to support spark3.0, we expect spark-nlp would have a version based on scala2.12 that can used for spark3.0...

[maziyarpanahi]

Hi,
We planned for Apache Spark 3.x which comes with Scala 2.12 and support of Java 11. But this won't happen this year. We have over 330+ pretrained models and pipelines which are supported in Apache Spark 2.3.x and Apache Spark 2.4.x. These two major versions are fully supported by almost all cloud providers such as AWS, Azure, Google, Cloudera, HDP, Databricks, etc.

In order to support Apache Spark 3.x, we have to retrain and maintain over 330+ models and pipelines for Apache Spark 3.x and the same amount for 2.3.x and 2.4.x since those won't go anywhere anytime soon. We need to see a market adaptation before going through this long and painful process. At the moment, only people who do pip install pyspark without knowing the version, direct download, and Databricks support Apache Spark 3.x when it comes to the major providers/platforms.

That's bein said, we have planned the limited support of Apache Spark 3.x (in terms of models/pipelines) in the first quarter of 2021 if everything goes as plan. I suggest following this issue as Scala 2.12 only happens with Apache Spark 3.x in Spark NLP: https://github.com/JohnSnowLabs/spark-nlp/issues/940

[koertkuipers]

databricks supports spark 3, aws (emr) support spark 3, google (dataproc) support spark 3.
scala 2.12 has become the default for most big data projects (spark, kafka, flink). scala 2.11 is sometimes still supported (flink i think) or more often dropped in latest version (spark, kafka). this trend will now accelerate rapidly as newer versions of libraries are no longer available for scala 2.11.