Spark-nlp: Pretrained Model not working due to missing Amazon dependencies

[VincentRoma]

Description

When running a pretrained model such as shown in the documentation, an error occurs due to com.amazonaws.services.s3 dependencies missing.

I've been trying to use this lib in a Spark Streaming job, but after trying to debug the dependency issue, I try running the simple exemple from the documention, and I'm getting this error using spark 1.6 or spark 2.2

scala> import spark.implicits._
import spark.implicits._

scala> import com.johnsnowlabs.nlp.pretrained.pipelines.en.BasicPipeline
import com.johnsnowlabs.nlp.pretrained.pipelines.en.BasicPipeline

scala> val data = Seq("hello, this is an example sentence").toDF("mainColumn")
data: org.apache.spark.sql.DataFrame = [mainColumn: string]

scala> BasicPipeline().annotate(data, "mainColumn").show()
java.lang.NoSuchMethodError: com.amazonaws.services.s3.S3ClientOptions.builder()Lcom/amazonaws/services/s3/S3ClientOptions$Builder;
  at com.amazonaws.services.s3.AmazonS3Builder.resolveS3ClientOptions(AmazonS3Builder.java:404)
  at com.amazonaws.services.s3.AmazonS3ClientBuilder.build(AmazonS3ClientBuilder.java:64)
  at com.amazonaws.services.s3.AmazonS3ClientBuilder.build(AmazonS3ClientBuilder.java:28)
  at com.amazonaws.client.builder.AwsSyncClientBuilder.build(AwsSyncClientBuilder.java:46)
  at com.johnsnowlabs.nlp.pretrained.S3ResourceDownloader.client$lzycompute(S3ResourceDownloader.scala:47)
  at com.johnsnowlabs.nlp.pretrained.S3ResourceDownloader.client(S3ResourceDownloader.scala:35)
  at com.johnsnowlabs.nlp.pretrained.S3ResourceDownloader.downloadMetadataIfNeed(S3ResourceDownloader.scala:62)
  at com.johnsnowlabs.nlp.pretrained.S3ResourceDownloader.resolveLink(S3ResourceDownloader.scala:73)
  at com.johnsnowlabs.nlp.pretrained.S3ResourceDownloader.download(S3ResourceDownloader.scala:85)
  at com.johnsnowlabs.nlp.pretrained.ResourceDownloader$.downloadResource(ResourceDownloader.scala:91)
  at com.johnsnowlabs.nlp.pretrained.ResourceDownloader$.downloadPipeline(ResourceDownloader.scala:123)
  at com.johnsnowlabs.nlp.pretrained.ResourceDownloader$.downloadPipeline(ResourceDownloader.scala:118)
  at com.johnsnowlabs.nlp.pretrained.pipelines.PretrainedPipeline.modelCache$lzycompute(PretrainedPipeline.scala:11)
  at com.johnsnowlabs.nlp.pretrained.pipelines.PretrainedPipeline.modelCache(PretrainedPipeline.scala:10)
  at com.johnsnowlabs.nlp.pretrained.pipelines.PretrainedPipeline.annotate(PretrainedPipeline.scala:14)
  ... 50 elided

Steps to Reproduce

1. run a spark-shell
   spark-shell --packages JohnSnowLabs:spark-nlp:1.5.3
2. run documentation exemple:

import spark.implicits._
import com.johnsnowlabs.nlp.pretrained.pipelines.en.BasicPipeline
val data = Seq("hello, this is an example sentence").toDF("mainColumn")
BasicPipeline().annotate(data, "mainColumn").show()

Your Environment

Running on a CDH Cluster (EDH 5.11, Cloudera Certified) with Spark 1.6.1 & Spark 2.2.0

[albertoandreottiATgmail]

@VincentRoma , we will look into this and let you know. Thanks for reporting.

[haimco10]

I'm having the same issue.

[albertoandreottiATgmail]

@haimco10 thanks for reporting your situation. Are you also on CDH?

[GreGGus]

Same issue.
Spark Standalone on OVH.

[danilojsl]

Hi all, I will try to reproduce the issue and will let you know my findings

[saif-ellafi]

I have reached out Cloudera to work together into solving this issue with CDH/CDSW.
@GreGGus Can you point us to OVH website? I am not sure I heard of it.

[VincentRoma]

Hi @saifjsl, thanks for looking this up.
OVH is an european IAAS provider (www.ovh.com) that why @GreGGus having the same issue would demonstrate that's it's not linked to Cloudera maybe...

[haimco10]

@albertoandreottiATgmail . Yes, I'm also on CDH.

[VincentRoma]

@saifjsl, I made some test using a Spark standalone 2.2.0 on the same cluster and it's working. I now assume something is wrong with the spark2.2.0.cloudera1 version, I'm opening a ticket to Cloudera Support. I'll let you know

[saif-ellafi]

Hi all, if anyone can build this PR and test it on Cloudera to confirm it's resolved
https://github.com/JohnSnowLabs/spark-nlp/pull/222
It's a proposal fix by rolling back AWS dependencies by @apiltamang

[sshikov]

BTW, pretrained model example from docs also required internet connection (to download model probably). This is not obvious. Is it possible to download models for offline work?

[saif-ellafi]

Released.

@sshikov there is no way to do that now. But once you download them once through the API, you may have access to cache folder and may be able to read the models offline directly pointing to the path.

Cache folder by default is located in $HOME/cache_pretrained

[sshikov]

Thanks. What if I download it myself and manually put into cache_pretrained?

[Rachana-Baldania]

@saifjsl Thank you, I want to download POS models all trained text files folder (for example, small pos trained tags https://github.com/JohnSnowLabs/spark-nlp/tree/master/src/test/resources/anc-pos-corpus-small) As per my knowledge which you have uploaded in amazon s3 server I don't know how to access it and also about the privilege it is private or public So how can I access it?

[saif-ellafi]

It is public. I may have access to give you some links to download them offline. Lets continue on Slack to do so.