Spark-nlp: Upgrade aws-java-sdk to a higher version than 1.11.603

[bbodnar]

I've got the NPE https://github.com/aws/aws-sdk-java/issues/2070 running a Spark-NLP test-case. The error is fixed long ago, Spark-NLP should use a newer version of aws-java-sdk (at least 1.11.610, preferably the newest one 1.11.904).

[maziyarpanahi]

Could you please complete the template so we understand the issue? We cannot upgrade the aws-sdk to any desired version. The sdk has a direct effect on EMR compatibility and we have lots of users on 5.26.0 and 5.27.0 which won't support the newer aws-sdk.

Please complete the template fully and provide steps to reproduce the error so we can suggest a fix or a workaround if possible.

• https://github.com/JohnSnowLabs/spark-nlp#emr-support
• https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-release-5x.html
• https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-release-6x.html

[bbodnar]

Description

NPE occurs in the Amazon AWS library attempting to download resources required by Spark-NLP.

An exception or error caused a run to abort. 
java.lang.NullPointerException
    at com.amazonaws.ClientConfiguration.getProxyUsernameEnvironment(ClientConfiguration.java:874)
    at com.amazonaws.ClientConfiguration.getProxyUsername(ClientConfiguration.java:902)
    at com.amazonaws.http.settings.HttpClientSettings.getProxyUsername(HttpClientSettings.java:90)
    at com.amazonaws.http.settings.HttpClientSettings.isAuthenticatedProxy(HttpClientSettings.java:182)
    at com.amazonaws.http.apache.client.impl.ApacheHttpClientFactory.addProxyConfig(ApacheHttpClientFactory.java:96)
    at com.amazonaws.http.apache.client.impl.ApacheHttpClientFactory.create(ApacheHttpClientFactory.java:75)

(full stacktrace.txt)

Steps to Reproduce

Start the test _com.johnsnowlabs.nlp.annotators.parser.dep.DependencyParserModelTestSpec_ behind a corporate-proxy specified as environment-variable (_http_proxy_ and _https_proxy_) of the form _http://host:port/_.

Context

Test is not able to start, it is obviously generally not possible to access resources on S3.

Your Environment

• Spark NLP version: Master (2.6.3)
• Java version (java -version): java version "1.8.0_201"
  Java(TM) SE Runtime Environment (build 1.8.0_201-b09)
  Java HotSpot(TM) 64-Bit Server VM (build 25.201-b09, mixed mode)
• Setup and installation (Pypi, Conda, Maven, etc.): source-project Spark-NLP (master) opened in Intellij IDEA v2020.2.3
• Operating System and version: Ubuntu 20.04 LTS

[maziyarpanahi]

Hi @rohit-nlp
How do you set the config for machines behind the proxy? (I forgot the actual config to be set)