Aws-sdk-java: Java SDK issues on EMR cluster : java.lang.IllegalAccessError: tried to access class

[StayerX]

java.lang.IllegalAccessError: tried to access class 
com.amazonaws.services.s3.AmazonS3ClientConfigurationFactory from class 
com.amazonaws.services.s3.AmazonS3Builder     at com.amazonaws.services.s3.AmazonS3Builder.<clinit>(AmazonS3Builder.java:30)
````

when trying to use 

s3Client = AmazonS3ClientBuilder.standard().withCredentials(new
AWSStaticCredentialsProvider(credentials)).build()

with the jars bellow at the class-path.

aws-java-sdk-1.11.111.jar,\

aws-java-sdk-core-1.11.111.jar,\

aws-java-sdk-s3-1.11.111.jar,\

aws-java-sdk-kinesis-1.11.111.jar

`` EMR 5.4`, on AWS, and this is a scala 2.11 - mvn project (Apache Maven 3.3.9)

Is there any way to use the new, and not the deprecated SDK on the EMR cluster?

[kiiadi]

Do you have any other libraries on your classpath? It's possible that you have a conflicting version? Can you see where the s3 jar's are being loaded from? Can you see what you get when you run this:

AmazonS3Builder.class.getProtectionDomain().getCodeSource().getLocation()

[satz03]

I am facing the same issue , did you find any fix ?

[EXPEgdube]

I also have the same issue.

doing: logger.info(classOf[com.amazonaws.services.s3.AmazonS3Builder[AmazonS3ClientBuilder, AmazonS3]].getProtectionDomain.getCodeSource.getLocation)

gave me: file:/mnt1/yarn/usercache/hadoop/appcache/application_<my-account-id>_0069/container_<my-account-id>_0069_01_000001/__app__.jar

I assume __app__.jar is a renamed version of my compiled code...

[wolodarsky]

I'm running into the same issue submitting an assembly jar to EMR 5.1.0 with the following sbt dependencies:

libraryDependencies ++= Seq(
  "mysql" % "mysql-connector-java" % "6.0.6",
  "org.apache.spark" %% "spark-core" % "2.1.0" % "provided",
  "org.apache.spark" %% "spark-sql" % "2.1.0" % "provided",
  "com.amazonaws" % "aws-java-sdk" % "1.11.108",
  "com.typesafe.slick" %% "slick" % "3.2.0",
  "org.slf4j" % "slf4j-nop" % "1.6.4",
  "org.json4s" %% "json4s-jackson" % "3.5.1",
  "com.typesafe.slick" %% "slick-hikaricp" % "3.2.0",
  "org.scalaj" %% "scalaj-http" % "2.3.0",
  "com.github.scopt" %% "scopt" % "3.5.0"
)

Tried including "com.amazonaws" % "aws-java-sdk-core" % "1.11.115" with same result.

Also getting java.lang.NoClassDefFoundError when using AmazonDynamoDBClientBuilder.defaultClient()

[EXPEgdube]

I was able to get things to work using an older version of aws-sdk, 1.10.77. So I'm guessing this has to do with some incompatibility between my project's dependencies and what the EMR is using.

[wolodarsky]

@EXPEgdube thanks for the update, that's good to know.

I got my app working using the deprecated com.amazonaws.services.s3.AmazonS3Client class on EMR.

Agreed, it seems the Spark dependencies provided by EMR are the issue. The application will run fine locally or on an EC2 when I provide the Spark 2.1.0 jars.

[millems]

I have forwarded this feedback to the EMR service team.

[wolodarsky]

@millems thanks!

[satz03]

Hi, If I understood it right are you suggesting I try an
older version of aws-sdk, possibly 1.10.77.
Thanks
Sathi

On Apr 6, 2017, at 10:40 AM, Tyson Navvab-Safavi [e-mail] wrote:

@millems thanks!

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

[StayerX]

I tried to get classOf[AmazonS3ClientBuilder].getProtectionDomain().getCodeSource().getLocation() but it threw java.lang.IllegalAccessError. My understanding is that spark-submit ignores the jars submitted by the user and uses the jars under /usr/share/aws/aws-java-sdk/ which for EMR 5.4 are of version 1.10.75.1. spark-submit has a parameter which can override the server jars with the user jars, however this can cause other issues. All the paths are configured here : /etc/spark/conf/spark-defaults.conf.

Possible solutions to the issue I found [I think they are all irrelevant to this repository, and more relevant to the EMR cluster setup and the way spark-submit works]:

• Create a custom /etc/spark/conf/spark-defaults.conf that either adds the new jars path or removed the old one (you can do this in the bootstrap phase)
• Use the spark submit option for forcing the server to use the user jars first.
• Use the old API with the old cluster jars.

This should disable the old jars in bootstrap: sed -i.bak "s|:/usr/share/aws/aws-java-sdk/*||g" /etc/spark/conf/spark-defaults.conf (I have not tested it yet)

[ursvasan]

I got the same error

Exception in thread "main" java.lang.IllegalAccessError: tried to access class com.amazonaws.services.s3.AmazonS3ClientConfigurationFactory from class com.amazonaws.services.s3.AmazonS3Builder
    at com.amazonaws.services.s3.AmazonS3Builder.<clinit>(AmazonS3Builder.java:30)
    at org.apache.nutch.indexer.s3.S3WriterPlugin.<init>(S3WriterPlugin.java:39)

S3WriterPlugin.java:39

AmazonS3 s3client = AmazonS3ClientBuilder.standard().withRegion("us-west-2").build();

In this case I verified that appropriate class is loaded and not the wrong one. I was running this code from AWS EMR. My code ran successfully when i run the same code in local mode, but it failed with the "IllegalAccessError" when running in distributed mode.

Finally when i replaced the above code with the deprecated AmazonS3Client it worked in both local and distributed mode.

[sadowski]

I'm also experiencing this same error. Downgrading to 1.10.77 also fixed my issue.

I did have to switch AmazonS3ClientBuilder.defaultClient() with the currently deprecated new AmazonS3Client() to get things working

[varunnvs92]

EMR team have a task in their backlog to upgrade to 1.11.x version. There are working on it but don't have a definite timeline yet. Until then, using v1.10.77 seems to fix the issue as mentioned in previous comments.

[jung-kim]

This is super annoying as we planned on using Athena SDK from EMR, 1.10.x does not have Athena package. Although we can individually add the package or work around but this is disappointing.

[mkanchwala]

+1 Faced same issue with 1.11.163 version of DynamoDB on Amazon EMR vemr-5.7.0

[varunnvs92]

Hi everyone,
Thanks for your patience on this issue. EMR has released a new version 5.8.0 which uses SDK v1.11.160. Using this version of EMR should fix the issue.
Release guide - http://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-whatsnew.html
Announcement - https://aws.amazon.com/about-aws/whats-new/2017/08/updates-to-apache-spark-and-apache-hive-in-amazon-emr-release-5-8-0/

Please try it out and let us know if there are any issues. Thank you.

[millems]

As an extra note: Going forward, you should be able to get the near latest AWS SDK on EMR class path with frameworks like Hadoop and Spark by keeping up with the latest releases of EMR.