Aws-sdk-java: Latest AWS Java SDK is taking old version of jackson jar

[anuragmishra1991]

Currently I am working to add AWS Java SDK in my existing application to perform operations related to AWS cloud. While importing the aws java sdk for latest version i.e.1.11.113 I am getting run time exception. After researching about exception the reason is java sdk using deprecated version or incompatible version for jackson jar.

The sdk is using jackson jar of version 4.1 while required version must at least 5. Even I am giving latest jackson jar but still it is taking reference of jar present in SDK. Below code snapshot will help to identify this easily

Caused by: java.lang.NoSuchMethodError: com.fasterxml.jackson.databind.ObjectMapper.enable([Lcom/fasterxml/jackson/core/JsonParser$Feature;)Lcom/fasterxml/jackson/databind/ObjectMapper;
at com.amazonaws.partitions.PartitionsLoader.(PartitionsLoader.java:54) [aws-java-sdk-core-1.11.113.jar:]
at com.amazonaws.regions.RegionMetadataFactory.create(RegionMetadataFactory.java:30) [aws-java-sdk-core-1.11.113.jar:]
at com.amazonaws.regions.RegionUtils.initialize(RegionUtils.java:64) [aws-java-sdk-core-1.11.113.jar:]
at com.amazonaws.regions.RegionUtils.getRegionMetadata(RegionUtils.java:52) [aws-java-sdk-core-1.11.113.jar:]
at com.amazonaws.regions.RegionUtils.getRegion(RegionUtils.java:105) [aws-java-sdk-core-1.11.113.jar:]
at com.amazonaws.regions.Region.getRegion(Region.java:43) [aws-java-sdk-core-1.11.113.jar:]

While running I am getting mainly two exceptions : -

1. Caused by: java.lang.NoClassDefFoundError: Could not initialize class com.amazonaws.partitions.PartitionsLoader

1. Caused by: java.lang.NoSuchMethodError: com.fasterxml.jackson.databind.ObjectMapp
   How can I avoid code to use this particular jar and make it use latest jar. I am using gradle as build tool.

I am facing exception while performing operation to send string through AWS SNS. I am using gradle for dependencies.

[varunnvs92]

The SDK is depending on jackson v2.6.6. The jackson dependency cannot be upgraded to 2.7 or later because from v2.7 JDK baseline will be Java 7 / JDK 7. But SDK still supports JDK 6.

Not sure what you meant by "The sdk is using jackson jar of version 4.1 while required version must at least 5". Can you clarify? You can provide explicit dependency on higher level of jackson if you are using maven.

[anuragmishra1991]

@varunnvs92

As per stack trace it is pointing towards use of jackson jar of version 4.1 but we require higher version.

I have included the latest jakson version in my pom.xml but for this particular operation it is taking that from sdk jar.
To confirm below code gave me output of older version rather then newer version that means it is internally taking reference from aws-sdk jar.
(com.fasterxml.jackson.databind.ObjectMapper.class.getProtect‌​ionDomain().getCodeS‌​ource().getLocation(‌​))

[varunnvs92]

Can you share your pom.xml file?
Can you also provide the output of "mvn dependency:tree" command?

[anuragmishra1991]

I am using gradle in my project and dependencies as as follows :-
runtime/compile group: 'com.amazonaws', name: 'aws-java-sdk', version: '1.11.113'

Stracktrace is :-

d by: java.lang.NoSuchMethodError: com.fasterxml.jackson.databind.ObjectMapper.enable([Lcom/fasterxml/jackson/core/JsonParser$Feature;)Lcom/fasterxml/jackson/databind/ObjectMapper;
at com.amazonaws.partitions.PartitionsLoader.(PartitionsLoader.java:54) [aws-java-sdk-core-1.11.113.jar:]
at com.amazonaws.regions.RegionMetadataFactory.create(RegionMetadataFactory.java:30) [aws-java-sdk-core-1.11.113.jar:]
at com.amazonaws.regions.RegionUtils.initialize(RegionUtils.java:64) [aws-java-sdk-core-1.11.113.jar:]
at com.amazonaws.regions.RegionUtils.getRegionMetadata(RegionUtils.java:52) [aws-java-sdk-core-1.11.113.jar:]
at com.amazonaws.regions.RegionUtils.getRegion(RegionUtils.java:105) [aws-java-sdk-core-1.11.113.jar:]
at com.amazonaws.regions.Region.getRegion(Region.java:43) [aws-java-sdk-core-1.11.113.jar:]
.......more

[varunnvs92]

I don't see explicit dependency on jackson in your dependencies:
runtime/compile group: 'com.amazonaws', name: 'aws-java-sdk', version: '1.11.113'

I am not that familiar with gradle dependency management. You need to exclude jackson transitive dependency from sdk and add your own version of jackson. Following links might help:
http://stackoverflow.com/questions/30728533/gradle-override-transitive-dependency-by-version-classifier
https://discuss.gradle.org/t/how-to-exclude-transitive-dependency/2119

This looks like gradle dependency question rather than Java SDK specific issue. If above links doesn't work, you can involve broader developer community by asking the question on stack overflow. Thank you.

[varunnvs92]

Closing due to inactivity. Feel free to reopen if you still have the issue.

[brainstorm]

I'm facing this one exactly right now with AWS Java SDK v2, here's what happens in runtime given this dependency list:

https://github.com/umccr/igv/blob/fix_aws_jtree_regression/build.gradle#L134

This is a runtime error:

$ ./gradlew createMacAppDist && ./build/IGV-MacApp-dist/IGV_user.app/Contents/MacOS/IGV
(...)
ERROR [2020-05-28T15:58:32,077]  [DefaultExceptionHandler.java:49] [Thread-4]  Unhandled exception
java.lang.NoClassDefFoundError: com/fasterxml/jackson/core/JsonFactory
    at software.amazon.awssdk.protocols.json.internal.AwsStructuredPlainJsonFactory.<clinit>(AwsStructuredPlainJsonFactory.java:34) ~[aws-json-protocol-2.13.24.jar:?]
    at software.amazon.awssdk.protocols.json.BaseAwsJsonProtocolFactory.getSdkFactory(BaseAwsJsonProtocolFactory.java:157) ~[aws-json-protocol-2.13.24.jar:?]
    at software.amazon.awssdk.protocols.json.BaseAwsJsonProtocolFactory.<init>(BaseAwsJsonProtocolFactory.java:73) ~[aws-json-protocol-2.13.24.jar:?]
    at software.amazon.awssdk.protocols.json.AwsJsonProtocolFactory.<init>(AwsJsonProtocolFactory.java:30) ~[aws-json-protocol-2.13.24.jar:?]
    at software.amazon.awssdk.protocols.json.AwsJsonProtocolFactory$Builder.build(AwsJsonProtocolFactory.java:46) ~[aws-json-protocol-2.13.24.jar:?]
    at software.amazon.awssdk.services.cognitoidentity.DefaultCognitoIdentityClient.<init>(DefaultCognitoIdentityClient.java:123) ~[cognitoidentity-2.13.24.jar:?]
    at software.amazon.awssdk.services.cognitoidentity.DefaultCognitoIdentityClientBuilder.buildClient(DefaultCognitoIdentityClientBuilder.java:29) ~[cognitoidentity-2.13.24.jar:?]
    at software.amazon.awssdk.services.cognitoidentity.DefaultCognitoIdentityClientBuilder.buildClient(DefaultCognitoIdentityClientBuilder.java:22) ~[cognitoidentity-2.13.24.jar:?]
    at software.amazon.awssdk.core.client.builder.SdkDefaultClientBuilder.build(SdkDefaultClientBuilder.java:124) ~[sdk-core-2.13.24.jar:?]
    at org.broad.igv.util.AmazonUtils.GetCognitoAWSCredentials(AmazonUtils.java:126) ~[igv.jar:?]
    at org.broad.igv.google.OAuthProvider.fetchTokens(OAuthProvider.java:229) ~[igv.jar:?]
    at org.broad.igv.google.OAuthProvider.setAuthorizationCode(OAuthProvider.java:173) ~[igv.jar:?]
    at org.broad.igv.google.OAuthProvider.setAuthorizationCode(OAuthProvider.java:166) ~[igv.jar:?]
    at org.broad.igv.google.OAuthUtils.setAuthorizationCode(OAuthUtils.java:167) ~[igv.jar:?]
    at org.broad.igv.batch.CommandListener.processClientSession(CommandListener.java:203) ~[igv.jar:?]
    at org.broad.igv.batch.CommandListener.run(CommandListener.java:126) ~[igv.jar:?]
    at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: java.lang.ClassNotFoundException: com.fasterxml.jackson.core.JsonFactory
    at jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581) ~[?:?]
    at jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178) ~[?:?]
    at java.lang.ClassLoader.loadClass(ClassLoader.java:522) ~[?:?]
    ... 17 more
^CINFO [2020-05-28T16:02:00,212]  [ShutdownThread.java:46] [Thread-1]  Shutting down

And here's what happens when I add this line to the build.gradle above:

        [group: 'com.fasterxml.jackson.core', name: 'jackson-core', version: '2.11.0']

This is a build time error after adding the bumped jackson-core library above (AWS JAVA SDK v2 seems to have 2.10.4 still at the time of writing this):

$ ./gradlew createMacAppDist && ./build/IGV-MacApp-dist/IGV_user.app/Contents/MacOS/IGV
(...)
Using system JDK.
openjdk version "11.0.7" 2020-04-14 LTS
OpenJDK Runtime Environment Corretto-11.0.7.10.1 (build 11.0.7+10-LTS)
OpenJDK 64-Bit Server VM Corretto-11.0.7.10.1 (build 11.0.7+10-LTS, mixed mode)
Error occurred during initialization of boot layer
java.lang.module.FindException: Two versions of module com.fasterxml.jackson.core found in ./build/IGV-MacApp-dist/IGV_user.app/Contents/MacOS/../Java/lib (jackson-core-2.11.0.jar and jackson-core-2.10.4.jar)

On the other hand, both IntelliJ IDE and ./gradlew run do not exhibit the issues above, so my guess is that the createMacAppDist Gradle task above is missing something? Certainly does not look so on the IDE Gradle cache nor the bundled Mac App though (before putting in the jackson-core 2.11.0 gradle.build compile group):

So would you mind reopening this one @varunnvs92 ? Happy to investigate the underlying root cause, I'm curious :)

/cc @reisingerf @millems