Aws-sdk-java-v2: Unable to load an HTTP implementation from any provider in the chain. You must declare a dependency on an appropriate HTTP implementation or pass in an SdkHttpClient explicitly to the client builder

[jobetdelima]

Hi,

I'm trying to do a GetObject() call from S3 using an SBT project. When I try to run the project within my IDE, or do "sbt run", things work -- that is, I'm able to read a file from S3, and do what I need to do with it.

However, when I do "sbt assembly" to get an uber jar (which I plan on uploading to an AWS Lambda), I end up with the below exception:

software.amazon.awssdk.core.exception.SdkClientException: Unable to load an HTTP implementation from any provider in the chain. You must declare a dependency on an appropriate HTTP implementation or pass in an SdkHttpClient explicitly to the client builder.

I tried doing something like below, but to no avail. Exact same behaviour as before I tried to explicitly create an HttpClient.

val apacheClient: SdkHttpClient = ApacheSdkHttpClientFactory.builder()
                                                                .build()
                                                                .createHttpClient()

    val client = S3Client.builder()
                          .httpConfiguration(ClientHttpConfiguration.builder()
                                                                    .httpClient(apacheClient)
                                                                    .build())
                         .region(Region.US_EAST_1)
                         .build()
    val stream: ResponseInputStream[GetObjectResponse] = client.getObject(GetObjectRequest.builder()
                                                            .bucket(s3Bucket)
                                                            .key(s3EventKey)
                                                            .build())

the lib dependencies look like this on my build.sbt:

libraryDependencies ++=
  Seq(
    "org.apache.kafka" %% "kafka" % "1.0.0",
    "com.typesafe" % "config" % "1.3.1",
//    "com.fasterxml.jackson.core" % "jackson-core" % "2.9.1",
//    "com.fasterxml.jackson.core" % "jackson-databind" % "2.9.1",
    "org.apache.avro" % "avro" % "1.8.2",
    "io.confluent" % "kafka-avro-serializer" % "3.3.1",
//    "com.typesafe.play" %% "play-json" % "2.6.7",
//    "com.typesafe.akka" %% "akka-http" % "10.0.11",
    //"software.amazon.awssdk" % "dynamodb" % "2.0.0-preview-2",
    "software.amazon.awssdk" % "dynamodb" % "2.0.0-preview-9",
    "software.amazon.awssdk" % "core" % "2.0.0-preview-9",
   // "com.amazonaws" % "aws-java-sdk" % "1.11.297",
    "com.amazonaws" % "aws-lambda-java-core" % "1.2.0",
    "com.amazonaws" % "aws-lambda-java-events" % "2.1.0",
   // "com.amazonaws" % "aws-java-sdk-s3" % "1.11.301",
    "software.amazon.awssdk" % "sts" % "2.0.0-preview-9",
    "software.amazon.awssdk" % "s3" % "2.0.0-preview-9",
    "software.amazon.awssdk" % "aws-http-client-apache" % "2.0.0-preview-1"
  )

Your Environment

• AWS Java SDK version used: 2.0
• JDK version used: 1.8
• Operating System and version: Mac OS High Sierra (although I get the same exception after uploading and running the uber jar on an AWS lambda function).

[dagnir]

I'm not super familiar with sbt but I suspect it might have to do with the fact that the HTTP clients are declared as runtime dependencies of the service client modules:

https://github.com/aws/aws-sdk-java-v2/blob/521015e59b04059ac28ac3b6414231a4513af480/services/pom.xml#L163-L174

Can you check to ensure that the runtime classpath is being included in your assembly?

This looks related to https://github.com/sbt/sbt-assembly/issues/120

[ethul]

I had a similar issue using Proguard.

In order to resolve the problem I had to ensure that the appropriate
META-INF/services/software.amazon.awssdk.http.SdkHttpService
file was being included in the resulting jar.

For example, to use the apache-client, the following file should be present in the jar:

https://github.com/aws/aws-sdk-java-v2/blob/f0b1a54bdcdd7168c676bee7e110becf7d724b31/http-clients/apache-client/src/main/resources/META-INF/services/software.amazon.awssdk.http.SdkHttpService

[oripwk]

@dagnir

I'm experiencing the exact same issue. Checked the final zip file, and it indeed does not include apache-client dependency.

What's the reason behind making it a runtime dependency? The developer cannot know it advance that the code will fail at runtime.

[warrenronsiek]

Solved the problem by including an http client and passing a client builder (as opposed to a HttpConfiguration). E.g.

import software.amazon.awssdk.http.nio.netty.NettyNioAsyncHttpClient

S3AsyncClient.builder
    .credentialsProvider(credentials)
    .region(Region.US_EAST_1)
    .httpClientBuilder(NettyNioAsyncHttpClient.builder()
      .maxConcurrency(10)
      .maxPendingConnectionAcquires(1000))
    .build()

I think this forces sbt-assembly to include the http library during compile time.

[khalil-jibran06]

Checkout below link as well where explicit httpClient setting solves the issue in normal STS role assumption scenario through AWS SDK 2.x.
https://github.com/khalil-jibran06/StsAssumeRoleCredentialsProvider

[yaswanth2216]

I am running into the same issue when invoking Transcribe Streaming Client
TranscribeStreamingAsyncClient client = TranscribeStreamingAsyncClient.builder().credentialsProvider(getCredentials()).region(Region.US_WEST_2).build(); CompletableFuture<Void> result = client.startStreamTranscription(getRequest(), new com.amazon.alexa.aws.TranscribeStreaming.AudioStreamPublisher(getStreamFromMic()), getResponseHandler()); result.get(); client.close();

software.amazon.awssdk.core.exception.SdkClientException: Unable to load an HTTP implementation from any provider in thechain. You must declare a dependency on an appropriate HTTPimplementation or pass in an SdkHttpClient explicitly to theclient builder.

[oripwk]

This error happens because of wrong configuration of SBT assembly.
In your build.sbt whenever you have

case PathList("META-INF", _*) => MergeStrategy.discard

it will also discard the important services dir which includes info on the HTTP client. To resolve this, just do:

case PathList("META-INF", "MANIFEST.MF") => MergeStrategy.discard

[user20161119]

I have the jars http-client-spi-2.14.7.jar,netty-nio-client-2.14.7.jar in the classpath
but when import the SdkAsyncHttpClient httpClient = software.amazon.awssdk.http.nio.netty.NettyNioAsyncHttpClient.builder().maxConcurrency(100).maxPendingConnectionAcquires(10_000).build();
get the package can't be resolved error as follow,

but navigate from the above code in Eclipse,success to the right class:

any suggestion on the issue,thanks in advance.

[rjrb]

I have the same issue as @user20161119. Any solutions?

[user20161119]

I have the same issue as @user20161119. Any solutions?

check your maven dependency tree,i solved it to add netty dependency and work

[rjrb]

I have the same issue as @user20161119. Any solutions?

check your maven dependency tree,i solved it to add netty dependency and work

You're right... I had already the dependecy declared, but as runtime:

runtime group: 'software.amazon.awssdk', name: 'netty-nio-client', version: '2.15.32'

I changed it to compile and it worked:

compile group: 'software.amazon.awssdk', name: 'netty-nio-client', version: '2.15.32'

Thanks.

[debora-ito]

Hello! This issue has been open for some time now, I'm marking to auto-close soon. If there's anything further pending from the SDK team please reach out with a comment or a new issue.

[orozcoadrian]

fwiw, adding an explicit http client worked for me: https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/lambda-optimize-starttime.html
Using: software.amazon.awssdk:url-connection-client:2.15.66