I have recently re-written a service to use the newer AWS SDK (v2), but I am struggling with an error I just can't seem to figure out.
javax.net.ssl.SSLPeerUnverifiedException: Certificate for <s3s-nv.s3.amazonaws.com> doesn't match any of the subject alternative names: [*.s3.amazonaws.com, s3.amazonaws.com]
This service communicates a lot with a few other AWS services, and everything there is fine, but when it is running in production, it seems to have issues writing to customer buckets with the error above.
I have gone ahead and changed this project to only use AWS SDK v1 for S3 with almost identical commands, and it works fine.
I am running in an EC2 instance, using JDK 8, and using the latest version of this library.
javax.net.ssl.SSLPeerUnverifiedException: Certificate for <s3s-nv.s3.amazonaws.com> doesn't match any of the subject alternative names: [*.s3.amazonaws.com, s3.amazonaws.com]
at org.apache.http.conn.ssl.SSLConnectionSocketFactory.verifyHostname(SSLConnectionSocketFactory.java:507)
at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:437)
at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:384)
at software.amazon.awssdk.http.apache.internal.conn.SdkTlsSocketFactory.connectSocket(SdkTlsSocketFactory.java:113)
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142)
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376)
at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at software.amazon.awssdk.http.apache.internal.conn.ClientConnectionManagerFactory$Handler.invoke(ClientConnectionManagerFactory.java:80)
at com.sun.proxy.$Proxy58.connect(Unknown Source)
at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
at software.amazon.awssdk.http.apache.internal.impl.ApacheSdkHttpClient.execute(ApacheSdkHttpClient.java:72)
at software.amazon.awssdk.http.apache.ApacheHttpClient.execute(ApacheHttpClient.java:232)
at software.amazon.awssdk.http.apache.ApacheHttpClient.access$500(ApacheHttpClient.java:98)
at software.amazon.awssdk.http.apache.ApacheHttpClient$1.call(ApacheHttpClient.java:213)
at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.executeHttpRequest(MakeHttpRequestStage.java:66)
at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.execute(MakeHttpRequestStage.java:51)
at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.execute(MakeHttpRequestStage.java:35)
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:73)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:42)
at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:77)
at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:39)
at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:64)
... 31 common frames omitted
Wrapped by: software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request: Certificate for <s3s-nv.s3.amazonaws.com> doesn't match any of the subject alternative names: [*.s3.amazonaws.com, s3.amazonaws.com]
at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:98)
at software.amazon.awssdk.core.exception.SdkClientException.create(SdkClientException.java:43)
at software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper.setLastException(RetryableStageHelper.java:201)
at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:66)
at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:34)
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:56)
at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:36)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:80)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:60)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:42)
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26)
at software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:189)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:121)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute(BaseSyncClientHandler.java:147)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:101)
at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45)
at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:55)
at software.amazon.awssdk.services.s3.DefaultS3Client.copyObject(DefaultS3Client.java:970)
at software.amazon.awssdk.services.s3.S3Client.copyObject(S3Client.java:1503)
at com.sednanetwork.ingest.receiver.Receiver.copyEmailToTenantBucket(Receiver.kt:131)
at com.sednanetwork.ingest.receiver.Receiver.doReceive(Receiver.kt:86)
at com.sednanetwork.ingest.receiver.Receiver.receive(Receiver.kt:52)
at com.sednanetwork.ingest.IngestApplication$run$handler$1.invoke(IngestApplication.kt:53)
at com.sednanetwork.ingest.IngestApplication$run$handler$1.invoke(IngestApplication.kt:19)
at com.sednanetwork.sqs.SqsConsumer$launchWorker$1$1$1.invokeSuspend(SqsConsumer.kt:78)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:56)
at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:738)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678)
at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665)
@inssein did some investigation and I believe this is related to a regression found in Apache's httpclient version 4.5.9. _DefaultHostNameVerifier_ stopped matching certificates for hostnames with wildcards: https://issues.apache.org/jira/browse/HTTPCLIENT-1997
We will work in upgrading the httpclient version in the SDK but in the meantime you can override the version in your project.
@debora-ito ahh thats good to know, thanks.
@debora-ito rather late update, but I just tried pinning one of my services to 4.5.12 (same version used in dropwizard), and I am still getting the same error.
@inssein The exactly same error? According to the Jira issue I linked above it was fixed in 4.5.10.
Can you check if your environment is resolving the dependency version to 4.5.12? If you are using maven you can run mvn dependency:tree.
Yup, exactly the same error. I have ran mvn dependency:tree and ensured all of them are pointing to 4.5.12 (and they are, because we use dropwizard as a framework, and it has the httpclient pinned at that).
The only thing I am going to do next is try the url-connection http builder and see if it resolves the issues.
I have the code running with the url connection http service, and everything seems to be running smooth, which points to an issue with the apache http client.
@debora-ito it does seem like it should be fixed in the new version, but mvn dependency:tree for sure shows 4.5.12. Anything else I can do to confirm that it is using the right dep?
Can you share your dependency tree? Just the part regarding Apache httpcomponents -
mvn dependency:tree -Dverbose -Dincludes=org.apache.httpcomponents
[INFO] ---------------------< com.sednanetwork:sedna-db >----------------------
[INFO] Building sedna-db 1.0-SNAPSHOT [13/14]
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- maven-dependency-plugin:2.8:tree (default-cli) @ sedna-db ---
[INFO] com.sednanetwork:sedna-db:jar:1.0-SNAPSHOT
[INFO] +- com.sednanetwork:elasticsearch:jar:1.0-SNAPSHOT:compile
[INFO] | \- org.elasticsearch.client:elasticsearch-rest-client:jar:6.4.3:compile
[INFO] | +- org.apache.httpcomponents:httpclient:jar:4.5.12:compile (version managed from 4.5.2)
[INFO] | +- org.apache.httpcomponents:httpcore:jar:4.4.13:compile (version managed from 4.4.5)
[INFO] | +- org.apache.httpcomponents:httpasyncclient:jar:4.1.2:compile
[INFO] | \- org.apache.httpcomponents:httpcore-nio:jar:4.4.5:compile
[INFO] \- software.amazon.awssdk:s3:jar:2.13.11:compile
[INFO] \- software.amazon.awssdk:apache-client:jar:2.13.11:runtime
[INFO] +- (org.apache.httpcomponents:httpclient:jar:4.5.12:runtime - version managed from 4.5.9; omitted for duplicate)
[INFO] \- (org.apache.httpcomponents:httpcore:jar:4.4.13:runtime - version managed from 4.4.11; omitted for duplicate)
[INFO]
[INFO] ----------------< com.sednanetwork:sedna-veson-service >----------------
[INFO] Building sedna-veson-service 1.0-SNAPSHOT [14/14]
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- maven-dependency-plugin:2.8:tree (default-cli) @ sedna-veson-service ---
[INFO] com.sednanetwork:sedna-veson-service:jar:1.0-SNAPSHOT
[INFO] +- com.sednanetwork:sedna-db:jar:1.0-SNAPSHOT:compile
[INFO] | \- com.sednanetwork:elasticsearch:jar:1.0-SNAPSHOT:compile
[INFO] | \- org.elasticsearch.client:elasticsearch-rest-client:jar:6.4.3:compile
[INFO] | +- org.apache.httpcomponents:httpclient:jar:4.5.12:compile (version managed from 4.5.2)
[INFO] | +- org.apache.httpcomponents:httpcore:jar:4.4.13:compile (version managed from 4.4.5)
[INFO] | +- org.apache.httpcomponents:httpasyncclient:jar:4.1.2:compile
[INFO] | \- org.apache.httpcomponents:httpcore-nio:jar:4.4.5:compile
[INFO] \- software.amazon.awssdk:sqs:jar:2.13.11:compile
[INFO] \- software.amazon.awssdk:apache-client:jar:2.13.11:runtime
[INFO] +- (org.apache.httpcomponents:httpclient:jar:4.5.12:compile - version managed from 4.5.9; scope updated from runtime; omitted for duplicate)
[INFO] \- (org.apache.httpcomponents:httpcore:jar:4.4.13:compile - version managed from 4.4.11; scope updated from runtime; omitted for duplicate)
This is very interesting, I'm also getting the same exact issue with 4.5.12. I saw some promising mods that people have done, but nothing fixes this problem, even replacing the trust store to trust all certs. But I can issue my REST request via Postman and in the Firefox 76.0 browser with no problems.
Yup, I gave up in the end and just used the URLConnectionClient for now as the service didn't require high performance, but have a todo to switch it out once this dependency is upgraded.
Hi @debora-ito , I've observed endpoints .s3.amazonaws.com and .s3.us-east-1.amazonaws.com return different certificates.
Global endpoint:
$ true | openssl s_client -connect some-bucket.s3.amazonaws.com:443 2>/dev/null | openssl x509 -noout -text
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
08:2d:f6:8e:e9:c6:93:15:be:bf:72:07:9b:38:10:fd
Signature Algorithm: sha256WithRSAEncryption
Issuer: C=US, O=DigiCert Inc, OU=www.digicert.com, CN=DigiCert Baltimore CA-2 G2
Validity
Not Before: Nov 9 00:00:00 2019 GMT
Not After : Mar 12 12:00:00 2021 GMT
Subject: C=US, ST=Washington, L=Seattle, O=Amazon.com, Inc., CN=*.s3.amazonaws.com
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
...
X509v3 extensions:
X509v3 Subject Alternative Name:
DNS:*.s3.amazonaws.com, DNS:s3.amazonaws.com
...
Regional endpoint:
$ true | openssl s_client -connect some-bucket.s3.us-east-1.amazonaws.com:443 2>/dev/null | openssl x509 -noout -text
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
0d:64:50:6b:45:f3:0c:e3:5a:6c:2d:df:2c:18:b4:37
Signature Algorithm: sha256WithRSAEncryption
Issuer: C=US, O=DigiCert Inc, OU=www.digicert.com, CN=DigiCert Baltimore CA-2 G2
Validity
Not Before: Aug 4 00:00:00 2020 GMT
Not After : Aug 9 12:00:00 2021 GMT
Subject: C=US, ST=Washington, L=Seattle, O=Amazon.com, Inc., CN=s3.amazonaws.com
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
...
X509v3 extensions:
X509v3 Subject Alternative Name:
DNS:s3.amazonaws.com, DNS:*.s3.amazonaws.com, DNS:*.s3.dualstack.us-east-1.amazonaws.com, DNS:s3.dualstack.us-east-1.amazonaws.com, DNS:*.s3.us-east-1.amazonaws.com, DNS:s3.us-east-1.amazonaws.com, DNS:*.s3-control.us-east-1.amazonaws.com, DNS:s3-control.us-east-1.amazonaws.com, DNS:*.s3-control.dualstack.us-east-1.amazonaws.com, DNS:s3-control.dualstack.us-east-1.amazonaws.com, DNS:*.s3-accesspoint.us-east-1.amazonaws.com, DNS:*.s3-accesspoint.dualstack.us-east-1.amazonaws.com, DNS:*.s3.us-east-1.vpce.amazonaws.com
....
So, this code:
import software.amazon.awssdk.regions.Region
import software.amazon.awssdk.services.s3.S3Client
import software.amazon.awssdk.services.s3.model.DeleteObjectRequest
import software.amazon.awssdk.services.s3.model.DeleteObjectsRequest
import software.amazon.awssdk.services.s3.model.GetObjectRequest
import software.amazon.awssdk.services.s3.model.ListObjectsV2Request
import software.amazon.awssdk.services.s3.model.ObjectIdentifier
import software.amazon.awssdk.services.s3.model.S3Object
import software.amazon.awssdk.services.s3.presigner.S3Presigner
fun main() {
val bucket = awsConfiguration.bucket
val prefix = "foo/bar"
val listObjectsV2PaginatorResult = s3Client.listObjectsV2Paginator(
ListObjectsV2Request
.builder()
.bucket(bucket)
.prefix(prefix)
.build()
)
val keys: List<String> = listObjectsV2PaginatorResult
.contents()
.stream()
.map { it.key() }
.toList()
logger.info { "==================>>>>> KEYS: $keys" }
}
Worked in EU-CENTRAL-1 , but returned SdkClientException: Unable to execute HTTP request: Certificate for
Full stack trace:
software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request: Certificate for <some-bucket.s3.amazonaws.com> doesn't match any of the subject alternative names: [*.s3.amazonaws.com, s3.amazonaws.com]
at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:97)
at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage$RetryExecutor.handleThrownException(RetryableStage.java:137)
at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage$RetryExecutor.execute(RetryableStage.java:95)
at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:63)
at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:43)
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:57)
at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:37)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:81)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:61)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:43)
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26)
at software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:198)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:122)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute(BaseSyncClientHandler.java:148)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:102)
at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45)
at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:55)
at software.amazon.awssdk.services.s3.DefaultS3Client.listObjectsV2(DefaultS3Client.java:4926)
at software.amazon.awssdk.services.s3.paginators.ListObjectsV2Iterable$ListObjectsV2ResponseFetcher.nextPage(ListObjectsV2Iterable.java:147)
at software.amazon.awssdk.services.s3.paginators.ListObjectsV2Iterable$ListObjectsV2ResponseFetcher.nextPage(ListObjectsV2Iterable.java:138)
at software.amazon.awssdk.core.pagination.sync.PaginatedResponsesIterator.next(PaginatedResponsesIterator.java:58)
at software.amazon.awssdk.core.pagination.sync.PaginatedItemsIterable$ItemsIterator.<init>(PaginatedItemsIterable.java:58)
at software.amazon.awssdk.core.pagination.sync.PaginatedItemsIterable.iterator(PaginatedItemsIterable.java:48)
at java.lang.Iterable.spliterator(Iterable.java:101)
at software.amazon.awssdk.core.pagination.sync.SdkIterable.stream(SdkIterable.java:34)
Setting AWS_S3_US_EAST_1_REGIONAL_ENDPOINT to regional fixed it.
But I believe this is a bug because the exception is raised even if the region is being explicitly passed to the S3Client builder:
S3Client s3Client = S3Client.builder()
.region(Region.of(region))
.credentialsProvider(DefaultCredentialsProvider.create())
.build();
@raonitimo that is the expected behavior when providing us-east-1 as a region, the SDK defaults to the S3 global endpoint for legacy reasons. It would be a breaking change to make the SDK hit the us-east-1 regional endpoint by default, so using the AWS_S3_US_EAST_1_REGIONAL_ENDPOINT flag is the right way to do it.
I'm having the exact same issue as described here.
I have got it working using the UrlConnectionHttpClient instead, but would like to use the ApacheHttpClient for its performance.
@debora-ito, @raonitimo would this be the correct way to get the SDK to pick up the AWS_S3_US_EAST_1_REGIONAL_ENDPOINT setting:
AWS_S3_US_EAST_1_REGIONAL_ENDPOINT=regional java -jar my_app.jar
For a Spring app with a bean configuration like:
@Bean
public S3Client s3Client() {
return S3Client.builder()
.httpClient(ApacheHttpClient.builder().build())
.build();
}
This is my stack trace:
SdkClientException: Unable to execute HTTP request:
Certificate for <my-app.s3.amazonaws.com>
doesn't match any of the subject alternative names: [*.s3.amazonaws.com, s3.amazonaws.com]
The S3 bucket I'm getting the above error from is in US East (N. Virginia).
I also have made sure that the apache client i've installed is the version stated as having the fix:
[INFO] | +- software.amazon.awssdk:apache-client:jar:2.15.0:compile
[INFO] | | +- org.apache.httpcomponents:httpclient:jar:4.5.12:compile