Spring-cloud-aws: EC2 metadata resolution related exception thrown when running application locally

[nanachimi]

Describe the bug
I used spring-cloud-starter-aws (version 2.2.1.RELEASE) to send email using Amazon SES. I'm able to send email but I have the following error when starting the app. I'm not dealing with EC2 but it looks like the underline aws-java-sdk-ses (version 1.11.787) is trying to load EC2 resources. How can I get rid of this misconfiguration?

2020-05-22 17:41:09.647  WARN 47453 --- [           main] i.InstanceMetadataServiceResourceFetcher : Fail to retrieve token 

com.amazonaws.SdkClientException: Failed to connect to service endpoint: 
    at com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:100) ~[aws-java-sdk-core-1.11.787.jar:na]
    at com.amazonaws.internal.InstanceMetadataServiceResourceFetcher.getToken(InstanceMetadataServiceResourceFetcher.java:91) ~[aws-java-sdk-core-1.11.787.jar:na]
    at com.amazonaws.internal.InstanceMetadataServiceResourceFetcher.readResource(InstanceMetadataServiceResourceFetcher.java:69) ~[aws-java-sdk-core-1.11.787.jar:na]
    at com.amazonaws.internal.EC2ResourceFetcher.readResource(EC2ResourceFetcher.java:66) ~[aws-java-sdk-core-1.11.787.jar:na]
    at com.amazonaws.util.EC2MetadataUtils.getItems(EC2MetadataUtils.java:402) ~[aws-java-sdk-core-1.11.787.jar:na]
    at com.amazonaws.util.EC2MetadataUtils.getData(EC2MetadataUtils.java:371) ~[aws-java-sdk-core-1.11.787.jar:na]
    at org.springframework.cloud.aws.context.support.env.AwsCloudEnvironmentCheckUtils.isRunningOnCloudEnvironment(AwsCloudEnvironmentCheckUtils.java:38) ~[spring-cloud-aws-context-2.2.1.RELEASE.jar:2.2.1.RELEASE]
    at org.springframework.cloud.aws.context.annotation.OnAwsCloudEnvironmentCondition.matches(OnAwsCloudEnvironmentCondition.java:37) ~[spring-cloud-aws-context-2.2.1.RELEASE.jar:2.2.1.RELEASE]
    at org.springframework.context.annotation.ConditionEvaluator.shouldSkip(ConditionEvaluator.java:108) ~[spring-context-5.2.6.RELEASE.jar:5.2.6.RELEASE]
    at org.springframework.context.annotation.ConfigurationClassParser.processConfigurationClass(ConfigurationClassParser.java:225) ~[spring-context-5.2.6.RELEASE.jar:5.2.6.RELEASE]
    at org.springframework.context.annotation.ConfigurationClassParser.processImports(ConfigurationClassParser.java:599) ~[spring-context-5.2.6.RELEASE.jar:5.2.6.RELEASE]
    at org.springframework.context.annotation.ConfigurationClassParser.access$800(ConfigurationClassParser.java:110) ~[spring-context-5.2.6.RELEASE.jar:5.2.6.RELEASE]
    at org.springframework.context.annotation.ConfigurationClassParser$DeferredImportSelectorGroupingHandler.lambda$processGroupImports$1(ConfigurationClassParser.java:811) ~[spring-context-5.2.6.RELEASE.jar:5.2.6.RELEASE]
    at java.base/java.util.ArrayList.forEach(ArrayList.java:1540) ~[na:na]
    at org.springframework.context.annotation.ConfigurationClassParser$DeferredImportSelectorGroupingHandler.processGroupImports(ConfigurationClassParser.java:808) ~[spring-context-5.2.6.RELEASE.jar:5.2.6.RELEASE]
    at org.springframework.context.annotation.ConfigurationClassParser$DeferredImportSelectorHandler.process(ConfigurationClassParser.java:779) ~[spring-context-5.2.6.RELEASE.jar:5.2.6.RELEASE]
    at org.springframework.context.annotation.ConfigurationClassParser.parse(ConfigurationClassParser.java:192) ~[spring-context-5.2.6.RELEASE.jar:5.2.6.RELEASE]
    at org.springframework.context.annotation.ConfigurationClassPostProcessor.processConfigBeanDefinitions(ConfigurationClassPostProcessor.java:319) ~[spring-context-5.2.6.RELEASE.jar:5.2.6.RELEASE]
    at org.springframework.context.annotation.ConfigurationClassPostProcessor.postProcessBeanDefinitionRegistry(ConfigurationClassPostProcessor.java:236) ~[spring-context-5.2.6.RELEASE.jar:5.2.6.RELEASE]
    at org.springframework.context.support.PostProcessorRegistrationDelegate.invokeBeanDefinitionRegistryPostProcessors(PostProcessorRegistrationDelegate.java:280) ~[spring-context-5.2.6.RELEASE.jar:5.2.6.RELEASE]
    at org.springframework.context.support.PostProcessorRegistrationDelegate.invokeBeanFactoryPostProcessors(PostProcessorRegistrationDelegate.java:96) ~[spring-context-5.2.6.RELEASE.jar:5.2.6.RELEASE]
    at org.springframework.context.support.AbstractApplicationContext.invokeBeanFactoryPostProcessors(AbstractApplicationContext.java:706) ~[spring-context-5.2.6.RELEASE.jar:5.2.6.RELEASE]
    at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:532) ~[spring-context-5.2.6.RELEASE.jar:5.2.6.RELEASE]
    at org.springframework.boot.web.reactive.context.ReactiveWebServerApplicationContext.refresh(ReactiveWebServerApplicationContext.java:62) ~[spring-boot-2.3.0.RELEASE.jar:2.3.0.RELEASE]
    at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:758) ~[spring-boot-2.3.0.RELEASE.jar:2.3.0.RELEASE]
    at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:750) ~[spring-boot-2.3.0.RELEASE.jar:2.3.0.RELEASE]
    at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:397) ~[spring-boot-2.3.0.RELEASE.jar:2.3.0.RELEASE]
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:315) ~[spring-boot-2.3.0.RELEASE.jar:2.3.0.RELEASE]
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:1237) ~[spring-boot-2.3.0.RELEASE.jar:2.3.0.RELEASE]
    at org.springframework.boot.SpringApplication.run(SpringApplication.java:1226) ~[spring-boot-2.3.0.RELEASE.jar:2.3.0.RELEASE]
    at com.packnmoov.webappserver.WebAppServerRunner.main(WebAppServerRunner.java:9) ~[main/:na]
Caused by: java.net.ConnectException: No route to host (connect failed)
    at java.base/java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:na]
    at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:399) ~[na:na]
    at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:242) ~[na:na]
    at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:224) ~[na:na]
    at java.base/java.net.Socket.connect(Socket.java:591) ~[na:na]
    at java.base/sun.net.NetworkClient.doConnect(NetworkClient.java:177) ~[na:na]
    at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:474) ~[na:na]
    at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:569) ~[na:na]
    at java.base/sun.net.www.http.HttpClient.<init>(HttpClient.java:242) ~[na:na]
    at java.base/sun.net.www.http.HttpClient.New(HttpClient.java:341) ~[na:na]
    at java.base/sun.net.www.http.HttpClient.New(HttpClient.java:362) ~[na:na]
    at java.base/sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1242) ~[na:na]
    at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1221) ~[na:na]
    at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1075) ~[na:na]
    at java.base/sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:1009) ~[na:na]
    at com.amazonaws.internal.ConnectionUtils.connectToEndpoint(ConnectionUtils.java:52) ~[aws-java-sdk-core-1.11.787.jar:na]
    at com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:80) ~[aws-java-sdk-core-1.11.787.jar:na]
    ... 30 common frames omitted

Sample

@Configuration
public class AwsMailConfig {

    private final String awsAccessKey;
    private final String awsSecretKey;
...
    @Bean
    public AmazonSimpleEmailService amazonSimpleEmailService() {
        return AmazonSimpleEmailServiceClientBuilder.standard()
                .withCredentials(new AWSStaticCredentialsProvider( new BasicAWSCredentials(awsAccessKey, awsSecretKey)))
                .withRegion(Regions.EU_CENTRAL_1)
                .build();
    }
...
}

[giova333]

This is how OnAwsCloudEnvironmentCondition works. In order to understand if the application is executed on AWS environment it tries to get instance metadata https://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/instancedata-data-retrieval.html The issue is more related to the exception handling/logging as I see.
@maciejwalkowiak I can check this

[maciejwalkowiak]

Thanks @giova333. Indeed it's just a log message triggered during @ConditionalOnAwsCloudEnvironment evaluation. It happens only with new version of AWS SDK v1 (checked with 1.11.791). It would be nice if we find away to detect AWS env without having this exception logged. A workaround for today is adding to application.properties:

logging.level.com.amazonaws.util.EC2MetadataUtils=error

[giova333]

@maciejwalkowiak I don't think we will be able to manage that on our side.
EC2 metadata service is the recommended way to identify whether an application is running on AWS environment. Of course, we can call this service without using EC2MetadataUtils and this will solve the problem in this particular case but not everywhere.

Aws sdk uses DefaultAwsRegionProviderChain as default class for region discovery and
DefaultAWSCredentialsProviderChain for credentials lookup.

Both classes have a dependency on EC2MetadataUtils

DefaultAwsRegionProviderChain -> InstanceMetadataRegionProvider -> EC2MetadataUtils
DefaultAWSCredentialsProviderChain -> InstanceProfileCredentialsProvider -> EC2MetadataUtils

This means that whenever developers run our code outside ec2 instances and use some clients such as AmazonS3Client, AWSSecretsManagerClient ... without providing the region explicitly, sdk will try to load the region from env variables, system properties, aws config. If the region is not provided in any of the sources that I mentioned above, as the last resort, DefaultAwsRegionProviderChain
will try to load the region from ec2 metadata service using EC2MetadataUtils. And this is where the same issue with the logging will occur and the same flow for credentials lookup.
It is easy to reproduce, just call the perform s3 GetObjectRequest, catch the exception and you will see the same problem (checked with 1.11.791):

 try {
            AmazonS3Client.builder()
                    .build()
                    .getObject("bucketName", "key");
        } catch (Exception e) {  
        }

I would say that the issue has a global level and as for me the most appropriate way to manage it is to report it to aws sdk saying that the developers should be able to manage such cases themselves, handling the thrown exception, without having the original exception logged. That is what I am going to do.

[fransflippo]

For me the problem is not just the exception being logged, but the several seconds this adds to my startup time. Is there no way to set a property that will override this so this check doesn't need to be done? Just like cloud.aws.region.static=eu-west-1 specifies the region instead of it being autodetected.

[psrmx]

-> _This means that whenever developers run our code outside ec2 instances and use some clients ... without providing the region explicitly_...
Problem is that the warning/ exception still happens when providing the region explicitly. I have set cloud.aws.region.static=eu-central-1 and cloud.aws.region.auto=false, but I still get the autodetected warning.
Hiding a log warning feels more like tackling the symptoms instead of addressing the cause to me.

[maciejwalkowiak]

@psrmx I understand the sentiment but the underlying issue is in AWS SDK. We will consider changing the way cloud environment is detected (as fetching the metadata is anyway very slow) in Spring Cloud AWS 3.0.

[raja-baz]

So this happens as part of the evaluation of @ConditionalOnAwsCloudEnvironment which feeds into two pieces of auto-configuration: ElastiCacheAutoConfiguration and ContextInstanceDataAutoConfiguration. If you don't care about those being autoconfigured, you can exclude them and this issue goes away.

Like so(in kotlin):

@SpringBootApplication(exclude = [
    ElastiCacheAutoConfiguration::class,
    ContextInstanceDataAutoConfiguration::class,
])

[edudar]

Or even better, in properties like this:

spring:
  autoconfiure:
    exclude:
      - <full package path>.ContextInstanceDataAutoConfiguration
      - <full package path>.ElastiCacheAutoConfiguration

and have this exclusion only in dev or tests profile but keep them in prod or whatever that runs in AWS

However, there might be more uses for AwsCloudEnvironmentCheckUtils so I'd still like to vote for a new property like cloud.aws.metadata.enabled or spring.cloud.aws.metadata.enabled. If it's false that AwsCloudEnvironmentCheckUtils .isRunningOnCloudEnvironment() would return false preemptively without even asking AWS about it. This can obviously be implemented with a custom property but then one has to drag it around from project to project.

[maciejwalkowiak]

In 2.3.0 we introduced a property cloud.aws.instance.data.enabled which is false by default.