Spring-cloud-aws: Feature Request - Configurable endpoint for AWS clients for testing

[davidgoate]

When testing, I often use things like localstack to run local API's that mock AWS services like S3, SQS, SNS. I don't think there are properties that allow the endpoint to be configured as there are for things like regions and credentials.

Would be great if I could do something like:

cloud.aws.s3.endpoint=http://localhost:4321
cloud.aws.sqs.endpoint=http://localhost:4322
cloud.aws.sns.endpoint=http://localhost:4323

localstack uses a different port per service so unfortunately I don't think this property could be one setting globally.

Currently i define my own client as follows:

@TestConfiguration
    @EnableSqs
    public static class SQSConfiguration {

        @Primary
        @Bean(destroyMethod = "shutdown")
        public AmazonSQSAsync amazonSQS() {
            final String serviceEndpoint = String.format("http://%s:%s", LOCAL_STACK_BINDING.hostIp(), LOCAL_STACK_BINDING.hostPort());
            final AwsClientBuilder.EndpointConfiguration endpoint = new AwsClientBuilder.EndpointConfiguration(serviceEndpoint, REGION);
            return new AmazonSQSBufferedAsyncClient(AmazonSQSAsyncClientBuilder
                    .standard()
                    .withRegion("eu-west-1")
                    .withCredentials(new AWSStaticCredentialsProvider(new BasicAWSCredentials("access", "secret")))
                    .withEndpointConfiguration(endpoint)
                    .build());
        }
    }

If the auto configuration could handle this for me, I think it'd be pretty useful!

[engrun]

I ran into this just today, this would be very useful.

[Vity01]

Anyone from the Spring Cloud AWS team?

[FedorRomanov]

guys,

any idea how to do similar @TestConfiguration for S3?
I've tried

    @TestConfiguration
    @EnableContextResourceLoader
    public static class S3Configuration {

        @Primary
        @Bean(destroyMethod = "shutdown")
        public AmazonS3 amazonS3() {
            return AmazonS3ClientBuilder
                    .standard()
                    .withEndpointConfiguration(localstack.getEndpointConfiguration(S3))
                    .withCredentials(localstack.getDefaultCredentialsProvider())
                    .build();
        }
  }

and it does not work. In the prod code, I access S3 via ResourceLoader:

    public String storeOnS3(String filename, byte[] data) throws IOException {
        String location = "s3://" + bucket + "/" + filename;
        WritableResource writeableResource = (WritableResource) this.resourceLoader.getResource(location);
        FileCopyUtils.copy( data, writeableResource.getOutputStream());
        return filename;
    }

[davidgoate]

@FedorRomanov a little off topic from the SQS config, but yes, we do this in our projects like this:

    @TestConfiguration
    public static class S3Configuration {

        @Bean
        public AmazonS3 amazonS3() {
            final String serviceEndpoint = String.format("http://%s:%s", LOCAL_STACK_BINDING.hostIp(), LOCAL_STACK_BINDING.hostPort());
            return AmazonS3Client.builder()
                .withEndpointConfiguration(new AwsClientBuilder.EndpointConfiguration(serviceEndpoint, REGION))
                .withCredentials(new AWSStaticCredentialsProvider(new BasicAWSCredentials("dummyKey", "dummySecret")))
                .build();
        }
    }

This assumes you have localstack running (or some other S3 compatible API) but we directly use the S3 API in our production code to write the file(s):

@Service
@RequiredArgsConstructor
public class S3UploadServiceImpl implements S3UploadService {


    private final AmazonS3 amazonS3;


    public void doUpload(final String text, final String key, final ObjectMetadata metadata) {
        try (ReaderInputStream in = new ReaderInputStream(new StringReader(text), UTF_8)) {
            final PutObjectRequest request = new PutObjectRequest(BUCKET_NAME, key, in, metadata);
            amazonS3.putObject(request);
        } catch (final IOException ioe) {
         ....
        }
    }

}

[FedorRomanov]

@davidgoate thx for prompt response! Indeed, That's an off-topic.

Could you, pls, maybe provide complete S3 unit test at https://stackoverflow.com/questions/53092222/spring-cloud-testing-s3-client-with-testcontainters

As you can see I am also creating AmazonS3 in TestConfiguration, but it is not picked up during test execution.

[Lucas3oo]

Since we can't specify the endpoint for S3 access the AWS GovCloud can't be used for S3. That AWS cloud is isolated from the rest of the AWS cloud.

[maciejwalkowiak]

Thanks for reporting. Support for custom endpoints will be added in 2.3. This issue is superseeded by #570.