Spring-cloud-config: map AbstractScmEnvironmentRepository semantics to S3 versions

[mwnorman]

? spring-cloud-config-serverless-s3 ?
Instead of requiring a git server in the production environment, would it be possible to map
the semantics of AbstractScmEnvironmentRepository to S3 versioning?

I see in the Spring Cloud Incubator an example of a database-backed config server (MongoDB)
and somewhere else I saw a discussion of Cassandra; however, I would like to examine
the possibility of going "serverless", using AWS S3 (perhaps need Lambda?) object versioning.

Any help/hints would be greatly appreciated,

Mike Norman

[spencergibb]

Not sure AbstractScmEnvironmentRepository would be ideal. You'd have to experiment with the S3 apis. I think in general using S3 or other storage services could be useful.

[marcellodesales]

@mwnorman We have developed a proprietary serverless solution called Config Publisher that publishes all known/required app/profiles to S3. That allows our customers can use Hystrix-enabled clients against S3-compatible storage services plus the Config Service itself. We may Open-source it this year...

Just HTTP PUT the endpoint ${config-server-host}/[${label}/]${appName}/${profiles} to S3 as is and u can use an app, with the Config Client and ${spring.cloud.config.uri} against the client S3 URL.

[mwnorman]

I started my implementation: spring-cloud-config-s3: it depends on spring-cloud-config-server and aws-java-sdk-s3 and has an auto-config class in META-INF/spring.factories:
org.springframework.boot.autoconfigure.EnableAutoConfiguration=\ org.springframework.cloud.config.s3.CloudConfigS3AutoConfiguration
which looks like

@Configuration
@ConditionalOnBean(CloudConfigS3Configuration.Marker.class)
@EnableConfigurationProperties(CloudConfigS3Properties.class)
@Import({S3EnvironmentRepositoryConfiguration.class})
public class CloudConfigS3AutoConfiguration {}

The S3EnvironmentRepositoryConfiguration class looks like

@Configuration
public class S3EnvironmentRepositoryConfiguration {
    @Autowired
    protected ConfigurableEnvironment environment;
    @Autowired
    protected CloudConfigS3Properties s3Properties;
    @Bean
    @Primary
    public EnvironmentRepository environmentRepository() {
        return new S3EnvironmentRepository(environment, s3Properties);
    }
    @Bean
    public CloudConfigS3Properties buildCloudConfigS3Properties() {
        return new CloudConfigS3Properties();
    }
}

I built a simple client:

@SpringBootApplication(
    //Hmm I think I shouldn't have to do this ... haven't figured it out yet
    exclude = {
        org.springframework.cloud.config.server.config.ConfigServerAutoConfiguration.class
    }
)
@EnableS3CloudConfig
public class Client {

    public static void main(String[] args) {
        ConfigurableApplicationContext applicationContext = new SpringApplicationBuilder(Client.class)
            .web(false)
            .run(args)
        ;
        applicationContext.getEnvironment().getProperty("foo.bar");
    }
}

My properties are correctly auto-wired (bucketName and prefix) and my S3EnvironmentRepository is built as the 'primary' EnvironmentRepository bean.

My issue comes with the findOne API - it is never called. In the client when I debug inside
the context's Environment, I can see Standard property sources, but no where is the source
I build in S3EnvironmentRepository - a wrapper around NativeEnvironmentRepository with the objectSteam contents of application.yml found in the S3 bucket/prefix.

Any help would be greatly appreciated

Mike Norman

[spencergibb]

I'd prefer to see a PR or repo rather than trying to comment on code in a comment.

[mwnorman]

I will create repo ...

[mwnorman]

My repo: https://github.com/mwnorman/spring-cloud-config-server-s3.git

[mwnorman]

@spencergibb Any help would be appreciated

[marcingrzejszczak]

Is it still a problem with Finchley.RELEASE?

[spencergibb]

Closing due to lack of requested feedback. If you would like us to look at this issue, please provide the requested information and we will re-open the issue.

[marcellodesales]

@spencergibb I'm interested in this feature...

• We'd like to remove the load to our Github Enterprise from Config Server instances used as Side-cars
• S3-compatible storage is a viable option to Github as a backend.

If Config Server supports S3 as a backend, this would be a great win!

thank you!

[ozcanoguz]

Hello;

Firstly, thanks for adding #1425 to this project, this will be really helpful for many people. But, one short question or request from my side, could we create one use-case example how to use this option for a spring-cloud-config app, one example application would be really beneficial about usage of this functionality, or you can refer me a document that i should read.

Would be appreciated! Many thanks in advance.

[spencergibb]

https://cloud.spring.io/spring-cloud-static/spring-cloud-config/2.2.0.M3/reference/html/#_aws_s3_backend