Spring-cloud-config: Multi tenancy support

[lejeunen]

We are considering using the config server for our multi-tenant SaaS application, but have trouble finding an elegant way to do it with the actual state of the config server. Or maybe we are missing something, if so please tell us :)

Our application serves multiple customers, or tenants, and each of them have specific configuration.

Examples of tenant aware configuration are

• Structure of analytical levels
• Activation of optional modules (e.g. project, sales...)
• Configuration of widgets
• Injections to run and their configuration
• Emails of people to notify of various events

With spring cloud config server, there are 2 options identified

• A branch per tenant, specify it in the label part. Can use a list of labels to "inherit" standard properties, e.g. label=tenantXXX,standard. Another option is to git merge manually.
• A properties file with a different name, e.g. accounting-service-tenantXXX.properties. Inherit mechanism can easily be set up by using 2 variables : one for standard property (static), and another optional one for tenant specific (dynamic based on tenant).

In any case the client side cannot be used as-is due to its inability to dynamically set the label (option 1) or the application name (option 2).

Instead, a simple config client based on RestTemplate could be developed to get tenant aware configuration through the config server, but this means bypassing the client part of config server entirely.

Some kind of support to achieve in a more elegant way would be nice.

Any comment appreciated

[dsyer]

I'd like to understand the use case a bit more. What does "Configuration of widgets" mean?

Note that the config client is designed to initialize a single ApplicationContext (or more specifically its Environment). How is that relevant to your multi-tenant scenario? Wouldn't you be able to create a new ApplicationContext per tenant? It seems like you would need some way to separate their concerns anyway, so why not in that way?

[lejeunen]

The widget thing means data analysis components (mostly charts) of the application are sometimes deactivated for some tenant, and activated for others. Similar to a configurable dashboard.

I hadn't thought about having an _ApplicationContext_ per tenant. An instance of our service might serve hundreds of different tenants, do you think it would be adequate to have hundreds of contexts?

More generally, do you think using the config server to manage business configuration like ours is adequate?

Thank you for your feedback.

[dsyer]

do you think it would be adequate to have hundreds of contexts

Your use of the word "adequate" is probably just a translation from something in French that wouldn't help me. Certainly an ApplicationContext can be very lightweight, so it's not a problem to have them lying around, or even to create them quickly and throw them away (as long as you are very careful with what's in them - you don't want to reboot Hibernate every time you call a method for instance). Spring Cloud (Context and Config) already uses a lot of throwaway ApplicationContexts mainly just to load property sources. If you design how they are created and used carefully, I don't suppose there would be any technical issues at the Spring level.

do you think using the config server to manage business configuration like ours is adequate

It depends a lot what you mean by "adequate". If I were you I'd do a spike and see what you can learn.

[lejeunen]

By adequate I mean appropriate; would the resulting design be elegant and robust.

Thanks for your comments

[dsyer]

I don't see why not. But the devil as usual will be in the detail.

[gonzalad]

Hello,

I also need a Configuration Server with multi-tenant support.

I'm adding our scenario (in case it can bring some ideas/requirements/needs related to multi-tenancy) :

Scenario

• a tenant is created whenever we register a new customer in our platform.
• the creation step will create tenant-specific resources (i.e. db-schemas, credentials, ...).

At boot-time, our Spring Application will only need classic Configuration data (application/profiles/label as usual).

But at runtime (i.e. when serving a request), our application will need sometimes tenant-specific data (i.e. when accessing a tenant-specific db schema).

Our tenant information is available from the Thread-Context (TenantContextHolder -> TenantContext -> tenantId).

Having a multi-tenant friendly Spring Cloud Config Server would help here.

But Perhaps it would require quite a lot of change in Spring Config Server.
i.e. Config Server supports lookup based on application/profile/label, and we would need to add lookup based on tenant (we can even imagine adding multiple lookups to allow anyone to add custom search criterias for config).

Also, we'll need to handle lookup key search order.

Otherwise we can implement a hacky-solution (i.e. prefixing or suffixing config key with tenant specific information).

Sample

Tenant Creation Step

Let's say we register a new customer, which create a new tenant.
Tenant creation triggers a tenant-specific mongo db creation.

We'll register into our Configuration Server:

• tenant-specific MongoDb URL (i.e. key app1.mongo.url).
• tenant-specific MongoDb Databasename (i.e. key app1.mongo.databaseName).

Accessing tenant-info from our Spring app

Somewhere in our code, we'll have something like :

public class SomeHelperClass {

    @Autowired
    private Environment ;

    public DB getTenantDb() {
        final MongoClientURI databaseURI = provider.getDatabaseURI();
        MongoClient mongoClient = new MongoClient(environment.getRequiredProperty("app1.mongo.uri"));
        return MongoDbUtils.getDB(mongoClient, environment.getRequiredProperty("app1.mongo.databaseName"));
    }
}

We could even imagine having :

public class SomeHelperClass {

    @Autowired
    private Environment ;

    @ConfigurationProperties(prefix = "app1.mongo")
    @TenantScope
    @Bean
    public TenantDbProperties tenantDbProperties() {
        ...
    }
}

Let me know if it can be worthwhile to add this kind of support in Spring Config Server, or if I'm out of the road.

Cheers,
Adrian

[spencergibb]

So far, only two people have asked for this. Config Server isn't meant to be accessed at request time. Whatever is done would have to work with normal spring boot properties (since that is all config server uses). My guess is that it would be a large change.

[ghoddg]

Same requirement is for me also to have config server which should the multi-tenancy support. The requirement is like each tenant can have its own configuration details which should be available when request came for tenant.
e.g. Tenant A: key=message Value = I am Tenant A
Tenent B: key=message Value = I am Tenant B

here when request come for Tenant A for key=message, then it should provide Value = I am Tenant A.
The same client (one Application Context) should server this and there is no client deployment per Tenant

If have the project of such feature, please provide github link.

[fjmpaez911]

I also need to have the multi-tenancy support but I think that it's already there. This is my way to achieve it:

spring:
  cloud:
    config:
      server:
        git:
          uri: https://github.com/company/default-config-repo
          username: abcd
          password: 1234
          clone-on-start: true
          search-paths: '*'
          repos:
            tenant1:
              pattern: '*/tenant1*'
              uri: https://github.com/company/tenant1-config-repo
              username: zzzz
              password: 1234
              clone-on-start: true
              search-paths: '*'
            tenant2:
              pattern: '*/tenant2*'
              uri: https://github.com/company/tenant2-config-repo
              username: yyyy
              password: 1234
              clone-on-start: true
              search-paths: '*'

The idea is explained in the documentation.
The approach here is to have different repositories per tenant and use different profiles in order to know which tenant applies in each case

[snowe2010]

Late to the party, but using different repos for tenant config doesn't work because then you must modify your bootstrap whenever you add a new tenant. This is not acceptable for many reasons, the point of config server is that you can refresh configuration just by adding configuration to your source. It would be ideal to have a tenant solution, as splitting your configuration in this way is not really desirable.

Some other reasons that the bootstrap method doesn't work is that you can't then share default configurations between these different tenants, you must duplicate that config across many repos. You also duplicate environment profiles and labels across the repos.

[spencergibb]

can't you use placeholders?

[snowe2010]

@spencergibb what do you mean by placeholders?

[snowe2010]

something like tenant-*?

[spencergibb]

https://cloud.spring.io/spring-cloud-static/spring-cloud-config/2.2.2.RELEASE/reference/html/#_placeholders_in_git_uri

[snowe2010]

@spencergibb I had updated my comment right before you commented, so you probably didn't see, but that doesn't solve the problem of using defaults. Those won't merge with your tenant specific stuff so now you have to maintain significantly more configuration that keeps growing based on the number of tenants you have.

[spencergibb]

how about composite configuration?

[snowe2010]

Hm. Looking at that it seems it might interfere with our Vault config?

Currently have this:

spring:
  profiles:
    include:
      - git
      - vault
  application:
    name: configuration-management
  cloud:
    config:
      server:
        vault:
          ...
        git:
          uri: [email protected]:company/config.git
          search-paths: "private/,private/{application},global/,public/"
          ...
          repos: 
            public:
              pattern: factory*
              search-paths: "global/,public/,public/{application}"
              uri: [email protected]:company/config.git
              ...
            private:
              pattern: differentpattern*
              search-paths: "private/,private/{application},global/"
              uri: [email protected]:company/config.git
              ...
        bootstrap: false
      fail-fast: true

note that the private pattern just searches different search paths, since we were trying to keep our config all in a single application.

Wondering how I would modify this to accomplish a composite repository with a vault configuration and this split search path stuff.

[snowe2010]

@spencergibb even with a composite configuration, spring doesn't provide placeholders for things like tenant. We'd still have one extra variable that spring can't handle. So say we did a composite config like so

composite:
  - type: git
    uri: [email protected]:promontech/base-config.git
  - type: git
    uri: [email protected]:promontech/config-tenant-{whatgoeshere?}.git

what placeholder do I use in the uri?

[spencergibb]

how do you define a tenant?

[snowe2010]

What do you mean by define? Our tenants are companies that buy our software. We are working towards making it so that they can add themselves to our platform, add their own configuration, etc. Essentially we don't want to have to push code changes to add a new tenant. So we have several applications, that will have different configurations based on the environment _and_ the tenant. If one tenant wants a certain feature turned on but another tenant does not, then we need SCCS to support that. We would like to have this all present in a single repository, so that our frontend and backend can share certain global toggles, like whether to turn certain features on, but if that's not possible at the moment I understand.

If you are asking how we add a new tenant, currently we have a bunch of json files that are environment, tenant, and application specific that we're moving to be in SCCS. We have to manually update about 90 files every time we make a single configuration change, so you can see why this is kind of important to me, as that number is based on the number of clients/tenants we have.

[spencergibb]

there's some property on an app that defines a tenant? can't that go in as a spring.cloud.config.profile?

[snowe2010]

That's currently the solution we're looking at, but we're still overloading the concept because we then need to make the profile environment specific. so something like application-tenant_a_dev.yml and application-tenant_b_dev.yml.

We tried nesting the tenants in a app/profile/app-envprofile.yml kind of situation, but SCCS doesn't seem to like that very much

search-paths: "global/,public/,public/{application},public/{application}/{profile}/" <- didn't work. I actually managed to get this working.

I've been trying to figure it out using the multiple profile solution for almost the whole day now. We need to pass two separate profiles, the environment and the tenant. So something like dev and tenant_a. I tried a structure like /application/tenant_a/application-dev.yml as well as using single file yaml docs like /application/application-tenant_a.yml where the yaml had separated profiles like

defaults:
  TENANT_NAME: Tenant A

---
##### DEV ######

spring.profiles: dev & tenant_a

additional: config

---
##### PROD ######

spring.profiles: prod & tenant_a

additional: config

but only the default document at the top of the file is picked up.

edit:
I managed to get the nested {application}/{profile}/application-{envprofile}.yml style working, so I guess that's what we'll have to go with until (if) spring decides to support this.