Quarkus: Multiple per-profile configuration files or includes

Created on 27 Jan 2020  ·  13Comments  ·  Source: quarkusio/quarkus

Several users have asked us to put their config in multiple files. This is not good generally, but there's no good reason to prevent people from doing it if they must.

There are several strategies possible:

Nr.|Strategy|Pros|Cons
---|---|---|---
1|application-%profile.properties, fall-back to application.properties if not found|Clear mapping with fallback|No composition (single file)
2|application.properties + application-%profile.properties if it exists|Composition possible|Users are bound to ask for reversing the order of composition, or how to unset a config value set by the previous file
3|Support @include=application-%profile.properties directives|We don't take sides, composable and extendable|No default guideline/convention to follow

Am I missing options, pros, cons?

areconfig kinenhancement

Most helpful comment

The difficulty with such an approach is that it is not always possible to iterate a directory on the class path, so detecting these files would be difficult. If however you instead were able to derive the potential file name from the configuration property somehow, perhaps a scheme like this could be used to spread configuration across files in a structured manner. But there could be performance implications for probing for many files per property, so that would have to be balanced somehow.

I have a problem about polluting application.properties with specific %test properties, which is not easily resolvable with environment variables in my CI/CD pipeline.

How about looking for properties from application.properties and application.${quarkus.profile}.properties, where all properties defined in the latter would be automatically prepended with %${quarkus.profile}.?

Example: when the test profile is active, quarkus would look in:

  • application.properties
  • application.test.properties, if exists

The property quarkus.http.port defined in application.test.properties would become %test.quarkus.http.port.

All 13 comments

@dmlloyd @gsmet

Are we sure that % is allowed as a file name character on all platforms? Also note that it's pretty weird when you URI-encode it (some config sources use URLs internally, and resources use URLs as well), and might be a source of bugs. Perhaps the file name should exclude the %.

"No composition" (number 1) seems like a deal-breaker to me. It essentially means asking users to duplicate their entire configuration even if only one value is changed.

I don't think users would want to reverse the order of composition (after all, nobody has requested that foo take precedence over %profile.foo). We'd always want the profile configuration to be layered on top. (Number 2)

Unsetting a configuration value is already generally supported; using multiple files should not affect this one way or the other. If a per-profile file takes precedence, then values can be explicitly unset in that file by giving an empty property value.

@include is not ideal because it's not clear that it can be extended to other common formats such as YAML, HOCON, etc.

I feel like the only viable option is number 2. That said, there are some questions:

  • What about YAML and other formats? Is it OK to simply duplicate the logic that applies to .properties to YAML (and, in the future, to other configuration sources), though that is not ideal?
  • We support profiles today by encapsulating all config sources. This means that some unusual constructs would be possible if we carried this forward, for example %dev.foo=xx will take precedence over foo=yy in application-dev.properties, however %dev.foo=xx will not be seen in application-prod.properties. Is that OK?

I think that the solution should be simple, and somewhat future proof and general. So for example, implementing complex logic within the properties config source to support this by directly managing multiple files would not be ideal; it would be much simpler to instead (for example) add an additional config source which layers over a given properties config source for the profile-specific one. This way the logic can be extended to YAML or other applicable formats fairly easily. But this does raise the above questions in a direct way.

Are we sure that % is allowed as a file name character on all platforms? Also note that it's pretty weird when you URI-encode it (some config sources use URLs internally, and resources use URLs as well), and might be a source of bugs. Perhaps the file name should exclude the %.

I meant to show that the value would be substituted, so I'm actually referring to application-dev.properties. Sorry for the confusion.

This means that some unusual constructs would be possible if we carried this forward, for example %dev.foo=xx will take precedence over foo=yy in application-dev.properties,

I feel like if the current profile is dev then %dev.foo= and foo= are equivalent, and only their order determines the final value of the property (last one wins). Is that not the case currently?

however %dev.foo=xx will not be seen in application-prod.properties. Is that OK?

Given that the only way for this file to be loaded is on the prod profile (no include), it's indeed always going to be ignored. So sure.

Are we sure that % is allowed as a file name character on all platforms? Also note that it's pretty weird when you URI-encode it (some config sources use URLs internally, and resources use URLs as well), and might be a source of bugs. Perhaps the file name should exclude the %.

I meant to show that the value would be substituted, so I'm actually referring to application-dev.properties. Sorry for the confusion.

:+1:

This means that some unusual constructs would be possible if we carried this forward, for example %dev.foo=xx will take precedence over foo=yy in application-dev.properties,

I feel like if the current profile is dev then %dev.foo= and foo= are equivalent, and only their order determines the final value of the property (last one wins). Is that not the case currently?

No, order is not significant in any of the configuration sources. If this were implemented in the simplest/most obvious way, then the behavior would be as I described.

however %dev.foo=xx will not be seen in application-prod.properties. Is that OK?

Given that the only way for this file to be loaded is on the prod profile (no include), it's indeed always going to be ignored. So sure.

:+1:

No, order is not significant in any of the configuration sources. If this were implemented in the simplest/most obvious way, then the behavior would be as I described.

Well we have a different idea of "simplest/most obvious" because I totally expected order to be significant with last-one-wins.

But if it's as you say "most-specific-wins" then fine, as long as it's documented because it's certainly not obvious to me ;)

No, order is not significant in any of the configuration sources. If this were implemented in the simplest/most obvious way, then the behavior would be as I described.

Well we have a different idea of "simplest/most obvious" because I totally expected order to be significant with last-one-wins.

I was talking about simplicity from an implementation perspective.

But if it's as you say "most-specific-wins" then fine, as long as it's documented because it's certainly not obvious to me ;)

OK. Since it's consistent with the current behavior of %foo.bar overriding bar, I think we can probably extend the documentation at that point to explain that the rule extends to per-profile files.

I was talking about simplicity from an implementation perspective.

From an implementor's perspective, I would have done a s/%profile//g on the file before parsing it, so… :man_shrugging: ?

Actually, we put all the configs within application.properties (or yaml/yml) which acts like a god config (similar to a god class). At first it is good to have all configs in one place. But this can lead at some time to too much unrelated configs in one file. It would be nice if quarkus provides the ability to split the configs to different files with the following convention:

Folder "src/main/resources" can have the following application.{name}.properties:

  • application.db.properties
  • application.quarkus.properties
  • application.xyz.properties

where the {name} is any user defined name (not needed to correspond to a quarkus-extension name). The marker for quarkus would be the prefix (application.) and the suffix (.properties or .yaml or .yml). And all such configs must be within the folder "src/main/resources".

I was talking about simplicity from an implementation perspective.

From an implementor's perspective, I would have done a s/%profile//g on the file before parsing it, so… 🤷‍♂ ?

The profile support is agnostic to configuration source, which is to say that the configuration sources themselves have no specific knowledge of profiles. We implement it by wrapping each config source with a decorator which first queries the delegate configuration source for a profile-specific property value for the given name, and then looks for the profile-agnostic value for the same name. Had we not implemented it this way, then every config source implementation would have to reimplement profile handling.

So we would only be able to do it the way you suggested if we had replaced the property implementation with our own which reads the file into memory, performing the substitutions as it goes. Obviously this doesn't help with YAML or other config source implementations.

Actually, we put all the configs within application.properties (or yaml/yml) which acts like a god config (similar to a god class). At first it is good to have all configs in one place. But this can lead at some time to too much unrelated configs in one file. It would be nice if quarkus provides the ability to split the configs to different files with the following convention:

Folder "src/main/resources" can have the following application.{name}.properties:

  • application.db.properties
  • application.quarkus.properties
  • application.xyz.properties

where the {name} is any user defined name (not needed to correspond to a quarkus-extension name). The marker for quarkus would be the prefix (application.) and the suffix (.properties or .yaml or .yml). And all such configs must be within the folder "src/main/resources".

The difficulty with such an approach is that it is not always possible to iterate a directory on the class path, so detecting these files would be difficult. If however you instead were able to derive the potential file name from the configuration property somehow, perhaps a scheme like this could be used to spread configuration across files in a structured manner. But there could be performance implications for probing for many files per property, so that would have to be balanced somehow.

The difficulty with such an approach is that it is not always possible to iterate a directory on the class path, so detecting these files would be difficult. If however you instead were able to derive the potential file name from the configuration property somehow, perhaps a scheme like this could be used to spread configuration across files in a structured manner. But there could be performance implications for probing for many files per property, so that would have to be balanced somehow.

I have a problem about polluting application.properties with specific %test properties, which is not easily resolvable with environment variables in my CI/CD pipeline.

How about looking for properties from application.properties and application.${quarkus.profile}.properties, where all properties defined in the latter would be automatically prepended with %${quarkus.profile}.?

Example: when the test profile is active, quarkus would look in:

  • application.properties
  • application.test.properties, if exists

The property quarkus.http.port defined in application.test.properties would become %test.quarkus.http.port.

I have a problem about polluting application.properties with specific %test properties, which is not easily resolvable with environment variables in my CI/CD pipeline.

I could not agree more!

Btw, #9895 is partially related in the sense that I would love to see multiple files being loaded, not just the default one + a single profile specific one.

Actually it's very good idea to keep separate config files per profile.

There's no reason we would want a production password inside a dev config file and viseversa. Unless we try to do all that stuff through CI/CD and scripting. Not all of us are pulling from places like vault etc... Or have the resources to so yet.

Was this page helpful?
0 / 5 - 0 ratings