Configuration-as-code-plugin: Support a conf.d/ directory for placing YAML snippets

[rtyler]

For some Jenkins Essentials use-cases, I believe it would be really useful if we could configure Jenkins to use YAML files loaded from a directory rather than one large monolithic YAML file for the entire configuration.

For example, having cloud.yaml, jenkins.yaml, `security.yaml, etc, would make managing our configuration sprawl much eeasier.

FYI @batmat

[ewelinawilkosz]

Interesting idea, we should definitely consider that
once we have root elements well defined (#110) we can maybe have separate file for each root element
we can require the files to have specific name, or we can accept any and just check which root element is inside...
Any special requirements @rtyler ?

[daniel-beck]

Not knowing the details of this requirement, I expect that a file per root element wouldn't really help, root elements should be distributable across multiple files, and just be merged.

I expect that cloud.yaml and security.yaml as mentioned by @rtyler would need to share the same root element jenkins based on https://github.com/jenkinsci/configuration-as-code-plugin/tree/master/demos/build_agents and https://github.com/jenkinsci/configuration-as-code-plugin/tree/master/demos/global-matrix-auth

[rtyler]

@ewelinawilkosz As you can tell, @daniel-beck has gotten quite good at parsing my vague suggestions over the years :smile:

But yes, he's pretty spot on, the ideal scenario for me would be to slice up my .yaml files in a manner which makes sense for my needs, and then have the Configuration as Code plugin merge conf.d/*.yaml together before applying configuration.

No special requirements otherwise. I am expecting to drive 100% of the initial configuration via YAML for Jenkins Essentials, so I'm already a bit worried about configuration sprawl :face_with_thermometer:

[batmat]

+1, gave it some thoughts. And I agree having a file per root wouldn't fly. How about creating a JEP for this part?

For instance, we'd need some sort of documented ordering logic:

Imagine two files

jenkins:
  systemMessage: "Welcome to Jenkins Essentials!"
  numExecutors: 0

jenkins:
  numExecutors: 5
...

What would be the result? Exception? numExecutors=5? numExecutors=0?

Maybe a simple naming rule, that files should have 00, 01, or just the filename itself, guaranteeing that the alphanumerical order will be used, and the last value would win (or not)?

[buep]

I like your suggestions as well, and I would find myself comfortable using this approach like on Linux with all the conf.d directories.
So I suggest to lean on that approach. I couldn't dig up the load order, or what happens when redefining things.

But if I redefine say numExecutors I would like to at least see a warning being written somewhere.

[batmat]

I would like to at least see a warning being written somewhere

:+1: I definitely agree it is (at least) desirable in the short term, as CasC is under heavy development it could help detect wrong behaviour and so on. Then, _possibly_, in the future, offer a way to silence this warning if people give good reason for this use case.

[MadsNielsen]

I like this idea of a standard conf.d folder for configuration files, so what we could do as a start would be:

1. Sort the list of configuration sources by filename.
2. The last always wins
3. We'll create a merge function to merge configuration files

If anyone has any objections to i'll see to making this.

[ewelinawilkosz]

I think we could start with your proposal @MadsNielsen, is sums up what we've discussed above

[ndeloof]

I also think conf.d would be a nice option. Trivial to implement (if CASC_JENKINS_CONFIG points to a folder, then load all files from inside) but need to manage conflicting attributes. Maybe we will need to pre-process the yaml contents and merge all sources into a single tree. Doing so we could also detect conflicts as noticed by @batmat and just "fail fast".

[ndeloof]

I just noticed using a configuration folder to group yaml files is already implemented !
https://github.com/jenkinsci/configuration-as-code-plugin/blob/master/src/main/java/org/jenkinsci/plugins/casc/ConfigurationAsCode.java#L178

[jetersen]

And the glob allows for multiple folders: https://github.com/jenkinsci/configuration-as-code-plugin/blob/master/src/main/java/org/jenkinsci/plugins/casc/ConfigurationAsCode.java#L53

[batmat]

@ndeloof so does it mean this issue could be closed?

AIUI, this got added (or refactored, IIUC) in https://github.com/jenkinsci/configuration-as-code-plugin/pull/119

[ndeloof]

Moving to a specific issue for collision detection