Hugo: Add a /config dir

[bep]

This is issue is vaguely described other places but is long over due. This description should be concrete enough to implement:

• Add a configDir setting, default /config.
• Add a --environment flag (and setting), which defaults to production for hugo and development for hugo server
• Configuration keys in /config will override keys in config.toml

A sample tree could look like this:

config
├── _default
│   ├── config.toml
│   ├── languages.toml
│   └── mediaTypes.toml
└── production
    ├── config.toml
    ├── mediaTypes.toml
    └── related.toml

• config.toml allows top level keys (baseURL) etc. to be set per environment.
• The other filenames maps to a configuration key, so languages.toml could look like this:

  [en]
    contentDir = "content/en"
    languageName = "English"
    weight = 1
  [zh]
    contentDir = "content/zh"
    languageName = "中文"
    weight = 2

There are no clever merging inside /config -- we pick the most specific file (for production and mediaTypes in the example above, we pick the file below /production).

This will (in its first iteration) be a project only thing.

/cc @regisphilibert @budparr @kaushalmodi

[regisphilibert]

Was just thinking about it yesterday having to manage many menu items from the config.yaml. "Boy I wish there was a native way to isolate those menu items in their own file..."

I assume we can add as many files as we want in order to create a config objects. This is not limited to the one you mentioned?

There are no clever merging inside

What do you mean by clever ? I just wish the config/production/config.yaml could overwrite some values of config/_default/config.yaml such as the baseURL or publishDir while using its many "shared" parameters.

[bep]

This is not limited to the one you mentioned?

No, the scope is endless.

What do you mean by clever ?

I should be more clear (and less clever):

• In /config we pick the most specific file (config.toml, mediaTypes.toml etc.)
• mediaTypes.toml wlil be a complete configuration that will replace any in config.toml (there are some special rules for themes here ... but that is little bit on the side)
• Keys inside config.toml below /config will override the root config, so you can override single values there (baseURL etc).

[budparr]

Love this. Slightly uncomfortable with environment variable defaults, but I suppose that if one went to the trouble of using them at all, they'd be aware of the defaults.

Question (hopefully not too off topic). Would it be possible to do a build with no config at all? Right now I believe Hugo looks for a config file, no matter how minimal (even empty). Would be cool to be zero config with nothing required other than a minimal layouts/index.html file. Then Hugo could be "The world’s fastest ZERO CONFIG framework for building websites" :)

[bep]

Slightly uncomfortable with environment variable defaults, but I suppose that if one went to the trouble of using them at all, they'd be aware of the defaults.

You mean the development/production logic?

Yes, it's strictly a little outside the scope of this issue, but doing it at the same time is considerably cheaper. And the number of overrides should be very small (my example is bad) -- and I think this makes it more explicit than the if production we have today.

[regisphilibert]

Yes I suspect environment only key/value pair will be limited to baseURL, googleAnalytics and a few params. So most of the config settings will happen in _default.

This will be an awesome improvements on the way Hugo currently deals with config! 🤩

[budparr]

Yes, I meant that I explicitly set environment variables and that control is predictable to me. Not a big deal though.

[bep]

Yes, I meant that I explicitly set environment variables and that control is predictable to me

Note that if we add environment as a new config key, any value in HUGO_ENVIRONMENT will still work.

[bep]

An important addendum to the above, having thought about this a little. One of the big benefactors of this feature is sites with many languages (and with that really a long languages section with menu definitions etc.).

So, for the languages config I think we need to allow "one dot" in the filename, e.g.:

config
├── _default
│   ├── config.toml
│   ├── languages.en.toml
│   ├── languages.toml
│   └── mediaTypes.toml
└── production
    ├── config.toml
    ├── mediaTypes.toml
    └── related.toml

So in the above, languages.en.toml represents the English configuration and will replace any languages.en key in languages.toml.

/cc @regisphilibert @budparr

[regisphilibert]

But could we still have "standalone file" config for each languages (menus.en.toml, menus.fr.toml) ?

[bep]

But could we still have "standalone file" config for each languages (menus.en.toml, menus.fr.toml) ?

menus isn't a configuration key in Hugo, I think. This issue is getting scope creeped as it is, so we're not adding config options that isn't already there.

I think it helps to think about this issue as "very technical".

[bep]

@regisphilibert I've dived into this issue now, and it does not look too hard. There is one issue that you can help me with. I will think out loud a little. It is related to your last comment; I think we need to be a little more clever when it comes to language handling.

Taken from the Hugo docs site:

languages.zh.menu.main  => array
languages.zh.params => map

(I have made a note to create menus (plural) as an alias -- to get it in line with the others)

I have put the above to wrap my head around and make sure we can represent this in all the config formats. So, only JSON allows arrays without a root, so we need to avoid (or work around) that.

With that, I was thinking that the 2 above examples could look like:

config/menus.zh.toml:

[[main]]
name = "函数"
weight = 30
[[main]]
name = "变量"
weight = 35
[[sidebar]]
name = "变量"
weight = 35

This means that if you have several menus, you need to put them in the same file for that language.

config/params.zh.toml:

description = "A description in that language"

The above makes sense to me ...?

[regisphilibert]

This means that if you have several menus, you need to put them in the same file for that language.

Of course. The important part here is that multilingual projects will not loose this "file isolation" feature by having to dump all their config objets into one "language config" file.

(I have made a note to create menus (plural) as an alias -- to get it in line with the others)

Agree! Current key is counter-intuitive (I always forget it's supposed to be singular).

There is one issue that you can help me with.

Not sure how I can help beside supporting this logic (config/menus.zh.toml) as it is exactly what I would wish as a user (https://github.com/gohugoio/hugo/issues/5422#issuecomment-439037811).

[bep]

Not sure how I can help beside supporting this logic

That's the help I needed.

[bep]

@budparr and @regisphilibert -- given the "config tree" below:

config
├── _default
│   ├── config.toml
│   ├── languages.toml
│   ├── menus
│   │   ├── menu.en.toml
│   │   └── menu.zh.toml
│   ├── params.1.toml
│   └── params.2.toml
└── development
    └── params.2.toml

Do you understand the meaning of params.1.toml etc or is it too clever?

The thing is, the main rule has to be that if languages etc. is set for development, we don't use the one in _default. I cannot know if the user wants to merge without some additional information. There are, however, situations where you have a lot of common params, but where only some of them is different for a given environment, hence the numbering.

Also note that the menus sub folder is only a way to organize your files, it can be named anything, and I will fix "menu vs menus".

[regisphilibert]

Do you understand the meaning of params.1.toml etc or is it too clever?

I'm guessing this is a way of defining a group of params to be merged vs another group to be left as is.

Let me share a personal yet very common (I think) use case to make sure I'm following.

Using the CLI command hugo --config config.yaml,config.prod.yaml, I have a config.yaml with a lot of settings common to both environement, and a config.prod.yaml with a distinct baseURL and params.api_url. Those will override the one present in config.yaml.
If a key present in config.yaml needs to be absent in production I suppose I could set it to false in config.prod.yaml.

By what I understand this is how I would replicate this behaviour using the new config directory feature:

├── _default
│   ├── config.toml (baseURL set here)
│   ├── params.1.toml (params common to both env. set here)
│   └── params.2.toml (api_url set here)
└── production
│   ├── config.toml (baseURL overriden here)
    └── params.2.toml (api_url overriden here)

Am I following?

[bep]

Am I following?

Yes.

The main motivation here is that if you have 120 generic params + 1 environment specific param, you don't want to repeat the 121 params in every environment.

Also note that this is a general pattern; you could also do config.5.toml-- the number is a key element, but it also controls the order (params.2.toml is added after 1).

[regisphilibert]

So still with the previous use case I could not do the following:

├── _default
│   ├── config.toml (baseURL set here)
│   └── params.toml (params common to both env. set here + default api_url)
└── production
│   ├── config.toml (baseURL overriden here)
    └── params.toml (default api_url overriden here with production api_url)

And expect the params common to both env to be set in the production env. as well.

If this is a technical decision then I'm happy to trade this very light complexification to be able to manage configuration with directories and filenames.

If this is a usability decision on the other hand, I'd like to understand the logic behind it as it presently escapes me.

[bep]

Your last example will not work. This is merged on "file level": We pick the most specific file, e.g. production/params.toml.

Note that it would be technically possible to implement it so your particular example would work. But that would break other examples/use cases, examples being that I would want a different set of languages, output formats, whatever, in the different environments, so we cannot merge blindly.

[regisphilibert]

But that would break other examples/use cases

But I think this could be solved differently by having Hugo merge blindly while having the user exercise a much expected discipline of never putting in _default something not shared by every environment, unless it can be overridden by a false value.
I would not add a "production only" Output Format in _default, and I don't expect any one would.

Again my suggestion may not be sustainable, but as I cannot think of a use case for which this basic discipline would not suffice, I must insist on asking you or other users if this "blind merging" could not be OK, providing it does not complicate your task of course.

This would avoid introducing and documenting the file numbering system and make the new config management all the more intuitive by matching the behaviour of hugo --config's blind merging logic even though I suspect it is currently underused.

[regisphilibert]

Here's the use case I can think of to defend the numbering file implementation:

I have 5 environments, I need a certain output format in all of them but one. This means with blind merging, I cannot put it in _default and therefore have to copy and paste this output format settings in the 4 environments needing it.

I'm suspecting this use case will be rare and I'm afraid we may complexify a feature for the vast majority of use cases only to benefit a very few.

[bep]

I will sleep on it.

[bep]

@regisphilibert I have slept, and you are right. I was thinking too hard about this. If we for some really good reason need "my variant" we could add some naming standard for that case later. Thanks for the input, always appreciated.

[regisphilibert]

Anytime, glad I could help.