Terminal: Reconsider using JSON for configuration

[flying-sheep]

Description of the new feature/enhancement

Having tried out quite a few editors and IDEs as well as programming languages and their infrastructure, I’m pretty sure I can argue from experience:

JSON, while being a decent serialization language, is a bad configuration language.

It has a few traits that make it nice to write parsers for, but unnecessarily restrictive for manual editing as configuration:

• No comments. This is such a pain that many extend it to have this.
• No trailing commas. That simply makes it annoying to add list or dict entries at the end, and makes VCS diffs unnecessarily noisy.
• Mandatory quotes. Config keys are often chosen to be valid identifiers, so allowing a subset of many languages’ identifiers ([A-Za-z_][A-Za-z0-9_]*) to work as dict keys without quotes would remove a lot of visual noise.
• No multiline strings. Sometimes you want one of these (e.g. for a long regex or a template) without having to squeeze it all into one line with \ns.
• Lack of visual structure. There’s only dicts and arrays, while config files often benefit from being split into sections (I.e. a nicer syntax for the top level dict)
• No NaN or Inf. That comes straight from its JavaScript heritage and has no justification outside of it.

JSON isn’t designed to be a config language and it shows. (Actually I googled “json as configuration” and found this article which happens to be a long form of what I just typed)

Proposed technical implementation details (optional)

If your configuration is (or can be made) flat enough that TOML works, I’d recommend it. It’s basically a more sane, standardized version of INI. It fixes all of the above and more. After all, it’s designed to be a config language!

If you really need deep nesting (I assume almost nobody does and they just didn’t try finding a schema that’s flatter), CSON or a YAML subset would work, but there’s nothing that’s both simple and widely used.

[CoolDadTx]

Like it or not JSON has become a configuration language for a lot of apps now. VS Code uses it as an example. The reason why it has become this way is because the app needs to read it and JSON is very easy for an app to read.

The point you're arguing I generally agree with however I think the solution isn't to replace JSON. If you did then the app would be reading that format instead and JSON, honestly, has won for now. I think the better solution is to provide a GUI for editing the settings and then it doesn't matter what format is used.

Honestly, most of your complaints are tied to the serialization settings being used. Comments do work in the current JSON. The default file has them already. The trailing comma is a serializer setting and can be easily changed in the code. The concept of sections already exists in JSON and is the basis for objects. A "section" is a JSON object so you can have any # of sections you want. For example in the existing config there is a section for profiles, a section for schemes, a section for key bindings, etc. Your remaining points are your personal preference and I'd argue that they are not useful to others. However if we go back to the solution of providing a GUI then the UI can provide you all this and store it in JSON correctly. Provided you're not editing the JSON directly the underlying format and concerns you raised aren't really relevant. Hence the app can quickly read JSON (like every other app) and you get the features you want.

[flying-sheep]

VS Code uses it as an example […] Comments do work in the current JSON

I know, this is why I’m writing this. I like VS Code, but using it shows me that its choice of config language was a mistake.

JSON is a standard, without version number. VS Code is therefore not using JSON anymore. Creating wild west dialects like this is what leads down the path of INI and CSV – nobody knows what a correctly formatted file is because every serializer does things differently. It also defeats your other argument about there being tooling: Both libraries and editor syntax highlighting are already annoying to wrangle with just the variation that added comments. (many editors and language libraries don’t support it, and those that do make you manually select if you have them, including VS Code)

And you’d need a lot of deviations to get JSON to the usability and signal-to-noise ratio of TOML. Which has one standard. Nobody’s trying to fork it because it has been designed as a config language (unlike JSON) and does that job well (unlike JSON).

---

There’s absolutely no discussion to be had if the above is true. If you want to be productive, you can instead talk about

• how MS has specific tooling for JSON and how it integrates into powershell and how that’s relevant here. (if any of that is true)
• Or that there is too much momentum and too many people are using the terminal already. (I doubt that migrating to TOML automatically would be hard though)
• Or that people are accustomed to JSON-with-comments and know how to write it. (Learning all of TOML takes <15 minutes)
• Or that the config schema is by definition too nested to work in TOML and can’t be changed to fit. (if that is true)

But there’s no question that JSON sucks as a config language and that it’s a bad idea to retrofit it with comments and other niceties that dedicated config languages already have.

[CoolDadTx]

I had to google what TOML was. It doesn't look like it is used by any of the major players in the industry. Looking at the list of projects using it seems small. Furthermore it isn't at 1.0 release yet. Compare that to JSON that is used for "configuration" by every major player in the space - Amazon, Google, Microsoft, Oracle, Apache, Chrome, Firefox. The list goes on and on. JSON has won configuration files for now. Having Terminal use another format just because you believe it is better qualified doesn't make good business sense to me.

Irrelevant of the JSON standard, every serializer that I'm aware of supports the standard features of comments, trailing commas, etc that you mentioned as an issue. Again, if we just had a GUI to edit the settings then it wouldn't matter.

I believe your enhancement request is just going to get rejected. There is no value add here. But I do agree with your concerns about editing JSON directly and would rather see your request be implemented as a GUI instead of changing the underlying format. But we can disagree here. I've conveyed everything I feel I can so I'll leave this conversation to others to weigh in.

[flying-sheep]

The only thing we’re apparently disagreeing on is what a “config file” is.

I’m talking about the kind you edit by hand: .vimrc, pyproject.toml. You’re talking about config databases.

TOML wins for the kind I’m talking about (adopted by e.g. Python, Rust, GitLab), sqlite and JSON win for the kind you’re talking about.

I filed this request because the latest blog post contains config snippets, so it’s clearly (for the time being) intended to be edited by hand. The question is: Is a GUI that handles everything planned (e.g. a nice interface for the most common stuff and something like Firefox’ about:config for the rest)? If so, sqlite or JSON is the better choice. If not, TOML is the better choice.

[DHowett-MSFT]

Sorry, this thread deserves a longer response than I'm about to give it. I apologize.

Settings UI is planned for post-v1: #1564
Previous discussion about json/why/why not: https://github.com/microsoft/terminal/issues/475#issuecomment-493706114 https://github.com/microsoft/terminal/issues/475#issuecomment-493672168 https://github.com/microsoft/terminal/issues/475#issuecomment-504708282 https://github.com/microsoft/terminal/issues/3949

[flying-sheep]

Hmm, what tooling is there that doesn’t exist for TOML?

Now that you told me it has a name (I didn’t know), everything I can find about JSONC is a tiny personal page with a broken editor and a handful of implementations, none of which define a grammar. Seems like you at Microsoft had to write the node parser yourselves. TOML is implemented in every language, used e.g. by standard Python packaging (so it’s a fixed, big part of Python itself!), and has a Wikipedia page.

From a cursory glance, it looks like a) JSONC is a Microsoft in-house thing without much relevance in the wider world and b) competes with other efforts like JSON5. But I’m happy to learn about a huge world of JSONC I didn’t know about :wink:

Sad to see that what I feared above about editor highlighting is also a trap you ran right into since you decided in #3949 to not use .jsonc.

Good to hear about the UI! If it completely replaces the need to touch the config file even for advanced settings, it’s a fix for the whole situation.

[ghost]

No comments. This is such a pain that many extend it to have this.

I see this is not true. See the issue I linked

[flying-sheep]

@cup read the thread. We’re all aware that JSONC isn’t JSON. What I’m saying is that JSONC is more esoteric than TOML, and not much better as a config language than JSON.

[ghost]

@flying-sheep personally I think TOML is awful, YAML would be a better choice. But the fact that comment are allowed with JSONC is a huge blow against anyone complaining AFAIC.

[oising]

Sorry, this thread deserves a longer response than I'm about to give it. I apologize.

Settings UI is planned for post-v1: #1564
Previous discussion about json/why/why not: #475 (comment) #475 (comment) #475 (comment) #3949

IF the settings UI is not planned until after 1.0, then I _strongly_ recommend adding an environment variable with the path to the json settings file, so it can be edited with the editor of choice (vi, notepad, nano, emacs, whatever) from within the shell of choice without burning brain calories trying to find it buried in AppData. Also: Relying on the gui menu is, well, unreliable.

Hmm, I'll make an issue for this independently. We've already got WT_SESSION env var -- something like WT_PROFILE would be useful. Yes, from within WSL it would have to be munged into /mnt/c/users/blah, but I can live with that - or maybe add WT_PROFILE_WSL also?

[DHowett-MSFT]

munged

We can actually tell WSL to path-convert it on entry for us :smile:

[oising]

I frequently use notepad $profile from within powershell. Being able to type notepad $env:wt_profile would be nice too.

[oising]

https://github.com/microsoft/terminal/issues/4566

[akulbe]

How do you _get_ to the settings in the latest release? Neither trying to find the profiles.json nor Ctrl-, do anything for me.

[DHowett-MSFT]

@akulbe we've not changed how settings is launched. If you open %LOCALAPPDATA%\Packages\Microsoft.WindowsTerminal_8wekyb3d8bbwe\LocalState and double-click profiles.json, what happens?

[akulbe]

So very weird thing… but VS Code wouldn't open. I had to close all the
instances of it and reopen it and automagically there it was. So please
disregard my comment. 😳

[glen-84]

My 👍 is for YAML, which is a superset of JSON, meaning that those who prefer JSON could continue using it.

[kumarharsh]

I think I'm probably commenting in vain here, but please consider using TOML for configuration. If you absolutely do want to use JSON, please at least use JSON5 - the lack of support for dangling comma on last line is mildly frustrating every time I edit the settings.

[zadjii-msft]

Oh man, it looks like support for trailing commas was added to jsoncpp in https://github.com/open-source-parsers/jsoncpp/pull/1098. From the looks of that, it was only included in their latest 0.11 release, which only came out weeks ago. Whenever we ingest that update (#5297), we'll probably get the trailing comma fix for free*.

*: We'll need to add builder.settings_["allowTrailingCommas_"] to the json reader but that's easy.

[kimsey0]

@zadjii-msft: Yep. I honestly made that patch for jsoncpp just to get support for trailing commas in the configuration here in Windows Terminal. 😄 It's just now released in the mainline v1.9.3 release. Should I do anything to get it incorporated here (submit a pull request) or will it happen in #5297?

[zadjii-msft]

@kimsey0 sorry I lost this comment in my inbox - I'm sending a PR now for updating the dependency, and once that's approved I've got another commit queued for adding the trailing comma support. Thanks for the (indirect) contribution to the Terminal!

[kimsey0]

Thanks a lot!

[WSLUser]

All it takes is time for a good project and people speaking about such good projects for it to become widely adopted. So getting TOML supported as the config for Windows Terminal is simply a matter of someone volunteering time to port it from the currently existing json. We do now have trailing commas coming to Preview for Json at least.

[DHowett]

I'm fairly certain it's CLI11 that supports TOML; it would be a massive retooling to run our settings model _through our commandline argument parser._ :smile:

[WSLUser]

Ha true. Still could be done with time and effort. I fixed my comment.

[plastikfan]

Please remove non standard JSON features in settings, more specifically comments, which are not permitted in JSON. I know why you may have taken the decision, supposedly giving inline assistance to users to help them configure Terminal settings (owing to lack of settings UI), but, this could have been documented elsewhere. Using non standard features makes it difficult if not impossible for 3rd parties to develop stop-gap solutions (as I have attempted to do with TerminalBuddy.)

[davidchisnall]

@DHowett, I think I've mentioned this to you before, but if you're looking for an alternative to JSON that is good for configuration files, I can't recommend the universal configuration language highly enough. Advantages:

• A mature library for serialising / deserialising the config.
• Used in some large projects (e.g. rspamd)
• Supports multiple files with priority and overrides, so it's trivial to provide default, system-wide or organisation-wide (which can be useful for providing default configs for internal systems to all users), and user configurations with no custom code in the applications.
• Human-friendly language
• Macro support
• Decopuled object model and syntax (supports JSON, YAML and Nginx-like file formats)
• Can load JSON, so can be backwards compatible with existing config files.
• Supports a full range of useful types without ad-hoc extensions (e.g. dates / times / 64-bit integers)
• Supports comments (the current config file comments are not valid in standard JSON).

I can't imagine starting a new project that needs complex hand-written configuration today and using anything else.

[mischkl]

As mentioned in an issue in the jsonc library repo, JSONC (unlike JSON5) isn't meant to be anything more than "JSON with comments". It seems like using additional extensions such as trailing commas, while user-friendly for these specific use cases, is not really helping to move the JSON ecosystem forward, since it's creating incompatibility with other libraries and tooling. I'd say the approach of JSON5 is a much better fit for the future of human-editable/readable files, as it has a well-defined specification and not only supports comments and trailing commas, but also unquoted keys. And unlike any of the other proposals it's still a strict subset of JavaScript, meaning its syntax is already well-understood by a huge number of users.