Toml: Proposal: Allow newlines and trailing commas in inline tables

[JelteF]

Overall I really like toml and its syntax feels very obvious to me for the most part. The only thing that doesn't is the expclicit cripling of inline tables, i.e. inline tables cannot have newlines or trailing commas. I've read the reasoning behind this in the existing issue and PR. However, I don't think that the reason given (discouraging people from using big inline tables instead of sections) weighs up against the downsides. That's why I would like to open up a discussion about this.

There's three main downsides I see:

1. It's unexpected for most people using the language. Most popular languages that have {} style mappings allow newlines in them (JSON, Python, Javascript, Go). Also newlines and trailing commas are allowed in lists in the toml spec, so it is inconsistent in this regard.
2. To me even small inline tables are much more readable at first glance when split over multiple lines:

# Single line
person = { name = "Tom", geography = { lat = 1.0, lon = 2.0 } }

# Multi line
person = { 
    name = "Tom", 
    geography = { 
        lat = 1.0, 
        lon = 2.0,
    },
}

1. A deeply nested list of tables are forced to have a lot of repeated keys in the section headers. Compare this version with list of tables:

[main_app.general_settings.logging]
log-lib = "logrus"

[[main_app.general_settings.logging.handlers]]
  name = "default"
  output = "stdout"
  level = "info"

[[main_app.general_settings.logging.handlers]]
  name = "stderr"
  output = "stderr"
  level = "error"

[[main_app.general_settings.logging.handlers]]
  name = "access"
  output = "/var/log/access.log"
  level = "info"

To the one with inline tables with newlines:

[main_app.general_settings.logging]
log-lib = "logrus"

handlers = [
    {
        name = "default",
        output =  "stdout",
        level = "info",
    }, {
        name = "stderr",
        output =  "stderr",
        level = "error",
    }, {
        name = "access",
        output =  "/var/log/access.log",
        level = "info",
    },
]

Finally, extending current toml parsers to support this is usually really easy, so that also shouldn't be an argument against it. I changed the the https://github.com/pelletier/go-toml implementation to support newlines in tables and I only had to change 5 lines to do it (3 of which I simply had to delete).

[eksortso]

Maybe I'm off-base, but I'm not yet sold on this proposal.

Regarding the first point, considering that arrays and tables are _different things,_ the perceived inconsistency in syntax is not a problem, is perfectly acceptable, and sets these different things off nicely.

Skipping down to point 3, isn't the following equivalent? It's already legal TOML, it's readable, and it's space-efficient, or so I like to think. You may disagree with me (especially since I swapped two of the keys) but at least take a look:

[main_app.general_settings.logging]
log-lib = "logrus"

handlers = [
    {name = "default", level = "info",  output = "stdout"},
    {name = "stderr",  level = "error", output = "stderr"},
    {name = "access",  level = "info",  output = "/var/log/access.log"},
]

Also consider this. Any more readable?

[person]
name = "Tom"
geography = {lat = 1.0, lon = 2.0}

Inline tables are fully intended to be _small_ tables, with multiple key/value pairs on one line. If the tables in your (quite readable) example were any larger, then double-bracket notation would make much more sense, even with repeated keys, and you'd get the one-line-one-pair that you seem to find aesthetically appealing.

In either case, we don't need to add a pseudo-JSON to get readability, no matter whether it would be simple to implement.

[JelteF]

@eksortso I can see where you're coming from on the first point. I do disagree though, because IMHO they are very much similar because they're both inline datastructures. I don't know how to make that argument more convincing though. I think my main point there is: both the ararys and the inline tables have pseudo-JSON syntax, but the inline tables are missing some features right now that you would expect coming from JSON (or any other language that has similar syntax).

On the other two examples you make some good points. I took the second example because it was mentioned in the original issue. I see now though that there was discussion on that issue if it was even a good example.

The third one is an issue I actually have myself with my configs. I think you made some good points there as well. Especially the aligning of keys in the third one helps quite a lot with readability. I do think you indeed cheated in a smart way a bit by moving the keys around a bit. I'll will expand on that point and hopefully make my arguments there a bit stronger, but of course you're still allowed to disagree:

Modified point 3

I'll show the same piece of config in different ways below and list some of disadvantages and advantages with each one.

With double brackets

[main_app.general_settings.logging]
log-lib = "logrus"

[[main_app.general_settings.logging.handlers]]
  name = "default"
  output = "stdout"
  level = "info"

[[main_app.general_settings.logging.handlers]]
  name = "stderr"
  output = "stderr"
  level = "error"

[[main_app.general_settings.logging.handlers]]
  name = "http-access"
  output = "/var/log/access.log"
  level = "info"

[[main_app.general_settings.logging.loggers]]
  name = "default"
  handlers = ["default", "stderr"]
  level = "warning"

[[main_app.general_settings.logging.loggers]]
  name = "http-access"
  handlers = ["default"]
  level = "info"

Advantages:

• Diffs are extremely clear, a line changed means that value changed.
• Short lines

Disadvantages:

• Lot's of times repeated main_app.general_settings.logging
• Hard to see at first glance that there's two distinct arrays handlers and loggers
• Quite a lot of vertical space is taken

With inline tables unaligned

[main_app.general_settings.logging]
log-lib = "logrus"

handlers = [
    {name = "default", output = "stdout", level = "info"},
    {name = "stderr", output = "stderr", level = "error"},
    {name = "http-access", output = "/var/log/access.log", level = "info"},
]
loggers = [
    {name = "default", handlers = ["default", "stderr"], level = "warning"}, 
    {name = "http-access", handlers = ["http-access"], level = "info"},
]

Advantages:

• Very little vertical space is used

Disadvantages:

• Looks messy, which makes it hard to compare the different tables in a single list.
• Line based diffs don't show easily what value changed.

With inline tables without newlines without reordered keys

[main_app.general_settings.logging]
log-lib = "logrus"

handlers = [
    {name = "default",     output = "stdout",              level = "info"},
    {name = "stderr",      output = "stderr",              level = "error"},
    {name = "http-access", output = "/var/log/access.log", level = "info"},
]
loggers = [
    {name = "default",     handlers = ["default", "stderr"], level = "warning"}, 
    {name = "http-access", handlers = ["http-access"],       level = "info"},
]

Advantages:

• Looks quite pretty
• Very little vertical space is used

Disadvantages:

• Quite long lines because of the added white space.
• Changing the length of a value requires a some effort. You have to change the spacing in that line or in the other lines appart from changing the value itself.
• Line based diffs don't show easily what value was changed.
• Changing one value can even show other lines as changed in diffs because the spacing had to be chanegd.

With inline tables without newlines with reordered keys

[main_app.general_settings.logging]
log-lib = "logrus"

handlers = [
    {name = "default",     level = "info",  output = "stdout"},
    {name = "stderr",      level = "error", output = "stderr"},
    {name = "http-access", level = "info",  output = "/var/log/access.log"},
]
loggers = [
    {name = "default",     level = "warning", handlers = ["default", "stderr"]}, 
    {name = "http-access", level = "info",    handlers = ["http-access"]},
]

Advantages:

• Lines are less long than without reordering
• Looks quite pretty

Disadvantages:

• Still quite long lines.
• You have to reorder the keys, possibly having to choose between a logical order and order in which the whitespace is minimised.
• Changing the length of a value requires a some effort. You have to change the spacing in that line or in the other lines appart from changing the value itself.
• Line based diffs don't show easily what value changed.
• Changing one value can even show other lines as changed in diffs because the spacing had to be chanegd.

With newlines

[main_app.general_settings.logging]
log-lib = "logrus"

handlers = [
    {
        name = "default",
        output =  "stdout",
        level = "info",
    }, {
        name = "stderr",
        output =  "stderr",
        level = "error",
    }, {
        name = "http-access",
        output =  "/var/log/access.log",
        level = "info",
    },
]
loggers = [
    {
        name = "default",
        handlers = ["default", "stderr"]
        level = "warning",
    }, {
        name = "http-access",
        handlers = ["http-access"]
        level = "info",
    },
]

Advantages:

• Diffs are extremely clear, a line changed means that the value changed.
• Short lines

Disadvatages:

• Needs to indent twice
• Quite a bit of vertical space is used.

Conclusion

I think ultimately it's a matter of taste what looks better. And a matter of tradeoffs between, repeated keys, vertical space, line length, diff clarity and logical vs visually pleasant key ordering. I think my main point with this example is that it would be nice if users could choose what they find more important.

[seeruk]

I'm all for this. Just started to look into TOML properly for the first time as I was planning on using it for the configuration file for a tool I'm writing. I really like TOML overall, but this one thing makes some specific things really nasty. The bit I'm working on is actually sort of like the Docker Compose syntax in some ways.

Take this YAML for example:

version: "3"

services:
    elasticsearch:
        container_name: metrics_elasticsearch
        image: docker.elastic.co/elasticsearch/elasticsearch:5.5.3
        network_mode: host
        environment:
          discovery.type: single-node
          http.cors.enabled: true
          http.cors.allow-origin: "*"
          xpack.security.enabled: false
        ports:
        - 9200:9200
        - 9300:9300
        volumes:
        - elasticsearch-data:/usr/share/elasticsearch/data

    kibana:
        container_name: metrics_kibana
        image: docker.elastic.co/kibana/kibana:5.5.3
        network_mode: host
        environment:
          ELASTICSEARCH_URL: http://localhost:9200
          XPACK_MONITORING_ENABLE: false
        ports:
        - 5601:5601

volumes:
    elasticsearch-data:
        driver: local

And then compare it to the equivalen TOML:

version = "3"

[services]

  [services.elasticsearch]
  container_name = "metrics_elasticsearch"
  image = "docker.elastic.co/elasticsearch/elasticsearch:5.5.3"
  network_mode = "host"
  ports = [
    "9200:9200",
    "9300:9300"
  ]
  volumes = [
    "elasticsearch-data:/usr/share/elasticsearch/data"
  ]

    [services.elasticsearch.environment]
    "discovery.type" = "single-node"
    "http.cors.enabled" = true
    "http.cors.allow-origin" = "*"
    "xpack.security.enabled" = false

  [services.kibana]
  container_name = "metrics_kibana"
  image = "docker.elastic.co/kibana/kibana:5.5.3"
  network_mode = "host"
  ports = [
    "5601:5601"
  ]

    [services.kibana.environment]
    ELASTICSEARCH_URL = "http://localhost:9200"
    XPACK_MONITORING_ENABLE = false

[volumes]

  [volumes.elasticsearch-data]
  driver = "local"

That extra level of nesting just makes TOML that much less nice to use in this case. If the environment could be on the same level as the rest of the service configuration it'd tidy it right up.

version = "3"

[services]

  [services.elasticsearch]
  container_name = "metrics_elasticsearch"
  image = "docker.elastic.co/elasticsearch/elasticsearch:5.5.3"
  network_mode = "host"
  environment = {
    "discovery.type" = "single-node",
    "http.cors.enabled" = true,
    "http.cors.allow-origin" = "*",
    "xpack.security.enabled" = false,
  }
  ports = [
    "9200:9200",
    "9300:9300"
  ]
  volumes = [
    "elasticsearch-data:/usr/share/elasticsearch/data"
  ]

  [services.kibana]
  container_name = "metrics_kibana"
  image = "docker.elastic.co/kibana/kibana:5.5.3"
  network_mode = "host"
  environment = {
    ELASTICSEARCH_URL = "http://localhost:9200",
    XPACK_MONITORING_ENABLE = false,
  }
  ports = [
    "5601:5601"
  ]

[volumes]

  [volumes.elasticsearch-data]
  driver = "local"

[eksortso]

At the heart of your issue, you have a large subtable that you wish to keep in the middle of your configurations. Not before it, and not after it. Some relief exists with inline tables and key-path assignments. But with a table nested a few layers deep, the keys would grow very long.

I still find multiline tables that look like JSON offputting. But I think I have an idea for a TOML-friendly syntax that could get you what you're wanting. I don't have time to write it down now, but I'll be back later on.

[pradyunsg]

Hi @JelteF!

Thanks for filing this issue. I'm deferring any new syntax proposal as I try to ramp up my effort to get us to TOML 1.0, which will not contain any new syntax changes from TOML 0.5.

This is definitely an idea I want to explore more -- personally, I still haven't finalized how much TOML should be flat (INI-like) vs nested (JSON-like). Both approaches have their trade-offs and we'll know what we want to do for this specific request, once we finalize that overarching idea. However, I'd appreciate if we hold off that discussion until TOML 1.0 is released.

[polarathene]

The earlier example:

[services]
  [services.kibana]
  container_name = "metrics_kibana"
  image = "docker.elastic.co/kibana/kibana:5.5.3"
  network_mode = "host"
    [services.kibana.environment]
    ELASTICSEARCH_URL = "http://localhost:9200"
    XPACK_MONITORING_ENABLE = false

Could instead be:

[services]
  [.kibana]
  container_name = "metrics_kibana"
  image = "docker.elastic.co/kibana/kibana:5.5.3"
  network_mode = "host"
    [.environment]
    ELASTICSEARCH_URL = "http://localhost:9200"
    XPACK_MONITORING_ENABLE = false

The example just takes advantage of the dotted keys notation, in that if the key starts with a dot, it would inherit the parent table keyspace. I went with a dot as it is has related meaning for a relative path as well ./, although another symbol may stand out better?(or could instead be prepended to the table syntax, .[environment])

The above deals with the issue of table keys getting progressively longer, where the actual table unique name gets offset to the right(potentially requiring scrolling) and/or lost in the noise of similar table keys as shown earlier in the thread.

Personally, for nested config the table keys or dotted keys can get quite long/repetitive. It's one area that I think JSON and YAML handle better.

I still find multiline tables that look like JSON offputting. But I think I have an idea for a TOML-friendly syntax that could get you what you're wanting. I don't have time to write it down now, but I'll be back later on.

@eksortso I take it later on never came, or did you raise it in another issue? What do you think about the above?

I did find it odd that inline tables have this special syntax for single lines, unable to break to multi-line with trailing commas like arrays can. Most new comers to TOML will be familiar with a table/object being defined this way and it'd click, until they realize it breaks should you want to go to multiple lines, yet arrays don't share this restriction.

I personally prefer curly brackets for additional clarification of scope. TOML appears to rely on name-spacing allowing for a flat format should you pay attention to the keys. Some try to indicate the scope a bit more via the optional indentation as shown earlier but that uncomfortable/detached to me.

I like that end of lines don't need commas in TOML, although they're required for arrays(and inline tables), they could be dropped/optional for multi-line variants?:

[services]
  [.kibana]
  container_name = "metrics_kibana"
  image = "docker.elastic.co/kibana/kibana:5.5.3"
  network_mode = "host"
  environment = {
    ELASTICSEARCH_URL = "http://localhost:9200"
    XPACK_MONITORING_ENABLE = false
  }
  ports = [
    "5601:5601"
  ]

  [.kibana_2]
  container_name = "metrics_kibana2"
  image = "docker.elastic.co/kibana/kibana:5.5.3"
  network_mode = "host"
  environment = {
    ELASTICSEARCH_URL = "http://localhost:9201"
    XPACK_MONITORING_ENABLE = false
  }
  ports = [
    "5602:5601"
  ]

---

This example from the project readme is a good case of verbosity/noise that gave me a double take of trying to make sense of what was going on:

[[fruit]]
  name = "apple"

  [fruit.physical]  # subtable
    color = "red"
    shape = "round"

  [[fruit.variety]]  # nested array of tables
    name = "red delicious"

  [[fruit.variety]]
    name = "granny smith"

[[fruit]]
  name = "banana"

  [[fruit.variety]]
    name = "plantain"

This is probably not much better, and might be asking for too much?(strays too far from what TOML currently is?):

[[fruit]]
name = "apple"
physical { # Scoped table
  color = "red"
  shape = "round"
}
variety [ # Scoped array of tables
  name = "red delicious"
  --- # A separator between objects
  name = "granny smith"
]

[[fruit]] # Still useful as a `---` above may not be distinct enough
name = "banana"
variety [
  name = "plantain"
]

Applied to the earlier example for arrays of tables:

[main_app.general_settings.logging]
log-lib = "logrus"

handlers [
    name = "default"
    output =  "stdout"
    level = "info"
    ---
    name = "stderr"
    output =  "stderr"
    level = "error"
    ---
    name = "http-access"
    output =  "/var/log/access.log"
    level = "info"
]
loggers [
    name = "default"
    handlers = ["default", "stderr"]
    level = "warning"
    ---
    name = "http-access"
    handlers = ["http-access"]
    level = "info"
]

The use of --- as a separator between elements allows for avoiding unnecessary{ }(which are useful for a single instance assigned to a key), as those add noise along with , that @eksortso I believe found offputting?

Note the lack of assignment =, that would probably lead to some mishaps with array elements as you'd need ,(instead of inferring from \n) on single lines and objects/tables would need to be wrapped with { }..

[eksortso]

@eksortso I take it _later on_ never came, or did you raise it in another issue? What do you think about the above?

Ouch...

_Later on_ came and went. See #525 for discussion, and #551 for the now-closed PR.

I'll be back in a few hours.

[polarathene]

Ouch...

Oh, I didn't mean it that way! :stuck_out_tongue_closed_eyes:

Later on came and went. See #525 for discussion, and #551 for the now-closed PR.

Ah, that's unfortunate.. :disappointed: I liked the multi-line approach you proposed, substituting commas with new lines. HJSON ended up offering a good enough solution for me offering this feature in the meantime.

[eksortso]

Ouch...

Oh, I didn't mean it that way! 😝

No worries. But there is a link to #525 up there.

Later on came and went. See #525 for discussion, and #551 for the now-closed PR.

Ah, that's unfortunate.. 😞 I liked the multi-line approach you proposed, substituting commas with new lines. HJSON ended up offering a good enough solution for me offering this feature in the meantime.

Thanks! That's good how HJSON implemented it. I've see similar patterns in other config formats, whose names I've forgotten.

But keep in mind that HJSON is based on JSON, and TOML was originally inspired by informal INI formats. What that means, philosophically, is that nesting in TOML is possible, but deep nesting is, and _ought to be_, discouraged. By that philosophy, shallow nesting is ideal for a configuration format, and it also works for simple data exchange uses. Over time, I've come to adopt this philosophy myself. I'm still interested in bringing back a little bit of nesting, a la #551, but unless it gains traction, I won't push for it.

Other proposals have been offered to use [.subtable] syntax for nesting. But it can get confusing if you can't keep track of your absolute path. In fact, your first example suggests that each [.subtable] nests inside its parent, but your second example suggests that each [.subtable] is a subtable of a common parent. Is [.kibana_2] actually [services.kibana_2], or [services.kibana.kibana_2]?

But there's another problem that [.subtable] syntax doesn't solve, which was my impetus for #525: it can't be used to put subtable definitions _in the middle_ of other tables. That was relieved with the introduction of dotted keys in key/value pairs. Again, it works best for shallow nesting.

Regarding commas in arrays and in inline tables, I do feel like the rules for placing those commas ought to be strict, to prevent confusion. It's already decided that arrays require commas between elements, and that a trailing comma is fine. For inline tables, commas must separate the key/value pairs, since they're on the same line. If #551 were reintroduced, newlines could be used to separate the key/value pairs in multi-line inline tables, same as they are used for regular tables. But commas would not be allowed between lines.

I'm intrigued by some of your other proposals, particularly the --- separator in arrays of tables (which could be used with regular table-array notation, actually). But I'll suggest that you simplify their presentations, and open a new issue for each to present them. Long posts like these are often hard to follow, so less really would be more.

[brunoborges]

Perhaps worth connecting this proposal with #744 as well, for use of placeholders/shortcuts to outer tables names.

Example:

[servers]
mask = "255.255.255.0"

[*.server1] # subtable 2
ip = "192.168.0.1"

[*.server2] # subtable 3
ip = "192.168.0.2"

The above is the same as explicit/verbose keys servers.server and servers.server2 in tables 2 and 3.

[oovm]

I agree to support line break.

---

Yes, there exists a form that makes the final result look good and easy to read.

But the problem is that the conversion tools and serde tools can’t do it.

The conversion tool can only convert a long line of things that cannot be read.

If line breaks are allowed, these tools can adjust the indentation to make the results look better.

[dmbaturin]

For me as a user, the fact that newlines aren't allowed inside inline tables was _extremely_ surprising.

For me as an implemented, that's a special case in the parser that I wish I could get rid of.

I'm all for allowing it.