Toml: Clarify that a table defined by dotted keys can not be overridden

[pradyunsg]

This clarification should be added after the example of super-tables in "Tables" section.

Dotted keys can't be overridden via a regular table but addition of new sub-tables is allowed.

[LongTengDao]

That means this is invalid:

a.b = {}
a = {}

But we know this is valid:

[a.b]
[a]

So is this valid? I think this case should be clarifed:

a.b = {}
[a]

[pradyunsg]

@LongTengDao Could you clarify what you mean by includes / excludes, by using valid / invalid classification instead?

[LongTengDao]

@pradyunsg Oh, sorry

[pradyunsg]

a.b = {}
[a]

This is valid, as long as [a] doesn't define b.

Essentially, if you see an inline table, that key will be exactly that and no more.

[ChristianSi]

@pradyunsg That's not my understanding of the "strict interpretation" (see #499).

By writing a.b = ..., you don't just define a.b (as whatever the value is), but also a (as a table). So you might still add other values to it within the same table block (e.g. a.c, a.d.f) but you are no longer allowed it define it outside of that block. Otherwise a would be defined twice, which is forbidden by TOML.

````toml
a.b = {}
a.c = 1

[a] # ILLEGAL, a has already been defined

[a.b] # ILLEGAL, obviously

[a.d] # legal, we haven't seen a.d yet
````

[ChristianSi]

Also, isn't that exactly the topic this issue is meant to address? (The other case, first mentioned by @LongTengDao, should be covered by #630).

For clarity, however, the subject of this issue might be changed to "Clarify that a table defined by dotted keys can not be overridden" rather then using the singular ("a dotted key").

[workingjubilee]

My understanding is that the strictest interpretation, in the end, was not the one that was chosen.

a.b = {}
[a]
c="One! One little example! Ah ah ah ah ah!"

is extremely inelegant, but is legal, for the same reason that

[a.b]
d="Why does everyone use these alphabet examples?"
[a]
c="One! One little example! Ah ah ah ah ah!"

is legal, no matter which order you put [a] and [a.b] in. This has some advantages for the language in one way, and the highly obvious disadvantages you've already noted. But since it is at all legal, the maintainers have expressed an unwillingness to change that and break compatibility, per the final decision in https://github.com/toml-lang/toml/issues/499#issuecomment-501659274

[profile]
release.debug = true

[profile.release.misc]
alpha = "A"

would risk being invalidated if we did alter the definitions of tables "opening" and "closing" such that heretofore undescribed supertables in dotkeys cannot have keys appended to them in later table definitions, when this was listed as an example that was intended as being valid. I do not think a more restrictive syntax would be automatically bad, mind, I just also do not believe reopening [discussion] will allow me to change the keys or values residing therein. So, I have opened a PR elaborating my understanding of the maintainers' selected interpretation. I would be quite happy to write the exact opposite PR, also.

[workingjubilee]

If people still feel this needs to be consulted on, though, I'd be happy to ask literally everyone in the Rust ecosystem that I get face-to-face time with for their opinions on how best to handle this bit of the grammar, since it really is on their heads to implement the .toml spec.

[ChristianSi]

@workingjubilee:

My understanding is that the strictest interpretation, in the end, was not the one that was chosen.

That's a misunderstanding. The comment you mention refers to the out-of-order definition of dotted keys, i.e. stuff like this:

a.b.value1 = 1
a.c.value1 = 2
a.b.value2 = 3

That'll remain legal in TOML 1.0, but the strict interpretation was never about the order of keys within a single table block but about how dotted keys might interact with inline tables and stuff defined in other tables blocks.

[ChristianSi]

@workingjubilee: Also, please note that your example

[profile]
release.debug = true

[profile.release.misc]
alpha = "A"

remains valid under the strict interpretation. The strict interpretation merely forbids the following (since it would amount to defining the table "profile.release" twice):

[profile]
release.debug = true

[profile.release]
misc.alpha = "A"

[pradyunsg]

@pradyunsg That's not my understanding of the "strict interpretation"

Indeed. My bad -- I likely hadn't had enough sleep, back when I made that comment. :(

[workingjubilee]

Both of those statements would otherwise contain an identical meaning, and at no point does either explicitly contain a [table] declaration with the same path. So, I think either both should be valid, or neither should be, instead of the current situation. Because even if you are correct, it allows what are essentially "hidden" table declarations e.g.

[ninja]
hiding.under.a.tatami.mat.name = "半蔵" # What? It's UTF-8!
[ninja.hiding.under.a.tatami.mat] # invalid, table was already defined
[ninja.hiding.under.a.tatami] # valid... because... wait what?

In addition, it creates extremely odd cases when it comes to defining inline tables, where they have a very different behavior to other definitions, because I believe

[ninja]
hiding.under.a.tatami.mat.name = { given="半蔵" }# What? It's UTF-8!
[ninja.hiding.under.a.tatami.mat] # is valid

[eksortso]

I'd thought that every part of a dotkey _(nice word!)_ defines a subtable in the current block.

Is this right or is this wrong?

[ninja]
hiding.under.a.tatami.mat.name = "半蔵" # What? It's UTF-8!

[ninja.hiding.under.a.tatami.mat] # INVALID, defined in [ninja]
[ninja.hiding.under.a.tatami]     # INVALID, defined in [ninja]
[ninja.hiding.under.a]            # INVALID, etc.
[ninja.hiding.under]              # INVALID, etc.
[ninja.hiding]                    # INVALID, etc.
[ninja]                           # INVALID, but that's obvious

[ChristianSi]

@eksortso: You're absolutely right.

Also (re @workingjubilee 's final remark) it doesn't matter whether the value of a dotted key is an inline table or anything else.

[workingjubilee]

Gathering based on what @pradyunsg said, no, given

a.b = {}
[a]

This is valid, as long as [a] doesn't define b.

Essentially, if you see an inline table, that key will be exactly that and no more.

was apparently intended, which would be consistent with the usual syntax of

[ninja.hiding.under.a.tatami.mat]
[ninja]

being valid. Describing them and locking them out of having more key/value pairs attached to them does make the differences between the two versions of

[profile]
release.debug = true

[profile.release.misc] # valid
alpha = "A"

[profile.release] # invalid
misc.alpha = "A"

easier to reason around, and I'd be happy with it. Yet, I would still somewhat prefer to take it further to declare that all the dotkeys in a table declaration, combined, could be collectively considered as a single inline table, and refuse appending of any kind later, as I think that would maximally preserve the intent of dotkeys (quickly describing a small nested table) while making it easier to stop worrying about them. I could go either way, as I believe both are much more consistent, and my preference might arguably be too inflexible.

But I guess we are reopening the [discussion]? Well, as long as we are, before backwards compatibility comes up, I did some research and I don't think having a much clearer or harder definition of dotkeys is much of a concern... remember, all the Golang parsers for TOML seem to still be on 0.4, and I found that most of the existing major .toml configuration files, such as pyproject, Cargo, etc., Netlify, don't use dotkeys, even where it would be sane to do so. In fact I don't think I saw a single usage of them, though I only spent a few hours thumbing through Github. And I've even seen things disregarding basic TOML rules that have been around since probably 0.1, and some other fascinating errors that I'm guessing were permitted by the parser. It would be better to risk a few minor edge-case incompatibilities now and clarify this, with sharp edges, than compromise on the format overall and not only deal with a headache explaining it a thousand times over but also probably inflicting it on others.

I could collect more actual statistics, rather than a mere anecdote from a couple hours of research, but I would need a bit of time to assemble them.

[ChristianSi]

@workingjubilee: @pradyunsg's comment was a mistake, as they themselves have since (https://github.com/toml-lang/toml/issues/631#issuecomment-522074896) pointed out.

In my view, the discussion is indeed closed and I haven't yet heard any compelling arguments for opening it again. Let's just document and implement the decision made in #499.

Also, for what it's worth, Go is just one of dozens of languages for which TOML parsers exist and where TOML is used.

[pradyunsg]

Thanks for getting things moving again @eksortso @ChristianSi @workingjubilee @LongTengDao! ^>^

I haven't had much time for TOML (actually, all my OSS stuff) lately. I intend to make time for TOML 1.0 stuff early next month, but I'm not sure how that looks yet. :3

---

Apologies again for the incorrect statement earlier, @ChristianSi's comment here is indeed what we'd decided.

@workingjubilee Does the clarification above address your concerns? Thanks for offering to ask around and research into actual uses of TOML in the wild. :)

[workingjubilee]

Mmmalrighty! That does clear things up.

As promised, I did take a crack at writing that (@ChristianSi's definition).