Toml: Formalize using +++ for TOML document streams

[ukiahsmith]

TOML should formalize use of +++ in a way similar to how YAML uses --- to separate distinct YAML structures in a stream or file. [1]

This form is already in widespread use by the Hugo Project for document metadata in the frontmatter. [2]

Formalizing the use of +++ for TOML document separation would improve compatibility between desperate projects using TOML. Formalization would also encourage TOML use as a frontmatter data language for projects that currently use only YAML.

[BurntSushi]

I feel like this issue assumes everyone is already familiar with, and understands, what +++ actually is. I don't. (Even though I do use Hugo.) Could you please explain what it is, its purpose and what its semantics should be?

[ukiahsmith]

Sure.

The --- on a single line by itself is the demarcation of a new YAML document. In this way several independent data structures can exist either in a stream of text, or in a text file on disk. When parsed each separate YAML document is used as an independent item.

---
Ranking of 1998 home runs:
- Mark McGwire
- Sammy Sosa
- Ken Griffey
---
Team ranking:
- Chicago Cubs
- St Louis Cardinals

How it would look in TOML

+++
"Ranking of 1998 home runs" = ["Mark McGwire", "Sammy Sosa", "Ken Griffey"]
+++
"Team ranking" = ["Chicago Cubs", "St Louis Cardinals"]

The Hugo project uses +++ to denote demarcation for TOML documents. It is a way for them to have a single text file that consists of TOML in the "frontmatter" (document meta-data), and a second "document" that is the body of page content.

Now Hugo does use this in a slightly different way, as their second "document" is not YAML or TOML but a free form text that is body of a blog post or content page.

The purpose, for the Hugo project, is to have a single file on disk that includes the data of the file and meta-data describing the file in one place.

Another benefit would be to allow multiple independent TOML documents to be in a single file on disk, reducing the overall number of files needed to represent data. As well as indicate new TOML documents in a stream of data flowing into a parser.

[pradyunsg]

I think if we want to allow multiple TOML documents to be in a single file/stream, splitting them with +++ is the Most Obvious™ way to do it.

I think there's merit in doing this but not sure about how it feels with the minimal part.

[jongiddy]

The Hugo frontmatter example uses +++ mainly as a type indicator for the frontmatter, to distinguish TOML from YAML or JSON. It seems that the Even More Obvious way of using the familiar --- was not chosen because it does not distinguish between YAML and TOML.

Distinguishing the enclosed format is a task for the enclosing format, so for the Hugo use case, the use of ---, +++, or { to identify enclosed formats should go into the Markdown spec and be handled first by the Markdown parser. That parser can then hand the enclosed document to the appropriate TOML, YAML, or JSON parser for parsing.

It would be useful for these sub-parsers to allow a configurable terminator other than End-Of-File, but the terminator's value is still a property of the enclosing format.

Another problem with using +++ is confusion when looking at diff's, which use +++ as a delimiter. A diff could contain +++, the diff delimiter, (space)+++, a TOML delimiter, ++++, an added TOML delimiter, and -+++, a deleted TOML delimiter.

[pradyunsg]

All of what @jongiddy points out makes me think it might be better to let the application decide how it wants to split multiple TOML documents when processing.

[guai]

For toml to be a drop-in replacement for yaml it requires many more changes, many of which already was declined. For example, toplevel keys to not be mandatoryly on the top of the file, toplevel arrays and so on.
If you ask me, I think all of them should be done.

[pnathan]

This seems complex and beyond the intentionality of TOML as a simple human readable config file.

[homeworkprod]

@jongiddy lists some strong and insightful points. Thanks for that. They made me reconsider my original support of this proposal.

That said, I wonder if it wouldn't still be valuable to formalize such a document delimiter regarding the need (or lack thereof) to escape the delimiter if it appears inside of such a sub-document. But maybe I'm overthinking this.

[pradyunsg]

Coming back around to this now, I've not seen more support for users requesting a single file on disk representing multiple TOML documents.

I think this is something that can be handled at the application level even today and doesn't need to be a part of the spec.

[pradyunsg]

Whoops. I didn't intend to close this but then I also don't think there's much merit in doing this either so, I'm going to let this stay closed.

[LongTengDao]

toplevel keys to not be mandatoryly on the top of the file

What does this mean?

[eksortso]

toplevel keys to not be mandatoryly on the top of the file

What does this mean?

Well, within context, @guai was listing aspects of TOML that they wanted to modify for TOML to be more YAML-like. In this case, they did not want to force the top-level keys to the top of the document. Issues like #456, #565, and others touched upon this particular topic by proposing syntax changes that would allow top-level keys to be specified elsewhere in the document.

Likewise, they mentioned top-level arrays. This came up in #535.

In response to @guai's comment, I'll just say that TOML won't do all of the stuff that YAML does, even if TOML begins to evolve from a configuration format into a general data representation format. We'll never have arbitrary explicit type tags, for instance.

Anyway, these topics can be discussed elsewhere. This issue's particular topic was discussed here and in #583.