Dep: Move to TOML for manifest and lock

[sdboyer]

DECISION IS MADE - we'll be using TOML.

---

At the time dep went public, the committee was generally in agreement that we would prefer to use TOML or YAML over JSON for the manifest and lock files. There are a few reasons for this, but the primary one is that those formats, unlike JSON, allow users to include arbitrary comments. Because the manifest is the locus of dependency decisionmaking for teams, such comments can add a lot of value:

# we pinned this to a rev because of <link to bug>. remember to unpin when that's fixed!

However, in looking through the existing TOML and YAML implementations in Go, we found that none of them actually support preserving arbitrary comments when regenerating the files. dep is constantly rewriting the manifest and lock files; without support for comment preservation, each rewrite would blow away any user comments. So, without this feature, TOML and YAML lose much of their advantage over JSON.

There are other arguments for using TOML or YAML, and it is possible that the we may still elect to use them even without support for comment preservation. However, we are putting a call out - if anyone feels particularly itchy at the thought of using JSON for these files, then please, contribute comment preservation support to an existing implementation, and we'll make the switch! The deadline for this change is the relevant milestone, after which the format and structure of the manifest and lock must be stable.

Alternatively, if we've missed an implementation of TOML or YAML that does support comment preservation, or there's some other crafty way around this problem, please tell us here!

---

UPDATE: we've decided to generally not have the tool update the manifest, which relieves us of the need for a comment-preserving implementation.

[kardianos]

In what instances would users be expected to manually modify any of the files, manifest or lock?

[freeformz]

FWIW: I don't think it should generally be necessary with the exception of adding comments or the possibility that there are some seldom used features that are (at least to start) implemented through directly modifying the manfiest file. That's my 2c anyway.

[jessfraz]

agreed, don't think there _should_ be much reason other than comments to directly edit the manifest, but _if_ there is we should think about if that is the user experience we desire

[adg]

I vote for sticking with JSON.

[sdboyer]

In what instances would users be expected to manually modify any of the files, manifest or lock?

echo what @freeformz and @jessfraz said. it's also possible that, when we start looking at direct integration into the go toolchain, we have to trim the command set down.

[kardianos]

when we start looking at direct integration into the go toolchain, we have to trim the command set down.

@sdboyer Is there a design document for a go tool integration? Are there docs from internal discussions? Are you just hypothesizing?

[jessfraz]

somewhere around here: https://github.com/golang/dep/issues/25#issuecomment-270460908

[sdboyer]

Is there a design document for a go tool integration?

Nope, there are no more design docs hiding anywhere.

Are there docs from internal discussions? Are you just hypothesizing?

I'm basing this on some email chains with @rsc. He set out some desired constraints for how he wants the overall go toolchain to work, and the first thought I had was, "it would be easier to meet these if we weren't trying to provide a command for ever possible manifest change." I shared that thought in an email about two weeks ago, and there's been no further discussion since. That's where we're at.

[kevinburke]

Funny, I wrote about this recently - this is out of left field, but the only Go parsers I found that preserved comments were for XML, HCL and Go source files themselves, and they are about equally difficult to work with. https://kev.inburke.com/kevin/more-comment-preserving-configuration-parsers/

Here's a parser that finds a version number in a Go source file and bumps it: https://github.com/shyp/bump_version. I've also written code to manipulate HCL files and it's a little finicky but the resulting file is recognizable. https://github.com/hashicorp/hcl

[nathany]

There is also JSON5, which does allow comments, though I don't think it's anywhere near a standard.

Though I prefer the readability of TOML > YAML > JSON, another consideration is the licensing and CLA if dep is later incorporated into the go toolchain. As far as I understand, the TOML or YAML libraries would need to be contributed under the Google CLA or rewritten (with comment preservation as well). /cc @willnorris

[kevinburke]

I'll just put this out there: I'm available for hire as a contractor, and would love to work on a CLA-compliant, Go TOML parser that can preserve comments and allow AST manipulation, if anyone's interested. https://burke.services

[andrew]

+1 for a non-executable format for manifest+lockfile, this will allow third party services (like https://libraries.io) to analyse the dependencies without needing to execute arbitrary code download from the internet.

Also YAML is basically a superset of JSON, so you could mix and match, but just because you can doesn't mean you should,

Aside from that, allowing comments to explain _why_ you switched to a particular fork or pinned to a weird tag would be ✨

[kr]

There is a format that supports arbitrary comments, with regeneration preserving the comment positions, is supported by the Go standard library, and has syntax familiar to all Go programmers: Go source code.

Imagine a manifest file (with no .go filename extension) in a subset of Go syntax:

dependencies = {
        "github.com/Masterminds/semver": {
            "branch": "2.x",
        },
        "github.com/Masterminds/vcs": {
            "version": "^1.8.0",
        },
        "github.com/pkg/errors": {
            "version": ">=0.8.0, <1.0.0",
        },
        "github.com/sdboyer/gps": {
            "branch": "master",
        },
}

Something along these lines wouldn't be too hard to put through ParseExpr and get an AST out, which can then be modified and pretty-printed.

Is that too bizarre?

[jessfraz]

Your level of bizarre reminds me of myself :)

On Tue, Jan 24, 2017, 11:33 Keith Rarick [e-mail] wrote:

There is a format that supports arbitrary comments, with regeneration
preserving the comment positions, is supported by the Go standard library,
and has syntax familiar to all Go programmers: Go source code.

Imagine a manifest file (with no .go filename extension) in a subset of
Go syntax:

dependencies = {
"github.com/Masterminds/semver": {
"branch": "2.x",
},
"github.com/Masterminds/vcs": {
"version": "^1.8.0",
},
"github.com/pkg/errors": {
"version": ">=0.8.0, <1.0.0",
},
"github.com/sdboyer/gps": {
"branch": "master",
},
}

Something along these lines wouldn't be too hard to put through ParseExpr
and get an AST out, which can then be modified and pretty-printed.

Is that too bizarre?

—
You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub
https://github.com/golang/dep/issues/119#issuecomment-274912263, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ABYNbITdrboTAIPBES5QJg7-hwW-jZikks5rVlINgaJpZM4LrVlt
.

[pquerna]

FWIW, I had the same issues with preserving comments in config files for a project, and ended up using HCL with some wrapper code for my use case.

[freeformz]

@kr Something similar was discussed early on, here was my attempt at the time (which was a little more aggressive than yours)....

package main

// +build deps

import (
    "deps"
)

var (
    // RootPackagName of the Project
    RootPackageName = "github.com/freeformz/tdeps"

    // IgnoreTags that should be ignored when analyzing deps
    IgnoreTags = []string{"appengine", "test"}

    // Dependencies stated for the package tree starting at RootPackage
    Dependencies = []deps.Dependency{
        {
            Name:    "github.com/Sirupsen/logrus",
            Version: ">= v0.9.0",
        },
        {
            Name:     "github.com/heroku/private",
            Version:  "1.0.0",
            Location: "git-hub-enterprise.herokai.com/privatething",
        },
    }

    // Lock of resolved dependencies which may be in vendor/
    Lock = []deps.Dependency{
        {
            Name:    "github.com/Sirupsen/logrus",
            Version: "v0.9.0",
        },
        {
            Name:    "golang.org/x/net/context",
            Version: "someVer",
        },
        {
            Name:    "golang.org/x/oauth2",
            Version: "someVer",
        },
        {
            Name:    "golang.org/x/oauth2/internal",
            Version: "someVer",
        },
    }
)

I don't remember why we didn't go down this route, possibly because my example wasn't a limited subset, but trying to do way more.

I do think @kr suggestion would be interesting to explore.

[kardianos]

@kr @freeformz I'm skeptical of that route, but if it were to be pursued, make those top level vars part of a type so the fields can be looked up / autocomplete type things can assist.

[mitsuhiko]

I hate toml but I must admit that the choice of Toml was good in Rust land. It diffs very well which makes resolving conflicts much nicer than a json file would allow.

[typeless]

The composability with other existing tools does make JSON desirable, which also makes it easier to be 'scriptable' (with the help of jq for example).

Not sure if that really matters in the end though.

[vasi]

An awful but effective workaround I've used in the past: Just have a key that is specced to be used for comments, and nothing else. Eg:

dependencies:
  github.com/Masterminds/semver:
    comment: "2.x is incompatible for reasons XYZ"
    branch: 1.x

This comment will clearly be preserved even if the structure is programmatically modified.

Caveats:

• Can get awkward if keys are reordered. But usually it's easier to preserve key order than to preserve comments.
• Only one comment is possible per dictionary. Extensions (comment2, commentXYZ…) can be allowed, but that's ugly.
• Comments can't be added to arrays, only dictionaries. Not a big problem in practice. Arrays are typically small, or contain dictionaries—either of which allows comments to be close to where they should be.

[4ad]

@kr I like your suggestion (although I have no interest in Go dependency management) because it would encourage and make it easy for other projects to use the same configuration language.

[kegsay]

I would love to see TOML used instead of JSON or YAML.

Hand-rolling JSON files is incredibly fiddly: I find I always miss a , or a :. The lack of being able to justify your versions (as mentioned by the OP) are infuriating when working with package managers like NPM which use package.json. It does parse easily though.

YAML is a lovely format to write in, but extremely complicated to parse. I think devising a completely compliant YAML implementation is very tricky, and even if dep manages to use one, what about 3rd parties who want to parse these files?

TOML gets the balance right IMHO: easy to parse, easy to write and easy to read. The most popular TOML parser for Golang AFAIK is https://github.com/BurntSushi/toml - so perhaps @BurntSushi or @cespare may have some thoughts on adding comment preservation support? I'd be interested in helping out.

[BurntSushi]

@Kegsay I'd expect comment preserving transformations to be a pretty significant change. I haven't done well with keeping the TOML implementation up to date with the latest spec either. I would personally support a replacement fork (i.e., we retire or move BurntSushi/toml) if you found it useful to start with the existing code.

[jrick]

It may be too late in the design phase to change something like this, but I am completely ok with dropping the requirement of being able to modify the manifest through the tool instead of just hand editing it. Especially if the manifest is in an easily readable and writable format such as TOML. If the manifest is never rewritten by the tool, then there's no need to make it preserve comments.

[kegsay]

@BurntSushi thanks for the quick reply! I feel that your TOML parser is the most "battle-worn" out of the available parsers in that it superficially seems to have the most usage, and therefore has been tested in a large amount of real-world scenarios. For that reason, I'd be reluctant to do a complete rewrite unless the change was sufficiently large that you'd end up doing a rewrite in order to implement comment preservation. Given you wrote the parser, do you believe that we'd need to start from scratch?

[BurntSushi]

Given you wrote the parser, do you believe that we'd need to start from scratch?

It's not just the parser that needs attention. It's also the printer too. You need some way to carry comments through both parsing and commenting. There may be other issues, like preserving the order of keys, but I can't quite remember. I can't really be any more specific than "significant change" as to what's required. Comments were given precisely zero thought other than dropping them at lexing time.

[groob]

Has anyone considered HCL yet? https://github.com/hashicorp/hcl
It's a bit like JSON but supports human readable comments.

[jbrodriguez]

dep is awesome !

FWIW, I think the current manifest syntax is too verbose.

if the comments requirement is dropped, I'd prefer something more minimalistic

Node (package.json)

  "dependencies": {
    "classnames": "2.2.5",
    "font-awesome": "4.6.3",
    "history": "3.0.0",
    "isomorphic-fetch": "2.2.1",
    "react": "15.2.1",
    "react-dom": "15.2.1",
    "react-router": "2.6.0",
    "react-tree-menu": "jbrodriguez/react-tree-menu#683858e",
    "reactorx": "0.6.5"
  }

[cespare]

Hi! I help maintain github.com/BurntSushi/toml and I use TOML in a fair number of projects. I also have a strong interest in Go dependency management (I even wrote a few tools in the past, though these days I just use govendor + the vendor directory).

I support using TOML as the configuration language for this tool, but if dep needs to be able to rewrite the config file while supporting comments it will make things hard.

I have experience writing a parser+formatter (in Go, for another language) that supports comments. Handling them requires coming up with a bunch of heuristics about how to group comments and attach them to AST nodes so that they will be preserved in sensible ways in the output.

You can also see all the logic in the go/* packages (everything gofmt uses) to implement similar heuristics.

The main issue here is that most languages (including TOML and, I think, YAML) don't define how comments fit into the AST structure. (Total off-topic aside: nim is the only language I'm aware of that does.) So you're left with writing a bunch of semi-arbitrary rules and you keep coming across corner cases that don't feel right to the end user (they still crop up in gofmt from time to time).

All that said, I'd like to push back a bit on the premise:

dep is constantly rewriting the manifest and lock files

In other two-file dependency management systems I've used (mainly Ruby's bundler, but I think cargo and others work this way as well), the manifest file is only touched by humans and the lock file is only generated by machine. These properties make this problem go away: the manifest is only ever parsed, not generated, so comments are left untouched, and the lock file doesn't contain comments and isn't altered by hand.

Is it really impossible to restrict this manifest rewriting in dep? Here are two ways to avoid regenerating the manifest:

1. Only ever generate a manifest initially. Future dependency changes must be done by hand.

2. Only ever append new dependencies to the manifest. For example, if the format looks like this:

   someglobalconfig = "blah"
   
   [[dep]]
   path = "github.com/fsnotify/fsnotify"
   branch = "master"
   
   [[dep]]
   # This is a comment.
   path = "golang.org/x/sys/unix"
   branch = "master" # some other comment
   

   Then it is easy to append a new dependency programmatically by adding a few lines while leaving the rest of the file untouched:

   [[dep]]
   path = "github.com/kr/pty"
   branch = "master"
   

I've read the dep help text and played around with it briefly; I apologize if I missed some fundamental design reason why it must be able to fully rewrite the manifest.

Finally, as much as I like TOML, I think that avoiding the need to programmatically rewrite hand-written TOML is important enough that JSON may still be the better choice if manifest-rewriting is a hard requirement. Even just allowing for a single comment string attached to each dependency gets a lot of the value here:

{
    "dependencies": {
        "github.com/fsnotify/fsnotify": {
            "comment": "we pinned this to a rev because of <link to bug>. remember to unpin when that's fixed!",
            "commit": "abc123"
        }
    }
}

Sorry for the wall of text. I will gladly pitch in to help out if TOML changes (or even a new TOML library) would be useful.

[seh]

I agree with @cespare here: the manifest is _my_ artifact; the lock file is derived from it. No tool should touch my manifest without me explicitly requesting that it do so as a convenience.

For most operations Glide works this way. When I decide to invoke an operation that might have it change my manifest, I have to manually restore the comments that it drops (not to mention reordering entries to match my subjective preference of what belongs where). Out of such suffering comes avoidance of those mutative operations in favor of manual editing.

[freeformz]

FWIW: dep only alters the manifest based on commands the user issues as a convenience, aside from dep init anyway which does some discovery of existing deps and writes out the initial manifest file. You could alter it manually and run dep ensure.

My personal opinion is that I'd much rather have the tool do the alteration when I run a command and the work to enact the alteration instead of having to edit a file and then run a command to enact the alteration.

[BurntSushi]

In case it's not clear: I completely defer to @cespare with respect to the fate of BurntSushi/toml.

[kegsay]

The problem with the JSON-style comments of the form:

{
    "dependencies": {
        "github.com/fsnotify/fsnotify": {
            "comment": "we pinned this to a rev because of <link to bug>. remember to unpin when that's fixed!",
            "commit": "abc123"
        }
    }
}

is that it breaks down a lot the moment you want to have any kind of extended prose (you can't have literal new lines), and you need to escape characters (e.g "), making it rather frustrating to put anything more than a quick one-liner. One can argue that is a good thing since it encourages people to only put comments if they really have to, but discouraging documentation isn't a good thing IMO.

[punya]

If we used YAML but with a comment field as opposed to an actual comment, we'd get preservation for free as well as support for multi-line prose

dependencies:
  "github.com/fsnotify/fsnotify":
    comment: >
      we pinned this to a rev because of <link to bug>.
      remember to unpin when that's fixed!
    commit: abc123

The downside is that this will confuse people who are used to the native comment syntax.

[sdboyer]

catching up...

A new/custom format

I think it would be unwise to invent any new format for this purpose, a la @kr's proposal. Comment preservation is nice, but it's not required, and IMO is less important than any one of the following:

1. This tool already involves creating a lot of new things that are essential to the problem. New things are risky. We should not make new things for non-essential problems.
2. It's hard to predict who else might want to read this data for what purpose, but @andrew already showed up with a use case. Creating a custom format would make that harder to do.
3. The information we need to keep in a manifest is easily expressed declaratively. Comment preservation isn't worth sacrificing that, or even the appearance of it, with a [pseudo]-imperative format.

Manifest r/w

Brought up by a few people, but quoting from @cespare:

In other two-file dependency management systems I've used (mainly Ruby's bundler, but I think cargo and others work this way as well), the manifest file is only touched by humans and the lock file is only generated by machine. These properties make this problem go away: the manifest is only ever parsed, not generated, so comments are left untouched, and the lock file doesn't contain comments and isn't altered by hand.

This is also my preferred approach. I even have a cute graphic more or less to this effect:

(humans to the left, machines to the right)

We opted for this approach, though, because the dominant opinion amongst those who showed up was to have the tool handle updating the files. And I can't really argue with @freeformz's point that it's clearly more of a pain to edit a file THEN run a command, rather than simply run a command. If we were to go back to hand-editing the file, though, then this would be mostly solved - init can generate the manifest to start with, and from there on, it's hand-modified.

...with one exception: we also want to nudge users towards semver, and specifically carat range constraints, as a sane default. Without that sane default, users who don't care will have to provide their own default, which I worry could lead us to a project ecosystem with lots of unnecessary version constraint conflicts. So, being able to write new deps to the manifest is pretty key...but as @cespare noted, is probably workable just by appending the information.

So, if we were to drop commands that basically mutate the manifest for you, we might be able to get away without comment preservation.

Comment preservation

Again, @cespare:

The main issue here is that most languages (including TOML and, I think, YAML) don't define how comments fit into the AST structure. [..] So you're left with writing a bunch of semi-arbitrary rules and you keep coming across corner cases that don't feel right to the end user (they still crop up in gofmt from time to time).

On the one hand, I'm loathe to "meh" at problems like this, and I don't have any sense of what the corner cases look like, or how absurd the unexpected outcomes can be. For our needs here, though, perhaps we'd be able to tolerate it? My thinking on that:

1. We won't be encoding any real semantics within comments, so there's no functional need for precision.
2. The only truly intended audience is other contributors who are looking at the file. There isn't even something like godoc; even if we do end up creating a registry, we can just show the manifest file in its entirety. So, if a comment gets misgrouped and moved around, then it's still fine as long as a human is able to sorta figure out what happened.
3. Manifest changes generally tend to be small and come a few at a time. If the machine did something weird, the user (or reviewer) is more likely to notice it and take corrective action than with an oddity buried in some chunky diff of actual code.

[kr]

I agree with @sdboyer's response. (But note that I was not proposing an imperative format.)

[nathany]

@kr If I saw that the manifest looked like Go code, I would expect to be able to use imperative code as I could with Ruby Bundler or Elixir Mix. Then be disappointed when it didn't work, or worse, confuse what the dep tool doesn't allow in the manifest with what Go allows. The familiarity is nice, but probably best to avoid (IMO).

[sdboyer]

@kr glad you agree :) and yeah, sorry, I know you weren't proposing imperative - I was lumping things together for brevity. As @nathany points out, it's much more about the expectation. That the conversation after your comment slid so quickly towards real .go files/syntax is evidence of how strong that expectation would be.

[nathany]

If the manifest is intended to be human-editable or not, the location (#207) and name (#168) could help suggest which is the case, even without reading the documentation.

[nathany]

we also want to nudge users towards semver, and specifically carat range constraints, as a sane default

With a human-editable format, I'd most likely specify constraints similar to the existing ones, either provided by dep init or others working on the project.

If the commands are simple, the CLI wins, but editing a file has the advantage of being self-documenting. I can see the existing constraints and copy-pasta to get things working, rather than memorizing the UI to the CLI. That, and the ability to add/see comments such as "keep at version x.y due to incompatibility with z".

I admit I also like organizing the deps into logical groups with short descriptions of what they do. Of course dep status or similar could probably do that for me.

[nathany]

After more discussion and consideration, my opinion is that the "manifest" (#168) should be a human-editable format (preferably not JSON) that can be generated on init but shouldn't be modified or mucked with by the tool.

That means comments and order is preserved because the file is only ever parsed.

[nathany]

This issue is a big deal for how dep operates in practice.

Is the dep UI primarily a CLI that maintains a machine-readable manifest for you?

Or is the primary UI a human-editable manifest file with a few simple CLI commands that enforce it?

In both cases, the third and currently most important input into dep is the import graph derived from the code it is operating on. I ponder this more in #213.

[peterbourgon]

Great, foundational questions! They deserve answers:

Is the dep UI primarily a CLI that maintains a machine-readable manifest for you?

Yes.

Or is the primary UI a human-editable manifest file with a few simple CLI commands that enforce it?

No.

[nathany]

@peterbourgon Well, that's almost what it is now. But is that what it should be?

Why have two separate .json files not intended for the user to touch?

Using a CLI requires memorizing syntax and consulting help, whereas a file-based UI provides example usage just by having constraints already specified, not to mention comments for documentation right there.

Personally, I think a simpler CLI has advantages. Fewer commands could allow easier integration into the Go toolchain.

If it's a CLI-only UI, then fine, but push further that way and remove the whole notion of a separate manifest that someone might expect to edit. Then a single dep.json should be enough.

[peterbourgon]

There are reasonable technical arguments for every position you've enumerated. We've gone through a lot of deliberation internally, and settled on separate manifest and lock files, and the tool as the primary mechanism of interaction.

For the record, the most important justification for separate files, as I recall, is to have a very clear separation between user declaration of intent (manifest) and computed declaration of reality (lock), especially for git diffs.

It's on me for not documenting that reasoning process sufficiently, and I'll work on that in the coming weeks in a Wiki.

[nathany]

@peterbourgon Having a separate file for the user to express their intent is great.

What I'm really not so sure about is having that be a machine-modified manifest file modified exclusively by CLI commands.

Please see #213 for some example workflows that utilize a human-editable file.

[ukiahsmith]

We can not be sure, with 100% certainty, that a a manifest file, the goal of which is user intent, actually expresses user intent when that file is updated by CLI commands.

It is far easier to accidentally run an unwanted CLI command, or to run a command twice, than it is to manually edit the manifest file.

Having a manifest file untouched by automation means that there is a very clear separation between user declaration of intent and computed declaration of reality.

[peterbourgon]

Your point is taken, but putting manifest files in unexpected or undesired states is just as easy with bad edits as it is with bad commands, especially given the state of config file formats.

[sdboyer]

Big update here: as described in https://github.com/golang/dep/issues/213#issuecomment-278831924, we're going to start moving away from having the CLI UI be the main manipulator of the manifest, and instead move the ongoing experiment that is dep towards hand-editing of the manifest (outside of dep init) and a terser command set.

In other words, the new direction mostly obviates the need for comment preservation :)

[nathany]

Hand-editing the manifest gets around the need for comment preservation while allowing comments, which is great.

Comments are handy for several reasons. One is to explain why a particular user intent is declared. For example, locking "foo/bar" at an older version due to an incompatibility, LICENSE change, etc.

Comments also could be used by dep init to provide some basic documentation or URLs to further information for the manifest format.

While the lock file is perfectly fine as JSON without any comments, I personally think it would be worth exploring alternatives for the manifest file.

[nathany]

Perhaps of note, one of the more popular YAML packages is from Canonical which is under an Apache license. https://github.com/go-yaml/yaml/tree/v2.

Kubernetes, Docker's libcompose, and countless other projects import it. That suggests it's had a fair bit of use (battle tested).

Perhaps it's too early to worry about licensing, but I wonder if @niemeyer, et al. would be interested in contributing a BSD-licensed YAML parser under the Google CLA towards this project/Go?

[nathany]

We can not be sure, with 100% certainty, that a a manifest file, the goal of which is user intent, actually expresses user intent when that file is updated by CLI commands.

💯

Comment-preservation may still be a consideration if the manifest has a gofmt-like tool. For example, HCL can be reformatted via Terraform's terraform fmt. /cc @mitchellh

[nathany]

As a next step, I think it would be worth doing a comparison of various formats that the manifest/lock could use (JSON, YAML, TOML, HCL). I'm willing to take this on.

Possible factors to consider:

• Human readability, comments, what might it look like in use.
• Pretty-printing (fmt) with comment preservation.
• Widespread usage of format, familiarity with the format, simplicity of the format, impact on adoption of dep?
• Available implementations, usage, stability, reliability
• Current license and ease of inclusion in the Go toolchain in the future (internal/ package)
• other?

[omeid]

Comment persistence from my understanding requires some form of AST-like IR which rules out majority of encoding that follow in the paths of encoding/json et al that takes a direct route of data to Go type.
That would mean working with an AST-like representation of Manifest and Lock instead of a simple Go struct, is that acceptable?

The other issue to consider is that all the encodings discussed (other than @kr's Go subset) lacks any concept of comment association, consider this example:

# is this comment releated to semver?
[[dep]]
pkg ="github.com/Masterminds/semver"
# version = "2.x"
# well, what happens to version being comment out?
version = "fix-constraints" # [instructions on when to switch back] but which comment 
#will be removed when version is changed?

# should this comment stay after alt is gone?
alt = "gitlab.internal/3rd-party/semver" #This one?
# How about this one?

[[dep]]
pkg = "github.com/pkg/errors"
version = ">=0.8.0, <1.0.0"
# is this comment about semver or errors?
[[dep]]
pkg = "github.com/sdboyer/gps"
version = ">=0.14.0, <1.0.0"
alt = "gitlab.internal/3rd-party/gps2" #This one?
# is this comment for gps? should it be removed with gps? or with alt?

So one must come up with rules along the lines of Go's CommentMap for any format that is being considered.

I came up with this concerns when considering to write a TOML codec, once these concerns are addressed, I am keen to contribute.

On a side note, considering that Go sytanx solve's this problems, maybe @kr is onto something here.

[nathany]

Based on https://github.com/golang/dep/issues/119#issuecomment-278839601, comment preservation isn't a requirement. It would be nice for a fmt like tool, which may or may not be implemented as part of the dep toolchain (e.g. text editors have pretty printing for JSON, etc.).

I think a *.go derived format was sufficiently debunked already. https://github.com/golang/dep/issues/119#issuecomment-275590619

[sdboyer]

As a next step, I think it would be worth doing a comparison of various formats that the manifest/lock could use (JSON, YAML, TOML, HCL). I'm willing to take this on.

This would be great, but I want to make a decision here and close this issue, soon. So, let's keep that analysis brief and strictly factual?

lacks any concept of comment association, consider this example:

Ah, let's back this up a sec - with the changes to our requirements, we can safely assume that the tool needn't write out the manifest, except on init. That means comment preservation is not currently a need.

On a side note, considering that Go sytanx solve's this problems, maybe @kr is onto something here.

We closed the door on a custom format a ways back. Let's keep it shut, please 😄

[omeid]

My two cents,
comments are really useful, so JSON is out.
YAML is extremely complex, and since there is no yaml codec in standard library, the cost of contributing one is pretty high.
HCL is not that common, and not many people use it.
TOML seems like the winner.

[peterbourgon]

I see more HCL than TOML in my daily life.

[flibustenet]

Comments can be preserved in all formats if we consider it like any parameter:
comment="my comment..."

[willfaught]

Can't the JSON format allow an optional comment attribute that gets parsed and preserved automatically?

Please don't choose TOML. It's weird and verbose and no one knows it. Use YAML instead of TOML.

[jostyee]

@willfaught maybe it just you don't know toml only, plz don't represent everyone except yourself.

[willfaught]

Everything I write only represents myself unless I explicitly say otherwise. That's true...implicitly :)...for everybody.

I know all 3 formats. I hadn't even heard of TOML until a few months ago.

[fatih]

I also think HCL is great. I'm biased of course because I've wrote the initial Go parser and printer for HCL. It's now transfered to the great people of @hashicorp and used for months and is also battle tested in various projects, such as Terraform, Vault, Nomad, etc.

The most I like is that HCL is also JSON compatible. So you can always use a JSON instead of an HCL file as well if you don't like using HCL. It makes it easy to switch formats. It also supports comments and has also other extensions (such as HIL) that further adds powerful scripting features. Because it has a printer it also has commands like hclfm (gofmt equivalent). Of course this is not much required, but it's there if you need it.

[flibustenet]

json is not perfect but it just work now with stdlib. When no other toml/hcl/yaml... parser will be include in the stdlib it seems more reliable to stay with json and don't worry about compatibility promise.
Imagine the situation in some years where we must say that dep files should be read with this version of this parser !
Keep it simple, we should work with what we have instead of looking at what we could have if... I think about the tools but also about the dev. It's enough that we will have a config file and not only the source code, we should not have to learn a new language !

[willfaught]

I'm not sure depending on a non-stdlib package to parse is an issue, since
dep would (hopefully) be built into the Go tool. Even if it's not, go get
would handle the dep seamlessly.

On Tue, Mar 7, 2017 at 8:59 AM Wilk [e-mail] wrote:

json is not perfect but it just work now with stdlib. When no other
toml/hcl/yaml... parser will be include in the stdlib it seems more
reliable to stay with json and don't worry about compatibility promise.
Imagine the situation in some years where we must say that dep files
should be read with this version of this parser !
Keep it simple, we should work with what we have instead of looking at
what we could have if... I think about the tools but also about the dev.
It's enough that we will have a config file and not only the source code,
we should not have to learn a new language !

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/golang/dep/issues/119#issuecomment-284786598, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAD5VgyuxfOqPc_zVfUq3PAsuXpwyjAnks5rjYzogaJpZM4LrVlt
.

[jrick]

@flibustenet

Imagine the situation in some years where we must say that dep files should be read with this version of this parser !

This is precisely the problem that dep solves.

[ukiahsmith]

@flibustenet

Comments can be preserved in all formats if we consider it like any parameter:
comment="my comment..."

I am not wild about overloading data to include comments. I can see issues arising where these comments become required data elements, though as a comment they would not appear so. I believe this could lead to confusion and complexity. An example would be a 3rd party tool that uses the comment field to have meaning when it processes the manifest and lock file.

@flibustenet

json is not perfect but it just work now with stdlib. When no other toml/hcl/yaml... parser will be include in the stdlib it seems more reliable to stay with json and don't worry about compatibility promise.

It is more important to choose the best format, regardless of having a parser in the stdlib or not. In the future, it will be easier to introduce a new parser into the stdlib than it would be to change the format that dep uses.

We have an opportunity to build the best dep tool. There is no reason I can see to choose a sub-optimal format.

[flibustenet]

@jrick we should not forget that the best tool for go must also prevent us to abuse of dependencies. I remember that it was in the goal isn't it ?

[jrick]

That's not something the tool can stop you from doing.

[ukiahsmith]

My personal preference is to use TOML for the manifest and lock file format.

I like the simple, direct, and uncluttered nature of the syntax. This is not a technical preference, rather an aesthetic one.

[lclarkmichalek]

I'm going to try and sum up what people are saying about each format, and offer a brief opinion at the bottom of this (sorry, I realise this is an issue that already has far too many opinions).

JSON

+ stdlib support
+ universally understood
+ simple
- suboptimal ergonomics
- no comments without hacks

Prior art: npm, elm, probably countless others

YAML

- no stdlib support
+ broadly understood
- complex
+ good ergonomics
+ comments
- no comment preserving parser

Prior art: glide

HCL

- no stdlib support
- moderate to niche understanding
+ good ergonomics
+ simple
+ comments
- no comment preserving parser

TOML

- no stdlib support
- niche understanding
- moderately complex
+ good ergonomics
+ comments
- no comment preserving parser

Prior art: cargo

Opinion

If files read by this tool are to be written/modified by users, then ergonomics and widespread understanding should be prioritised. A dependency manager is something someone deals with quite early on when using a new language, and introducing a second DSL to learn isn't great.

[sdboyer]

That's a great summary @lclarkmichalek, thank you.

I think we've probably had all the necessary thoughts on this topic. I'm going to give this till the end of the week for more discussion, then we'll make a decision and run with it. Time's a-wastin!

[tonglil]

This exploration has also occurred for the Crystal Language:

• https://github.com/crystal-lang/shards/issues/25
• https://github.com/crystal-lang/crystal/issues/1357

[nathany]

Excellent summary @lclarkmichalek.

It may be worth splitting it up between the pros/cons for the user vs. for the implementation. Here are a few more points for consideration.

On the user side:

JSON
+ 1st ed. spec http://www.ecma-international.org/publications/files/ECMA-ST/ECMA-404.pdf

YAML
- significant whitespace (the Go language is very specific in choosing not to use significant whitespace)
+ mature, YAML 1.2 spec http://yaml.org/spec/
~ JSON compatible

TOML
~ "Be warned, this spec is still changing a lot." (not sure if this is really true). Spec is 0.4. https://github.com/toml-lang/toml
+ validator tomlv

HCL
- No official specification (yet)
- Not sure if there is an official file extension (e.g. Terraform use *.tf)
+ Pretty printer written in Go hclfmt
~ JSON compatible

Another thing to check: how easy it is to avoid incidental merge conflicts with the format? Can it be used to specify a dependency/constraint in one line? The same line of thinking as Go's trailing commas in multi-line composite literals, which is beneficial for diffs.

On the implementation side:

Clearly JSON and XML :trollface: have the big advantage of already being in the std library. Here are popular Go implementations for the other formats:

YAML

• https://godoc.org/gopkg.in/yaml.v2 (1908 imports) (Canonical) @niemeyer
• Apache License, Version 2.0
• 7595 lines of code (cloc .)

TOML

• https://godoc.org/github.com/BurntSushi/toml (963 imports) @BurntSushi
• WTFPL license
• 4294 lines of code

HCL

• https://godoc.org/github.com/hashicorp/hcl (192 imports) (HashiCorp) @fatih
• Mozilla Public License, version 2.0
• 7012 lines of code

In the future it may become necessary to have the parser as an internal/ package used by the go toolchain. My understanding is doing so will require the implementation to be under the Go license and Google CLA (or at least the later). From that perspective, TOML looks to be winning, but it's quite possible that this could be worked out for the other options too.

[BurntSushi]

"Be warned, this spec is still changing a lot." (not sure if this is really true). Spec is 0.4. https://github.com/toml-lang/toml

To clarify: we are on the cusp of 1.0 and I don't foresee any breaking changes or major additions.

cc @mojombo

[nathany]

This may be of interest: Why does Cargo use toml?

[nathany]

A dependency manager is something someone deals with quite early on when using a new language, and introducing a second DSL to learn isn't great.

@lclarkmichalek In my opinion, new users learning a file format shouldn't be weighted heavily against the selection. Here's why:

• dep is based primarily on the import graph, which means the Manifest could be mostly optional (#233), in which case learning the file format could be deferred until later
• at some point users will need to learn the syntax for specifying constraints, how the dep tool CLI works, etc. some learning is to be expected, whether the file format is already known or not
• using a human-readable format with comments makes it possible for dep init to provide a standard comment block in the Manifest file with usage examples and urls to further instructions

Instead I would put more weight on ergonomics, as you call it.

• comments to say why a constraint is specified is a huge benefit IMO
• if it's at all possible to avoid incidental merge conflicts (unrelated to an intentional change) in the Manifest and lock file, that would be hugely appreciated from the developer happiness perspective
• a pretty printer like gofmt would be icing

[nathany]

To give a feel of ergonomics, compare JSON to TOML:

{
    "dependencies": {
        "github.com/gorilla/mux": {
            "version": "^1.3.0"
        },
        "github.com/st3fan/gocrypto": {
            "branch": "master"
        },
        "golang.org/x/crypto": {
            "branch": "master"
        },
        "golang.org/x/net": {
            "branch": "master"
        }
    }
}

TOML with an inline table (map):

# Comment header

[dependencies]
"github.com/gorilla/mux" = { version = "^1.3.0" } # stay with 1.x
"github.com/st3fan/gocrypto" = { branch = "master" }
"golang.org/x/crypto" = { branch = "master" }
"golang.org/x/net" = { branch = "master" }

Or a simplified form (something like Cargo.toml)

[dependencies]
"github.com/gorilla/mux" = "^1.3.0"
"github.com/st3fan/gocrypto" = "master"
"golang.org/x/crypto" = "master"
"golang.org/x/net" = "master"

By relying on the import graph and good defaults, the constraints could be reduced even further.

[mibk]

Could this simplified JSON manifest be also considered?

{
    "dependencies": {
        "github.com/gorilla/mux": "^1.3.0",
        "github.com/st3fan/gocrypto": "master",
        "golang.org/x/crypto":  "master",
        "golang.org/x/net": "master"
    }
}

[willfaught]

To be fair for all three formats:

TOML:

[dependencies]
"github.com/gorilla/mux" = "^1.3.0"
"github.com/st3fan/gocrypto" = "master"
"golang.org/x/crypto" = "master"
"golang.org/x/net" = "master"

JSON:

{
  "dependencies": {
    "github.com/gorilla/mux": "^1.3.0",
    "github.com/st3fan/gocrypto": "master",
    "golang.org/x/crypto": "master",
    "golang.org/x/net": "master"
  }
}

YAML:

dependencies:
  github.com/gorilla/mux: ^1.3.0
  github.com/st3fan/gocrypto: master
  golang.org/x/crypto: master
  golang.org/x/net: master

HCL:

dependencies {
  github.com/gorilla/mux = ^1.3.0
  github.com/st3fan/gocrypto = master
  golang.org/x/crypto = master
  golang.org/x/net = master
}

YAML is the most concise, readable, and "ergonomic", IMO. Just look at it! So pretty. ;)

Technically, YAML is a superset of JSON, so if you pick YAML, you also get JSON.

[kegsay]

Technically, YAML is a superset of JSON

This is not "technically" true at all. There are a few differences, only one of which is practically relevant in Go-land: tabs. This is a valid JSON file:

{
    "foo": "bar"
}

Because the JSON spec says:

Whitespace can be inserted between any pair of tokens.

This is "technically" NOT valid YAML because the YAML spec explicitly disallows tabs:

To maintain portability, tab characters must not be used in indentation, since different systems treat tabs differently.

That all being said, in practice a lot of YAML parsers will happily accept tabs, so "practically" it is a superset, but "technically"? Nope. It's entirely possible I'm reading the YAML spec incorrectly here, but the explicit rejection of tabs in this section of the spec means some of the YAML parsers I've used do not parse that JSON object as valid YAML.

EDIT: My mistake. It looks like YAML will accept tabs provided they're wrapped with appropriate braces like { or [, which would make JSON files with tabs all valid YAML files. However, the following YAML file won't parse:

foo:
    bar: baz

which is annoying given how most people's editors will be configured to use tabs on go projects.

[flibustenet]

Theses last examples doesn't show more parameters for each dependency.

[nathany]

An HCL example would be nice too.

[ukiahsmith]

From the thread on why Cargo uses TOML.

They are discussing the complex nature of the YAML specification.

But if you're using a subset of YAML, you're not using YAML. You're using a custom, specific format, that's poorly specified.

Then go on to say.

As soon as you start subsetting YAML, you run into the same problem. If you use a tool that assumes one YAML feature is present when writing, but your subset doesn't accept it, then you can't use that tool to produce YAML that Cargo can read.

TOML covers many of the same use cases, but is much simpler. JSON is also simpler, but not a good format for hand-written config files. TOML seems like a decent happy medium; well specified, not too complex, reasonably familiar ...

I agree with this sentiment. I feel a format such as TOML or HCL is less complex, and could prevent unforeseen issues in the future.

I wish HCL had a more formal spec, and a larger community of support around it.

TOML is well supported in both library implementations and editor support. Making it more likely to be familiar to users who are new to the dep tool.

[willfaught]

Kegsay:

From http://www.yaml.org/spec/1.2/spec.html:

The primary objective of this revision is to bring YAML into compliance with JSON as an official subset.

It's technically a superset, at least as of v1.2.

Even if it wasn't, in most cases they are compatible, as you pointed out, so I think the point still stands.

On Thu, Mar 16, 2017 at 7:30 AM Ukiah Smith [e-mail]
wrote:

From the thread on why Cargo uses TOML.

They are discussing the complex nature of the YAML specification.
https://users.rust-lang.org/t/why-does-cargo-use-toml/3577/17

But if you're using a subset of YAML, you're not using YAML. You're using
a custom, specific format, that's poorly specified.

Then go on to say.

As soon as you start subsetting YAML, you run into the same problem. If
you use a tool that assumes one YAML feature is present when writing, but
your subset doesn't accept it, then you can't use that tool to produce YAML
that Cargo can read.

TOML covers many of the same use cases, but is much simpler. JSON is also
simpler, but not a good format for hand-written config files. TOML seems
like a decent happy medium; well specified, not too complex, reasonably
familiar ...

I agree with this sentiment. I feel a format such as TOML or HCL is less
complex, and could prevent unforeseen issues in the future.

I wish HCL had a more formal spec, and a larger community of support
around it.

TOML is well supported in both library implementations
https://github.com/toml-lang/toml#implementations and editor support
https://github.com/toml-lang/toml#editor-support. Making it more likely
to be familiar to users who are new to the dep tool.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/golang/dep/issues/119#issuecomment-287074955, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAD5VhlNsZKLhNzvGPyILX-mXDZjD1sjks5rmUdugaJpZM4LrVlt
.

[azr]

Can't it be a simple golang slice of struct ?

#file vendor/dep.go
package vendor

import "dep"

var Dependencies = dep.List{
    dep.Basic{ Pkg: "golang.org/x/crypto", Branch: "master" },
    dep.Basic{ Pkg: "golang.org/x/net", Tag: "v42" },
    dep.Basic{ Pkg: "golang.org/x/http2", MajorVersion: 30 },
    // etc ...
}

It can be dumped to json or just used as a package by dep

[sdboyer]

@azr no - please see earlier parts of the discussion related to using an actual .go file. Such an approach is not on the table.

[nathany]

@willfaught HCL is also compatible with JSON.

Is that an advantage or disadvantage? It would be great for a tool transitioning from one format to another, but for dep, it can be clean break.

Sure, people who are more familiar with JSON could use that syntax instead, but is that a good thing?

It makes the format more complex. There is more than one way to do it. There is an extra burden for developers needing to parse (in their heads) multiple formats of the same data.

Not to say that is a strike against YAML or HCL. Just not necessarily an advantage for this use case IMO.

[azr]

@sdboyer Ah sorry, I only saw the first "kinda .go" proposal not the second one but I still think it was ruled out a bit fast, and with not such certainty.
And I think it's just being ruled out because its looks too creative/out of the box thinking.
As far as I can tell the arguments against a .go/executable file are:

it's new/different from what already exists, therefore it's dangerous

-> well, go is go and we would not have to JSON

it cannot be parsed by external tools, like https://libraries.io

-> a generated file could be

it's not scriptable at the moment

-> it will be

All additional tool could better the cool tools we already have to dev in go.
All the formatting tools could also play with this file/pkg and would only have to do go.

And to take @freeformz's example it could look not so bad:

...
Dependencies = []dep.Dependency{
        { "github.com/Sirupsen/logrus", ">= v0.9.0" },
        { "github.com/heroku/private", "someVer" },
}

Or even

...
Dependencies = dep.Map{
        "github.com/Sirupsen/logrus": ">= v0.9.0",
        "github.com/heroku/private": "someVer",
}

Every dep entry could also be typed and know how to upgrade a dependency as well. ( like for private repos )

Please, at least make it a probable solution !

[sdboyer]

I'm sorry, but no, it's been definitively ruled out.

[willfaught]

@nathany:

HCL is also compatible with JSON.

Forgot about HCL. Added an HCL example (at least, to the best of my understanding) to my post above. :)

Is that an advantage or disadvantage? It would be great for a tool transitioning from one format to another, but for dep, it can be clean break.

It's an advantage. JSON is great for program reading and writing, but not ideal for humans. YAML is great for humans, but not great for program reading and writing. Supporting both with equivalent formats has the most flexibility, allowing both humans and programs to easily read and write the configuration. Most people get JSON and YAML fundamentals, so I don't see an issue with complexity. If you run across some "crazy" YAML, it's trivial enough to simplify it with a tool.

Sure, people who are more familiar with JSON could use that syntax instead, but is that a good thing?

Alternatively: "Sure, people could use one syntax only, but is that a good thing?" Can't you use that argument against anything?

It makes the format more complex. There is more than one way to do it. There is an extra burden for developers needing to parse (in their heads) multiple formats of the same data.

If, say, YAML were chosen, then it would be just one format—YAML—wouldn't it? Whether it's JSON or YAML or TOML, you're using just one library to read and write it, so I don't see any extra burden for developers there.

I don't really have a strong opinion except against TOML. I really don't like INI-like syntax with the bracketed headers, so it's just a taste thing I guess; but I don't see much downside to picking YAML over JSON.

Regarding preserving comments: it seems to be the case that most YAML libraries today don't preserve comments? If that's the case, then while it might be fine for the dep impl to use a library that does preserve them, it doesn't guarantee that other developers who write tools will use the same library. So those comments are getting lost anyway, whatever dep does. The only way to have useful preserved comments, as I see it, is to either pick a format where most tools support comment preservation, or make the comments part of the data schema. I don't think it's a big deal to support an extra comment attribute on stuff. Most of the time there won't be one anyway. Arguably a comment attribute could be useful for tools anyway.

[nathany]

@willfaught Thanks for adding the HCL example. I was having trouble coming up with what it would look like.

Installing and running https://github.com/fatih/hclfmt reported some errors until I quoted all the keys and values like so:

dependencies {
  "github.com/gorilla/mux"     = "^1.3.0"
  "github.com/st3fan/gocrypto" = "master"
  "golang.org/x/crypto"        = "master"
  "golang.org/x/net"           = "master"
}

As @flibustenet pointed out, we should consider what multiple parameters would look like.

The following is valid HCL. The hclfmt tool will split it across multiple lines, which wouldn't be great for diffs, but maybe hclfmt could be changed.

// comment header

dependencies {
  "github.com/gorilla/mux"     = { version = "^1.3.0" }
  "github.com/st3fan/gocrypto" = { branch = "master" }
  "golang.org/x/crypto"        = { branch = "master" }
  "golang.org/x/net"           = { branch = "master" }
}

Notice that this looks almost identical to the TOML version, except it nests dependencies in a map instead of having the special INI-style format for the same thing (tables vs. inline tables in TOML).

+ It's nice that HCL uses Go-style // comments

[nathany]

YAML has an inline hash so it could look like this:

dependencies:
  github.com/gorilla/mux: { version: ^1.3.0 }
  github.com/st3fan/gocrypto: { branch: master }
  golang.org/x/crypto: { branch: master }
  golang.org/x/net: { branch: master }

[nathany]

@willfaught I agree that YAML has a nice aesthetic. The use of colons for key: value mappings is even reminiscent of composite literals.

Personally, any of HCL, TOML, YAML would be fine with me.
+ comments
+ one line per dependency for diffs and merging (JSON is adverse to extra trailing commas)
+ readable/ergonomics

YAML is the most mature, but there are a few strikes against it too -- in my opinion.

Significant Whitespace

"structure is determined by indentation" http://yaml.org/spec/1.2/spec.html#space/indentation/

I don't personally have a problem with significant whitespace, but I observe that Go intentionally doesn't rely on it.

"Some observers objected to Go's C-like block structure with braces, preferring the use of spaces for indentation, in the style of Python or Haskell. However, we have had extensive experience tracking down build and test failures caused by cross-language builds where a Python snippet embedded in another language, for instance through a SWIG invocation, is subtly and invisibly broken by a change in the indentation of the surrounding code. Our position is therefore that, although spaces for indentation is nice for small programs, it doesn't scale well, and the bigger and more heterogeneous the code base, the more trouble it can cause. It is better to forgo convenience for safety and dependability, so Go has brace-bounded blocks." https://talks.golang.org/2012/splash.article

Even if it turns out to be less of an issue for a dependency tool config file, a consistent stance does have some value IMO.

Complexity

Compare http://yaml.org/spec/1.2/spec.html to https://golang.org/ref/spec. YAML has way more than necessary for such a simple need. I do remember the remote code execution vulnerabilities the Ruby community experienced partially due to YAML. That may not be a problem for the Go parsers, but it was enough for the Rust (Cargo) folks to avoid the format.

[willfaught]

Re: whitespace:

Code is different than config. It's copied, rewritten, transformed, edited,
etc. a lot. I don't think anyone would be working with dep config that way.
Do you foresee there being deep nesting in the config?

Re: complexity:

You don't need to read the full spec to learn YAML, and the YAML dep would
be writing would be pretty simple, I think.

My understanding of the YAML-related Rails security flaw was that Ruby code
was reading YAML values and "eval"-ing code with them. So someone basically
did an injection attack like you would for SQL. The same thing could happen
with reading from JSON.

On Fri, Mar 17, 2017 at 3:48 PM Nathan Youngman [e-mail]
wrote:

@willfaught https://github.com/willfaught I agree that YAML has a nice
aesthetic. The use of colons for key: value mappings is even reminiscent
of composite literals.

Personally, any of HCL, TOML, YAML would be fine with me.

• comments
• one line per dependency for diffs and merging (JSON is adverse to extra
  trailing commas)
• readable/ergonomics

YAML is the most mature, but there are a few strikes against it too -- in
my opinion.
Significant Whitespace

"structure is determined by indentation"
http://yaml.org/spec/1.2/spec.html#space/indentation/

I don't personally have a problem with significant whitespace, but I
observe that Go intentionally doesn't rely on it.

"Some observers objected to Go's C-like block structure with braces,
preferring the use of spaces for indentation, in the style of Python or
Haskell. However, we have had extensive experience tracking down build and
test failures caused by cross-language builds where a Python snippet
embedded in another language, for instance through a SWIG invocation, is
subtly and invisibly broken by a change in the indentation of the
surrounding code. Our position is therefore that, although spaces for
indentation is nice for small programs, it doesn't scale well, and the
bigger and more heterogeneous the code base, the more trouble it can cause.
It is better to forgo convenience for safety and dependability, so Go has
brace-bounded blocks." https://talks.golang.org/2012/splash.article

Even if it turns out to be less of an issue for a dependency tool config
file, a consistent stance does have some value IMO.
Complexity

Compare http://yaml.org/spec/1.2/spec.html to https://golang.org/ref/spec.
YAML has way more than necessary for such a simple need. I do remember the
remote code execution vulnerabilities the Ruby community experienced
partially due to YAML. That may not be a problem for the Go parsers, but it
was enough for the Rust folks to avoid the format.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/golang/dep/issues/119#issuecomment-287489933, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAD5Vh8KY3sJ6PRT9f4Pwn7UAyi9f9wXks5rmw2_gaJpZM4LrVlt
.

[sdboyer]

🔨 🔨 🔨 _(why is there no gavel emoji)_

We're going to go with TOML. 😄

As I think discussion of their various characteristics here has demonstrated, there's no clear winner between YAML, TOML, or HCL in the general case. I say "various characteristics" to emphasize that I'm not talking about personal views (my view: "all formats are terrible and I will hate them at least sometimes - life sucks, buy a helmet").

The one thing that does stick out with TOML is, being _not_ tree-structured, it's possible for us to append constraints to the manifest without rewriting it. That may turn out to be a very important factor in applying sane defaults that help guard us (that is, the entire public Go ecosystem) against nasty exponential growth in solver running time.

This would be a great moment for anyone with time to jump in with a PR that converts us to use TOML for the manifest and lock. For now at least, let's rely on BurntSushi/toml.

🎉 🎉 🎊 🎊

---

EDIT: due to licensing concerns, we're gonna at least start from https://github.com/pelletier/go-toml.

[carolynvs]

If it's okay, I have time this week to jump over to TOML.

[willfaught]

Can't HCL also be non-structured?

foo bar {
baz = 42
}

etc. HCL at least can be both, but TOML can't:

foo bar {
baz = 42
boz {
foo = 101
}
}

You get the best of both worlds.

On Mon, Mar 20, 2017 at 7:45 AM sam boyer [e-mail] wrote:

🔨 🔨 🔨 (why is there no gavel emoji)

We're going to go with TOML. 😄

As I think discussion of their various characteristics here has
demonstrated, there's no clear winner between YAML, TOML, or HCL in the
general case. I say "various characteristics" to emphasize that I'm not
talking about personal views (my view: "all formats are terrible and I will
hate them at least sometimes - life sucks, buy a helmet").

The one thing that does stick out with TOML is, being not
tree-structured, it's possible for us to append constraints to the manifest
without rewriting it. That may turn out to be a very important factor in
applying sane defaults that help guard us (that is, the entire public Go
ecosystem) against nasty exponential growth in solver running time.

This would be a great moment for anyone with time to jump in with a PR
that converts us to use TOML for the manifest and lock. For now at least,
let's rely on BurntSushi/toml.

🎉 🎉 🎊 🎊

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/golang/dep/issues/119#issuecomment-287781062, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAD5VuXyTn_J0QCFGTz-lwe0dVd13K8eks5rnpEGgaJpZM4LrVlt
.

[sdboyer]

@willfaught decision's made - let's move on, please.

[sdboyer]

@carolynvs absolutely!

[sdboyer]

Actually, let's maybe start from https://github.com/pelletier/go-toml to avoid dealing with the WTFPL right now. Or, another one. Honestly, idc, I'm just trying to make suggestions to break logjams before they happen 😄

[carolynvs]

I don't have strong feels on the TOML library, happy to use whichever you prefer. I've never run into the WTFPL though. 😃 I'll give go-toml a try first.

[ghost]

What about HCL? I know this has been mentioned before. Given that it has JSON interop, you could also allow certain JSON producing scripts as an alternative. For example, dep would recognize that the man file has a .go, .rb or .py extension. When those extensions are recognized, dep runs the file and collects the output as JSON etc.

This could be amazing for people who want to do something different with their project or simply want some logic determining what deps they are using.

dep "github.com/gin-gonic/gin" {
    version = "1.1.0"
}

job "build-client" { ...

or

require 'go-dep'

def build do
    puts dep.conf ...

[sdboyer]

@rucuriousyet decision's made - let's move on, please.

[ghost]

Please close the issue then... theres so much content here I didn't notice...

[sdboyer]

Fair enough - I've updated the issue title and OP accordingly.

I'm keeping the issue open, though, because we still need to actually _do_ it 😄

[bep]

For now at least, let's rely on BurntSushi/toml.

TOML is a great choice!

But I want to chime in that we switched from BurntSushi/toml to go-toml in some of the @spf13 projects at one point, after a big license debate. go-toml have had some issues, but the maintainer is pretty responsive, and the bugs seem to have been ironed out.

[sdboyer]

@bep it's good to get some direct feedback on that, thank you!

[tro3]

@carolynvs - I'm happy to help here, if you want to take the lead on the conversion. Just let me know.

[carolynvs]

@tro3 I'm working on marshaling to TOML right now, using go-toml. Once I have something worth looking at, I'll start a WIP PR.

[tro3]

Okay. If there are any subtasks you want me to take on, just let me know.

[domgreen]

Also happy to help on sub tasks 👍

[rsc]

Please just keep the files with the names and formats they already have. They are perfectly fine for the dep experiment. They will probably change in the final go command integration based on that experience, and that's fine. But don't introduce unnecessary churn by changing them before that.

[peterbourgon]

@rsc The opportunity for comment has come and gone. We will move forward with TOML as the means to establish intermediate stability for users.

[kris-nova]

We will move forward with TOML as the means to establish intermediate stability for users.

I think @peterbourgon's words here bring up a good point. Decision is made, let's get to coding.

I also agree with @rsc in the sense that iteration is always possible (if necessary). But we need to start somewhere, and it looks like we are starting with TOML.

[aajtodd]

TOML is a good choice as is hand editing. I find this same workflow in Rust to be rather easy and intuitive. Hand edit manifest, tool generates the lock file. People can build tooling on top of it to automatically write a manifest if they really find that to be an issue.

[mitsuhiko]

May I suggest locking this issue to contributors? I do not want to unsubscribe from the issue because I'm interested in when it is being closed but I am less than thrilled by the continued inbox spam I get from it. And I do understand the irony of me just contributing to the problem with this comment.

[mibk]

@mitsuhiko May I suggest https://tellmewhenitcloses.com/session/new (although I haven't tested it myself...)

[pierrre]

May I suggest locking this issue to contributors? I do not want to unsubscribe from the issue because I'm interested in when it is being closed but I am less than thrilled by the continued inbox spam I get from it. And I do understand the irony of me just contributing to the problem with this comment.

+1 👍 https://github.com/golang/go/wiki/NoMeToo