Go: proposal: cmd/go: notify about newer major versions

Created on 21 Jul 2020  Β·  70Comments  Β·  Source: golang/go

Proposal: Major Module Version Alerts

This is a unification of two separate proposals (#38762 and #38502) that seek to alert users to the fact that a set of new major module versions may be available for inclusion in their project.

Problem

Semantic Import Versioning introduced the concept of a version suffix, which needs to be explicitly specified on your module path for major versions greater than 1. It is easy for users to consume a v0/v1 version of a module by default, even if they would be better served by selecting the most recent major version.

Discoverability is a key issue. The mechanisms for module authors to advertise recent major versions are inconsistent, and can be low-visibility (documentation? README.md?) or highly disruptive (printing deprecation warnings in init, broken builds to force an investigation).

Abstract

We propose a mechanism that notifies users of the latest major version of a module dependency when (a) that dependency is first added to the project; (b) they update to a newer minor or patch version; or (c) they list out dependencies that can be upgraded.

The proposed notification is informational; the user will have to decide the best course of action (stay with the v0/v1 version they fetched, or update their code to use a later version). We understand that automatic upgrades are not generally possible, as new major semantic versions are necessarily incompatible with previous ones.

Proposal

We propose notifying users of new major versions when:

  • They first add a requirement for an old major version of a module
  • They update a requirement for an old major version of a module
  • They list dependencies that can be updated

There are a few ways users add requirements to their modules:

  • By running go get github.com/user/repo
  • By adding a require github.com/user/repo line to their go.mod file manually
  • By adding an import line to a Go source file, and having goimports or gopls editor integration add the import line on save

For the latter two, the module isn't fetched until the go command is invoked within the module.

There are a few ways users update requirements:

  • By running go get [-u] github.com/user/repo
  • By running go get -u ./... from the module root

And there is one way users find out about available updates:

  • By running go list -m -u all

We propose to, in these cases, have the go tool print a note when a more recent major version of the module is available.

Examples

Consider a user fetching github.com/peterbourgon/ff with go get. We propose adding a notification to the output, alerting the user to a new major version:

$ go get github.com/peterbourgon/ff
go: finding github.com/pelletier/go-toml v1.6.0
go: finding gopkg.in/yaml.v2 v2.2.4
go: finding github.com/davecgh/go-spew v1.1.1
go: downloading github.com/peterbourgon/ff v1.7.0
go: extracting github.com/peterbourgon/ff v1.7.0
go: note: a more recent major version is available: github.com/peterbourgon/ff/v3 [v3.1.0]  πŸ‘ˆ

Consider a user listing all of the most recent versions of their dependencies. We propose printing the same note to standard error after any new minor or patch versions:

$ go list -m -u all
example.com/my-module
github.com/BurntSushi/toml v0.3.1
github.com/mitchellh/go-wordwrap v1.0.0
github.com/peterbourgon/ff v1.6.0 [v1.7.0]
go: note: a more recent major version is available: github.com/peterbourgon/ff/v3 [v3.1.0] πŸ‘ˆ

If the requirement is added to the go.mod file manually, the go tool would print the notification when it first fetches the new module, as part of a go build, go test, etc. run.

Integrations

pkg.go.dev

The Go package discovery website at pkg.go.dev shows modules and their versions. However, it obscures successive major versions when they exist, apparently treating major module versions as completely distinct. For example, the landing page for peterbourgon/ff shows v1.7.0 with a "Latest" bubble beside it. The versions tab does list other major versions, but under the heading "Other modules containing this package", which is confusing.
Instead, pkg.go.dev could feature a prominent indicator on the landing page for a v0/v1 module that there are two successive major versions (v2 and v3), to funnel the user toward the latter.

Editor integration (gopls, goimports)

Go text editor integrations typically include a feature that automatically adds import statements to source files based on the mentioned identifiers. Because of Semantic Import Versioning, this also gives those tools the responsibility of choosing the major version of the imported module. In the case where there is no suitable existing requirement in the project’s go.mod file, these editor integrations could alert the user to the availability of newer major module versions. How this works is outside the scope of this proposal.

Proposal Co-authors

  • @adg
  • @peterbourgon
Proposal Proposal-Hold

Most helpful comment

@peterbourgon:

I hope the proposed message doesn't imply, explicitly or implicitly, that the consumer should upgrade their version.

Given the general Go position of not printing warnings - only printing things that are actionable and worth acting on - I personally think a message would imply that users should upgrade.

Perhaps it would make sense to provide the information about a newer major version in go list -m -u and to make it available also in go list -m -u -json. Then other tooling can find out and show it when appropriate.

I tend to agree with the point raised that for other commands, like go get itself, printing the message about v2 without a clear signal from the module author that v1 should no longer be used is overstepping.

All 70 comments

The main problem with β€œon first add” warnings is that users often don't have control over when things are added: it's easy for a tool to accidentally do the thing that triggers the warning (and then not log the warning, because the command completed successfully), and then the warning is buried ~forever. (Compare rust-lang/cargo#5560.)

The editor integration seems more interesting, in that editors have the opportunity to surface low-priority information based on what the user is _viewing_ rather than based on what they are _doing_.

it's easy for a tool to accidentally do the thing that triggers the warning (and then not log the warning, because the command completed successfully), and then the warning is buried ~forever

Do you have a concrete example of this? I just use the go tool to manage my dependencies.

It seems a bit cavalier to dismiss the utility of this proposal just because some people or tools might ignore the output.

Do you have a concrete example of this?

Consider the following scenarios:

  • You run a command that produces a lot of output, such as go test all or go test -v ./..., then go back to editing your code (or go get a coffee or otherwise turn your attention from the go test command). By the time you look at the output (all tests passed!), the warning has scrolled well off the screen.

  • You receive a pull request from an external user who copied the change from a local patch, and didn't feel like rewriting their patch (or even mentioning the newer version) when they mailed you the change.

  • You added an import to a file, and now gopls is suggesting a change to your go.mod file to add the corresponding module (perhaps based on an internal, background go mod tidy run). You add it.

In all of these cases, the warning appeared at some point but was either ignored or missed entirely.

If it is important to surface out-of-date dependencies, I don't think a one-time warning is a good way to do that because it's so easy to miss the one time it appears. (Or, to put it another way: if it's unimportant enough that we shouldn't worry about the user missing the warning, then it's probably not important enough to emit the warning in the first place.)

CC @jayconrod @matloob

These proposals aren't intended to be perfect or exhaustive solutions to the complex problems they address β€” they're intended to be (substantial) improvements to the status quo. It is of course possible that a user could add a new module dependency to their project and then run the go tool in such a way that it would be easy to miss the messages we're proposing to add. YMMV, but I think it is much more likely, and far more common, that users add dependencies deliberately, and do pay attention to the results of the fetch.

YMMV, but I think it is much more likely, and far more common, that users add dependencies deliberately, and do pay attention to the results of the fetch.

I commented over in https://github.com/golang/go/issues/40357#issuecomment-662872827. I suspect in these days of gopls that the goimports-esque workflow is actually the most common.

Hence:

The editor integration seems more interesting, in that editors have the opportunity to surface low-priority information based on what the user is viewing rather than based on what they are doing.

is I think the best way to surface this information.

It seems a bit cavalier to dismiss the utility of this proposal just because some people or tools might ignore the output

@adg - I don't think this is a fair or particularly charitable interpretation of @bcmills' response. @bcmills made an observation about the problems associated with β€œon first add” warnings, cited an example, and then went on to _agree_ with the proposal in the context of surfacing this via editor integrations. I don't think I would characterise that as "dismiss(ing) the utility of this proposal".

As some context, the golang-tools group has been discussing problems/solutions such as this since GopherCon 2018 (the group was largely established in the wake of the vgo release). I might well be biased, but based on previous experience the monthly calls are a good opportunity for discussion of these problems/solutions/options in the pre-proposal (design) phase.

This overlaps a bit with #40357, but I assume if a module author explicitly deprecates a major version with a comment in go.mod, then the go command wouldn't also show this notice.

I think some signal from the module author should be required to show a notice though, for two reasons:

  1. If a new major version is available, it doesn't necessarily mean the old major version is deprecated. Even if no new features are added, an old major version may still receive bug fixes and security updates. Users might still depend on features in the old version that were removed. Basically, migrating to a new major version isn't necessarily the right choice for users. While the messages proposed here don't suggest an action, I feel like they imply that.
  2. If we're telling users about a new major version, we should provide an action they can take to migrate. Because major versions are incompatible changes, there's nothing we can do automatically, yet. It would be better for the author to provide migration instructions in a deprecation notice. Or ideally, if migrations were automated (#32816), we could tell people what command to run.

I think some signal from the module author should be required to show a notice

The problem we're trying to solve is package consumers who intend to take the latest major version of a module, but select an old version due to unfamiliarity with the nuances of SIV. Making package authors opt-in to this notification wold subvert the intent and impact of the proposal.

If a new major version is available, it doesn't necessarily mean the old major version is deprecated.

The notification that a new major version is available is in no way a signal of deprecation of the current version. There is another proposal for authors who want to opt in to that stronger messaging.

If we're telling users about a new major version, we should provide an action they can take to migrate.

We initially had the notification include a command to make the upgrade, but went with this version in the end. Happy to switch back if that's the consensus.

If a new major version is available, it doesn't necessarily mean the old major version is deprecated.

The problem is that they look like warnings and will create more and more noise as a project gets older. Instead of putting it on a separate line, it could just be combined with the existing output.

$ go list -m -u all
example.com/my-module
github.com/BurntSushi/toml v0.3.1
github.com/mitchellh/go-wordwrap v1.0.0
github.com/peterbourgon/ff v1.6.0 [v1.7.0] [latest v3.1.0]

The problem is that they look like warnings and will create more and more noise as a project gets older.

We propose only a single line of output per dependency, specifically with a "note" or "notice" (read: not "warning") prefix, output only the first time the dependency is added to a project, and identifying only the most recent version of that dependency. I don't think this looks like a warning, nor does it create more noise over time.

_edit_: Also, to me, putting the notification inline implies some level of compatibility between major versions that feels misleading.

Instead of putting it on a separate line, it could just be combined with the existing output.

By itself, this would be insufficiently obvious to solve the problem we're trying to solve with this proposal.

nor does it create more noise over time.

Over time, as new major versions of your dependencies are released, more notices are printed.

Over time, as new major versions of your dependencies are released, more notices are printed.

No, I don't think so. At least, the proposal doesn't intend so. Notices are only printed when a dependency is first added to a project at an older major version, or upgraded within an older major version tree. In other words, the trigger is not "a new major version of one of your dependencies is now available" but rather "you have added or upgraded a dependency to your project on a major version which is not the latest". And in all cases the notice only includes the most recent major version, not all major versions greater than the currently pinned major version.

@bcmills wrote:

The main problem with β€œon first add” warnings is that users often don't have control over when things are added

The "on first add" part is just one part of this proposal. We also suggest printing the message when the user is looking to upgrade their module dependencies with go list -m -u all. That seems like an opportune time to present such a message. Do you agree?

@myitcv wrote:

I don't think this is a fair or particularly charitable interpretation of @bcmills' response. @bcmills made an observation about the problems associated with β€œon first add” warnings, cited an example, and then went on to agree with the proposal in the context of surfacing this via editor integrations. I don't think I would characterise that as "dismiss(ing) the utility of this proposal".

I apologise for my reaction, as opposed to a response. As someone who doesn't use the various editor integrations, I read @bcmills's response as a dismissal of my workflow. I still read it that way, but I recognise that the onus is on me to explain myself better here. Mea culpa.

the golang-tools group has been discussing problems/solutions

A little off topic in this thread, but: I would like to be involved but 15:30 UTC is 01:30 in my time zone, and I am ~never awake then. Real time chat does not work for me either; I struggle enough to stay on top of my asynchronous obligations as it is. I appreciate that there is some focused discussion on these issues, though. I will try to find some time to look through the minutes from previous meetings.

@icholy wrote:

Instead of putting it on a separate line, it could just be combined with the existing output.

@peterbourgon wrote:

By itself, this would be insufficiently obvious to solve the problem we're trying to solve with this proposal.

FWIW, I am not opposed to this suggestion. If the user is looking to upgrade their modules with go list -m -u all, then they are likely already scrutinising each line to see whether they should be ugprading.

I would make a small change to @icholy's suggestion, which is to include the full module path of the new major version, as it is different to the one printed on the line. So instead of this

github.com/peterbourgon/ff v1.6.0 [v1.7.0] [latest v3.1.0]

we should print something like

github.com/peterbourgon/ff v1.6.0 [v1.7.0] latest major version: github.com/peterbourgon/ff/v3 v3.1.0

But I do note that at that point things become crowded, and perhaps the two-line version in the OP is preferable.

We also suggest printing the message when the user is looking to upgrade their module dependencies with go list -m -u all. That seems like an opportune time to present such a message. Do you agree?

I think the related proposal #40357 is probably a good idea.

However, especially given that proposal, I suspect that the log message from _this_ proposal would be too noisy, for the reasons described by @jayconrod in https://github.com/golang/go/issues/40323#issuecomment-663201650. I suspect that users will generally ignore the log message if it is not accompanied by instructions (or a tool!) to fix incompatible call sites, and even if they do know how to make the change it's not obviously worth the churn if the major version they are already using is itself still supported.

On balance if we had to choose one proposal or the other I'd go with #40357, but I do think both proposal serve different purposes, and would work nicely together.

Updating between major versions is almost always going to the user to read some documentation and write some code. Whether they want to make the change is a judgment that only the consumer can make. Yes, we can't necessarily provide an easy upgrade path, but is that a reason not to offer them the information?

If the overwhelming concern is noise, perhaps we could try putting together a functional prototype of the command and see just how noisy it is on typical projects. IMO, such information isn't "noise" when I'm specifically asking about updates. But maybe it would become overwhelming? My gut says no, but I can't say for sure without trying it out.

gopls is great, but it should be possible to query this info without using third party tooling.

edit: is the intent to discourage publishing new major versions? (Not being facetious).

@bcmills

I suspect that users will generally ignore the log message if it is not accompanied by instructions (or a tool!) to fix incompatible call sites,

One, this proposal is a response to the specific condition described in the problem statement, which is module consumers _inadvertently_ selecting an old, typically v0/v1, major version of a module, when they _intended_ to select the latest major version. This has happened to me, and to colleagues and peers, enough times that I was motivated to brainstorm, draft, iterate, submit, and re-submit this proposal with Andrew and Zach. Your critique seems to dismiss this initial condition as either invalid or so infrequent as to not be worth addressing. Is that your intent? Do you think this doesn't happen?

Two, the message is not printed only after the consumer has written code against the older major version which would need to be changed, it is also printed when the module consumer first imports the module. It's at that time that the proposal delivers the most value, in my opinion. Rather than wasting time integrating against an old version, the consumer can realize what happened and express their intent correctly.

Three, I personally have no expectation that module authors provide explicit upgrade instructions between major versions, and certainly not that there is any kind of tooling to do so automatically. Actually I've never heard this desire expressed by anyone I've ever interacted with, until now. Is this something that's common at Google? In any case, I don't think the lack of this theoretical tooling should impact the judgment of this proposal.

and even if they do know how to make the change it's not obviously worth the churn if the major version they are already using is itself still supported.

Is github.com/google/go-github v10.0.0 still supported, even though the latest version is currently v32.1.0? Is your position that someone who had inadvertently selected this ancient version would see the message suggested by this proposal as delivering net negative value?

@myitcv @adg

the golang-tools group has been discussing problems/solutions

A little off topic in this thread, but: I would like to be involved but 15:30 UTC is 01:30 in my time zone, and I am ~never awake then. Real time chat does not work for me either; I struggle enough to stay on top of my asynchronous obligations as it is. I appreciate that there is some focused discussion on these issues, though. I will try to find some time to look through the minutes from previous meetings.

For the record, I intend to join the next meeting/s. I'll keep an eye on the wiki page for the next scheduled date, but please poke me in Slack when it's coming up, if you don't mind.

If the overwhelming concern is noise, perhaps we could try putting together a functional prototype of the command and see just how noisy it is on typical projects.

I generally love this idea. We don't need to speculate as to how noise when we can directly observe (and maybe have others here try this).

I personally have no expectation that module authors provide explicit upgrade instructions between major versions, and certainly not that there is any kind of tooling to do so automatically.

Really big plus one here. A major version upgrade, to me, means that an author will spend time understanding what about the library has changed in addition to how to upgrade their code to support it. Whether they then perform the upgrade themselves or there is some automation to do perform it doesn't prevent the need for that initial understanding.

It sounds like the main part of the disagreement here is about whether the module author should be able to keep this from happening. As written, this proposal does not let a module author who is perfectly happy maintaining v1 and v2 indefinitely stop the (implicit) "you should update to v2" messages every time a user starts using v1. Especially if v2 is not as capable as v1, that could be quite annoying for users, who will then complain to the author.

That is, this proposal forces a particular usage of module versions that is not strictly required to date.

In contrast, #40357 puts control over whether users are told about the new version in the module author's hands.

Do I have that right?

@rsc

As written, this proposal does not let a module author who is perfectly happy maintaining v1 and v2 indefinitely stop the (implicit) "you should update to v2" messages every time a user starts using v1. Especially if v2 is not as capable as v1, that could be quite annoying for users, who will then complain to the author.

That is, this proposal forces a particular usage of module versions that is not strictly required to date.

I hope the proposed message doesn't imply, explicitly or implicitly, that the consumer should upgrade their version. I certainly hope the proposal isn't read as _forcing_ an interpretation of module versioning. If there is a way to lighten the message to avoid this interpretation, I'd be happy to change it.

(It does seem both obvious and uncontroversial to me that, absent extenuating circumstances, a larger/newer major version of a software artifact should be preferred to a smaller/older one β€” but that's a philosophical discussion that we don't need to have here.)

It sounds like the main part of the disagreement here is about whether the module author should be able to keep this from happening.

I think it's important that this message isn't opt-in, but I'd be fine with it being opt-out.

this proposal is a response to … module consumers _inadvertently_ selecting an old, typically v0/v1, major version of a module, when they _intended_ to select the latest major version. … Your critique seems to dismiss this initial condition as either invalid or so infrequent as to not be worth addressing. Is that your intent? Do you think this doesn't happen?

I think it does happen, but I think the appropriate point to intervene is usually earlier in the workflow. If a user has written an import statement, they got the idea to write that import from somewhere β€” a website (maybe pkg.go.dev or a code snippet from someone's blog), or a tool (perhaps goimports or gopls), or a snippet copied from some existing project.

If they got the import path from pkg.go.dev and didn't notice the version skew there, that's #37765 or perhaps #36969.

If they got it from goimports or gopls, then there may be a heuristic in those to fix. (If so, please file a separate issue.)

So that leaves a code snippet copied from some existing project, or perhaps a blog or tutorial. But in that case, why would you infer that they intended to use the latest version, rather than the version that the rest of their code snippet was written against? The rest of the code snippet is likely to not even work with a different major version.

Personally, my workflow is a) I find the github (or whatever) page of the package I'm interested in, b) I type go get github.com/foo/bar (by hand) into my shell and c) I use bar.Baz() in my code and let goimports add the import statement.

So personally, I'd benefit from a warning on go get. Arguably, my workflow is bad and I am trying to be better about remembering major versions and type a more specific module path into my shell. But notably, I don't copy it from anywhere - I just type the name I think is correct.

Three, I personally have no expectation that module authors provide explicit upgrade instructions between major versions, and certainly not that there is any kind of tooling to do so automatically. Actually I've never heard this desire expressed by anyone I've ever interacted with, until now. Is this something that's common at Google? In any case, I don't think the lack of this theoretical tooling should impact the judgment of this proposal.

See #32816 and https://abseil.io/docs/cpp/tools/api-upgrades.

@bcmills

If a user has written an import statement, they got the idea to write that import from somewhere β€” a website (maybe pkg.go.dev or a code snippet from someone's blog), or a tool (perhaps goimports or gopls), or a snippet copied from some existing project.

There are infinitely many ways that someone may know and express the identifier (e.g.) github.com/sirupsen/logrus as something they want to add to their new microservice, including just typing it directly from memory due to frequent exposure. All of those roads converge at the point where the dependency is first added to the project, and only there can a notification like this be consistently applied.

@Merovius wrote:

Personally, my workflow is a) I find the github (or whatever) page of the package I'm interested in, b) I type go get github.com/foo/bar (by hand) into my shell and c) I use bar.Baz() in my code and let goimports add the import statement.

I often do this too. With GitHub-hosted repos it is particularly easy to miss new major versions, since GH doesn't do a great job of surfacing branches and tags (they're hidden behind a drop down menu).

@peterbourgon:

I hope the proposed message doesn't imply, explicitly or implicitly, that the consumer should upgrade their version.

Given the general Go position of not printing warnings - only printing things that are actionable and worth acting on - I personally think a message would imply that users should upgrade.

Perhaps it would make sense to provide the information about a newer major version in go list -m -u and to make it available also in go list -m -u -json. Then other tooling can find out and show it when appropriate.

I tend to agree with the point raised that for other commands, like go get itself, printing the message about v2 without a clear signal from the module author that v1 should no longer be used is overstepping.

@rsc

I tend to agree with the point raised that for other commands, like go get itself, printing the message about v2 without a clear signal from the module author that v1 should no longer be used is overstepping.

This implies that all major versions of a software artifact are, by default, equally viable for consumers, and equally supported by authors. I'm not aware of any software project for which this is the case, much less that this is or should be the default position for tooling. Essentially every software project I'm aware of directs essentially all of their development effort to the latest major version, prefers their users to use that latest version, and "supports" only the most recently older major versions, and only in the weakest possible sense of the word.

~The obvious example here is the Go project itself. Major versions are "supported" only back a couple of releases, and even then, "support" means only that they will receive security and compatibility fixes, not that they will continue to receive valuable new features, or even non-critical bug fixes. Do we not want new users to be directed to the latest version? Would it be "overstepping" if the mechanism to download Go releases notified users fetching Go 1.11 that the latest version was actually Go 1.15? What if that mechanism would, by default, download Go 1.0, unless it was explicitly parameterized with the latest version (which was undiscoverable)?~ _edit_: It's been pointed out to me that Go is technically still on major version 1. Fair.

_edit_: I admit, I'm deeply confused by this resistance. As a software author, I can't think of any circumstance in which I wouldn't want someone fetching an older major version of my artifact to see a message telling them that a more recent version exists. As a software consumer, I can't think of any circumstance where I would be anything but thankful to see that message. I'll ask you what I asked @bcmills β€” do you think the problem described in the proposal text doesn't actually happen? Or maybe, if it happens, it's not actually a problem? Or maybe, that the harm caused doesn't justify the proposed change?

@rsc wrote:

Given the general Go position of not printing warnings - only printing things that are actionable and worth acting on - I personally think a message would imply that users should upgrade.

I remember that being the case a long time ago, but it appears that the module support added a lot more (welcome) output to the command when fetching dependencies (the old go get was mysteriously quiet, and go get -v too verbose). In the proposed output, we suggest adding one line to a series of informational lines:

$ go get github.com/peterbourgon/ff
go: finding github.com/pelletier/go-toml v1.6.0
go: finding gopkg.in/yaml.v2 v2.2.4
go: finding github.com/davecgh/go-spew v1.1.1
go: downloading github.com/peterbourgon/ff v1.7.0
go: extracting github.com/peterbourgon/ff v1.7.0
go: note: a more recent major version is available: github.com/peterbourgon/ff/v3 [v3.1.0]  πŸ‘ˆ

The other lines are not 'actionable' IMO. Do you agree?

@rsc wrote:

Perhaps it would make sense to provide the information about a newer major version in go list -m -u and to make it available also in go list -m -u -json. Then other tooling can find out and show it when appropriate.

Strong +1 from me on that.

Perhaps it would make sense to provide the information about a newer major version in go list -m -u and to make it available also in go list -m -u -json. Then other tooling can find out and show it when appropriate.

The index/proxy does not currently support listing other major versions of a module. That would be the first step to enable third party tooling.

@icholy, we know what form a newer major version would take if it _did_ exist, so (assuming no gaps in the major-version sequence) we could detect the presence of the immediate next version with an O(1) check, and find the highest major version in O(N). I think it's reasonable for us to miss the notification if authors skip straight from vN to vN+2. (Don't do that!)

We would need to be a little careful to check for both example.com/vN and example.com vN.0.0+incompatible, but it's not fundamentally a problem.

@bcmills Is your O(N) referring to http requests? That would be 31 requests to find the latest version for github.com/google/go-github

You can do it in O(log(N)) as well, if that's needed, by first determining an upper bound with consecutive doublings and then doing a binary search. Though it might also be reasonable to add a way to list major versions to modules (though the proxy might not always know about them, I think?)

@icholy, yes.

For github.com/google/go-github in particular, I would expect that at some point the maintainers will either decide to stop breaking user code ~8x per year, or decide to go back to the v0.*.* line as an explicit admission that the API is not actually stable. (Since that module appears appear to lack a v1.*.* release, that would likely provide a pretty reasonable user experience; see https://golang.org/doc/go1.14#incompatible-versions.)

For github.com/google/go-github in particular, I would expect that at some point the maintainers will either decide to stop breaking user code ~8x per year,

I don't quite understand this comment. From my perspective that repo isn't breaking user code at all, as anyone using it will have pinned to a version. On the contrary, from my perspective they're using SemVer exactly as it was intended: the major version is just an integer that gets incremented when a new version breaks backwards compatibility, and nothing more.

or decide to go back to the v0.*.* line as an explicit admission that the API is not actually stable.

Stability is not a boolean property which software artifacts have or don't have, it's I guess a description of the rate of change of major versions. More stability is not always better, less stability is not always worse β€” it all depends on the artifact, its users, its context, and so on. From my perspective there's no reason for this project to "go back to v0.x.x" because they're doing things exactly correctly right now.

I think the discussion about the merits of SemVer and appropriate major-version-bump cadence is off-topic here.
Except maybe, I guess, insofar as I would be pretty annoyed to get constantly nagged that I'm supposed to chase a module like that one - so arguably, this notification isn't a great idea after all.

From my perspective that repo isn't breaking user code at all, as anyone using it will have pinned to a version.

I would be very surprised if a module on API v32 is still supporting a significant number of trailing major versions, or vetting those versions for bugs or vulnerabilities. So while it may no longer be _actively_ breaking user code, it is de facto still breaking users' ability to pull in fixes for the dependencies of their existing code.

Change https://golang.org/cl/250797 mentions this issue: internal: create GetLatestMajorVersion function in DataSource

Side note, but speaking as someone who just updated from v12 to v32 of go-github to get access to a new GitHub API version, they are _definitely_ breaking user code and forcing work on users. I've been meaning to raise that with them.

@adg, I agree that the status prints in go get are not actionable. They are there to explain "what is going on and why is this command so slow". It's less than ideal and I'd rather not add to it by default. If anything, switching to not doing any network access except during "go get" (#40728) should make those happen less often and be less necessary. But a status print is different from a "hey look at this!" print. It doesn't imply that you should do anything the way that printing about a v2 does.

This discussion is still ongoing but it doesn't seem to be converging on a consensus that we should adopt the proposal.

The back and forth about minor versions is interesting, though. What about printing advice/warnings about there being much newer _minor_ versions? Should we do that? If not, why not? If so, why? Figuring that out might help us understand major versions as well. (I'm not arguing for any particular outcome on that question.)

@rsc

Side note, but speaking as someone who just updated from v12 to v32 of go-github to get access to a new GitHub API version, they are definitely breaking user code and forcing work on users. I've been meaning to raise that with them.

By updating a dependency from one major version to another, you're explicitly opting-in to changes that aren't backwards compatible. Try as I might, I can't understand how this is "breaking user code" in any meaningful sense. Can you explain your perspective a bit more? (I hate to keep harping on this point, and please tell me to buzz off if it's too much of a derail, but I feel like it's essential to the conversation...)

@peterbourgon my two cent is that semantic versioning should not be used as an excuse to break user code.

If you as a maintainer break your clients 32 times in a few years because you feel that it's safe to do it thanks to semantic versioning, then I don't want to be your user because you are not a maintainer who values stability and consistency; you're probably too much concerned about playing with your own library and doing refactorings for your own sake rather than providing a value to somebody else. That's ok, but I don't want to depend on your playground.

In fact, I was against introducing semantic versioning in Go modules: I felt the risk of the community shifting to a break-often model would have been too high. In my opinion it would have been enough to let a library simply fork to a new URL when they absolutely needed to break compatibility, once every few years. In fact, on the other hand, the absence of semantic versioning discourages library authors from breaking APIs, which is a very good thing in my opinion. Go itself (all its standard library and builtins) has been preserving v1 compatibility for more than 10 years, I don't see how a library wrapping GitHub API feels entitled to break their users 32 times. The only answer to that is that they think it's "normal" to do it "because semantic versioning allows me to do it". That's a bug, not a feature, in my opinion.

From your other comments in this issue, it seems like you feel it's perfectly acceptable to break an API because semantic versioning allows you to do it. I don't want to misrepresent your thoughts, this is just what I got. I guess we live in different worlds with different drivers, and I think this is the difference you're probably not appreciating.

I don't see how a library wrapping GitHub API feels entitled to break their users 32 times.

It simply isn't true that publishing a new major version of your software that isn't API compatible with older major versions "breaks users". The definition of "break" which allows that sentence to be true is so broad that it loses meaning. Breaking users would require the software artifact to make those non-API-compatible changes under the same major version, and thereby violate the semantics of semantic versioning. I don't know how to put it any plainer than that. And I certainly don't feel like I'm projecting my opinion here β€” I view this as a descriptive statement about software and software versioning as it exists today. If we can't agree on foundational semantics, then I'm not sure how to advance the conversation.

What about printing advice/warnings about there being much newer minor versions? Should we do that? If not, why not? If so, why?

This is interesting, actually. I think clearly not, because, unlike with major versions, there are already mechanisms to both identify newer minor (and patch) versions of an artifact, and to upgrade to them automatically. But the meta point I find most interesting is that you bring them up in the first place. I certainly see major and minor and patch as fundamentally the same thing: expressing different degrees of change in what is ultimately the same entity. But isn't it modules' position that major is fundamentally different than minor and patch?

It simply isn't true that publishing a new major version of your software that isn't API compatible with older major versions "breaks users". The definition of "break" which allows that sentence to be true is so broad that it loses meaning.

Then let's choose a different term. Publishing a new major version, without also maintaining ongoing support for existing major versions, _drops support for_ existing users. And if those users want to stay on supported versions of their dependencies, they must make essentially the same manual changes that they would need to make for a breaking API change.

(In other words: it is only substantially different from β€œbreaking” users if you believe that users do not want to stay on supported versions of their dependencies.)

Appropriate usage of semver aside, I think that @rsc made a good point when he said that no consensus is being reached. However, it does seem like there's some agreement that go list should be able to show newer available major versions.

It simply isn't true that publishing a new major version of your software that isn't API compatible with older major versions "breaks users". The definition of "break" which allows that sentence to be true is so broad that it loses meaning.

Then let's choose a different term. Publishing a new major version, without also maintaining ongoing support for existing major versions, _drops support for_ existing users. And if those users want to stay on supported versions of their dependencies, they must make essentially the same manual changes that they would need to make for a breaking API change.

I gave a look at go-github Release Page. The now-current latest release is v32.1.0 which does some non-breaking changes like adding the new field twitter_username to the User object. They did this fix only in the v32 major version, and they did not bother to backport this change to all the previous 31 major versions, releasing minor versions for all of them. If I am on any major version which is not v32, I'm unable to access the twitter_username field in the User object. I manually patched v3.0.0 to add TwitterUsername to User and guess what, it works. So there's no real technical reason why that change was not applied to all 33 major versions rather that just the last one, if not an explicit maintainer choice of not supporting previous major versions. So I think we can agree that go-github maintainers only support the latest major version and drops support for all previous major version as soon as a new one is released (multiple times per year).

If I were a user of that library, I would be forced to upgrade to v32 to access this new field, which in turns breaks my code. This is what Russ also mentioned: he needed to access a new GitHub API and was forced to upgrade go-github major version which in turns broke his code. The same would apply if we were discussing a bug-fix instead of a new feature: since the previous major versions are not maintained, having the bugfix would require me to upgrade to the last major version (or at least one new enough to contain the bugfix), breaking my code in the process.

Everything the maintainers are doing is technically well-defined and they correctly reported what they did through semantic versioning. On the other hand, I would say their behavior is hostile to users of their library, and I would be very sad if this became common in the Go community. I would never be willing of being a user of a library which requires me to upgrade to API-incompatible major versions (and thus break my code) multiple times a year to access new features and new bugfixes.

So in the end, I think this proposal boils down to this:

  • Scenario A: If you feel like it's normal and acceptable that a library maintainer releases a new major version multiple times a year, while dropping support for previous major versions at the same time, then I think this proposal makes sense because users must be aware as early as possible that they need to upgrade to a major version, so that they can plan accordingly to adjust their code for the API-incompatible changes that the library decided to do (rather than finding out when they're in the rush of fixing something and find out that they also need to upgrade the major version of a dependency to get their work done).
  • Scenario B: If you instead feel that it's normal for a library to stay on a major version for a long time (eg: years) and possibly after a v2 is eventually released, maintain v1 for even more years to help the users, then this proposal should be rejected, because it adds a console message that is not immediately actionable and is probably even against the wills of the library maintainers. It would be much better to let the maintainer add a deprecation notice to specific unsupported major versions when they eventually drops support of them (like another proposal is already discussing).

Since I'm in team B, my vote is to reject this proposal. Accepting this proposal would be an implicit hint that scenario A is acceptable and supported, if not even encouraged.

@bcmills

Then let's choose a different term. Publishing a new major version, without also maintaining ongoing support for existing major versions, _drops support_ for existing users.

Perfect! This sounds appropriate to me πŸ‘

(In other words: it is only substantially different from β€œbreaking” users if you believe that users do not want to stay on supported versions of their dependencies.)

That a given version of a software artifact has a finite support lifetime is well-understood by all stakeholders. Longer support lifetimes are not strictly superior to shorter ones β€” that calculation depends on a number of things that vary from project to project.

@rasky

Team A, Team B

The cost and benefit of different major version release cadences is not the same for all software projects. What is appropriate or optimal for one project, can be inappropriate or sub-optimal for another. Tooling should not bias for one class of project over others in this regard.

However, it does seem like there's some agreement that go list should be able to show newer available major versions.

I don't agree with this. The messages are repeated and not actionable. If we just printed a new version at the end of go list -m -u lines, that implies the user would be able to upgrade with go get, but there's more migration work involved than that. A user may not be able to upgrade at all if they depend on a feature removed in the newer version, or if another dependency requires the old version, and the old and new versions are incompatible.

There's some cost to implementing this in go list -m -u, too. If I'm on v10 of go-github, go list will do a bunch of sequential network fetches to find v32. That may put pressure on module proxies, looking up lots of non-existent modules. We can do some clever things to mitigate both problems, but it's still not free.

Overall, I feel that #40357 is a stronger approach. The main difference I see is that #40357 requires explicit action from module authors, i.e., adding a comment to go.mod and tagging a release. That seems very reasonable to me: an author releasing a new major version should provide guidance to their users on how to migrate.

@jayconrod

If we just printed a new version at the end of go list -m -u lines, that implies the user would be able to upgrade with go get

I don't believe this is implied, neither explicitly nor implicitly.

If I'm on v10 of go-github, go list will do a bunch of sequential network fetches to find v32.

Shouldn't it be able to do just one? I would expect "latest major version of module X" to be a single operation.

Overall, I feel that #40357 is a stronger approach.

40357 and this proposal are complementary, advancing the ball towards the same goal from different angles.

@peterbourgon

I don't believe this is implied, neither explicitly nor implicitly.

Not implied in this proposal, but nonetheless, I think if users see one of these outputs:

github.com/peterbourgon/ff v1.6.0 [v1.7.0] [latest v3.1.0]
github.com/peterbourgon/ff v1.6.0 [v1.7.0] latest major version: github.com/peterbourgon/ff/v3 v3.1.0

They will believe they can upgrade with go get.

Shouldn't it be able to do just one? I would expect "latest major version of module X" to be a single operation.

No, for each proxy it would be a series of requests like:

GET https://someproxy/github.com/google/go-github/v10/@v/list
GET https://someproxy/github.com/google/go-github/v10/@latest
GET https://someproxy/github.com/google/go-github/v11/@v/list
GET https://someproxy/github.com/google/go-github/v11/@latest
GET https://someproxy/github.com/google/go-github/v12/@v/list
GET https://someproxy/github.com/google/go-github/v12/@latest
...

Proxies should return 404 or 410 for non-existent modules, which will cause go to fall back to the next proxy in the list. With default settings, it will end up at direct. So you'd end up cloning the git repo locally. Each proxy would likely attempt the same.

40357 and this proposal are complementary, advancing the ball towards the same goal from different angles.

If we're heading toward printing in go list -m -u, the difference isn't clear to me. The key difference seems like whether the module author adds a // Deprecated comment in go.mod and tags a new release. From the user's perspective, it seems like a similar experience.

@jayconrod

I think if users see one of these outputs . . . they will believe they can upgrade with go get.

I don't know why you think this. A different MAJOR SemVer explicitly signals breaking changes β€” all users know this, and therefore no user would expect go get to just work.

No, for each proxy it would be a series of requests . . .

Perhaps the proxy should be enhanced to understand major versions not as wholly separate entities?

This has been a long discussion and it seems clear that there _isn't_ a consensus.
We have #40357 accepted, which addresses some of the same things, so it would make sense to do that first and see what's left.

Based on all that, it seems like this should be a likely decline.

Can we defer making a decision on this? I have a partially written followup to this discussion; I think we got a bit lost.

@adg, sure thing, moved to Hold. Please move back to the Active project column when you're ready.

User experience report, cross-posted here following a discussion on #general on Slack, with input from @seankhliao, @theckman, and @peterbourgon. tl;dr "is there an existing tool that tells you when there's a new major version of any of the Go modules that you use?"

I wrote:

@theckman My use case is that I have an app that has some dependencies that are end-user-visible and I was relying on go get -u to keep these dependencies up to date. I was surprised to discover that there was a new major version of a key library I was using, and then got suspicious that there might be other out-of-date dependencies and wanted to check.

Specifically, the app is https://github.com/twpayne/chezmoi and the key library is https://github.com/Masterminds/sprig. sprig contains a bunch of useful text/template functions which chezmoi makes available to the user.

In practice, many open source projects can only afford to keep the most recent major version up to date with bug fixes, so it's generally worth making the effort to maintain your own code to use the most recent major version.

There's a deeper issue of course: the use of semantic versioning in Go modules assumes that module developers always know when they break backwards compatibility and need to tag a new major version. This assumption is obviously and demonstrably false (obligatory XKCD: https://xkcd.com/1172/) but this has already been widely discussed.

In short, in open source, you're almost always better off using the latest version, be that major, minor, or patch, and it would be nice if a tool existed to help you do that.

"is there an existing tool that tells you when there's a new major version of any of the Go modules that you use?"

@twpayne FWIW, I hacked together a tool which does that. https://github.com/icholy/gomajor

Thanks @icholy - I used your tool based on a recommendation from @seankhliao and https://github.com/twpayne/chezmoi/pull/889 was the result.

There's a deeper issue of course: the use of semantic versioning in Go modules assumes that module developers always know when they break backwards compatibility and need to tag a new major version. This assumption is obviously and demonstrably false (obligatory XKCD: https://xkcd.com/1172/) but this has already been widely discussed.

@twpayne go release is meant to help with that. There's an experimental version at golang.org/x/exp/cmd/gorelease.

If an author finds that they have unintentionally broken backward compatibility, one option is to revert the change, tag a new release, and retract releases containing the change (#24031).

In practice, many open source projects can only afford to keep the most recent major version up to date with bug fixes, so it's generally worth making the effort to maintain your own code to use the most recent major version.
…
In short, in open source, you're almost always better off using the latest version, be that major, minor, or patch, and it would be nice if a tool existed to help you do that.

I agree that in open source you're generally best off using the latest version, but it's also important to be able to migrate on your own time, when the benefits outweigh the cost of migration. For that reason, authors should communicate both their support policy and their intended migration path for users, generally in a README and/or a deprecation notice. (The latter is slated for support in the Go toolchain as #40357.)

For the example you mentioned, github.com/Masterminds/sprig, the README says (emphasis mine):

There are two active major versions of the sprig package.
…

  • v2 is the previous stable release series. It has been more than three years since the initial release of v2. You can read the documentation and see the code on the release-2 branch. Bug fixes to this major version will continue for some time.

In this case it appears that v2 remains supported as well β€” so, while you might want to upgrade to the v3 API in order to use new features or in anticipation of some future deprecation of the v2 API, that migration does not appear to block your ability to pull in fixes for critical bugs today. To me, that indicates that the v2-to-v3 migration does not have the same importance as, say, pulling in fixes from the latest v2 minor or patch release.

If they got it from goimports or gopls, then there may be a heuristic in those to fix. (If so, please file a separate issue.)

That appears to be #41800.

41501 is the analogous issue for cmd/doc.

I just want to mention, in case any of you missed it, that pkg.go.dev now surfaces later major versions (#37765). For instance, if you visit https://pkg.go.dev/github.com/russross/blackfriday you'll see "The latest major version is v2" near the top.

you'll see "The latest major version is v2" near the top.

It's there, but even with its distinguishing information icon, I still missed it in my first two scans of the page. There's no other deviation I see from the site's white, black, and blue palette, but is it too controversial to propose using a color like orange or red to make it stand out more?

@seh, it really shouldn't be orange or red or blink. The point that we've made repeatedly in this discussion is that the mere existence of v2 does not mean that everyone using v1 needs to be coerced into updating.

Now, if the module was flagged as deprecated (#40357), then that would be appropriate to make orange or red or blink.

That's the fundamental difference between that issue and this one. This issue takes as a given the proposition that if an author has published v2, v1 is deprecated and should no longer be used. That's just not always true, and we should not act as though it is true in the absence of a clearer signal from the module author. #40357 provides that clearer signal.

I see your point, Russ. Please keep my suggestion in mind for the deprecated case.

Usually when I'm browsing the documentation like this, I'm already trying to use the latest major version available, so the mere existence of a new version is enough to trigger me to take a closer look, and makes me start trying to figure out if it's worth upgrading.

@rsc

This issue takes as a given the proposition that if an author has published v2, v1 is deprecated and should no longer be used. That's just not always true, and we should not act as though it is true in the absence of a clearer signal from the module author.

I'm sorry that this proposal fails to communicate its intent in such a way that you can draw this conclusion β€” that's our fault as the authors. To be clear: this isn't an assumption the proposal makes, nor is it what we want to communicate with the notification. Instead, the assumption is that if an author has published v2, _it should be preferred_ to v1, absent any signal suggesting otherwise. I believe that's true in the general case, and I hope it's noncontroversial.

@bcmills

@icholy, we know what form a newer major version would take if it _did_ exist, so (assuming no gaps in the major-version sequence) we could detect the presence of the immediate next version with an O(1) check, and find the highest major version in O(N). I think it's reasonable for us to miss the notification if authors skip straight from vN to vN+2. (Don't do that!)

We would need to be a little careful to check for both example.com/vN and example.com vN.0.0+incompatible, but it's not fundamentally a problem.

I ran into the "assuming no gaps in the major-version sequence" scenario with https://github.com/coreos/go-systemd. Prior to modules, it was tagged with v1 to v21 which the proxy does not support. Version v22.0.0 is the first version that supports modules. However, there's no way to discover that because the intermediate versions are missing: https://proxy.golang.org/github.com/coreos/go-systemd/@v/list

AFAIK this isn't a common issue, but I thought it was worth documenting.

edit: related https://github.com/coreos/go-systemd/issues/346

Was this page helpful?
0 / 5 - 0 ratings