Go: cmd/go: automatically check and use vendored packages

[bcmills]

Abstract

This is a proposal to enable the use of the main module's vendor directory, by default, in module mode. This proposal replaces #30240 and #29058, and subsumes #27227.

Background

The Go 1.5 release included support for a vendor subdirectory for each package subtree within GOPATH. A vendor directory, if present, implicitly changed the interpretation of import statements within that subtree to always refer to packages within the vendor directory if present.

In module mode, the meaning of each import is instead determined by the dependencies of the main module used for the build. However, from the first release of module mode (in Go 1.11), the go command has included a go mod vendor subcommand, which populates a vendor directory at the root of the main module with source code from those dependencies; and a -mod=vendor flag, which instructs the go command to use that source code for building.

In module mode, unlike GOPATH mode, the vendor subdirectories of other packages and modules are ignored — only the vendor directory at the root of the main module is used. This gives the author of the main module complete control over their dependencies via require and replace directives.

Go users have built a variety of workflows around vendored dependencies for a variety of use-cases — including self-contained builds, language-agnostic code review, and ephemeral patches — and since the launch of module mode in Go 1.11 they have made it clear that those workflows remain important in module mode.

However, vendoring in module mode today has a few rough edges:

• A maintainer who introduces a new import statement may easily and accidentally introduce skew between the versions in the go.mod file and those in the vendor directory (#29058).

  • In CL 174528, we added a simple check for this skew in the std and cmd modules in the standard library, but it relies on some assumptions about the dependencies of std and cmd that do not hold in general.

• The maintainers of projects that use vendoring expect the vendor directory to be used by default (#27227), but today in module mode that requires users to set -mod=vendor, either explicitly on the command line or in a GOFLAGS variable. For users working on multiple modules with different vendoring strategies, setting the variable may require repeated intervention over the course of the day.

This proposal attempts to address those rough edges.

Proposal

The concrete proposal is posted in a comment below. (Any updates will be linked from here.)

Latest update: https://github.com/golang/go/issues/33848#issuecomment-537222782

(CC @jayconrod @ianthehat)

[bcmills]

This comment contains my original proposal.

An updated version can be found below (in https://github.com/golang/go/issues/33848#issuecomment-537222782).

Proposal

I propose to enable -mod=vendor automatically, and to fail build operations when -mod=vendor is enabled if the vendor/ directory is out of sync with the go.mod file.

In order to detect when the vendor/ directory is out of sync with the go.mod file, we need to know the constraints imposed by the main module — that is, its require and replace directives. Given those, the remaining module requirements are computed deterministically, so we do not need to store or check the complete module graph, which may be much larger than the explicit requirements or even the final list of selected versions.

Concretely, the changes are:

• In the vendor/modules.txt file, add an annotation indicating which modules were listed explicitly in a require or replace directive in the go.mod file as of the last invocation of go mod vendor. (This implies that every module path and version that appears in the main module's go.mod file must be listed in modules.txt, even if no packages are vendored from within that module or version.)

  • To maintain compatibility with the Go 1.11 modules.txt parser, this annotation must have the following properties:

    • Does not change the number of fields in the # <module-path> <version> lines, which must be equal to 3 in order for the existing parser to recognize the line.
    • Does not occur as a line with exactly one field (which would be interpreted as a package path within the preceding module).
    • Either does not have the prefix #, or does not occur _in between_ the # <module-path> <version> lines and the associated <package-path> lines. (Otherwise, it would break the association between modules and packages.)

  • To comply with the above requirements, I propose that we use a line of the form:

    ## explicit
    

    or

    ## replaced by example.com/some/module vX.Y.Z
    

    or

    ## explicit; replaced by example.com/some/module vX.Y.Z
    

  • To allow for future expansion, the updated parser should check lines beginning with ## for _any semicolon-delimited field_ matching the expected annotation, ignoring any whitespace surrounding the field. (All other tokens within such comments are reserved for future use.)

• When a top-level vendor/modules.txt file exists and contains at least one ## annotation, then for all build commands except go get:

  • Change the default value of the -mod build flag to vendor.

  • Verify that the set of modules annotated as explicit in vendor/modules.txt is exactly equal to the set of modules (and versions) found in require directives in the main module's go.mod file.

  • Verify that the set of modules annotated as replaced by a module in vendor/modules.txt is exactly equal to the set of replace directives (including both path and version, if applicable) in the main module's go.mod file.

• Add a new value for the -mod flag, -mod=mod, to explicitly request the non-vendor behavior: that is, to ignore the contents of the vendor directory and to automatically update the go.mod file as needed.

  • The go get command today treats an explicit -mod=vendor flag as an error. With this change, go get should instead always provide the behavior of -mod=mod, treating the -mod flag as unrecognized (see https://github.com/golang/go/issues/26850#issuecomment-411903910).

    In particular, go get should ignore any implicit -mod=vendor from the presence of a vendor/modules.txt file or from a -mod flag set via GOFLAGS (see also #32502), and treat an explicit -mod=vendor, -mod=readonly, or -mod=mod as an error (see #30345).

The proposed approach allows the following commands to work from a vendor directory with an empty module cache and no network access:

• go list -deps ./...
• go build ./...
• go test ./…

Other commands _may_ fail without an explicit -mod=mod override:

• go list all
• go test all
• go list -test $(go list -deps ./…)
• go list -m example.com/some/module

Note that the go mod subcommands do not have a -mod= flag and therefore never look at the vendor directory for information. They always require access to a module cache or module proxy. The check in this proposal ensures that the results produced by any go mod subcommand remain accurate even when the corresponding build commands use vendored sources.

Example workflow

# Upgrade the golang.org/x/sys module and fetch it from $GOPROXY.
# Also ensure that the rest of the module dependencies are consistent
# with the requirements of the upgraded module.
# This step updates the go.mod file, but not the vendor directory.
$ go get -d golang.org/x/sys

# A plain 'go test ./…' at this point may fail,
# because vendor/modules.txt may be out of sync with the go.mod file.
# However, 'go test -mod=mod ./…' should succeed,
# because it explicitly ignores the vendor directory.
$ go test -mod=mod ./...

# Download any newly-upgraded dependencies to the module cache,
# and copy them from the module cache to the vendor directory.
$ go mod vendor

# Test the packages within the module
# using the newly-updated dependencies in the vendor directory.
$ go test ./...

Caveats

Intentional modifications

The above checks do not detect intentional modifications to the vendored sources. Users who want to ensure that no intentional modifications are present will need to re-run go mod vendor and look for diffs, or (after #27348 is addressed) run go mod verify.

Local filesystem changes

The above checks also do not detect skew due to changes in local files within a directory specified in a replace directive. I do not recommend the use of filesystem replace directives in general, but the potential for skew makes them particularly important to avoid when using a vendor directory.

If we believe that changes in the local filesystem will be a significant source of skew in practice, we could expand the replaced by annotation to include a hash of the replacement directory. However, computing that hash could significantly increase the cost of the consistency check, which in the current proposal only requires us to read the go.mod and vendor/modules.txt files.

Dependency analysis

Because the approach proposed here does not retain go.mod files for modules that do not contribute packages to the build (including older-than-selected versions of packages that _do_ contribute packages), the vendor directory will not support analysis of the module graph (go list -m all, go mod why -m, and go mod graph).

That seems like an acceptable tradeoff: adding those go.mod files would require some scheme for encoding their versions within the vendor directory, which we do not have today, and would introduce diffs that are largely irrelevant to the build during code review.

[bcmills]

One point I'm a bit uncertain on: above I proposed to trigger the automatic -mod=vendor behavior based on the presence of a ## annotation in the vendor/modules.txt file.

Would it be better to instead trigger it based on a go 1.14 directive in the go.mod file (plus the existence of vendor/modules.txt? (Why or why not?)

[thepudds]

That seems like it would be simpler to understand, including because I think it has been previously stated that the format of the modules.txt file is an internal and undocumented implementation detail?

Could you even take it a step further to have it trigger off of the presence of the vendor directory itself plus go 1.14 in the go.mod, rather than vendor/modules.txt plus go 1.14?

It would then be triggering off of things that are both more visible and actually documented, and hence would likely be easier to understand.

[ardan-bkennedy]

I propose to enable -mod=vendor automatically, and to fail build operations when -mod=vendor is enabled if the vendor/ directory is out of sync with the go.mod file.

When I build in a container, I will be copying the entire projects source tree of code into that container manually. This will cause things to be out of sync. So now, I will have to make sure that I don't copy the go.mod|sum files into the container.

This also means that anyone else on my team working with this project, will need to run throught the proxy server to validate their vendor folder is "correct". I am concerned that in the end, the vendor folder is not really acting as a vendor folder. If the module system doesn't like what it sees in vendor, the build breaks. The deps needs to be cached anyway. So might as well not even use vendor.

[bcmills]

When I build in a container, I will be copying the entire projects source tree of code into that container manually. This will cause things to be out of sync. So now, I will have to make sure that I don't copy the go.mod|sum files into the container.

I think I'm missing something. How would copying the entire source tree into the container cause the vendor directory to get out of sync with the go.mod file? Presumably you'd be copying in the same go.mod file that you used to generate the vendor directory in the first place.

[ardan-bkennedy]

How are you going to verify that the vendor folder and the go.mod file is in sync? This is going to require network calls against the proxy server I assume? I am wondering if the deps are going to be downloaded as well during this process?

[bcmills]

@ardan-bkennedy, that's all covered in the text of the proposal above (https://github.com/golang/go/issues/33848#issuecomment-524982109).

[ardan-bkennedy]

I read that several more times and I think it’s starting to make sense. What the above proposal is saying is, given a vendor folder with a modules.txt file, just verify that this file is in sync with the go.mod file? If they are, build against the vendor folder. If they are not, issue a build error? Therefore no calls to the proxy server are required. The original workflow is maintained?

[bcmills]

Yep. Being able to do that without hitting the network at all requires that we include some extra information in the vendor/modules.txt file compared to what we're writing today, but the amount of extra information seems reasonable to me, and the resulting file should be backward-compatible to earlier Go releases.

[ardan-bkennedy]

That's brilliant and allows vendoring to be an option again. Which is much appreciated. So the next question is, will this make a point release for 1.13 or do we need to wait for 1.14?

My concern is that projects may just give up on vendoring because of the current breakage in workflows. Which is a shame because I think vendoring has strong advantages when it's reasonable to use it.

[bcmills]

This would not be backported to 1.13. (Per the minor release policy, we only backport “security issues, serious problems with no workaround, and documentation fixes”.)

In the interim, users who rely on vendoring workflows can enable vendoring for all build commands by setting GOFLAGS=-mod=vendor, explicitly disable module mode in Go 1.13 by setting GO111MODULE=off, or implicitly disable module mode within GOPATH/src by staying on Go 1.12 until they're ready to try a 1.14 pre-release.

[rsc]

It sounds like the general consensus here is that we should accept this proposal for Go 1.14.
This is probably important to do before we set GO111MODULE=on for people using the dev branch.

Am I reading this wrong? Does anyone _object_ to accepting this proposal? Thanks.

[bcmills]

An interesting wrinkle: if a replaced module does not contribute any imported packages, we omit it from vendor/modules.txt. However, that replacement can still have an effect on the overall build list, because the replacement go.mod file may differ from the original.

This implies that we need to duplicate (and verify) all replace entries found in the go.mod file to modules.txt, even if they do not affect any module in the actual build list.

If we're not loading transitive dependencies, we don't know whether a replaced module actually exists in the build list, so we can't add an entry of the form # <module> <version>: otherwise, older versions of the go command will erroneously interpret them as being in the build list.

Fortunately, the old modules.txt parser skips module entries that have additional text after the version, so the following styles of lines should be ignored (and thus safe to use).

# example.org/unused v1.0.0 (unused)
## replaced by github.com/unused v1.0.0
# example.com/other/unused * (unused)
## replaced by ./other-unused

[bcmills]

Well that's funny. The current go mod vendor implementation already writes replacements, using the form # MODULE VERSION => MODULE [VERSION]:
https://github.com/golang/go/blob/bdf0fe54480034cd21e36cfed6e44f10f4cb5c92/src/cmd/go/internal/modcmd/vendor.go#L72

I think that means that go mod vendor is currently writing vendor/modules.txt files that it isn`t capable of parsing correctly.

[gopherbot]

Change https://golang.org/cl/198319 mentions this issue: cmd/go: automatically check and use vendored packages

[bcmills]

I have updated the proposal below, based on information learned during prototyping. This version supersedes the one in https://github.com/golang/go/issues/33848#issuecomment-524982109.

Proposal

I propose to enable -mod=vendor automatically for operations that do not explicitly query or manipulate the build list, and to fail build operations when -mod=vendor is enabled if the vendor/ directory is out of sync with the go.mod file.

In order to detect when the vendor/ directory is out of sync with the go.mod file, we need to know the constraints imposed by the main module — that is, its require and replace directives. Given those, the remaining module requirements are computed deterministically, so we do not need to store or check the complete module graph, which may be much larger than the explicit requirements or even the final list of selected versions.

Concretely, the changes are:

• In the vendor/modules.txt file, add an annotation indicating which modules were listed explicitly in a require or replace directive in the go.mod file as of the last invocation of go mod vendor. (This implies that every module path and version that appears in the main module's go.mod file must be listed in modules.txt, even if no packages are vendored from within that module or version.)

  • To maintain compatibility with the Go 1.11 modules.txt parser, this annotation must have the following properties:

    • Does not change the number of fields in the # <module-path> <version> lines [edit: for unreplaced modules only], which must be equal to 3 in order for the existing parser to recognize the line.
    • Does not occur as a line with exactly one field (which would be interpreted as a package path within the preceding module).
    • Either does not have the prefix #, or does not occur _in between_ the # <module-path> <version> lines and the associated <package-path> lines. (Otherwise, it would break the association between modules and packages.)

  • [edit: Adopted the existing syntax generated by go mod vendor for replacements, as discovered above .]
    To comply with the above requirements, I propose that we use a line of the form:

    # example.com/some/module vA.B.C
    ## explicit
    

    to annotate explicit dependencies, and

    # example.com/some/module vA.B.C => example.com/some/other/module vX.Y.Z
    

    for replacements.

  • To allow for future expansion, the updated parser should check lines beginning with ## for _any semicolon-delimited field_ matching the expected annotation, ignoring any whitespace surrounding the field. (All other tokens within such comments are reserved for future use.)

• [edit: Adopted @thepudds' suggestion for trigger condition.]
  When the go.mod file specifies go 1.14 and a top-level vendor directory exists, then for all build commands except go get:

  • Change the default value of the -mod build flag to vendor.

  • Verify that the set of modules annotated as explicit in vendor/modules.txt is exactly equal to the set of modules (and versions) found in require directives in the main module's go.mod file.

  • [edit: Expanded this point to ensure that the replacements in the go.mod file account for all of the replacements reported in vendor/modules.txt.]
    Verify that the set of replacements indicated in vendor/modules.txt includes the complete set of replace directives (including both path and version, if applicable) in the main module's go.mod file, and that the replacements in the go.mod file account for all of the module replacements reported in vendor/modules.txt.

• Add a new value for the -mod flag, -mod=mod, to explicitly request the non-vendor behavior: that is, to ignore the contents of the vendor directory and to automatically update the go.mod file as needed.

  • The go get command today treats an explicit -mod=vendor flag as an error. With this change, go get should instead always provide the behavior of -mod=mod, treating the -mod flag as unrecognized (see https://github.com/golang/go/issues/26850#issuecomment-411903910).

    In particular, go get should ignore any implicit -mod=vendor from the presence of a vendor/modules.txt file or from a -mod flag set via GOFLAGS (see also #32502), and treat an explicit -mod=vendor, -mod=readonly, or -mod=mod as an error (see #30345).

  • [edit: Added discussion of go list.]
    go list -m command today accepts -mod=vendor, and displays a truncated version of the build list (restricted to what was found in the vendor/modules.txt file) without further warning. Implicitly truncating the build list seems like a mistake.

    Since -mod=vendor will now be enabled implicitly, that difference will become even more subtle, especially since go mod graph will not similarly truncate the graph. I propose that go list -m should instead fail with a helpful error if -mod=vendor is set.

The proposed approach allows the following commands to work from a vendor directory with an empty module cache and no network access:

• go list -deps ./...
• go build ./...
• go test ./…

Explicitly build-list related commands will continue to download missing modules to the cache:

• go mod graph
• go list -m example.com/some/module

Other commands _may_ fail without an explicit -mod=mod override:

• go list all
• go test all
• go list -test $(go list -deps ./…)

Note that the go mod subcommands do not have a -mod= flag and therefore never look at the vendor directory for information. They always require access to a module cache or module proxy. The check in this proposal ensures that the results produced by any go mod subcommand remain accurate even when the corresponding build commands use vendored sources.

Example workflow

# Upgrade the golang.org/x/sys module and fetch it from $GOPROXY.
# Also ensure that the rest of the module dependencies are consistent
# with the requirements of the upgraded module.
# This step updates the go.mod file, but not the vendor directory.
$ go get -d golang.org/x/sys

# A plain 'go test ./…' at this point may fail,
# because vendor/modules.txt may be out of sync with the go.mod file.
# However, 'go test -mod=mod ./…' should succeed,
# because it explicitly ignores the vendor directory.
$ go test -mod=mod ./...

# Download any newly-upgraded dependencies to the module cache,
# and copy them from the module cache to the vendor directory.
$ go mod vendor

# Test the packages within the module
# using the newly-updated dependencies in the vendor directory.
$ go test ./...

Caveats

Intentional modifications

The above checks do not detect intentional modifications to the vendored sources. Users who want to ensure that no intentional modifications are present will need to re-run go mod vendor and look for diffs, or (after #27348 is addressed) run go mod verify.

Local filesystem changes

The above checks also do not detect skew due to changes in local files within a directory specified in a replace directive. I do not recommend the use of filesystem replace directives in general, but the potential for skew makes them particularly important to avoid when using a vendor directory.

If we believe that changes in the local filesystem will be a significant source of skew in practice, we could expand the replaced by annotation to include a hash of the replacement directory. However, computing that hash could significantly increase the cost of the consistency check, which in the current proposal only requires us to read the go.mod and vendor/modules.txt files.

Dependency analysis

Because the approach proposed here does not retain go.mod files for modules that do not contribute packages to the build (including older-than-selected versions of packages that _do_ contribute packages), the vendor directory will not support analysis of the module graph (go list -m all, go mod why -m, and go mod graph).

That seems like an acceptable tradeoff: adding those go.mod files would require some scheme for encoding their versions within the vendor directory, which we do not have today, and would introduce diffs that are largely irrelevant to the build during code review.

[gopherbot]

Change https://golang.org/cl/198318 mentions this issue: internal/goversion: update to 1.14

[bcmills]

The (unused) annotations turn out to be very complex to implement, and for relatively little gain in usability. Probably we should leave them out.

[bcmills]

Updated https://github.com/golang/go/issues/33848#issuecomment-537222782 and the prototype CL to omit the (unused) annotations.

[bcmills]

The prototype implementation (CL 198319 and CL 198438) is ready for review.

[gopherbot]

Change https://golang.org/cl/198438 mentions this issue: cmd/go: remove the -mod flag from 'go get'

[rajender]

@bcmills Do we really need -mod=mod option? what are the workflows enabled by it?

[bcmills]

@rajender Users will typically not need to use -mod=mod day-to-day, but it can be handy if you want to bypass a temporarily-inconsistent vendor directory, or if you want to run tests for packages outside of your main module:

For example, you might have an interaction like:

$ go get -u example.com/some/dependency  # Upgrade some/dependency.
$ go test -mod=mod ./... example.com/some/dependency/...  # Make sure the new configuration is ok.
$ go mod vendor  # Tests passed, so vendor in the new code.

[rajender]

@bcmills What is the downside of making go get -mod=vendor as default instead, when vendor directory present?

Separately, -mod=novendor may be more readable than -mod=mod.

Also, Is it possible to drop -mod=vendor can be removed in some future release?

[rajender]

I mean, go get to update vendor directory and use it?

@bcmills What is the downside of making go get -mod=vendor as default instead, when vendor directory present?

[rsc]

Based on the discussion here, especially the replies since https://github.com/golang/go/issues/33848#issuecomment-533274286 asking if we had reached consensus, this seems like a likely accept.

Leaving open for a week for final comments.

[bcmills]

What is the downside of making go get -mod=vendor as default instead, when vendor directory present?

I mean, go get to update vendor directory and use it?

go get does not currently overwrite the contents of the vendor directory, and I don't think that it should: a user may have local edits to vendored packages (such as for debugging), and we should not discard those local edits without an explicit action — particularly if the user follows the go get with a -mod=mod command to investigate the result of the upgrade.

Local edits to the go.mod file are usually relatively minor — upgrading a dependency here or there but mostly not changing things. In contrast, a user in the middle of a debugging session may have a lot of debug prints in vendored packages that they won't want to have to reconstruct.

(Also note that if you _do_ want to update the vendor directory, it is straightforward to run go get […] && go mod vendor explicitly — but you still need to fetch things over the network or from the module cache to complete the go get step.)

[bcmills]

Separately, -mod=novendor may be more readable than -mod=mod.

novendor seems a bit ambiguous: both -mod=readonly and -mod=mod ignore the vendor directory, so lack of vendoring is not really a distinguishing feature.

[bcmills]

Also, Is it possible to drop -mod=vendor can be removed in some future release?

I'm not sure. Is there a particular reason we would need to drop it?

[rasky]

novendor seems a bit ambiguous: both -mod=readonly and -mod=mod ignore the vendor directory, so lack of vendoring is not really a distinguishing feature.

That's surprising to me. I think at some point after this proposal is accepted, it will be very surprising that -mod=readonly ignores vendor directory, as the defining feature of readonly is avoid modifying go.mod which seems completely orthogonal to using or not using the vendor directory.

What about having -mod=readonly/readwrite, and -mod=vendor/novendor, and then combinations like -mod=readonly,novendor

[bcmills]

it will be very surprising that -mod=readonly ignores vendor directory, as the defining feature of readonly is avoid modifying go.mod which seems completely orthogonal to using or not using the vendor directory.

Using the vendor directory and avoiding modifications to the go.mod file are, unfortunately, not orthogonal.

With the current vendor layout, it can contain — at best! — only the go.mod files for the selected version of each module, but a transitive dependency can (and sometimes does) arise only through an older-than-selected version. To ensure consistency in the go.mod file we must respect those transitive dependencies (see #31248 and #29773), so operations with -mod=vendor _necessarily cannot_ modify the go.mod file. So -mod=vendor effectively implies readonly, and there is no need to specify those two properties separately.

If you want to use locally-stored modules in conjunction with edits to the module graph, you'll need to use replace directives instead of (or in addition to) vendoring. (Note that today even replaced modules are copied to the vendor directory; #33789 is a feature request to allow mixing the two without duplication.)

[gopherbot]

Change https://golang.org/cl/199821 mentions this issue: cmd/go/internal/list: disallow 'list -m' with '-mod=vendor'

[bcmills]

One more revision: I now propose to make go list -m fail when -mod=vendor is set, rather than making the -m flag change the default -mod setting.

[rsc]

Various clarifying comments here, but no objections to accepting since https://github.com/golang/go/issues/33848#issuecomment-537604656.

Accepted.

[bcmills]

The implementation for this has landed at head. Please give it a try and let us know if you spot any problems!

[kevinburkemeter]

Adding ## explicit lines to vendor/modules.txt broke at least one tool, see https://github.com/goware/modvendor/issues/7 for details.

(We need that tool in order to copy a directory containing only header/C files into the vendor directory).

Fortunately the fix is easy, I think, I can submit a patch for it this afternoon.

[kevinburkemeter]

Found a second error... modvendor was trying to parse the replace directives to figure out when it needed to get a new module path for a directory. It did so by checking if the fourth token on a line was =>.

The last line in the modules.txt file - a replace directive - has the arrow as the third token so this now fails.

# github.com/goware/modvendor => github.com/meterup/modvendor v0.0.0-20191010172136-c2d7e3539f83

            // Handle "replace" in module file if any
            if len(s) > 3 && s[3] == "=>" {
                mod.SourcePath = s[4]
                mod.SourceVersion = s[5]
                mod.Dir = pkgModPath(mod.SourcePath, mod.SourceVersion)
            } else {
                mod.Dir = pkgModPath(mod.ImportPath, mod.Version)
            }

            if _, err := os.Stat(mod.Dir); os.IsNotExist(err) {
                fmt.Printf("Error! %q module path does not exist, check $GOPATH/pkg/mod\n", mod.Dir)
                os.Exit(1)
            }

I'm a little confused because my vendor/modules.txt also has this about halfway through the file:

# github.com/goware/modvendor v0.0.0-20190516042800-ce72b408a8fe => github.com/meterup/modvendor v0.0.0-20191010172136-c2d7e3539f83

I'm not sure which one takes precedence.

Open to ideas, I don't really understand how this stuff works, and I used this library because it was the easiest way to copy a directory containing only header/C files into the vendor directory.

[bcmills]

I used this library because it was the easiest way to copy a directory containing only header/C files into the vendor directory.

I suspect that we should copy over more header files that we're doing today anyway, but that won't get you the corresponding .c files.

[bcmills]

Adding ## explicit lines to vendor/modules.txt broke at least one tool, see goware/modvendor#7 for details.

Thanks for the heads-up. We intentionally didn't document vendor/modules.txt because it's an internal format, although we don't want to gratuitously break things if it's easy to avoid.

The more robust fix is probably to switch it to use go list. (go list -f '{{with .Module}}[…]{{end}}' should be portable to all go versions, or go list -mod=mod -json -m all should do the trick in 1.14.)

[bcmills]

I'm not sure which one takes precedence.

Neither are we! That's #26344.

[kevinburkemeter]

The more robust fix is probably to switch it to use go list. (go list -f '{{with .Module}}[…]{{end}}' should be portable to all go versions, or go list -mod=mod -json -m all should do the trick in 1.14.)

Yeah unfortunately I think switching is going to take more time than I have today so I might stick with the kludge.

[pstibrany]

I work on two projects, let's call them A and B, and A depends on B. Both have vendor copy of their dependencies. I use replace directive in A project to point to my local copy of a B project (replace B => /Users/peter/B), so that I can develop on both at the same time. This setup works fine with go 1.13, and go uses my local copy of B (in /Users/peter/B) when building A, and not vendored files.

When trying with latest tip (46b7557), gotip build in A complains about inconsistency between go.mod and modules.txt. So I run gotip mod vendor. This makes a copy of packages from my local B project to vendor.

Now if I edit files in B and run gotip build in A, it builds with vendored files. This is a change from go 1.13, which would use files in /Users/peter/B for building. To get latest changes from B in gotip, I need to run gotip mod vendor again. Furthermore, build doesn't even complain about changes between /Users/peter/B and vendored copy of B.

I would like to preserve my workflow, without a need to call go mod vendor each time I modify files in the B project. I would prefer if vendored files for module replaced with local directory were ignored, and replacement would be used instead.

I hope it makes sense. Thanks.

[bcmills]

@pstibrany

I would like to preserve my workflow, without a need to call go mod vendor each time I modify files in the B project.

Note that you can retain the default behavior of 1.13 by explicitly setting GOFLAGS=-mod=mod in 1.14.
(With the caveat that, as in 1.13, it will continue to fetch unreplaced modules using the network.)

I would prefer if vendored files for module replaced with local directory were ignored, and replacement would be used instead.

That is #33789, which we can pursue as a separate feature request (but not for 1.14).

[kevinburkemeter]

FWIW, I'm probably not going to be able to compile from tip for the rest of the release cycle. I'm the only one on my team that compiles from tip and any time I update the vendor file, I add the ## explicit lines to vendor/modules.txt and they remove them.

[bcmills]

I'm the only one on my team that compiles from tip and any time I update the vendor file, I add the ## explicit lines to vendor/modules.txt and they remove them.

Hmm. Perhaps we should only add the ## explicit lines in the first place if the go version in the go.mod file is 1.14? (But there may be other differences in general between the behavior of go mod vendor at tip and in the latest release, just like there may be differences in go fmt from time to time.)

[gopherbot]

Change https://golang.org/cl/201257 mentions this issue: cmd/go: omit new 'vendor/modules.txt' annotations if the go version is 1.13 or lower

[bcmills]

@kevinburkemeter, thanks for the feedback. If the go directive specifies 1.13 or earlier, we'll now omit the new annotations, so you can hopefully resume testing from tip as long as you leave that directive in place.

[kevinburkemeter]

Thank you! I will try to switch over my team once the beta is released, then we can test the new changes

[gopherbot]

Change https://golang.org/cl/202917 mentions this issue: cmd/go/internal/list: ensure that cfg.BuildMod is initialized before reading it in 'go list -m'

[gopherbot]

Change https://golang.org/cl/202977 mentions this issue: cmd/go: populate available module information for packages in vendor mode

[gopherbot]

Change https://golang.org/cl/203138 mentions this issue: cmd/go: re-enable 'go list -m' with -mod=vendor for limited patterns

[gopherbot]

Change https://golang.org/cl/205064 mentions this issue: cmd/go: derive TestExecutableGOROOT environment from tg.env instead of os.Environ()

[gopherbot]

Change https://golang.org/cl/207397 mentions this issue: cmd/go: document the -mod=mod option

[gopherbot]

Change https://golang.org/cl/210341 mentions this issue: cmd/go: include cfg.BuildModReason in 'import lookup disabled' errors

[gopherbot]

Change https://golang.org/cl/214081 mentions this issue: cmd/go: explicitly reject 'list -u' and 'list -versions' when '-mod=vendor' is set

[gopherbot]

Change https://golang.org/cl/222538 mentions this issue: vendor: migrate from govendor to go mod vendor

[gopherbot]

Change https://golang.org/cl/251159 mentions this issue: cmd/go/internal/modload: remove (*loader).forceStdVendor