Go: cmd/go: allow go.mod.local to contain replace/exclude lines

[maeglindeveloper]

Hi everyone,
I'm actually using a go.mod file for dependencies in my go project.

I saw that there is a way to use local packages instead of online repo, using the replace() features, which is really cool when developping multiple packages at the same time.

I was wondering if there is a way split in different files the require dependencies and the replace one. Something like that

go_replace.mod => define the replace part.
go.mod => define the module and the required dep, including the go_replace.mod.

the go_replace.mod will not be pushed on git, and each developer will create it if needed.

What version of Go are you using (go version)?

go version go1.11beta2 windows/amd64

Thanks a lot!

@rsc

[mvdan]

Could you clarify what is the purpose of separating the data into two files? So that developers don't commit temporary replace directives by mistake? I'd imagine that there could be tooling to help with that. For example, a tool to add and undo temporary directives, or having CI fail on unexpected replace directives.

I'm also worried about adding more special Go module files, even if they are in theory not to be committed into git.

[flibustenet]

It's dangerous as you can commit files that use something else than what said in go.mod.
With replace in go.mod you see immediately (just before commit) that you will commit with a dependency fork.

[bcmills]

when [developing] multiple packages at the same time

We certainly do need to address that use-case somehow, but splitting up the module definitions doesn't seem like an appropriate solution: ideally I don't want individual developers to have to edit the module definition to do that.

[cosban]

I fall into the use case of "building multiple packages concurrently" as well.
I need to be able to, without being forced to commit code that is not fully vetted, build and test our modifications.
There are multiple solutions to this problem, some more elegant than others.

• Having a release modifier or command (https://github.com/golang/go/issues/26420) that ignores local copies would be a simple (from the user perspective) solution and would not require additional modifications to the go.mod files.
• The solution proposed in this issue could also work. It could even be combined with the first proposal. @bcmills, when you say you "don't want individual developers to have to edit the module definition to do that", are you referring to them manualy modifying go.mod files? If not, could you please clarify your position a little more so I may understand it better? This could be resolved by allowing the developer to specify that the library is one which will be modified locally as well using something like go get -u <package@version> -local <relative/path/to/package>
• Allowing users to install local, uncommitted versions could also work for this, but probably isn't the most elegant solution. This is, users could go install their current copy of the package they are developing, and the module system would store it in some sort of development version folder. In this method, version v0.0.0-20180526204631-c40f734b202a could be the latest commit, but v0.0.0-20180526204631-c40f734b202a-devel could be the latest local copy. If the local copy is not found, the module system would be able to revert back to the last commit when using the go get command.

I believe that the first option is the best, but I'm interested in reading what direction you guys are learning toward in the interim.

[wsc1]

Hi all,

I'm not sure that my approach will help others, but in case it does, (taken from go-nuts)

There is a bare bones workaround without local go.mod nor replacements.

Suppose you need to work on module A and module B simultaneously (hopefully you set things up to avoid this, but hey, it's inevitable)

If you only need to edit one package at a time in at least one of A and B, then you don't need
to muck with local go.mod or versions tags or replacements or anything.

Let B be the module which has only one package to edit, and p the package.

Copy p somewhere in A, rename import path to p using existing go imports/ide,
edit both, build, test, etc. A's go.mod will be a little crazy and out of whack.
So what. Copy p back to where it belongs in B, re-run go imports, build/test or go mod tidy
Now commit as you like the two.

In terms of commands and tools, it's very simple: no tools, no replacements, no go.mod.local, no replacements which shouldn't be pushed/commited, no workflow restrictions. Simple "cp -r" and use with editor that understands go imports works fine.

Now, thanks to modules, if your edits correspond to releases, it doesn't even matter in what order
you commit/push the changes after the two modules have been edited.

Yes, it's still a PITA. For me it's much less painful than thinking about workflows and replacements.
But for me it also doesn't happen a lot.

That said, I do think it would be nice to have the go command do for a set of modules, say go build ./...
with ./ containing the modules in question. that would allow updates of all version in one shot for example.

[ohir]

bcmills> "ideally I don't want individual developers to have to edit the module definition to do that."

Pondering on how to "merge" go.mod with the hypothetical go.mod.local I came to the point where only simplest solution stayed viable: add a -local knob. Then, if -local flag is present, tools will use go.mod.local instead of go.mod.

I know that knobs breeding is unpopular among core team but consider it, please.

[benhalstead]

go.mod.local or similar would also help our use case. We have developers working on service 'A' which relies on library 'B'.

Our developers will make changes to both the service and the library and so have a replace in the service's go.mod file to use their local version of the library code instead.

The problem is that each developer will have checked the library out onto their filesystem with a different path so each of them need a different path in the 'replace' statement, meaning version control of go.mod is messy.

[zshbleaker]

Local dependency is useful when importing from a self-hosted system which does not support go-get(e.g. GitHub Enterprise).

[bradleyjames]

+1 to go.mod.local or just a separate file based approach. It would provide a clean logical separation and file based will be simple to ignore via .gitignore. Also overriding the mod file location (aka opt-in) during development would provide a sanity check in mitigating the risk of pushing to production.

[bradleyjames]

Local dependency is useful when importing from a self-hosted system which does not support go-get(e.g. GitHub Enterprise).

We’re using GitHub Enterprise which works for us. What specific issues are you having?

[pwaller]

I was pointed here by @bcmills on slack. Thought it was worth mentioning my use case since I can't see it obviously stated here.

I want to be able to do a git bisect with a replace directive in effect. For this to be effective, I want to avoid having a dirty working tree, so editing the go.mod isn't great for this.

I have found myself a few times wanting a replace directive without contaminating the git status output with changes, for example whilst just doing some development whilst working on multiple modules simultaneously.

[SamWhited]

I was pointed here by @bcmills on slack. Thought it was worth mentioning my use case since I can't see it obviously stated here.

Not everyone can see that link; if it's important, please copy any context needed into this discussion.

[bcmills]

@SamWhited, the discussion on Slack was literally just a link to this issue.
(I mostly treat Slack as a way to quickly route people to the relevant public issues.)

[thepudds]

As far I as understand the main proposal in this issue, a go.mod.local file residing in a module would be allowed to contain replace and exclude directives that apply when inside that module (with part of the benefit being that that go.mod.local file would not be checked in).

If so, a go.mod.local file containing replace statements would be fairly simple to setup if you had, say, two modules foo and bar that you needed to work on simultaneously. If foo depends on bar, it might be as simple as creating a single go.mod.local file inside foo that reads:

replace bar => ../bar

However, I think explicitly specifying each replace in go.mod.local files would be fairly challenging to manually setup if instead you have something like 6 interdependent modules with up to 6 go.mod.local files, which can translate to much more than 6 total replace statements, especially as you might sometimes want to work in module foo but sometimes work in module bar, and foo might depend on bar and baz while bar depends on baz and qux, etc. Dependency cycles between modules add to the total possible count of needed replace statements.

One could imagine the go tooling being able to take a set of modules that are laid down in a shared file tree containing M modules and automatically create the N needed replace statements to write to M go.mod.local files.

Even with help from the tooling, that seems a bit complex, and could get stale or otherwise allow for more user error.

Three alternatives:

1. Rather than having M go.mod.local files to describe the relationship between M modules, instead have a single go.mod.local file that is in a shared parent in the file tree of the M modules. That single go.mod.local file could have all the replace statements in that single location. This would at least centralize the information, while keeping it explicit. It would also mean there are fewer replace statements overall in typical cases (e.g., there could be a single replace baz => ./baz if the paths are relative to the location of the single go.mod.local file, rather than needing to repeat variations of replace baz => ../baz if there are multiple go.mod.local files). This single go.mod.local file could be created programmatically by the go tool (e.g., creating the single go.mod.local file based on running against a shared file tree containing M modules and populating the single go.mod.local file with the various replace statements needed for each of those modules to find each other's local copy).

1. A different approach would be to still have a single go.mod.local file that is in a shared parent in the file tree of the M modules, but rather than having N replace statements, the presence the file itself would be sufficient to imply that the M modules should be used locally, and the go.mod.local file itself could be empty. This is the same end result as if the go.mod.local file was created automatically with the N replace statements in 1., but in this case the replace statements would be implicit. Perhaps the file is called go.mod.local to mean "please use the local go.mod files", or perhaps the sentinel file is called something like go.workspace or go.space or some other better name.

2. Rather than treating 1. vs. 2. as mutually exclusive, both could be supported options, where the behavior in 1. could be triggered by explicit replace statements like replace foo => ./foo, and the behavior in 2. could be triggered by some other statement inside the file (perhaps replace all, or replace relative, or similar).

Perhaps those are not good approaches, but of those three options, option 2. makes the most sense to me personally. It would help with a particular set of pain points around working with multiple modules simultaneously, but would also allow people to more easily organize their code around projects, and could also retain some of the benefits of how at least some gophers enjoy the way GOPATH has encouraged them to organize their code. Someone could for example create ~/go/src/go.workspace, but also ~/project1/go.workspace and ~/project2/go.workspace.

[kortschak]

The second option also has the nice property that even if the go.mod.local/go.worskspace(/go.local?) were accidentally committed and pushed upstream, it would not leak internal information to the outside world. This is a concern that has driven some people to want a non-go.mod approach; the issue of accidentally pushing local filepaths to the outside world.

[akyoto]

As an author of ~30 Go packages where testing 1 new package version equals 5-6 replace statements in the current system, I highly appreciate the idea of having a single global file to indicate all your local copies, as @thepudds mentioned. This single file could then be used for any go build in any directory.

Advantages:

• Can't accidentally commit replace statements in go.mod anymore.
• It takes less time to test my new version in the other 29 local packages that depend on it.
• It opens up the possibility for a go link ~/projects/... style command where link finds all your git repositories inside ~/projects that have a go.mod file and automatically adds a global replace entry for each local copy.

[gunnsth]

As a developer in a fairly large library I frequently work with smaller applications that use this library. To be able to work with the local version of the library requires setting the replace directive for each small application. Setting a global replace directive would make this a lot easier. Otherwise it is definitely a regression from the GOPATH in terms of convenience.

It seems like a global replace directive could either be specified in a global file or through an environment variable, e.g. GOMODREPLACE=old[@v]=new[@v],... with multiple replacements comma separated.

[owais]

I use the replace directive quite a lot but this request scares me a little (at least the way it was requested originally). I fear people might forget that their local builds are using the .local file and end up testing against the wrong dependencies. Of course a good process with CI will catch such issues but IMO relying on external processes for correctness from the perspective of in-built Go tooling is wrong.

Imagine having to submit a very critical hotfix for something that needs to bypass the regular process. One might forget about the existence of a .local file or out of muscle memory use -local flag to build the program, test it locally and ship the code to prod. The remote build pipeline would not use the -local flag and would result in publishing an expected build. There can be many other such cases. For example, while reviewing a PR, I might want to checkout a branch to build and test the behavior locally but accidentally end up testing the branch with a local copy of the dependency. People learning Go might also find it frustrating to debug issues because of an old .local file lying around.

[myquant]

The problem of my team: say we have 3 coders developing same module, their go.mod might be as following:

coder1
replace my/program/module => c:/module

coder2
replace my/program/module => d:/module

coder3
replace my/program/module => e:/module

One guy might accidentally commit his go.mod to git, then others git pull and build...bomb! Guess go.mod.local is a nice idea to resolve this problem. It can be added to .gitignore.

[rwxrob]

I have regularly been struggling with this same thing and really do not like having to switch back and forth between having replace and not because I frequently do wip commits and every single one would require swapping it out. I really, really like the idea of something global. Now that $GOPATH/env is a thing perhaps that is the perfect place for it.

[jayconrod]

Closing in favor of #34506, which adds a -modfile flag to Go 1.14. It's not exactly what's proposed in this issue, but it's close, and it solves a lot of other problems.

For this workflow I'd recommend:

• Copy go.mod to go.local.mod. Add go.local.mod to .gitignore (or equivalent for your workspace).
• Run go env -w GOFLAGS=-modfile=go.local.mod. This tells the go command to use that file by default.
• Any any replace and exclude directives or other local edits.
• Before submitting and in CI, make sure to test without the local file: go env -u GOFLAGS or just -modfile=. Probably also go mod tidy.

[AndreKR]

@jayconrod I don't think #34506 is a replacement (no pun intended) for this one. From that issue:

Note however, that these local files are used _instead_ of the regular files, not _in addition to_, so some synchronization might be required.

So the modfile setting is not at all helpful for the scenario where you want the dependencies from the repository (and send any changes to the repository) and just some local replacements.

[jayconrod]

@AndreKR I understand it's not quite the same. With this, you can work out of the same code base without modifying the original go.mod.

We discussed whether -modfile should mean instead of or in addition to, I think on #34506 or one of the last few golang-tools calls. It seemed like we could serve more use cases by keeping the semantics simple and not touching the original go.mod file.

[bcmills]

I agree with @AndreKR: I don't think #34506 addresses the use-case here, which is to make a set of replace directives easy to revert (or to .gitignore) while still updating the require directives in the main go.mod.

[myitcv]

Agree, @bcmills.

Also to point out that

Run go env -w GOFLAGS=-modfile=go.local.mod. This tells the go command to use that file by default.

is a global environment change, whereas what is being discussed here is very much a project-local effect.

[pdf]

Allowing modfile to be specified multiple times, or accept multiple filenames, merge the contents, and fail silently if one of the files doesn't exist would solve this I think.

ETA: Though, it doesn't really standardise the mechanism (every developer/project may choose a different filename to .gitignore), and hence requires much more error-prone local modifications in a team or open source project.

[p554157atch]

Hello, please check this proposal as well: https://github.com/golang/go/issues/40933

We can combine two different mechanisms,
using a go.local.mod plus importing replace definitions into go.local.mod from an existing local file.

For example go.local.mod for 20 different packages can have:

module hello
go 1.14
replace_definitons /home/coder/replace_definitions.mod

If we made changes to /home/coder/replace_definitions.mod, then changes will be applied to all the 20 go.local.mod files that import the replace_definitions.mod file.
go mod tidy takes care of un-used definitions.

This way git repository won't have unnecessary commits, plus we will be able to specify replace definitions in one local file and changes will be reflected in all local modules that import this file.

Making simultaneous edits on local modules that require constant updated to replace definitions will become much easier if we follow this method.

Let me know what you think.

[sheen4n]

+1 for this feature.

Really helps developers develop locally and prevent unknowingly committing replace code into the remote repo.

We can add that to .gitignore.

[mvdan]

A reminder about https://github.com/golang/go/wiki/NoPlusOne - if you want to add your +1, just give a thumbs up reaction.

[pipe01]

An implementation I think could come in handy would be to look up on every parent directory for a replace.mod file, which contains replace and perhaps other directives.