Go: cmd/go: go build should be able to write multiple executables

[kardianos]

Right now there is no easy cross platform way to output multiple artifacts from multiple packages into a single directory.

Right now you can do env GOBIN=bin/ go install cmd/... on many platforms, but not when cross compiling or on windows.

Suggesting making go build -o bin/ pkg1 pkg2 build both packages into the "bin" folder.

@robpike thinks this would make the meaning different based on if it is a file or directory and suggests using a different flag like "-odir". That seems fine however I'm not sure how -o would interact with -odir. If they were forbidden from being set at the same time, I would say they should be the same flag, namely -o.

Reference thread: https://groups.google.com/forum/#!topic/golang-dev/a1wvIZdnic0

[robpike]

Flag -o would name the file and -odir would name the directory. Seems clear to me. If both are set one could complain if -o is a rooted path.

Not sure this is worth doing at all, but it seems easy to design and understand.

[kardianos]

@robpike That's a fair design. I think it would be useful to have, but I'm unsure if it would pay for a new flag.

I'm curious if others would also find it useful or if it is just me :).

[lmb]

FWIW I have been looking for something like this as well. For company policy reasons I need to build a .deb on a build box. There are several binaries, which I currently build individually and then package up. Not a terrible nuisance, but could probably save me some compile time since the binaries share most code.

[minux]

if you use go install instead of go build, then it doesn't matter if you're
building one at time or building everything together. The shared code will
only be built once.

I think a lot of problems come from the overuse of "go build" where "go
install" should be used instead.

[kardianos]

@minux There are times when I want the functionality of "go install", but placed into a local folder.

• project specific tools that I don't want to install globally, but are needed for a project.
• project specific binaries where due to the nature of resources or other design, should be built and ran locally.

The above may come up more often for me then others as I work on a contract basis and juggle different customers accounts. I use a single GOPATH, but often architect projects to run within a non-global context. (Sorry, that might not make complete sense).

[rsc]

This is so close to working. I'm not sure why it doesn't. We should fix it for Go 1.12.
Clearly -o can't be used with multiple binary targets, but the implicit "write to the current directory" should work, and instead it does nothing.

$ cd /tmp/zzz
$ go build cmd/doc cmd/gofmt
$ ls -l
$ go build cmd/doc
$ go build cmd/gofmt
$ ls -l
total 14808
-rwxr-xr-x  1 rsc  eng  4266200 Aug  9 15:00 doc
-rwxr-xr-x  1 rsc  eng  3310440 Aug  9 15:00 gofmt
$ 

[robpike]

You could imagine -o or a variant specifying a directory rather than or as well as a file.

[pd93]

This isn't exactly a necessity, but it would be a huge convenience in my workflow - not to mention it would bring it in line with the install command. I never looked this up in the go docs. I just assumed it would work, but it doesn't 😞

I have a few projects with numerous small tools/utils and it would greatly simplify my testing if I could build them all at once to my cwd. Currently I have a bash script that runs consecutive go build ./path/to/tool commands which is less than ideal.

[neilpa]

Same here, we do most of our local dev on mac but than build and package for our standard linux image so have have similar problems since the GOBIN=... go install ... trick doesn't work. I've hacked together Makefiles to support cross-compiling with explicit go build -o ... for each target binary as a workaround. You could probably take this further with wildcard magic over the cmd/ directory but I decided I like readable makefiles.

This works for now but would prefer a go build or go install solution.

[gopherbot]

Change https://golang.org/cl/143139 mentions this issue: cmd/go: make go build write multiple executables

[rasky]

One side effect of this CL is that it breaks the venerable "go build -a std cmd" for Go maintainers (when run from $GOROOT/src, it will complain that "go" is already a directory; when run from any different directory, it will drop many binaries in it). The current workaround is "go build -o /dev/null -a std cmd".

Comments? /cc @rsc @robpike

[kardianos]

There is a bit of discussion on the proposed CL, as well s the comment that the CL as implemented breaks go build -a std cmd. I would also comment the current CL breaks a personal use case I sometimes run go build ./cmd/... just to make sure everything my cmds rely on still build as a quick smoke test while in development.

I would make the following proposal:

Make go build without -o and multiple packages specified behave as before. Only output binaries from go build with multiple packages are specified when -o is specified. This prevents go build -a std cmd from breaking, preserves all existing behavior, while adding useful method such as go build -o ./bin/ ./cmd/.... Either that, or we could teach go install a -o switch that points to the output directory.

---

In the CL there are comments that suggest either:
go build $(go list -f '{{if eq .Name "main"}}{{.ImportPath}}{{end}}' ./...)
or for pkg in $(go list -f '{{if eq .Name "main"}}{{.ImportPath}}{{end}}' ./...); go build $pkg; end . Yeah, that's missing the point. I can get around this behavior, On *nix I can have a couple of helper scripts and on Windows another helper scripts, or I could make it part of my build system for individual projects. Again, that's kinda missing the point.

Sometimes it would be really useful to install a bunch of commands into a specific directory. Most of the semantics are already defined.

[rasky]

Another option is to add an explicit -dryrun option (or similar name) to avoid writing binaries; similar to the undocumented -o /dev/null but more explicit. This would help those who want to “just test it compiles”, in an explicit way.

Then, building multiple executables would just work, by default. This would make the behavior of go build more consistent; by default, it would always build executables, rather than changing behavior depending on how many packages were specified. It would be easier to explain and document, and less surprising for beginners.

[rasky]

Ping, can we get this unstuck? It just needs a decision on a small detail, we all want the main feature in.
/cc @andybons

[bcmills]

go build ./... is useful today, and its meaning should not change. We can either take the -o approach that @kardianos suggests, or require an explicit list of targets per my comment on the CL. I'll talk to @rsc to decide which.

[bcmills]

@kardianos In module mode, it may be useful to issue a go get command outside of any module (#24250) with the path to a binary plus some specific versions of dependencies to upgrade (or downgrade). In those cases, it seems redundant to require the user to pass -o $GOBIN or -o $GOPATH/bin explicitly just to get the binary to install in the usual place.

[kardianos]

@bcmills Thank you for working on defining how go get works outside of any module. I think I may be missing something. Here's how I could envision the go command working in module mode:

## Outside of a module

# In all cases, if $GOBIN is not defined, it assumes $GOBIN is $HOME/bin

# Outside a module. Fetches cmd1 and cmd2, compiles it, and installs it in $GOBIN.
go get github.com/user/repo/cmd/...

# (maybe) Outside a module. Fetches cmd1 and cmd2 installs in ./bin.
go get -o ./bin github.com/user/repo/cmd/...

# Outside a module. Fails.
go build github.com/user/repo/cmd/...

# (maybe) Outside a module. Builds cmd1 and cmd1 and outputs to ./bin
go build -o ./bin /home/user1/code/moduleroot/cmd/...

# Outside a module. Fails.
go install github.com/user/repo/cmd/...

# (maybe) Outside a module. Builds cmd1 and cmd1 and outputs to $GOBIN.
go install /home/user1/code/moduleroot/cmd/...

## Inside a module.

# Inside a module. Fetches cmd1 and cmd2, constrains get to local go.mod file, installs in $GOBIN.
go get github.com/user/repo/cmd/...

# (maybe) Inside a module. Fetches cmd1 and cmd2, constrains get to local go.mod file, installs in ./bin.
go get -o ./bin github.com/user/repo/cmd/...

# Inside a module, builds the local commands cmd1 and cmd2, and discards the results
go build ./cmd/...

# Inside a module. Installs cmd1 and cmd2 in ./bin.
go build -o ./bin ./cmd/...

# (maybe) Inside a module. Installs cmd1 in ./bin (assuming bin is a folder).
# This would be useful in reusing the same script for windows and linux.
go build -o ./bin ./cmd/cmd1

# Inside a module, builds the local commands cmd1 and cmd2, installs in $GOBIN.
go install ./cmd/...

Can you explain why you would need to pass in -o $GOBIN to go get with this change?

[rsc]

We may all want this "in", but it's too late for Go 1.12.

[rasky]

This is very sad.

It is the second release that this feature is postponed because the community doesn't get answers from the Go team regarding it in time for the release cycle (see also #23616 where I didn't get an answer for months). I have even sent a CL well in time, to try to unblock things.

How can we break this cycle? Is there a better channel than this? The community can't do development for things we care about if the Go team doesn't even prioritize answering to questions. /cc @andybons @bradfitz @bcmills @FiloSottile @cassandraoid

[ianlancetaylor]

I'm sorry this one got stuck. As you know there has been a great deal of work on cmd/go last cycle and this one, and that has made looking at CLs for cmd/go even slower than usual. Also it seems that this feature interacts with modules, which makes it even worse since modules is in a steady state of flux.

[kardianos]

@bcmills Let me know if there is something (proposal or code) that could make this move in 1.13.

Another use case I encountered today is building a main package in a different dir, without needing to worry about if I should append ".exe" at the end go build -o ./bin/ ./cmd/thecmd.

[DisposaBoy]

Another use case I encountered today is building a main package in a different dir, without needing to worry about if I should append ".exe" at the end go build -o ./bin/ ./cmd/thecmd.

@kardianos For that specific use-case, you might find go env GOEXE helpful.

~ $ env GOOS=linux go env GOEXE

~ $ env GOOS=windows go env GOEXE
.exe

[kardianos]

I am aware of that. But again, I'm talking about slightly nicer default behavior.

[rsc]

To move forward with this, let's make -o with an argument that is a directory write a new executable to that directory. Note that the key thing is that the argument _is_ a directory, not that it syntactically ends in slash. That is, go build -o becomes more like mv, not rsync.

Separately, go build with multiple command targets should be able to write multiple binaries to the current directory.

[gopherbot]

Change https://golang.org/cl/167679 mentions this issue: cmd/go: allow -o to point to a folder that writes multiple execs

[FiloSottile]

It looks like only -o behavior was changed, but go build doesn't yet build multiple packages to the current directory by default, so reopening this.

Separately, go build with multiple command targets should be able to write multiple binaries to the current directory.

[gopherbot]

Change https://golang.org/cl/190839 mentions this issue: cmd/go: fix "go help build -o" docs

[rasky]

For reference, my CL 143139 did work also without -o. Not sure why it was redone from scratch.

[kardianos]

Sorry for the scattered discussion. CL comment https://go-review.googlesource.com/c/go/+/143139/4#message-fb8adc4368e18ba081fe2fe221448bce17d990bc showed that go build ./... that outputs executables (no -o) breaks things both in current std lib tests, and I'm sure in many tools that rely on that to discard output. Requiring -o . to output to the current directory keeps the current command line compatibility, while still gaining the functionality.

[FiloSottile]

Makes sense, thanks for closing the loop.

[gopherbot]

Change https://golang.org/cl/190907 mentions this issue: doc/go1.13: mention '-o <directory>' support for 'go build'

[odiferousmint]

Is go build -o build/ ./cmd/foo ./cmd/bar main.go going to be supported? Workaround is:

$ go build -o build/
$ go build -o build/ ./cmd/foo ./cmd/bar

Note that it does not work without ./. Probably intentional.

[jayconrod]

@odiferousmint If a package is specified as a list of .go files on the command line, all the positional arguments must be .go files. So main.go can't be mixed with ./cmd/foo ./cmd/bar in the same command.

[Gregory-Ledray]

I just got 1.14 and I'm confused. This works great with regular GOOS and GOARCH on non-empty directories (nice work!):

gledr@LAPTOP-AH6IAHBO MINGW64 ~/Polyapp_Apps/gocode/src/gitlab.com/polyapp-open-source/poly/testdata (CI-expansion)
$ go list ./...
gitlab.com/polyapp-open-source/poly/testdata/cmd
gitlab.com/polyapp-open-source/poly/testdata/cmd/hello

gledr@LAPTOP-AH6IAHBO MINGW64 ~/Polyapp_Apps/gocode/src/gitlab.com/polyapp-open-source/poly/testdata (CI-expansion)
$ ls ./cmd
hello  main.go  wasm1  wasm2  wasm3

gledr@LAPTOP-AH6IAHBO MINGW64 ~/Polyapp_Apps/gocode/src/gitlab.com/polyapp-open-source/poly/testdata (CI-expansion)
$ go build -o ./cmd ./...

gledr@LAPTOP-AH6IAHBO MINGW64 ~/Polyapp_Apps/gocode/src/gitlab.com/polyapp-open-source/poly/testdata (CI-expansion)
$ ls ./cmd
cmd.exe  hello  hello.exe  main.go  wasm1  wasm2  wasm3

But doesn't work with GOOS=js GOARCH=wasm:

gledr@LAPTOP-AH6IAHBO MINGW64 ~/Polyapp_Apps/gocode/src/gitlab.com/polyapp-open-source/poly/testdata (CI-expansion)
$ GOOS=js GOARCH=wasm go list ./...
gitlab.com/polyapp-open-source/poly/testdata/cmd
gitlab.com/polyapp-open-source/poly/testdata/cmd/hello
gitlab.com/polyapp-open-source/poly/testdata/cmd/wasm1
gitlab.com/polyapp-open-source/poly/testdata/cmd/wasm2
gitlab.com/polyapp-open-source/poly/testdata/cmd/wasm3

gledr@LAPTOP-AH6IAHBO MINGW64 ~/Polyapp_Apps/gocode/src/gitlab.com/polyapp-open-source/poly/testdata (CI-expansion)
$ ls ./cmd
hello  main.go  wasm1  wasm2  wasm3

gledr@LAPTOP-AH6IAHBO MINGW64 ~/Polyapp_Apps/gocode/src/gitlab.com/polyapp-open-source/poly/testdata (CI-expansion)
$ GOOS=js GOARCH=wasm go build -o ./cmd ./...
go build gitlab.com/polyapp-open-source/poly/testdata/cmd/wasm1: build output "cmd\\wasm1" already exists and is a directory
go build gitlab.com/polyapp-open-source/poly/testdata/cmd/wasm2: build output "cmd\\wasm2" already exists and is a directory
go build gitlab.com/polyapp-open-source/poly/testdata/cmd/wasm3: build output "cmd\\wasm3" already exists and is a directory
go build gitlab.com/polyapp-open-source/poly/testdata/cmd/hello: build output "cmd\\hello" already exists and is a directory

It works great with GOOS=js GOARCH=wasm if I make a new directory:

gledr@LAPTOP-AH6IAHBO MINGW64 ~/Polyapp_Apps/gocode/src/gitlab.com/polyapp-open-source/poly/testdata
(CI-expansion)
$ ls ./cmd
hello  main.go  wasm1  wasm2  wasm3

gledr@LAPTOP-AH6IAHBO MINGW64 ~/Polyapp_Apps/gocode/src/gitlab.com/polyapp-open-source/poly/testdata
(CI-expansion)
$ ls ./bin
ls: cannot access './bin': No such file or directory

gledr@LAPTOP-AH6IAHBO MINGW64 ~/Polyapp_Apps/gocode/src/gitlab.com/polyapp-open-source/poly/testdata (CI-expansion)
$ GOOS=js GOARCH=wasm go list ./...
gitlab.com/polyapp-open-source/poly/testdata/cmd
gitlab.com/polyapp-open-source/poly/testdata/cmd/hello
gitlab.com/polyapp-open-source/poly/testdata/cmd/wasm1
gitlab.com/polyapp-open-source/poly/testdata/cmd/wasm2
gitlab.com/polyapp-open-source/poly/testdata/cmd/wasm3

gledr@LAPTOP-AH6IAHBO MINGW64 ~/Polyapp_Apps/gocode/src/gitlab.com/polyapp-open-source/poly/testdata
(CI-expansion)
$ mkdir ./bin

gledr@LAPTOP-AH6IAHBO MINGW64 ~/Polyapp_Apps/gocode/src/gitlab.com/polyapp-open-source/poly/testdata (CI-expansion)
$ GOOS=js GOARCH=wasm go build -o ./bin ./...

gledr@LAPTOP-AH6IAHBO MINGW64 ~/Polyapp_Apps/gocode/src/gitlab.com/polyapp-open-source/poly/testdata (CI-expansion)
$ ls ./bin
cmd  hello  wasm1  wasm2  wasm3

And it works great if I rebuild into the bin directory:

gledr@LAPTOP-AH6IAHBO MINGW64 ~/Polyapp_Apps/gocode/src/gitlab.com/polyapp-open-source/poly/testdata (CI-expansion)
$ GOOS=js GOARCH=wasm go build -o ./bin ./...

gledr@LAPTOP-AH6IAHBO MINGW64 ~/Polyapp_Apps/gocode/src/gitlab.com/polyapp-open-source/poly/testdata (CI-expansion)
$ rm ./bin/cmd ./bin/hello ./bin/wasm2

gledr@LAPTOP-AH6IAHBO MINGW64 ~/Polyapp_Apps/gocode/src/gitlab.com/polyapp-open-source/poly/testdata (CI-expansion)
$ GOOS=js GOARCH=wasm go build -o ./bin ./...

Is this intended to work with non-empty parent directories of the binaries? Why does it work with GOOS=windows GOARCH=amd64 but not GOOS=js GOARCH=wasm?

[bcmills]

@Gregory-Ledray, please open a new issue with steps to reproduce. Probably there is something special about wasm binaries that is interfering with the feature.