Go: cmd/go: go run pkg is significantly slower than running built binary

[myitcv]

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

go version devel +212c9479e3 Tue May 15 16:29:04 2018 +0000 linux/amd64

Does this issue reproduce with the latest release?

n/a: this relies on changes in tip.

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOCACHE="/home/myitcv/.cache/go-build"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/myitcv/gostuff"
GORACE=""
GOROOT="/home/myitcv/gos"
GOTMPDIR=""
GOTOOLDIR="/home/myitcv/gos/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build404056765=/tmp/go-build -gno-record-gcc-switches"

What did you do?

cd `mktemp -d`
export GOPATH=$PWD
mkdir -p src/example.com
cat <<EOD > src/example.com/main.go
package main

import (
        "fmt"
)

func main() {
        fmt.Println("Hello, world!")
}
EOD

cat <<EOD > src/example.com/main_test.go
package main_test

import (
        "fmt"
        "io/ioutil"
        "os"
        "os/exec"
        "strings"
        "testing"
)

const (
        testPkg = "example.com"
)

func BenchmarkGoRun(b *testing.B) {
        for n := 0; n < b.N; n++ {
                cmd := exec.Command("go", "run", testPkg)
                out, err := cmd.CombinedOutput()
                if err != nil {
                        panic(fmt.Errorf("failed to %v: %v\n%s", strings.Join(cmd.Args, " "), err, out))
                }
        }
}

func BenchmarkGoBuild(b *testing.B) {
        tf, err := ioutil.TempFile("", "")
        if err != nil {
                b.Fatalf("failed to create temp file: %v", err)
        }
        defer os.Remove(tf.Name())

        {
                cmd := exec.Command("go", "build", "-o", tf.Name(), testPkg)
                out, err := cmd.CombinedOutput()
                if err != nil {
                        b.Fatalf("%v failed: %v\n%s", strings.Join(cmd.Args, " "), err, out)
                }
        }

        b.ResetTimer()

        for n := 0; n < b.N; n++ {
                cmd := exec.Command(tf.Name())
                out, err := cmd.CombinedOutput()
                if err != nil {
                        b.Fatalf("%v failed: %v\n%s", strings.Join(cmd.Args, " "), err, out)
                }
        }
}
EOD

go test -test.bench . example.com

What did you expect to see?

The benchmark with go run example.com to be more comparable with the benchmark that runs a binary directly.

What did you see instead?

The go run example.com benchmark is ~180 times slower.

goos: linux
goarch: amd64
pkg: example.com
BenchmarkGoRun-8              10         163497281 ns/op
BenchmarkGoBuild-8          2000            893443 ns/op
PASS
ok      example.com     4.260s

I recall @rsc mentioning somewhere (can't recall where) that the result of go run pkg is not cached, which I think accounts for the difference above.

One of the major benefits of the go run pkg form is that it is possible to unambiguously identify the program in question. My particular use case is //go:generate directives, where it then becomes possible to calculate the go generate dependency graph.

Ideally I would like to replace all of my //go:generate abc directives with //go:generate example.com/p/abc, but the difference in speeds observed above makes this infeasible.

So I'm raising this as an issue to discuss whether it would be worth caching the output of go run pkg. I can't claim to understand any of the pros/cons here so would appreciate thoughts.

[champioj]

go run pkg is significantly slower than running built binary

It's normal for the first run, debatable for the next run.

Even if go run used the cache you would still have to read the source file to be sure that the cache is valid, no? Have you got any performance target to see if it's even possible?

[myitcv]

Even if go run used the cache you would still have to read the source file to be sure that the cache is valid, no?

Correct; that cost cannot be escaped. It is effectively (although not precisely) equivalent to go list -deps -json example.com.

My understanding is that the link step is the overhead I'm looking to eliminate by caching the result.

[agnivade]

Is this something which was added in tip ? Because the 1.10 release notes mention nothing about go run caching any output. Only go build, go install and go test is mentioned.

[myitcv]

@agnivade

Please see above:

Does this issue reproduce with the latest release?

n/a: this relies on changes in tip

[ianlancetaylor]

The problem is that the cache does not hold the results of a link action. When using go build, the go tool checks whether the output executable already exists, and, if it does exist, whether it is the correct executable for the inputs (by checking the build ID). If the executable exists and is correct, then the link step is fixed. When using go run this doesn't work, because go run always generates a temporary executable that never already exists. So the link action always has to be re-run.

It works this way because link actions are large and typical rebuilds do require changing the program and therefore linking again. We don't want to fill up the cache space with copies of programs that are going to exist somewhere else anyhow.

I added a NeedsFix label but as I think about it I'm increasingly inclined to close this as "working as expected." For the relatively unusual case of caching the results of go run, I suggest changing your procedure to go build -o /my/bin/x; /my/bin/x. That will let you hold your own cache. That seems a better choice than forcing everyone's cache to grow much larger for cases that will tend not to hit in practice.

[myitcv]

but as I think about it I'm increasingly inclined to close this as "working as expected."

...

That seems a better choice than forcing everyone's cache to grow much larger for cases that will tend not to hit in practice.

That sounds totally reasonable to me. Thanks for the explanation in any case @ianlancetaylor.

I'll update the tag to reflect "working as expected."

[myitcv]

@ianlancetaylor

The problem is that the cache does not hold the results of a link action.

As a quick follow up question, what does go install do that's different? Because it seems to have a "fast path" to do nothing if the target is up-to-date.

[ianlancetaylor]

go install looks at the installed binary, extracts the build ID, and compares it to the build ID that the go tool has generated from the input files. If they are the same, there is nothing to do. To put it a different way, go install in effect uses the previously installed binary, if any, as a cache.

[myitcv]

@ianlancetaylor thanks very much.

[rsc]

To be clear, while @ianlancetaylor explained the state of the world without expressing a preference on policy, I will express a preference on policy: we don't want to start caching binaries just so that people can "go run path/to/binary" instead of installing binaries. Installing binaries is good!

[myitcv]

Thanks @rsc. I think the the trick I've been missing all along is the use of GOBIN in combination with (v)go install. That serves my purposes entirely!