Go: doc: explain how to debug performance problems in Go programs

[rsc]

We should probably have a doc explaining how to debug performance problems in Go programs. There is the profiling blog post, but a larger survey of the field might be good.

/cc @aclements @alandonovan

[mvdan]

Would this be about both pprof and trace?

[dgryski]

Once upon a time I started https://github.com/dgryski/go-perfbook to collect resources for this.

[kevinburke]

Happy to contribute https://kev.inburke.com/kevin/real-life-go-benchmarking/ if it's at all helpful.

[smasher164]

I constantly reference examples from this talk: https://github.com/bradfitz/talk-yapc-asia-2015/blob/master/talk.md.

[dgryski]

Partially addressed by https://tip.golang.org/doc/diagnostics.html ?

[pciet]

@dgryski this could use a section on benchmarks. The first Google result for me is @davecheney https://dave.cheney.net/2013/06/30/how-to-write-benchmarks-in-go which I used before and @kevinburke has an article above.

Software testing methodologies help make isolated performance test cases too where those tools become more useful. The right setup is important to get good results. For example, a web server profile will be inaccurate when a test client is running on the same computer without limiting max procs, and virtual servers may compete with other instances on the same machine no matter what you do.

[MikeSpreitzer]

https://tip.golang.org/doc/diagnostics.html#profiling is the first place I found that lists all six of the built-in profiles! It would be great if that complete list were in a more central place. It would also be great to have a little more expansive definition of each of those profiles. For example, it wouldn't hurt to emphasize that the "block" profile includes time waiting on empty channels, selects with nothing to do, and so on; that is, time found here is not necessarily a problem, some if it is a symptom of a system eagerly waiting for something to do.