Go: proposal: runtime/pprof: NewCPUProfile + cpuProfile.Start to allow profile configuration

[rogerlucena]

I think SetCPUProfileRate is, albeit confusing, still functioning. It needs to be called in the right order with StartCPUProfile. Last time I looked (~a month ago), it probably prints some warnings, but it does change the rate.

The pprof package doesn't seem to have a way to set the profiling rate. Not sure if this is really a useful feature.

_Originally posted by @cherrymui in https://github.com/golang/go/issues/40094#issuecomment-655005554_

The pprof package indeed does not allow us to customize the CPU profiling rate at the moment. This was something critical for my application in https://github.com/google/badwolf/pull/155, since the default profiling rate of 100 was extremely low and did not allow me no see anything in my CPU profile.

To work around this, I had to make a call to runtime.SetCPUProfileRate(800) (with a rate of around 800 hz to see something meaningful and helpful in my CPU profile) before calling pprof.StartCPUProfile. Since there is a hard-coded call to runtime.SetCPUProfileRate(100) inside the pprof.StartCPUProfile, a warning is being printed to my terminal at the moment saying "runtime: cannot set cpu profile rate until previous profile has finished" as this call to runtime.SetCPUProfileRate(100) failed (the boolean cpuprof.on was already set to true in the previous call to runtime.SetCPUProfileRate(800)). This warning is annoying since it is misleading for someone who does not know the internals of pprof.StartCPUProfile, but the important here is that the CPU profiling is done with the first profiling rate that I manually set (800, as I wanted and as wanted from the user point of view).

Given this, what I am trying to say is that the public/exported method runtime.SetCPUProfileRate is still useful, and sometimes even critical for allowing us to use well pprof and to see something relevant on our profiles, depending on the application. I am not alone on this as you can see here (stack overflow) and here (google groups discussion). Even one principal software engineer (L8) I have been working with at Google found it useful when reviewing my PR https://github.com/google/badwolf/pull/155 above and copied the code to use it in one of his applications, on which setting a higher profiling rate was critical as well.

That said, I believe we could, from now on, follow three different paths:

1. The ideal would be for the pprof.StartCPUProfile method already receive as argument the CPU profile rate we want, allowing us to customize it, with this argument having its default value set to 100 if it was not specified by the function call. This would be the ideal, if we had support for default arguments in Go. But, we can achieve something similar if we had a similar method with signature pprof.StartCPUProfileWithSpecifiedRate(w io.Writer, hz int), in addition to the pprof.StartCPUProfile(w io.Writer) method we already have. This way, the runtime.SetCPUProfileRate method would not necessarily need to be an exported method anymore.

2. We could keep runtime.SetCPUProfileRate public/exported, but we would have to fix the problem with the misleading warning message "runtime: cannot set cpu profile rate until previous profile has finished" if we had already manually set the profiling rate before the call to pprof.StartCPUProfile. For this we could:

   2.1. Have an auxiliar boolean method runtime.isCPUProfileRateAlreadySet, and call it inside pprof.StartCPUProfile before the hard-coded call to runtime.SetCPUProfileRate(100) there, trying to set the profiling rate to 100 only if this rate was not already previously set. This way, we would not see any strange warning after a call to pprof.StartCPUProfile if we had already previously set the profiling rate manually. I like this solution because it makes minimal changes on your code, it does not change any signature and it resolves exactly the problem we are trying to solve here, while keeping the warning message for other calls to runtime.SetCPUProfileRate that were not from pprof.StartCPUProfile in the situation above.

   2.2. Make runtime.SetCPUProfileRate return an error in the case the profiling rate was already previously set, allowing the caller to handle it and decide what to do from then on (inside pprof.StartCPUProfile we could ignore this error for example). This would change the signature of runtime.SetCPUProfileRate.

3. We could move the SetCPUProfileRate public signature to the pprof package (with some changes as explained above to avoid the misleading warning message), and deprecate it / make it unexported inside the runtime package.

These were just some of the alternatives that I thought about for now. I would like to hear more from you on what you think about them and if you have any other suggestions for fixing what I explained above.

[gopherbot]

Change https://golang.org/cl/269437 mentions this issue: runtime/pprof: method StartCPUProfile adds a parameter to allow custom cpu rate.

[imxyb]

so we can add method StartCPUProfileWithSpecifiedRate for this issue? @ianlancetaylor

[ianlancetaylor]

I don't think that is the best name, but, yes, something along those lines could work.

[imxyb]

ok, let me think about it

[gopherbot]

Change https://golang.org/cl/269537 mentions this issue: runtime/pprof: method StartCPUProfile adds a parameter to allow custom cpu rate.

[ianlancetaylor]

I turned this issue into a proposal for the API change.

[hyangah]

@ianlancetaylor Thanks for turning this into a proposal.

The current proposal is specific to StartCPUProfileWithRate. If we decide to add an additional API, can we come up with a version that's flexible enough to be extensible in the future? As far as I remember we had a couple of API change requests before and most of them were rejected or put on hold because of the desire to avoid API changes.

1) PMU-based profiling: it seems one of the main blockers of the proposal https://github.com/golang/go/issues/36821 is currently the API change (in addition to the implementation details). This proposal was originally proposing pprof.StartCPUProfileWithConfig.
2) Symbolization: I once proposed an option to skip symbolization because the pprof tool may have a way to perform additional symbolization out of band.
3) Different policy for profiling rate setup: the example issue @rhysh reported (https://github.com/golang/go/issues/35057) made me wonder if we may end up with a completely different cpu profiling policies in the future that requires beyond a simple hz adjustment. For example, some users may want to try per-thread timer (https://github.com/golang/go/issues/14434) if the runtime supports. Some users may want to adjust the profile rate based on the GOMAXPROCS (https://github.com/golang/go/issues/35057).

[rsc]

@hyangah It sounds like you are suggesting some kind of

cpu := pprof.NewCPUProfile()
... configure prof ...
cpu.Start()

or even

cpu := new(pprof.CPUProfile)
...
cpu.Start()

?

[rsc]

It sounds like maybe the interface in the previous message is the answer if we want to solve this problem.
But do we need to solve this problem? How often do people need profile rates other than 100 Hz?
The full stack trace makes the profile event a bit expensive, and I wouldn't want people to set the rate very high and then complain about the overhead.

[hyangah]

@rsc Choosing the right profiling rate is hard. There are many theories and different opinion around choosing the right value - http://www.brendangregg.com/blog/2014-06-22/perf-cpu-sample.html is talking about a lockstep sampling issue, #35057 is discussing scaling the frequency based on the system's capability because 100hz is sometimes too high, and @rogerlucena wants higher than 100hz (aside from whether 800 hz is reasonable or not). It looks to me it's a parameter people still want to experiment and tune.

[rogerlucena]

But do we need to solve this problem? How often do people need profile rates other than 100 Hz?

@rsc For my application it was crucial to set a sampling rate higher than 100 Hz to see something useful in the profiling output, as I explained and detailed in the Issue description above. Apart from that, in this description I also referenced other people for which customizing the sampling rate can still be useful: stack overflow and google groups discussion.

[rsc]

OK, well then it sounds like the API in my earlier comment, which introduces the idea of a configurable-because-not-yet-running project, would be good for rate and maybe also leave open a way to handle things like other kinds of performance counters.

https://github.com/golang/go/issues/42502#issuecomment-729878103

Do I have that right? I will retitle this and see what people think.

[rogerlucena]

Do I have that right? I will retitle this and see what people think.

@rsc lgtm.

[rsc]

Based on the discussion above, this seems like a likely accept.
(NewCPUProfile + CPUProfile.SetRate + CPUProfile.Start).