Async-profiler: Trace output with timestamps of each sample

[ceeaspb]

Observing the profile over time is not possible/easy at the moment. I take a few profiles during the course of a benchmark to get a feel for the profile at know phases of tests. Not great though if things go wrong outside of those time ranges.

JMC/JFR captures the data over time then you can filter to get the cpu cost for specific time ranges.
Similarly the new flamescope tool allows for browsing the profile over time via linux perf script output.

obviously only if it is efficient enough
It would be useful to have a sample timestamped output to import into flamescope.

perf stores its data in a binary format for later post processing.

[apangin]

Unlike perf, async-profiler does not store every sample by design. Instead, it aggregates samples in memory. This is done intentionally to avoid saving huge amount of data on disk. One of the biggest obstacles in using perf is that it can generate several gigabytes of data per minute for a large enough application.

I guess the feature you are looking for is continuous profiling. This can be achieved without timestamping each sample. @stIncMale proposed a script for continuous profiling. Please have a look at #71 and tell what you think about it.

[nitsanw]

I think we have several features rolled into one. I would like to have:

1. Per event recording - this indeed leads to the problem @apangin is describing, but sometimes the overhead is appropriate, and it is usually computable. There's no need for the profiling to generate GBs worth of data necessarily if the period is short, sampling is infrequent, or the storage format is efficient enough. Of these aspects only the latter is controlled by AP. A binary format for profiling can be efficient, e.g. representing each sample as 2 longs (you can go for less) and so on.
2. Flight recorder like rolling profile - supporting a profile for the last rolling 1 hr etc. This is achievable by chunking summary output at a granularity you are happy to roll at and dumping at that frequency + trimming the oldest entries (which is what I assume #71 does). This too can benefit from a more efficient serialization scheme.

I think both are good features, and share a common supporting feature:

1. Support for dense binary format serialization

[ceeaspb]

Yes this isn't #71 - It is recording the timestamp of every sample. An efficient indexed binary format would make sense if possible.
I don't think need the rolling/chunking/continuous currently but I agree it would be useful. I have an internal continuous profiling script for automated benchmarks similar to #71 which although useful is only turned on when needed due to the storage requirements.

[apangin]

Any suggestions on how to implement this in async-profiler? (Having in mind that it contradicts the initial design).

[apangin]

Implemented Flight Recorder compatible output in jfr branch.
In this mode all samples with timestamps are recorded to a file in a compact binary format, and the file can be read directly by Java Mission Control.
Hope this helps.

[ceeaspb]

@apangin sorry combination of holiday and been busy - this looks great I will try it shortly!