Tidb: Call For Participation: add memory trace for all the AggFuncs

[b41sh]

Description

This issue is used to track the memory usage of AggFunc in HashAggExec as discussed in #14705 and #14103.
We need to implement the method of the AggFunc interface and return memDelta to identify memory usage of each function.

type AggFunc interface {
    AllocPartialResult() (pr PartialResult, memDelta int64)
    UpdatePartialResult(sctx sessionctx.Context, rowsInGroup []chunk.Row, pr PartialResult) (memDelta int64, err error)
    MergePartialResult(sctx sessionctx.Context, src, dst PartialResult) (memDelta int64, err error)
}

All aggregation functions are shown below, you can choose the ones you are interested in.
Feel free to join us in the #sig-exec to discuss.

• [x] avg agg functions @b41sh 18901
• [x] [bit agg functions](https://github.com/pingcap/tidb/issues/19733) @Lvnszn 19808
• [x] [count agg functions](https://github.com/pingcap/tidb/issues/19734) @SailerNote 19770
• [x] count distinct agg functions @b41sh 18901
• [x] [cume distinct agg functions](https://github.com/pingcap/tidb/issues/19735) @YangKian 19832
• [x] [first row agg functions](https://github.com/pingcap/tidb/issues/19736) @Howie66 19757
• [x] [group concat agg functions](https://github.com/pingcap/tidb/issues/19737) @William0423 20153
• [x] [json object agg functions](https://github.com/pingcap/tidb/issues/19738) @TszKitLo40 https://github.com/pingcap/tidb/pull/19819
• [x] [lead lag agg functions](https://github.com/pingcap/tidb/issues/19739) @blueseason #19966
• [x] [max min agg functions](https://github.com/pingcap/tidb/issues/19740) @xiaodong-ji 19783
• [x] [ntile agg functions](https://github.com/pingcap/tidb/issues/19741) @Howie66 19756
• [x] [percent rank agg functions](https://github.com/pingcap/tidb/issues/19742) @Patrick0308 19852
• [x] [rank agg functions](https://github.com/pingcap/tidb/issues/19743) @TszKitLo40 19766
• [x] sum agg functions @mmyj https://github.com/pingcap/tidb/pull/19376
• [x] [value agg functions](https://github.com/pingcap/tidb/issues/19744) @Hunsmore https://github.com/pingcap/tidb/pull/19811
• [x] [var pop agg functions](https://github.com/pingcap/tidb/issues/19745) @asiafrank 19755
• [x] [row number agg functions](https://github.com/pingcap/tidb/issues/19746) @asiafrank 19752

SIG slack channel

sig-exec

Score

• 600

Mentor

• @b41sh

Recommended Skills

• SQL Optimization
• Golang Profiling
• Code Refactoring

[ekexium]

/label sig/execution

[mmyj]

pick sum agg functions

[mmyj]

pick value agg functions

[SailerNote]

/pick-up count agg functions

[ti-challenge-bot[bot]]

Pick up success.

[ti-challenge-bot[bot]]

This issue os already picked by SailerNote.

[ti-challenge-bot[bot]]

This issue os already picked by SailerNote.

[ti-challenge-bot[bot]]

This issue os already picked by SailerNote.

[ti-challenge-bot[bot]]

This issue os already picked by SailerNote.

[ti-challenge-bot[bot]]

This issue os already picked by SailerNote.

[xiaodong-ji]

pick max min agg functions

[SailerNote]

pick count agg functions

[cloudhuang]

pick group concat agg functions

[ti-challenge-bot[bot]]

This issue os already picked by SailerNote.

[asiafrank]

/pick-up row number agg functions

[ti-challenge-bot[bot]]

This issue os already picked by SailerNote.

[asiafrank]

/pick-up count agg functions

[ti-challenge-bot[bot]]

This issue os already picked by SailerNote.

[TszKitLo40]

/pick-up max min agg functions

[ti-challenge-bot[bot]]

This issue os already picked by SailerNote.

[asiafrank]

/pick-up first row agg functions

[ti-challenge-bot[bot]]

This issue os already picked by SailerNote.

[TszKitLo40]

/pick-up value agg functions

[ti-challenge-bot[bot]]

This issue os already picked by SailerNote.

[asiafrank]

/pick-up var pop agg functions

[ti-challenge-bot[bot]]

This issue os already picked by SailerNote.

[TszKitLo40]

/pick-up rank agg functions

[ti-challenge-bot[bot]]

This issue os already picked by SailerNote.

[TszKitLo40]

/pick-up row number agg functions

[ti-challenge-bot[bot]]

This issue os already picked by SailerNote.

[Howie59]

/pick-up ntile agg functions

[ti-challenge-bot[bot]]

This issue os already picked by SailerNote.

[TszKitLo40]

/pick-up json object agg functions

[ti-challenge-bot[bot]]

This issue os already picked by SailerNote.

[TszKitLo40]

/pick-up percent rank agg functions

[ti-challenge-bot[bot]]

This issue os already picked by SailerNote.

[TszKitLo40]

/pick-up group concat agg functions

[ti-challenge-bot[bot]]

This issue os already picked by SailerNote.

[Howie59]

/pick-up first row agg functions

[ti-challenge-bot[bot]]

This issue os already picked by SailerNote.

[TszKitLo40]

/pick-up bit agg functions

[ti-challenge-bot[bot]]

This issue os already picked by SailerNote.

[TszKitLo40]

/pick-up var pop agg functions

[ti-challenge-bot[bot]]

This issue os already picked by SailerNote.

[TszKitLo40]

/pick-up cume distinct agg functions

[ti-challenge-bot[bot]]

This issue os already picked by SailerNote.

[hi-rustin]

/help
cc: @TszKitLo40 @Howie66 @asiafrank @xiaodong-ji

Pick Up Rules

[ti-challenge-bot[bot]]

Usage:

      /command [argument]

The commands are:

      /ping            ping the bot
      /pick-up      pick up the issue
      /give-up      give up the issue
      /reward score      reward the score to PR

---

You can find more details on challenge-bot-docs.

If you think there is a problem, you can report bug on report-a-bug.

If you think the bot is dead, please contact Rustin-Liu to fix it.

[XuHuaiyu]

@SailerNote @TszKitLo40 @Howie66 @asiafrank @xiaodong-ji

We'll create an individual issue for each item in the description, you may pick up the work later.

[SailerNote]

/give-up

[ti-challenge-bot[bot]]

Give up success.

[XuHuaiyu]

@TszKitLo40 @Howie66 @asiafrank @xiaodong-ji
Hi, you can pick up the sub-tasks now.

[Hunsmore]

/pick-up value agg functions

[ti-challenge-bot[bot]]

It is not a pickable issue!

[b41sh]

@Hunsmore value agg functions have picked up success

[hi-rustin]

/pick-up

[ti-challenge-bot[bot]]

It is not a pickable issue!

More

Tip : If you want this issue to be picked, you need to add a challenge-program label to it.

Warning: None

[qw4990]

Can we close this issue since all tasks have been finished? @b41sh

[b41sh]

Can we close this issue since all tasks have been finished? @b41sh

yes, we can finish it.

[SunRunAway]

@b41sh Could you provide a test report showing that your job work?

We can set up a test environment and run some testing aggregation queries. And the job can be proven by the following two points

1. During the execution of a long-running SQL, we can monitor the heap usage in grafana and SQL's memory usage in the show processlist at the same time. Give a comparison chart to show that the memory statistics shown in processlist are close to the memory used by the go language.
2. By setting an appropriate mem-query-quota, a query that consumes a lot of memory will eventually be killed.