Tikv: Region written stats is abnormal

[tabokie]

Bug Report

What version of TiKV are you using?

Versions that I know of: 4.0.10, 5.0.0-rc

What operating system and CPU are you using?

Steps to reproduce

What did you expect?

Value of region written bytes make sense, and is relatively stable.

What did happened?

In this case (5.0.0-rc), service is running normally, but region written bytes reach multiple PBs

In this case (4.0.10), two down servers worsen the abnormal state

[tabokie]

In one of the cases, the tikv_region_written_bytes_sum value is 1.8e19, it's likely a u64 overflow, possibly caused by written_bytes_delta calculation during heartbeat task.

[BusyJay]

Because the average written bytes keeps growing, and it's calculated by

let written_bytes = current_written_bytes - last_written_bytes;
last_written_bytes = current_written_bytes;

. If it's underflow, then current_written_bytes keeps decreasing for more than 5 hours, which doesn't seem reasonable.

[tabokie]

@BusyJay Are you referring to the second screenshot? I recheck the metrics, it seems like something wrong with grafana, the linear graph is a display error:

[BusyJay]

So it seems like the metrics is not updated for 5 hours, and then hit underflow?

[cosven]

Is the region scheduling affected by these stats?

/severity moderate

[BusyJay]

If metrics is not updated for 5 hours, then either the clock of the prometheus machine jumps or tikv/pd thread is stuck for 5 hours. If the latter is the case, and only pd thread is stuck, then scheduling will not work for the time; if tikv is stuck, then it should be completely out of service.

[tabokie]

If metrics is not updated for 5 hours, then either the clock of the prometheus machine jumps or tikv/pd thread is stuck for 5 hours. If the latter is the case, and only pd thread is stuck, then scheduling will not work for the time; if tikv is stuck, then it should be completely out of service.

I think during this 5 hours the two problematic servers stayed down, thus no metrics reporting.