Tidb: CLUSTER_INFO system table may not work after PD is scaled-in

[breeswish]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

Scale-in PD.

2. What did you expect to see? (Required)

Successfully query INFORMATION_SCHEMA.CLUSTER_INFO

3. What did you see instead (Required)

It tries to access a PD that no longer exists. dial tcp xxxx: connect: connection refused

4. Affected version (Required)

v4.0.4

5. Root Cause Analysis

[breeswish]

Friendly hint: According to current code base, the scaled-out PD node may be displayed improperly as well.

[jyz0309]

alkaid will fix it :)

[jyz0309]

when I try to scale in 3 tidb,I found tiup playground display and select * from INFORMATION_SCHEMA.CLUSTER_INFO; will return correct answer(have 3 tidb), but when I scale out 1 tidb, I found that tiup playground display will return correct answer(2 tidb) but select * from INFORMATION_SCHEMA.CLUSTER_INFO; will not, it still show me 3 tidb in cluster

[breeswish]

when I try to scale in 3 tidb,I found tiup playground display and select * from INFORMATION_SCHEMA.CLUSTER_INFO; will return correct answer(have 3 tidb), but when I scale out 1 tidb, I found that tiup playground display will return correct answer(2 tidb) but select * from INFORMATION_SCHEMA.CLUSTER_INFO; will not, it still show me 3 tidb in cluster

CLSUTER_INFO is based on some reporting items from TiDB with a lease, how about the result after 1 minute?

[jyz0309]

when I try to scale in 3 tidb,I found tiup playground display and select * from INFORMATION_SCHEMA.CLUSTER_INFO; will return correct answer(have 3 tidb), but when I scale out 1 tidb, I found that tiup playground display will return correct answer(2 tidb) but select * from INFORMATION_SCHEMA.CLUSTER_INFO; will not, it still show me 3 tidb in cluster

CLSUTER_INFO is based on some reporting items from TiDB with a lease, how about the result after 1 minute?

The select * from INFORMATION_SCHEMA.CLUSTER_INFO; result is still wrong after 5min :(

[breeswish]

@jyz0309 Good catch, go on and discover why

[breeswish]

Not fixed yet

[ti-srebot]

Please edit this comment or add a new comment to complete the following information

Bug

Note: Make Sure that 'component', and 'severity' labels are added
Example for how to fill out the template: https://github.com/pingcap/tidb/issues/20100

1. Root Cause Analysis (RCA) (optional)

TiDB uses etcd address in CLI, which is unchanged when PD has dynamic member change.

2. Symptom (optional)

Query INFORMATION_SCHEMA.CLUSTER_INFO will fail.

In TiDB Dashboard, cluster info page may fail.

3. All Trigger Conditions (optional)

Scale-in PD.

4. Workaround (optional)

1. Ensure latest PD is updated to TiDB (in TiUP, use cluster reload).

2. Restart TiDB.

5. Affected versions

[v4.0.1:v4.0.6]

6. Fixed versions

v4.0.7