Solana: Stake accounts stuck deactivating

[CriesofCarrots]

Problem

A collection of stake accounts on mainnet-beta are "stuck" in the deactivating state. All 14 of them happen to have been delegated and deactivated in epoch 16. The immediate cause of this issue is that the StakeHistory sysvar reports 0 lamports as deactivating for epoch 16, so that the share of deactivating stake allotted to each of these accounts is also 0.

Here is the output of solana stake-history for nearby epochs:

    19      5071038.34127776          400.39543424           15.31303968 SOL
     18      5020338.33040928        50700.01086848           15.31303968 SOL
     17       5020338.3381264            0.09771712           15.41847392 SOL
     16      4970322.82421824        50000.20086848                     0 SOL
     15        5020319.828784            2.99315136        49999.99771712 SOL

15.31303968 SOL corresponds to the amount of stake "stuck" in these 14 accounts.

Addresses:

[
    ZKbZsMhUJMp7MfcFmkYqtipcgcJxi33GiCyNMTQKUV9,
    2371aoQkSwkqoceYzA7ycSaLaW2takvLv52u8FDYmFJU,
    3Sxhqd87eGHzRCHW1wK8EbkVVpAqLeEoYSYQHpuKh9EZ,
    3fu3BEhLWP1Zm6PyjrSQWDkwMEmiUgnQy6xXDrRxeQg9,
    6uMvSMbEux6VLE2D4JY471hmG74PUDTvAqc2ggTi6Zg3,
    7SGCyBSwNCbhC1FxPW5FFTuyyoNqo1kWBN3P2ua881zJ,
    7cg9whPgWFZbCTn52AEjUaUAJk44n2jgpZWjzYPPtSVR,
    9S1gnNB1yQy3j1uTqrLVX6UqzQYKpHbdBtwHb4vjUfYd,
    9dUAdJcdKQ5ZDsJg4sWYTtCuYgtQbQwnXKFqJQGjTnEH,
    Bo1PzDvwSMorNW7dMsbT69aRuJox6aXKWeGLyu6FYikU,
    CVUysRJyjdRey56q92bTuzgNVot99uhdo5aSRH6qMBav,
    D59zLU646AvzXyhxMSZvFT67JiggCMYzUzWKB47qNhcn,
    DzHX8tfrE7uhSDpo9m1pcGHbH2sStmDuipRbdQYBRsKw,
    F6Ni2rYtY74h8ju7neMbUMC1afccDoELFEHhHiwY38ui,
]

@t-nelson reported seeing this issue locally when delegating and deactivating stake in the same epoch, but it does not always reproduce.

Proposed Solution

???

CC: @garious

[garious]

I don't know what's wrong here, but self-assigned to go debug it. Hints welcome.

[rwalker-com]

zero for the epoch in which the stake was deactivated is correct. what are the column headings for solana stake-history?

[t-nelson]

@rwalker-com Effective, activating, deactivating

[rwalker-com]

what's the history of these accounts? they look like they only have a few SOL each. did they originally add up to the 50k SOL activated in epoch 16?

[danpaul000]

Attaching the transaction history from my ledger-to-csv-dumper tool. I can't attach .csv's to github, so here are csv files disguised as .txt files. This data was parsed from the rocks db on the warehouse nodes. To make the structure spreadsheet friendly for accounting purposes, the transaction info is written to one sheet, and the instruction info (one or more per txn) is written to a second sheet. Only transactions that contain one of the addresses in the initial PR description are included, and all instructions from any relevant transaction are included. Use the txn_sig in the transactions_of_interest sheet as the key to filter and view the corresponding instructions in the instructions_of_interest sheet.

instructions_of_interest.txt
transactions_of_interest.txt

[rwalker-com]

ok, looks like these accounts were small when initialized

[rwalker-com]

@garious this small amount of deactivating stake should all have been activated in one epoch, and deactivated in one epoch.

What I'd try as a test is populating stake history with a large effective, then activate and deactivate a small amount in the same epoch. I can do this, but not until this weekend.

[garious]

Hey, thanks for the guidance @rwalker-com! I'll give it a shot.

[vidorge]

We also found 2 accounts showing this behavior:

Mainnet: https://explorer.solana.com/account/4Y183NztddoYcJJXDrcxrwbJAPHHxbRGEhgGKx5ChpzU

Devnet: https://explorer.solana.com/account/8sdDABzwj3XiFNZcNHAY36B9WCkFPrRwYekJrtW1bNh1?cluster=devnet

[CriesofCarrots]

4Y183NztddoYcJJXDrcxrwbJAPHHxbRGEhgGKx5ChpzU and 8sdDABzwj3XiFNZcNHAY36B9WCkFPrRwYekJrtW1bNh1 (devnet) both look fully deactivated to me

[vidorge]

@CriesofCarrots Sorry, turns out that is an explorer bug which provides a different result than the CLI.

[jstarry]

Sorry, turns out that is an explorer bug which provides a different result than the CLI.

FYI this bug is being tracked here: https://github.com/solana-labs/solana/issues/11186

[rwalker-com]

I'm wiping my brow. Had meant to get back to this last weekend but spaced.

BTW: stake history is finite. Once activation and deactivation epochs are clear of what's listed in stake history, warmup and cooldown percentages are ignored. I.e. had this been an actual stake bug, the stakeholders could have just waited until they were clear of stake history info.

[CriesofCarrots]

To be clear, I believe there is still a bug here. There is just also an explorer (display) bug.

[rwalker-com]

To be clear, I believe there is still a bug here. There is just also an explorer (display) bug.

My mistake.

[mvines]

Sounds like the only remaining action here is captured by #11186

[CriesofCarrots]

Actually, there is definitely still a bug here. Not sure if @garious made any progress tracking it down...?

[garious]

I have not

[ryoqun]

I think I've finally found the root cause of this.