Solana: Tds died 1.2.3

[carllin]

Problem

Tds stopped making roots at 2020-06-17T06:04:51.274160434Z 😢

Last root:
[2020-06-17T06:04:47.176098269Z INFO solana_core::replay_stage] new root 20605600

Forks at time of death from perspective of @aeyakovenko validator on merckx:
graph.pdf

Proposed Solution

[carllin]

I think we should do a coordinated restart with everyone at 1.2.3 and see where we land

[leoluk]

If your Solana node runs as a systemd service and logs to journald:

TZ=UTC journalctl -u solana.service -o cat --utc --since '03:45' --until '08:30' | xz | curl --data-binary @- 'http://logbin.certus.one:8080/solana-github-10653/<your node ID>.log.xz'

Or, if you have plain logfiles (make sure it contains the appropriate time range):

xz | curl --data-binary @- 'http://logbin.certus.one:8080/solana-github-10653/<your node ID>.log.xz' < your_logfile.log

(change the service/logfile name, if necessary, and your node ID)

Upload destination: http://logbin.certus.one:8080/Qui9iehu/solana-github-10653/

[JB273]

Hey @leoluk, uploaded my log: hQBS6cu8RHkXcCzE6N8mQxhgrtbNy4kivoRjTMzF2cA

[pkrasam]

Ta @leoluk just uploaded my sol.service logs

[leoluk]

Log upload for the failed restart with 1.2.4:

TZ=UTC journalctl -u solana.service -o cat --utc --since '2020-06-18 21:20' --until '2020-06-19 09:00' | xz | curl --data-binary @- 'http://logbin.certus.one:8080/solana-github-10653-2/<your node ID>.log.xz'

For plain logfiles:

xz | curl --data-binary @- 'http://logbin.certus.one:8080/solana-github-10653-2/<your node ID>.log.xz' < your_logfile.log

In case the Solana team has requested your ledger:

tar -I 'zstdmt' -c <path to your ledger> | curl --data-binary @- 'http://logbin.certus.one:8080/solana-github-10653-2/<your node ID>.ledger.tar.zst'

(you might have to install the zstd package)

Upload destination: http://logbin.certus.one:8080/Qui9iehu/solana-github-10653-2