Gridcoin-research: Suggest regular periodic snapshot

[tcblack]

I suggest we nominate a site (I will begin by nominating Startail/gridcoinstats who I believe is willing), to make a monthly official snapshot for distribution.

1. First day of the month (increase frequency if desired)
2. Check fork status.
3. If it's green compile a fresh master snapshot.
4. upload to server or start a torrent?

I suspect this is easily enough scripted via chron.

[startailcoon]

who I believe is willing

I'm willing to supply the community with this service, if desired.
I can supply a snapshot, with details regarding the last block height, hash and the chains last SB hash.

This would be fairly easy trough a cron, and could be supplied both trough a http URL and a torrent if it works.

[iFoggz]

wouldn't you need to shutdown and close the blockchain database before u can make a snapshot as to not get corrupted compressed file from database changing mid compression?

[startailcoon]

Yes, but that isn't an issue really

[iFoggz]

yea true.

[RoboticMind]

To make this slightly more decentralized we could get multiple people running these types torrents and you could compare hashes to try to find any bad actors

[fffffgggg54]

@RoboticMind yes I agree.

[startailcoon]

@RoboticMind would require a coordinated snapshot at predefined blocks, but I agree that would be a good course of action

[tomasbrod]

Snapshots are inherently insecure. Wallet does only very basic checks after snapshot is applied.
Can't compare file hashes as zip is non-deterministic, neither the txleveldb.

[RoboticMind]

The zip file format is deterministic but to get the same hashes you need to remove the timestamps and other metadata from the files. So if you got everyone running a snapshot to remove metadata and timestamps as well as having a snapshot on the same block, you could compare hashes of the files.

[xaminmo]

In theory, the need for snapshots would be moot most of the time. The wallet itself does all of this. It shares blocks with checksums, in a torrent-like fashion. The only reason it's not consistent right now is because of the protocol defects being worked on.

[xaminmo]

Brainstorming on systematic snapshots here.

Regarding checksum differences, we could use sha256sum or 7z h -scrcsha256 or similar to generate hashes of all of the files before packaging them up. Same for after, in comparison.

You'll still run into differences unless every snapshot is reorganized back to a specific block, then stopped before new blocks are pulled down. This might call for a new command that would snapshot up to a specific point, maybe a "generatesnapshot 1162500" or something like that.

It might be possible that we just truncate blk0001.dat at a nice block boundary, and share it out as a bootstrap.dat. From there, people could either import as-is, or wipe and import to let it build the txleveldb on startup. This does not help with the forking errors we have now. It still "downloads" all of the blocks from bootstrap.dat or blk0001.dat to generate the levelDB.

Another thing to ponder is including a prune option, or possibly pruning the blockchain at the last hardcoded checkpoint, and just sharing the levelDB and the blockchain after that checkpoint. I don't know that we have enough consistency among users for that to be a safe option though, though maybe the checkpoint blockchain could be saved separately. This seems not the best idea, but not the worst either.

Periodic snapshot sharing might also get into the consideration of storing the blk*.dat files with stream-compressed blocks, as long as it was lightweight enough.

[tomasbrod]

Snapshot includes all blocks in index and various other metadata. Index includes not only blocks from current best chain, but also blocks from some forks.
Bootstrap files are better at this, but if you make bootstrap from blk*.dat file, it would contain forked blocks too. Order of how these are added to the "history file" (blk) is not deterministic, so the file hash will not match. The best block hash will match, but that block is not marked in any special way.
However, the idea of sharing a deterministic bootstrap is good.
Pruning the database would be also a nice command. It is not easy. Blocks are stored in sequential file. Only few file-systems (none on windows) support deleting data from the middle of the file. This could be solved by storing the block data in database, from where records can just be deleted. I'm not sure wheter leveldb will release the space that was freed.

[jamescowens]

We are not going to do this. @iFoggz has done a wonderful job with the update check and snapshot PR (#1576) which has been merged into the dev branch. We will update the snapshot on the main website from time to time. With the improvements in the dev branch, the sync from zero time has improved to about 2 hours for a reasonably modern PC, which really eliminates most of the need for a snapshot anyway. And syncing from zero is always the most secure way to sync to the blockchain.